Agile: Organizational Balance

In my previous post Agile: Specialist vs Generalist, I recast the specialist vs generalist debate into broader understandings, considering that specialists and generalists naturally arise out of monolithic (waterfall-like) and startup-like (agile) organizations. One could say these are the default states, but these defaults do not preclude the other. The question that arose for me is this: What is the best balance between purely self-organizing (agile) and purely leader-based (waterfall) in larger organizations?

When I consider the experiences I've had with Agile, I found a very strong personal association with the Agile team and easily identify with roles, responsibilities and activities within an Agile team. I think it is an excellent way to organize a small group of 5-9 people. However that experience was embedded within an organization that had many groups of 5-9 people, all with varying ideas of what a team was responsible for and they should operate. I can't say that this is necessarily a problem, however I do not know what the best strategy the "layer above" could use to enable success for the entire group. Hence my question above - should the organization been structured as a recursive or self-similar entity where each of the "teams" were considered members of a meta-team, each group behaving as an individual developer in a simple "agile team"? Or should there be set of managers that treat the agile teams as members of their group and assign them tasks and mete out punishments?

Perhaps "punishment" is incorrect - consequences is probably better, but it is still something missing in the reading I've done on Agile. How does one handle a poorly performing team? How does one measure that output? From a team perspective, the team measures itself, but the team could be deluded or mistaken in its assumptions. I think I can see the comment postings now: "guide the team back to the right path by assigning a mentor or leader that can steer the team in the right direction." So maybe my initial recollections are not correct with respect to Agile and consequences, but it leads back to the idea leadership/management of many Agile teams: How does one make sure the projects stay on track?

One of the key items of Agile is communication between people. Probably the most critical communication junction is that between the implementors and the consumers. In a purely top-down model, the managers fill the role of "consumer" - the implementors talk to the people above themselves in the hierarchy and that's it. That abstraction can hurt the final product, but that abstraction (like any abstraction) provides flexibility: the manager can make absolute decisions and keep the project on track. I believe there must be a way to bring that flexibility into a large Agile organization without the loss of communication.

A hierarchical organization provides benefits - each layer can be thought of a leader with several "subordinates" that fulfill requests. This hierarchy does not preclude Agile - after all the best teams anticipate problems and act independently as much as possible. So I believe there must be a way to fit aspects of the hierarchy with Agile teams.

Why even bring this up? Consider a purely Agile environment with several (>= 10) teams. How would they organize the work? Each team would have to act as an individual member of a larger team. This larger team would define the overall backlog and each team member would take on work for the next iteration. Another way would be to designate one team to define the backlog and the rest to pull from the master backlog. What happens when disagreements occur? Disagreements can be resolved via communication between the effected teams, but what happens if one team is obstinate? More meetings and communication. The point is, to work in a consensus manner takes time, time which may not be available. Having a someone to that holds a "buck-stops-here" authority cuts down on this time. A decision has to be made and the quicker it is made, the better. Not everyone is going to agree with it, so just move on.

In fact, the Agile methodology is eminently suited for this quick-decision management - the organization must be able to take a new decision later if an old one proved incorrect. The benefit of this flexibility is lost if the initial decisions take too long. Have a single decision maker will speed the initial decision making process.

So what is the appropriate balance? My initial idea is that it requires someone who has authority within the company (as in the teams must heed the decisions and requests from this person), but who is not member of any team. This person would not define what needs to be done but would instead be responsible for the output of several teams. They could act on behalf of the teams in some instances, but would have to listen to the teams in others.

Yes very sketchy, but that is how many Agile things were described to me initially. The balance between authority and flexibility needs to be discovered within an organization. I believe this is a vital piece - one that was missing from the organization where I was first exposed to Agile. Moving to Agile, especially within a larger non-Agile organization, is a very tricky business. I'm just hoping to provoke some thought and comment.

Agile: Specialist vs Generalist

In his recent blog post, Peter Scheyen began a multi-part response to this comment. The blog post contrasted developer specialization and developer generalization in an agile context. The comment suggested that "... Agile robs us of specialization", and Peter's response described how specialization operates within an Agile team. I think the comment and the response come from different perspectives - one where Agile is not suitable (comment) and one where Agile is suitable (response). So allow me to step a little bit.

I wish to recast "specialist vs. generalist" to "Cog-in-the-machine vs. Member-of-Startup". Many large organizations have a decidedly non-Agile project management approach, and for many it works well. I think that specialization in a non-Agile environment is a side-effect of longevity or survival. If I were hired to be part of a large development organization, I would start off as a faceless cog - something that is interchangeable with any of the other cogs. The longer I stay in the organization, the specialized I will become, whether by accident (I worked on system X since day 1!) or design (I became the go-to guy for System X). Cast in pure evolutionary terms, this is a survival tactic, so the mere fact that I am still there (with the corresponding higher costs and "slowness") is because of something that makes me stand out from the faceless cogs. Specialization becomes inevitable.

The other extreme occurs if I were to be hired to a startup. In a startup, I may have some special skills, but everyone must work to pick up any work that needs to be done. In the startup, there is a chronic lack of staffing because the organization is short on resources. If things get left behind, the whole organization fails, so a weak team member is harder to accommodate. Generalization becomes inevitable.

This brings me back to the post: the ideal Agile team described is closer to a startup than anything else. The ideal team works like a startup, identifying and filling any gaps, regardless of specialty, but still leverages the team members with the greatest knowledge in any area.

I wrote a comment to summarize the above and it got me thinking about Agile organizations and more traditional organizations. The Agile team described in Peter Scheyen's post is ideal - the team has no distinct leader, just a set of developers with various skills and specialties that are engaged as needed. This works fantastically well when the entire development organization is "small" and the developers buy into the model. In the more traditional environments, there were team leaders, managers (at several levels), directors, etcetera, that had two roles: assignment of tasks and responsibility for the project. This is a very top-down model - the "top" decides to do something and the requests filter down to the ultimate workers who make it happen. If something goes wrong, there is a chain of responsibility up to the top that can easily be identified. Contrast this with the Agile team described - it is bottom up, so the team as a whole is responsible for getting a request done, but it has always been a bit murky to me on how a large Agile organization with many teams assigns tasks.

I don't want to make this post too long, so I will save this for my next post(Agile: Organizational Balance): How does an organization find the balance between purely distributed (self-organizing Agile teams) and purely leadership (top-down, waterfall) of large projects?