Agile: Organizational Balance

In my previous post Agile: Specialist vs Generalist, I recast the specialist vs generalist debate into broader understandings, considering that specialists and generalists naturally arise out of monolithic (waterfall-like) and startup-like (agile) organizations. One could say these are the default states, but these defaults do not preclude the other. The question that arose for me is this: What is the best balance between purely self-organizing (agile) and purely leader-based (waterfall) in larger organizations?

When I consider the experiences I've had with Agile, I found a very strong personal association with the Agile team and easily identify with roles, responsibilities and activities within an Agile team. I think it is an excellent way to organize a small group of 5-9 people. However that experience was embedded within an organization that had many groups of 5-9 people, all with varying ideas of what a team was responsible for and they should operate. I can't say that this is necessarily a problem, however I do not know what the best strategy the "layer above" could use to enable success for the entire group. Hence my question above - should the organization been structured as a recursive or self-similar entity where each of the "teams" were considered members of a meta-team, each group behaving as an individual developer in a simple "agile team"? Or should there be set of managers that treat the agile teams as members of their group and assign them tasks and mete out punishments?

Perhaps "punishment" is incorrect - consequences is probably better, but it is still something missing in the reading I've done on Agile. How does one handle a poorly performing team? How does one measure that output? From a team perspective, the team measures itself, but the team could be deluded or mistaken in its assumptions. I think I can see the comment postings now: "guide the team back to the right path by assigning a mentor or leader that can steer the team in the right direction." So maybe my initial recollections are not correct with respect to Agile and consequences, but it leads back to the idea leadership/management of many Agile teams: How does one make sure the projects stay on track?

One of the key items of Agile is communication between people. Probably the most critical communication junction is that between the implementors and the consumers. In a purely top-down model, the managers fill the role of "consumer" - the implementors talk to the people above themselves in the hierarchy and that's it. That abstraction can hurt the final product, but that abstraction (like any abstraction) provides flexibility: the manager can make absolute decisions and keep the project on track. I believe there must be a way to bring that flexibility into a large Agile organization without the loss of communication.

A hierarchical organization provides benefits - each layer can be thought of a leader with several "subordinates" that fulfill requests. This hierarchy does not preclude Agile - after all the best teams anticipate problems and act independently as much as possible. So I believe there must be a way to fit aspects of the hierarchy with Agile teams.

Why even bring this up? Consider a purely Agile environment with several (>= 10) teams. How would they organize the work? Each team would have to act as an individual member of a larger team. This larger team would define the overall backlog and each team member would take on work for the next iteration. Another way would be to designate one team to define the backlog and the rest to pull from the master backlog. What happens when disagreements occur? Disagreements can be resolved via communication between the effected teams, but what happens if one team is obstinate? More meetings and communication. The point is, to work in a consensus manner takes time, time which may not be available. Having a someone to that holds a "buck-stops-here" authority cuts down on this time. A decision has to be made and the quicker it is made, the better. Not everyone is going to agree with it, so just move on.

In fact, the Agile methodology is eminently suited for this quick-decision management - the organization must be able to take a new decision later if an old one proved incorrect. The benefit of this flexibility is lost if the initial decisions take too long. Have a single decision maker will speed the initial decision making process.

So what is the appropriate balance? My initial idea is that it requires someone who has authority within the company (as in the teams must heed the decisions and requests from this person), but who is not member of any team. This person would not define what needs to be done but would instead be responsible for the output of several teams. They could act on behalf of the teams in some instances, but would have to listen to the teams in others.

Yes very sketchy, but that is how many Agile things were described to me initially. The balance between authority and flexibility needs to be discovered within an organization. I believe this is a vital piece - one that was missing from the organization where I was first exposed to Agile. Moving to Agile, especially within a larger non-Agile organization, is a very tricky business. I'm just hoping to provoke some thought and comment.

Agile: Specialist vs Generalist

In his recent blog post, Peter Scheyen began a multi-part response to this comment. The blog post contrasted developer specialization and developer generalization in an agile context. The comment suggested that "... Agile robs us of specialization", and Peter's response described how specialization operates within an Agile team. I think the comment and the response come from different perspectives - one where Agile is not suitable (comment) and one where Agile is suitable (response). So allow me to step a little bit.

I wish to recast "specialist vs. generalist" to "Cog-in-the-machine vs. Member-of-Startup". Many large organizations have a decidedly non-Agile project management approach, and for many it works well. I think that specialization in a non-Agile environment is a side-effect of longevity or survival. If I were hired to be part of a large development organization, I would start off as a faceless cog - something that is interchangeable with any of the other cogs. The longer I stay in the organization, the specialized I will become, whether by accident (I worked on system X since day 1!) or design (I became the go-to guy for System X). Cast in pure evolutionary terms, this is a survival tactic, so the mere fact that I am still there (with the corresponding higher costs and "slowness") is because of something that makes me stand out from the faceless cogs. Specialization becomes inevitable.

The other extreme occurs if I were to be hired to a startup. In a startup, I may have some special skills, but everyone must work to pick up any work that needs to be done. In the startup, there is a chronic lack of staffing because the organization is short on resources. If things get left behind, the whole organization fails, so a weak team member is harder to accommodate. Generalization becomes inevitable.

This brings me back to the post: the ideal Agile team described is closer to a startup than anything else. The ideal team works like a startup, identifying and filling any gaps, regardless of specialty, but still leverages the team members with the greatest knowledge in any area.

I wrote a comment to summarize the above and it got me thinking about Agile organizations and more traditional organizations. The Agile team described in Peter Scheyen's post is ideal - the team has no distinct leader, just a set of developers with various skills and specialties that are engaged as needed. This works fantastically well when the entire development organization is "small" and the developers buy into the model. In the more traditional environments, there were team leaders, managers (at several levels), directors, etcetera, that had two roles: assignment of tasks and responsibility for the project. This is a very top-down model - the "top" decides to do something and the requests filter down to the ultimate workers who make it happen. If something goes wrong, there is a chain of responsibility up to the top that can easily be identified. Contrast this with the Agile team described - it is bottom up, so the team as a whole is responsible for getting a request done, but it has always been a bit murky to me on how a large Agile organization with many teams assigns tasks.

I don't want to make this post too long, so I will save this for my next post(Agile: Organizational Balance): How does an organization find the balance between purely distributed (self-organizing Agile teams) and purely leadership (top-down, waterfall) of large projects?

Opinions on Programming

Saw an interesting article on Slashdot (here's the full article). Of course I saw it's a great article because it lines up with some opinions I've held for several years. Guess it's more of a personal philosophy, probably shaped by learning how to program in text: you've got to know how things work if want to understand what is going on. It applies to so many different aspects of life too - it's shocking at times the trivial things that people don't know. Like how to change a tire - to understand what to do, you have to have an idea of how the tire is attached to the vehicle. Programming with pictures is all well and good, but there comes a point at which you have to reach deeper understanding of what goes on behind the pictures to ensure that things are operating as you wish.

I've had this discussion with one particular person and they pointed out, quite correctly, that there is a good reason why UML programming never took off. That's because the graphical development environments were always the encoding the wrong thing - the requirements. The "what" rather than the "how". The "what" parts are often contradictory, which is why I have a job. A software development professional is there to turn the "what" into the "how" correctly. Break the paradoxes and force the machine to do what is implied in the requirements, not do them literally.

So as one of the developers quoted in the article, graphical environments are good for learning. Small projects that show how things fit together. That first frustrating few experiences when the larger project is attempted and hours are spent to make two things mesh that won't work together. Then showing the text-based programming languages - the "how" behind it all. From what I've heard, any attempts at doing UML programming actually had two steps: First, design in the graphical environment and then second, tune the generated C source code to make things actually work. Very indicative of things to come.

I also appreciated what Herb Sutter had to say - that bare-metal programming and optimization will come back into vogue in the next 10 years. Waste is waste and graphical environments and elaborate abstractions are waste. And this waste can be translated into environmentally-relevant impacts. Smaller, efficient code will use less power, less space and be better all around. I still hope for the day when every chip-based interface responds instantaneously to my input. Even if it has to tell me that what I just requested will take a long time, that initial response shapes my interaction.

Code and stuff

This is post 191 - a nice 3-digit prime.  Don't really have enough posts to stretch the checking necessary - 191^(1/2) == 13.82... That means I'll have to wait until post 289 until the next prime past 13 needs to be checked.  Anyway, I spent the day working with code - something that hasn't happened in a while.  Always seems like a shock, but there you go.  People still don't believe me when I say that more time is spent debugging and testing than writing code.  Or designing code.  Or whatever with code.  Some stuff to look at though - something applicable even outside the small domain of our business - a thread pool.  I didn't find the bug, I didn't come up with the fix, I just applied the fix to the correct branch and added some comments.

Frankly, I'd like to be able to add some more tests before going forward, but I definitely didn't have the time now.  I know this sounds like an excuse - we are approaching a release - but there has to be balance between what should be done and what I'm being paid to get done.  In this case, the situation called for a targeted fix, so I relied on existing unit tests to verify the change.  Ideally, I'd like to spend more time checking the unit tests to make sure that items weren't missed, but that wasn't practical.  Next week is a different story however...

What's worse is that this issue is related to a low-frequency, high-impact bug that our team hasn't isolated yet.  I think that next week I'll try looking for it by writing more unit tests.  I have a feeling that may be the most reliable reproduce scenario - one that I create out of thin air.

Discovered something else near the end of the day, related to something I've seen often - improper shutdown/exit of software leads to problems.  This is an issue in embedded development, where restarting a problem subsystem is a good uptime strategy.  The Qnx operating system uses this philosophy, where the microkernel never goes down, but the real work is done by processes that can be restarted as appropriate.  Since everything, including device drivers, are "processes", a device driver upgrade consists of putting the software in the right place and "restarting" the driver.  Being able to shutdown and restart a small piece of functionality is akin to modular programming.  Unfortunately this modular, restartable philosophy was not something hammered home at the beginning of this project, so this issue keeps recurring.  I also think that being able to stop and restart a module without side effects demonstrates good modularization qualities - particularly isolation.  It means that the writer and designer of the piece of software has considered its interactions and resource requirements and they are well identified within the code.  Losing sight of these things means that module can be shutdown, but not restarted.  

Well, I guess that is something I'm going to have to talk-up at work.

Maven thoughts, then patterns thoughts

I'm still plodding away at Maven at work.  There was some discussion as to naming patterns, conventions, directory structure etc.  Most of these things weren't ground-breaking, just items that require agreement.  Without agreement, some benefits are lost.  The only point I wanted to see was that modules and Java packages corresponded.  Everyone in my group agreed with that idea, so I went to voice my concern with one of the main drivers behind the Maven efforts.  He said essentially the same thing, but pointed out an exception.  Imagine an interface is described in module I, with two concrete implementations in module A and module B.  If module I uses a factory, it must have special access to module A and module B.  Specifically, module A and module B should be non-public, package protected perhaps.  This leads to having the interface, factory and implementations in one Java package even though logically they three separate pieces.  I wasn't 100% comfortable with the example, but I couldn't identify any flaws in the reasoning.  Then I hit on why I didn't like it - I don't think I like factory classes.

I've used factory classes a few times in the recent past, but they aren't something I agree with as much.  They are hard to test and the classes that use factory are also hard to test.  I think I would prefer each of the modules be a separate Java package with a single public entry point in module A and module B.  The user would be able to determine which implementation to use, instead of putting all that logic in the factory class.  However, either implementation would work and are equally correct.  It also illustrated the point that the name conventions and layouts are guidelines, not rules.  The exceptions occur frequently enough that slavish adherence to particular rules will cause more problems than necessary.  Rather the rules are there to help remind people to really consider why they are using complex constructs.

This brings me to some thoughts on patterns.  The Abstract Factory pattern was mentioned above and my desire to avoid using it comes from a certain unease with ideas of software patterns.  I've previously had discussions with a co-worker who expressed the idea that there is something wrong with patterns, but he couldn't put his finger on it.  I suggested that it was because the patterns don't solve all problems (otherwise software development could have been fully automated by now) and the pattern requires adjustment all the time.  It seems like the problems are fit to the patterns, rather than using patterns to solve problems.  I think that patterns have been used by those that want to turn software development into more of an engineering discipline with common issues that simply need to be recognized and well-known (well-described?) solutions applied.  I agreed that these were my ideas on the topic, but didn't do the unease any justice.  My co-worker has been reading Christopher Alexander's book on pattern language, as Alexander is the source of the ideas adapted to software development.

I suppose the real issues come from those who apply the software design patterns too easily, without first considering the problem.  As time has gone on, I've been adding criteria and considerations to things I need to design.  Things like ease of understanding the result, simplicity of implementation, testability, current requirements.    Also, with the various patterns I've been involved in, they've been customized and adjusted in ways that suggest one of two things (or both): 1) pattern didn't fit and the problem is being adjusted to suit the pattern and 2) so many adjustments are made so that the pattern is only partially there.

Anyway, this is not a well-reasoned argument - it's barely written down at all!  Hopefully I will be able to devote some more thought and then more time to describing some of these things more coherently.  Not tonight though.

Build tools

I referred to the Maven change at my work the other day, but I didn't take the time to describe some of the basic principals that I was using.  These principals generate questions rather than exclude or include things.  As I mentioned in this post, any build tool can be configured to support your environment.  Using that logic, why would anyone move beyond shell scripts and batch files?  That's the first item in the philosophy:

1.  Can this project use the build tool in the way it was designed?

For make, this means try to use only the implicit rules and organize your pieces so that all the interdependent files are within the same directory.  For maven, this means organizing your project in a modular fashion where each of the configuration files describe only the dependencies between the modules.  When you move out of the comfort zone for the tool, you increase the likelihood of mistakes, or design cul-de-sacs.  Every design has certain features that shape it, and those features exclude other items or make them difficult to achieve.  Using Maven to build a project developed entirely in C raises a flag.

2. Can the build tool handle the corner case you know about now?

Pretty simple thing to remember - if the tool can't handle all your current requirements, how will you deal with the unexpected?

3. Can the tool be setup to run correctly from a sandbox?

By sandbox I mean can the tool be useful without referring to non-local items.  I understand that one of the features of Maven is that it can keep various libraries up-to-date by retrieving the latest version.  By "retrieve" it means connect via HTTP, I believe.  I've been informed that this behaviour can be disabled, which would be a plus in my view.  When building a particular project, it is preferable to be able to identify, before the build, the files that will be included.  That doesn't really provide any more or less protection from error, but it makes the process more repeatable and leads to the next point.

4. Can the tool be setup to run correctly from a source control?

Imagine you have a project in a source control system and you connect with a new machine.  Can  you do a single check, run the build tool and generate the proper output?  This is something that I value highly.  It speaks to repeatability and how fast you can get new project members up to speed.  It also means that as long as you have project source, you can build it.  Someone I knew at university was doing his PhD, but he said that he could no longer print or display his Master's thesis.  He wrote it in PostScript or LaTeX dialect and could no longer find the pieces the render the source as it appeared in the hard copy.  Many companies and individuals rely on open source tools.  If they do not keep the tools along with the project, they risk not being able to build the project from source in the future.  

5. Can developers operate effectively without modifying the configuration files?

One of the problems I've seen in various workplaces is build configuration differences.  If the build configuration files have to be checked out of source control and modified to build correctly, that is a large potential problem.  Ideally, build configurations would not need to be altered unless files are added or removed - even some tolerance of those changes would be good.  If these files have to be checked out, different groups produce slightly different builds, which leads to errors arising from these small changes.  These errors are very hard to track down and can be avoided with proper tool selection.

To summarize - the idea is that doing the right thing is easy and the wrong thing (unexpected things) is hard.  It should be easy to identify where everything came from and how something was built.  These things may take a large amount of configuration - a ramp up time learning how to use a tool properly.  As for Maven, so far I think it's right for what we are doing at work - but there are caveats.  The main one being the potential for cross-module changes, of which I know of one current example at work.  Items that touch many modules can be refactored by adding more indirection (or abstraction), but this has it's own cost.  Oh well - such complexity is what makes things difficult and what keeps me employed.  I just want to make sure that I don't forget the potential problem areas in the future.