Agile: Organizational Balance

In my previous post Agile: Specialist vs Generalist, I recast the specialist vs generalist debate into broader understandings, considering that specialists and generalists naturally arise out of monolithic (waterfall-like) and startup-like (agile) organizations. One could say these are the default states, but these defaults do not preclude the other. The question that arose for me is this: What is the best balance between purely self-organizing (agile) and purely leader-based (waterfall) in larger organizations?

When I consider the experiences I've had with Agile, I found a very strong personal association with the Agile team and easily identify with roles, responsibilities and activities within an Agile team. I think it is an excellent way to organize a small group of 5-9 people. However that experience was embedded within an organization that had many groups of 5-9 people, all with varying ideas of what a team was responsible for and they should operate. I can't say that this is necessarily a problem, however I do not know what the best strategy the "layer above" could use to enable success for the entire group. Hence my question above - should the organization been structured as a recursive or self-similar entity where each of the "teams" were considered members of a meta-team, each group behaving as an individual developer in a simple "agile team"? Or should there be set of managers that treat the agile teams as members of their group and assign them tasks and mete out punishments?

Perhaps "punishment" is incorrect - consequences is probably better, but it is still something missing in the reading I've done on Agile. How does one handle a poorly performing team? How does one measure that output? From a team perspective, the team measures itself, but the team could be deluded or mistaken in its assumptions. I think I can see the comment postings now: "guide the team back to the right path by assigning a mentor or leader that can steer the team in the right direction." So maybe my initial recollections are not correct with respect to Agile and consequences, but it leads back to the idea leadership/management of many Agile teams: How does one make sure the projects stay on track?

One of the key items of Agile is communication between people. Probably the most critical communication junction is that between the implementors and the consumers. In a purely top-down model, the managers fill the role of "consumer" - the implementors talk to the people above themselves in the hierarchy and that's it. That abstraction can hurt the final product, but that abstraction (like any abstraction) provides flexibility: the manager can make absolute decisions and keep the project on track. I believe there must be a way to bring that flexibility into a large Agile organization without the loss of communication.

A hierarchical organization provides benefits - each layer can be thought of a leader with several "subordinates" that fulfill requests. This hierarchy does not preclude Agile - after all the best teams anticipate problems and act independently as much as possible. So I believe there must be a way to fit aspects of the hierarchy with Agile teams.

Why even bring this up? Consider a purely Agile environment with several (>= 10) teams. How would they organize the work? Each team would have to act as an individual member of a larger team. This larger team would define the overall backlog and each team member would take on work for the next iteration. Another way would be to designate one team to define the backlog and the rest to pull from the master backlog. What happens when disagreements occur? Disagreements can be resolved via communication between the effected teams, but what happens if one team is obstinate? More meetings and communication. The point is, to work in a consensus manner takes time, time which may not be available. Having a someone to that holds a "buck-stops-here" authority cuts down on this time. A decision has to be made and the quicker it is made, the better. Not everyone is going to agree with it, so just move on.

In fact, the Agile methodology is eminently suited for this quick-decision management - the organization must be able to take a new decision later if an old one proved incorrect. The benefit of this flexibility is lost if the initial decisions take too long. Have a single decision maker will speed the initial decision making process.

So what is the appropriate balance? My initial idea is that it requires someone who has authority within the company (as in the teams must heed the decisions and requests from this person), but who is not member of any team. This person would not define what needs to be done but would instead be responsible for the output of several teams. They could act on behalf of the teams in some instances, but would have to listen to the teams in others.

Yes very sketchy, but that is how many Agile things were described to me initially. The balance between authority and flexibility needs to be discovered within an organization. I believe this is a vital piece - one that was missing from the organization where I was first exposed to Agile. Moving to Agile, especially within a larger non-Agile organization, is a very tricky business. I'm just hoping to provoke some thought and comment.

Agile: Specialist vs Generalist

In his recent blog post, Peter Scheyen began a multi-part response to this comment. The blog post contrasted developer specialization and developer generalization in an agile context. The comment suggested that "... Agile robs us of specialization", and Peter's response described how specialization operates within an Agile team. I think the comment and the response come from different perspectives - one where Agile is not suitable (comment) and one where Agile is suitable (response). So allow me to step a little bit.

I wish to recast "specialist vs. generalist" to "Cog-in-the-machine vs. Member-of-Startup". Many large organizations have a decidedly non-Agile project management approach, and for many it works well. I think that specialization in a non-Agile environment is a side-effect of longevity or survival. If I were hired to be part of a large development organization, I would start off as a faceless cog - something that is interchangeable with any of the other cogs. The longer I stay in the organization, the specialized I will become, whether by accident (I worked on system X since day 1!) or design (I became the go-to guy for System X). Cast in pure evolutionary terms, this is a survival tactic, so the mere fact that I am still there (with the corresponding higher costs and "slowness") is because of something that makes me stand out from the faceless cogs. Specialization becomes inevitable.

The other extreme occurs if I were to be hired to a startup. In a startup, I may have some special skills, but everyone must work to pick up any work that needs to be done. In the startup, there is a chronic lack of staffing because the organization is short on resources. If things get left behind, the whole organization fails, so a weak team member is harder to accommodate. Generalization becomes inevitable.

This brings me back to the post: the ideal Agile team described is closer to a startup than anything else. The ideal team works like a startup, identifying and filling any gaps, regardless of specialty, but still leverages the team members with the greatest knowledge in any area.

I wrote a comment to summarize the above and it got me thinking about Agile organizations and more traditional organizations. The Agile team described in Peter Scheyen's post is ideal - the team has no distinct leader, just a set of developers with various skills and specialties that are engaged as needed. This works fantastically well when the entire development organization is "small" and the developers buy into the model. In the more traditional environments, there were team leaders, managers (at several levels), directors, etcetera, that had two roles: assignment of tasks and responsibility for the project. This is a very top-down model - the "top" decides to do something and the requests filter down to the ultimate workers who make it happen. If something goes wrong, there is a chain of responsibility up to the top that can easily be identified. Contrast this with the Agile team described - it is bottom up, so the team as a whole is responsible for getting a request done, but it has always been a bit murky to me on how a large Agile organization with many teams assigns tasks.

I don't want to make this post too long, so I will save this for my next post(Agile: Organizational Balance): How does an organization find the balance between purely distributed (self-organizing Agile teams) and purely leadership (top-down, waterfall) of large projects?

Long day...

I went a bunch of days without posting again!  What kind of sad commitment is that?  At least I was blogging at work - took quite a while as well.  I posted something on Friday, but it was only "part 1".  The part 2 ended up being pretty big, so I probably should have made parts 2-5 and part 6 later, but the issue is current so I felt I had to get done.

The topic of how to manage releases comes up every now and again and it shouldn't.  We should have a system that provides benefits for all the user instead of just some and then we won't have to ask the same things over and over.  Hopefully my postings at work will help illuminate some of the choices and get more people thinking about them.  One can only hope...

Whoa

Guess I should use my computer at home more.  Or at least some kind of better mobile blogging software 'cause I haven't written anything for 5 days!  That's pretty horrible given how I was doing before...  I just haven't been up to using the computer for the past few days.  At home at least.  It's been pretty busy at work, what with meetings and just so much stuff to do.  That is a good thing, but sometimes it leaves one tired and out-blogged.

One of the things I've been involved in is around our branching/release strategy.  It's funny because we've had this conversation too often as far as I'm concerned.  In the meeting we had to discuss it, we talked about the same old set of arguments about the basic different ideas.  It took a few days, but I think I hit why that was.  We were not establishing the correct base questions - we didn't know what all the root issues were, so we argued over the proposed solutions.  Also the arguments also revolved around the benefits of the proposals, not the drawbacks.  I wrote up my ideas, but the analysis of the problem hasn't been done yet.  That's probably something for tomorrow, but I think it is absolutely necessary.  Otherwise we'll end up with the camps of "this is newer/better - let's do this" and the "this works, why mess with it?"  Of course neither camp is that extreme, but hey analogies aren't fun if you don't push them a bit.

Anyway, I'm going to construct an analysis of the proposed solutions and identify if they can each meet the requirements and then see which one does the least harm.  Maybe show which provides the best benefits with the least effort - both good criteria.  I've been saying for a while now that the best way to evaluate a process - because that's what these solutions will be - is to pick the one that makes it hardest to wrong thing and easiest to do the right thing.

That is kind of tricky in the software business because you can change so many parts at once.  For software processes, it's important to pick the process/tools that make it hard to do the wrong thing and then make your process fit the process implied by the tools.

You may have noticed something controversial a few paragraphs back as well - that we were picking a process.  It doesn't seem like it because the choices revolve around different tools, but each tool implies a different process.  Picking a process implies picking the tool to use.

Wow - I'm writing this in a very abstract way.  I hope it's clear to other people what I'm trying to say... It's all that math training over the years - I tend to remove the parts that are irrelevant to an argument and replace them with variables or generics (when writing).  The more involved the process or tools the more abstract the references become so we end up with this gobbledy-gook.  Whatever - I wrote up something interesting to me and that's why I'm doing this.  You can read for your own reasons.  Individuality springs up everywhere.

Tuesday's grey

No prize for who ever can place today's title quote.  I was going to reveal yesterday's reference here, but I'll add a comment on the post instead.  Not much happened today - I didn't make it to the gym but my throat is scratchy, so it kinda... Well I thought I'd pay down some bills today and maybe some debt, but didn't.  The debt didn't increase or decrease so it's an "unlose" situation, double-plus unlose in fact, according to my wife.  Not me though - I think it's pretty good.

The annoying thing related to paying bills were the polite letter I got for my line of credit.  Interest is charged to this line of credit by taking the bank prime rate plus some amount.  In recent months, the Bank of Canada overnight rate, the rate that banks are charged when they borrow money, has been falling.  As of this writing, this rate is at the historic low of 1%, while the bank prime rates are at 3%.  Well, according to this letter, the part that is added to the prime rate to determine the interest charged, is increasing.  I guess the banks really need to get a particular rate of return for their investments, so they can just nudge the rates up in such a way that the total interest that I'm charged remains constant.  So much for falling interest rates.  I probably shouldn't complain - if this was 1989 they'd probably be calling in the entire line instead of bumping the rates up a bit.  But still, it 'tis annoying.

An interesting article that I finally got some time to read was over at Geek In A Suit, the blog by Christian Gruber.  I talked with Christian a few times in his role as Agile consultant at my place of work.  The entry about testability and Object-Oriented development really hit some good points.  Especially how we, as developers, forgot all our O-O design education when we started working.  I remember one of the early classes (early in the morning, early in my university career, etc) on the software life cycle where the prof went out of his way to show that maintenance was by far the largest part of the cycle.  That initial development was small and that testing and what followed would dominate.  And that was the waterfall model.  So, over the past few years, my company has been exploring Agile methods and it's given me time to consider how we develop software and what I should be doing to create the best software.  It has shown me that testing, writing testable code, documentation, automation, KISS design are all very important.  One thing my feature team has discovered is that writing tests should take at least as much time as writing the code, especially given that there are two or three types of testing that have to be done.  Also, we cannot consider test code to be throw-away, second class or ignored.  We have one coding standard, which includes guidelines for design and documentation, and it applies to all code created.  Code reviews are done for all code.  Since tests are code, they are part of this process.

Something else that Christian touches on is the role of design in an agile process.  It's touched on a little bit, but not enough.  The emphasis in Agile is on iterative design, on incremental things, on people over process and I think people follow the buzzwords instead of doing "what's right".  Design is an important consideration, as I think many jump into the work because that seems like the right thing to do.  I believe what Agile is trying to impart is that design should be as simple and minimal as possible - that's the only way to have robust, flexible code.  The design should be map of what to do in a project.  It should be the five lines on a napkin map, not the satellite photo map.

Day 30 - The end of January

Pretty good day - we had a successful demo, though all I did was bring Timbits.  Something that our company adopted and I feel is an excellent part of the software lifecycle, is the demonstration of the completed feature.  This probably doesn't happen often enough, but whenever it does occur, I think it helps bridge the gap between the line-items that managers and higher-ups have spent so long talking about and what the ultimate user will see.

Something Agile-related that I heard this week, but less positive than demos, were questions from a status meeting.  The question centered around whether we (as a company) should mark a particular feature done or "at risk".  The reason for the question is the trouble - a defect in third-party software.  To make the issue more accessible, let me pose a question: Alice and Bob are writing software that must be combined to form a final product.  Alice's portion requires Bob's to work.  Is Alice done only when the final product works?

I'll let you think about that.  That was probably long enough, or you're impatient and don't want to think about it.  Either way, if Alice isn't finished until the final, combined product is working properly, then Alice is responsible for Bob's piece as well.  Not directly of course, but if Bill wants to buy the final product from Alice, then Alice is responsible for the whole product working correctly.  If Bill combines the piece from Alice and the piece from Bob, but Alice says her part isn't finished until the combination works, Alice is assuming responsibility for ensuring Bob's piece works, even if Alice has no influence over Bob.

That explanation isn't really any better than the original description, but what I'm trying to say is that the questions in my workplace tell me that there are some people that feel that our company can't be finished until the end product works.  I have a big problem with this because we are in the middle of a software sandwich, with one company building hardware and the first layer of software, us, and then another company builds the portion that sits on top of everything else.  If we can't say we're done until everyone else is, we've failed if any of the other two companies has a problem.  But our company can't alter the software from the other two or even influence them.  We'd be taking responsibility for things we can't change, like being responsible for bad weather.  Sure it makes things less nice, but we can't alter the weather - how can we be responsible for it?

After that rambling description of the problem, I'd like to state that I understand the motivation behind the statement.  It is motivated by a desire to make the best possible product.  However, being part of a compound product means that all we can do is our part as good as possible and help identify areas that need help in the other portions.

Had a pretty nice evening - brought some pizza over to my sister's place where the kids were having a good time, but not in the same room as the adults.  Makes life so much better when there is an appropriate amount of distance between the two groups - too close and you drive each other insane, too far and you wonder what's going on (or get lonely - haven't run into that yet).

Day 29 The End Approaches!

The end of the month that is.  Then I won't be labelling these posts with numbers... I almost feel lost just thinking about that.  At least I've been able to keep this up in a reasonable manner.  Like many things, practise makes the entire process smoother.  I was going to say easier, but that isn't exactly correct.  Maybe it is right - my typing speed is increasing and my initial spelling mistakes are decreasing.  The hard part, as always, is the content.  I sound like some kind of beleaguered television executive.  I like the description of the writing process that a recently ex-co-worker described: "Staring at the blank page until the blood squeezes out of your forehead..." It went on from there, but honestly I forget the exact details.  I think it was a matter of letting the blood hit the page and tracing the outlines with your pen and then it was done.  I assumed that editing involved a different bodily fluid, but I decided I'd have to find out on a different day.  A day when I had a lighter lunch...

Had two parts to my day at work - the beginning, which was depressing and the second half which was collaborative and productive.  The team discussed some of the upcoming work for the closing phases of the current project.  The work that was left wasn't the depressing part - it was the amount of work we saw ahead to convincing other teams that they may be missing some pieces of the puzzle.  Not that any one team is way off track, but attitudes may result in more work for everyone down the road.  Since this is the finishing stages of a project, we are doing more and more about testing.  I think some teams haven't hit the balance between things that need to be done all the time (testing-wise) and the final big-push testing phase.  We seem to be missing cross-team, integration-style testing.  Each team has a testing environment, a testing framework  and so, a testing infrastructure that is much better than even a year ago.  Some people seem to think that this testing is great and that this final concentrated testing is redundant.  There is still room for more direct testing, testing that breaks silos and testing of areas that are not a person's speciality.

I found it pretty daunting to contemplate how to convince that more work needs to be done.  It's not something that people like to hear.  The effort is not in convincing people, but in how to create a palatable argument that will lead to modest improvements.  Someone pointed out to me later in the day that the upcoming new project is like a light at the end of the tunnel, the promise of new work, the shiny porch light to mothy employees... New stuff is always more interesting than the final details, so I could understand why some sound so distracted finishing.

The productive, collaborative part of the day, which was happily about half of it, was working on the new project.  Going into these early planning stages for the new project, I was hopeful that I would be able to have a say in the design or planning of what was upcoming.  The team I'm on has a specialization in the current project that won't be in the new project, so we are working on things that are entirely new for everyone on the team.  The encouraging part was that all the people I've talked to about the new project has been easy to ask questions of, given great answers and accepted ideas and opinions readily.  So far, it feels like a team should - not everyone agrees, but everyone listens and problems are solved through consensus.  The test of such an environment comes later when there are other pressures, sort of like starting a business.  You can't say you have a good business until you run into hard economic times for your area - if you can survive that, you have a good business.  I believe there is an old saying "gold tested in fire" which is the same idea.  Three teams came together over the past week and we have sketched out a design "framework".  

I hate to call it a design or even a plan - more like a back-of-the-envelope drawing.  From that, the work that needs to be done can be identified and sized.  Now an iterative design approach takes over, with prototyping and measurement being the basis of what comes next.  So I don't consider this a plan or design - more like an agreement on the rules.  Rules in a game provide a framework with which to operate inside of.  Too many rules and there is basically only one way of doing things.  That's a heavily, tightly, specified design.  The designers do all the work, the implementers do little more than data entry.  Such designs and processes are very brittle - if they work, they can be very very good, but any missing requirements or specification flaw can shatter the result.  I believe that the best designs place trust in the implementers to create something that meets the requirements ("the ends") without specifying how to do that ("the means").  A minimal framework is necessary to allow many people to work in common purpose, the balance being how much detail is necessary to keep people together.  This is something that depends entirely on the specific group. 

To use some Agile terminology, evaluating the people is important to tailor the process.  One of the pillars of the Agile Manifesto is "People over process".  I believe that too many read this as "people no process", where process becomes a bad word.  I think the proper reading is "processes work for people, not the other way around".  Good processes help people work more efficiently, bad processes are actions that are mandated but don't advance the project.  Taking this analogy further, the skills of the "people" dictate the processes.  The more skilled, self-motivated, adaptive the "people" are, the less there is to the process.  Many Agile descriptions end up like this because best people can take a one sentence directive and turn that into a productive working environment.  Not everyone can do that, so more words are necessary.  Or there may be many words to start with, but eventually the descriptions (processes) are reduced to simple phrases to remind everyone where they came from.  The design of software should also follow this model - the design depth will be dependent on the people doing the work.  It is a mirror of the Agile methodology.  In Agile, the result is the running of a project and this is done by the people.  For software, the result is the program which is created by the people.  In both cases, the processes that lead to the result are dependent on the people, with lighter processes being used by more capable groups.

Having written all that out, I think that's why I was so pleased with how things were working today.  The design that came out of the meeting was enough to base the next 6 months off of, but can be written on a single piece of paper.  I think everyone left the meeting feeling like something had been accomplished and agreed to, but it was simple and more of a guideline or framework than anything else.  The ease with which this was achieved was great.  I hope things continue in this manner for the rest of the project.  Hope is important.

Day 6 - Is this a habit or a compulsion???

Working towards a habit with this here writing thang. I should really pursue better sleep habits, but whatever. Gotta hit the hay early tonight to be fresh for hockey tomorrow. It's really hard to concentrate when one hasn't had enough sleep. The worst part is that internal censor tends to go on extended leave when you're more tired.

Had an interesting team meeting today - my manager outlined what is going to be happening this year, as well as he knows it now. Like most things, the year has begun and the plan isn't finished, so we start with an outline and work our way towards having it done. Anyway, he gave some interesting insights to our "big boss", basically drilling into us that he likes to 'manage by data' instead of 'managing the data'. See that subtle difference there? Slick!

I really can't do sarcasm when I'm this tired, or when I'm writing - lacks the whole tone-of-voice and eye-rolling-boredom look that is necessary for sarcasm so obviously weak that it can do nothing but become an instant parody of itself.

Anyhoo, I've met this particular 'big boss' and really liked what he's had to say at various meetings. It didn't hurt that he was the one that delivered the "despite this whole market kablamo, every one's job is safe" talk. It's nice to be using the internet in your own house that one can continue making payments on. Nice. (Wow this sarcastic parody of this blog post keeps cropping up - my subconscious must be trying to tell me something...)  That wasn't behind my good opinion of him because I decided on that a long time ago.  He has an excellent track record working for highly technical companies and he started from an actual engineering position.  Plus he's sat on IEEE standard's committees, which is something I can respect.  Especially since he sat on the 802.11 series of committees.  Lasting through one of those would be quite an accomplishment simply because of the competing money, er companies that are trying to come up with something that is standard, but more like their standard than anyone else.  Everyone should be amazed that something emerges that everyone pretty much agrees to.  Anyone that can help move that kind of crazy mess forward seems like the kind of person that can sort out the mess that is the relationship between our customer & owner (they are the same entity).

I'm hoping that this 'big boss' can help our owner provide us with the information we need to create the products they want in a timely fashion.  There is lots of communication happening, but something is missing to make the whole thing really click.  Our company needs to change too and the fact that the 'big boss' is addressing both sides of the problem at the same time is really encouraging.

Did I mention that when I'm tired, I ramble on with crazy segues around the topic I was on?  I saw that Qnx has an embeddable Java VM that runs with its microkernel OS.  What I wouldn't give to slap that into some of the hardware we have at work...

As I was saying before, looks like the 'big boss' will be looking towards hard data to determine what is going on.  I think this is a good thing because it will bring us closer to repeatable and coherent processes.  This is the kind of thing that could be done in a scary way, but as I mentioned above, I like what the 'big boss' has done before and I'm optimistic.

I think some at my work may be worried about this 'big boss' because he isn't a big proponent of Agile - in fact, many of the ideas he's expressed are decidedly non-Agile.  I'm thinking of the 6-month product cycles as an example.  First, I'd like to point out that just because the entire company isn't 100% Agile doesn't mean our corner of it can't operate in an Agile fashion.  Secondly, I've made mention of processes a few times already and the Agile manifesto does say "value people over processes".  Some read that as "process == BAD", but I think that's limiting.  Processes can enable people - just like learning certain martial disciplines.  A student begins by copying the precise set of forms and movements.  Once those become instinctual, then the student is encouraged to explore the reasoning behind the precise steps.  Finally, the student moves beyond simple instruction and adapts the movements to themselves, expanding and refining.  Processes in an agile environment should serve the same purpose - to provide enough structure for people to operate in with the understanding that one day they may outgrow it.

Okay, guess it's going to be time for that whole sleep thing.  Soon.  Because if I keep this up much long and I'm just gonna pass out here and what little semblance of sense that these words have will evaporate.

Day 4 - Prepare for...

Wow - four posts, four days! It must be some kind of... pathetic attempt at something. Anyway, today is the day I'm preparing to get back to work and preparing to try out some new hockey equipment.

It's a new year at work. We've shipped product into an actual market, we have loose ends to tie up and I have to endeavor to make things better for all. Complacency is an enticing and dangerous state - we're an "agile" workplace, but I don't think I did enough retrospecting. Now is a good time to look back and think about I could have done better. Team-wise, I think things went quite well, except for a lack of retrospectives. The team is cohesive, adaptive and gets things done. We've tried, in our own silo fashion, to develop practices and methods that result in better output. Towards the end of the year, I gave a presentation on one of these practices. Another team member gave a presentation on the technical details of the features our team maintains. This kind of outreach is the type of company-wide connection that I'd like to promote. I haven't done enough to make that happen, other than complain - something I know I'm pretty darn good at already. It's time to branch out from complaint to action.

Optimistic words for an optimistic time of the year. Now on to my other preparations - hockey related. I got myself a new helmet and a new pair of skates. My current helmet, although quite nice, was beginning to react poorly during the course of a game. I suspect, (but can't prove), that certain pads were absorbing sweat and slowly crushing my head. Not good. I have the dents to justify these suspicions and the cage was getting rusty, so it was time to move on. I bought a cheaper version of the same helmet, with nice smooth pads. Hopefully this will provide the protection and lack of soft padding that was SQUEEZING MY BRAAAIN!!!

I also got new skates, which I've been promising myself for a long time. I wanted to get a pair of high-quality, modern skates. I asked around and Graf was touted as the best, so I've been investigating for a while now. I went to Pete's Sports to see if I could find someone that could fit skates properly. Graf is not a volume brand, so I was confident that any store that actually carried them would have suitably trained staff, and I was not mistaken. Graf is known for their comfortable skate and their range of shapes - capable of accomodating a very wide range feet. I have small feet, so they are hard to fit. Because they are so small. Anyway, this turned out to be an advantage as I fit best in a Juniour size, which cost less than the Senior sizes. Also, I ended up choosing a pair that I probably couldn't break in if they were a Senior pair. Turns out, it's not a great idea to go into the nearest sports store and buy the top-of-the-line skate. Usually they are so stiff that you'd have to play in a competitive league to break them in properly. So remember that when buying your next pair - save yourself some money and some pain!

Anyway, buying was good, but today I prepare to actually use them. It's going to be different because they are a size-and-a-half smaller than my old ones, so the blade area is much decreased. On the other hand, they are lighter, so maybe I'll be able to pump those legs a little faster. Now I just have to look into that extra padding 'round the middle...

Starting groups

My workplace moved to Agile development methodologies more than a year ago now, though it hardly seems that long. One of the techniques that was used in a few situations was to form a group and then allow the group to create the parameters, purpose and direction for the group.

This idea works well when used in the appropriate situation, but I think it had a less-than-optimal effect in some cases I was involved in. I'll describe those situations, propose an alternative group-start and finally end with where the "self-determining group" technique should be used.

The first time I encountered this technique was in the early introduction stages of Agile methods at our company. Some of the meetings were held using this technique to get the group to direct the presenter to the items that the group needed more information about. This did not work for two main reasons:

1) The audience was mainly people who knew nothing of agile methods, so had a poor understanding of the technique. Not knowing how to use the technique made things awkward.

2) The purpose of the meeting was deliver information to a group that is used to logic and structure (software developers). Going into the meeting, they all knew that they would be getting more information. Being asked to discover what the information was without knowing what it was seemed redundant.

Together the impasse is clear - the attendees were obliged to attend a presentation but were confronted with an unfamiliar process. I think such a meeting would work better a year later - the patterns are comfortably familiar.

The second time I encountered this technique was in the formation of an Agile "group". In this case, the technique was applied pretty successfully, but looking back I do not believe that it is the most efficient way to proceed. In the first meeting, this group had a purpose and dove into that head-long. By the second meeting, a split in the purpose became apparent. This was handled in an open fashion, quickly addressing the difference and the group moved on with an adjusted purpose. I think that the initial problems came partly from the fact that the group wasn't entirely self-selected. That is, some members were asked to attend instead of choosing to attend. The other reason for the slow start is that it seemed to take several meetings to work out all the details of what needed to be done.

What I hypothesized this was that a middle ground is needed to make things more efficient. The concept that the group needs to be self-determining is essential, so nothing should interfere with that. The only place left is in the initial group setup. Instead of proposing a group, organizing a meeting and then having the group decide everything from there, I think the initial proposal should include a sample framework. This would give more clarity to the purpose of the group and possibly save time if the group likes the initial ideas straight-away. If not, the group proceeds as before. I'll give a high-level example and a more concrete example.

The high level example is the idea of democracy. Some would like all decisions to be determined by vote, much like the earliest democratic ideals. Each citizen would have a vote and all decision would be taken based on voting results. Operating a modern democracy in this manner would be difficult because of the overhead. Instead, representative democracy allows representatives to work at some proposals and refine them before bringing them before the people. The one-vote-per-topic idea is like the self-organizing groups.

More concretely, let us say I'd like to start an Agile book club. The self determining method would see me send out a message saying "Agile book club is starting. First meeting Monday." The group would meet and begin the proposals for what the book club is about. Using the "proposed structure" method, I'd send out a message saying "Agile book club. Initial idea: Critique of sci-fi/fantasy books during weekly round table meeting. Please bring your proposals and ideas to the meeting Monday." This message conveys more information than the first, although it may bias what people think the purpose is. However I submit that if the attendees are comfortable in the Agile process, this would not prevent them from bringing their own proposals.

I think that the "proposed structure" methodology would work good in transition from a traditional work environment to an Agile one. With enough experience, the "self determining" method would work well, but until some additional skills are learned and techniques refined, it could be slower. The idea would be to encourage dialog and participation, but the mere lack of structure may turn many off. You need people there before you can accomplish anything.

If you've made it this far, you may have noticed how loosely tied this all is. I'll try and refine this idea in subsequent posts, but I apologize for my late-night incoherency. I did, however, feel it necessary to record the outline of the idea anyway.

Am I missing something?

Probably. Moving into the role of Feature Lead at work, but I keep getting the impression that I'm forgetting to do something, or that someone is expecting me to do something I'm not... Don't think that feeling will improve, as I have been able to rely on the previous Feature Lead. Until tomorrow because that's when he's incommunicado for a few weeks. Guess there is no learning like being cast into the thick of it.

Trying to tease out what exactly the role is. We've been told that a Feature Lead is a role, which is a fancy way of saying "do more stuff with no additional financial compensation". That part was actually spelled out a few times too, just in case some might think they'd be getting more out of it. No, the rewards are more in the opportunity itself - the chance to build soft skills, talk to more than the bottom of your coffee mug and possibly even make a difference in someone's life (note: someone that is not you). The tasks that I had a good idea about before have been easy to pick up. Funny how it is easier to do something when you know what that something is, but that's never really stopped anyone before. The things I worry about are the personal connections and implied relationships that the previous Feature Lead built up.

Guess I need to get a better feel for how to gauge priorities, who to talk to and when to talk to them. That's the entire role I suppose, so I still have much to learn. What I have been trying to do is to act more as a guide to authority. The pre-Agile culture had rigid hierarchys, so someone who told you what to do generally had authority (responsibility) over (for) you. The ideal Agile culture has a flat hierarchy with people seeking expertise and information directly. Roles like a Feature Lead are there to enable such flow - the Feature Lead needs to help the team connect with the right people or find the right info. They don't provide the information nor act as a proxy for other people. I think, but I'm not sure that, this boils down to the Feature Lead delegating (reflecting?) tasks.

For example, if a team member comes to the Feature Lead with a question about a component, the Feature Lead would suggest talking with a member of the group responsible for that component. The Feature Lead doesn't need to know the information so it isn't useful for the Feature Lead to find it out. The team member needs the information, so they discover it directly. This takes some time initially, but further queries can bypass all the people that aren't involved, making a nice, efficient work flow.

The catch is, the feature lead needs to keep track of what is going on, but at a level that doesn't require detail. That's the balance I'm still seeking. The other task of a Feature Lead is to interact with other teams or people that require status, information or work from the feature team. That's the other part I am unsure of yet. No problems interacting with the people that seek me out, I just worry that there is someone waiting for me to show up and provide information. And of course I don't know who or where they are... but they should work harder to seem me out gosh darn it!

Anyway, enough angst for now. Things are starting and haven't collapsed, so I'm happy. But yet strangely tired...

The Agile Emporer of Dune

Had a nice chat with a work colleague today, someone who generally is too busy to be interrupted by the likes of me. Happily today was not one of those days and we were able to converse about what we were doing, books and Agile. Turns out, Brian Herbert, son of Frank Herbert, has written another book in the "Dune" line and this time it's not a prequel. Guess I gotta get ahold of that sometime. However in preparation, my colleague was re-reading some of the older books and decided to start with "God Emperor of Dune", which is probably my favorite book.

I like this book because of the tale spun of the cycles of history - of oppression and bureaucracy and of leadership. That is one of the key themes of the book and I especially like the description of what separates a good leader from a great leader: a heartbeat. The great leader will make a decision in a heartbeat, while a good leader will hesitate. The great leader doesn't hesitate because they know that the decision taken is more important than if it is the correct decision. With the right people reporting to you as the leader, a less-than-perfect decision can be identified and corrected. The people under the great leader can all be trusted to make their own decisions and bring information up to the leader, good or bad. Some leaders will surround themselves with people that will bring "results", but that usually means the bad stuff will be avoided and hiding information is never good.

We both agreed that the principals behind an Agile workplace would seek to produce the great leader, where information is shared and decisions are made rapidly and can be changed easily when it becomes necessary.

One of the things that Agile seems to promote is a recursive pattern - a fractal, self-similar ideal. Each individual should be a great leader, but their underlings may be their email client, web browser and senses. But the feature lead, the product owner, the executive, the customer all these would ideal have a great leader in that position. There may be a chain of reporting, but each link looks like the one on either side. Information would flow up and down the chain. Decisions and compromise would flow as easily as information. What a blissful utopian dream (I gotta be a glass-half full guy).

A couple of things spring to mind - comfort and creating comfort. People, in general, would not feel comfortable to make decisions and operate in the ideal described above. Comfort being the key - they won't be able to perform at their best unless they are comfortable. To help that along, we need to provide a comfortable environment. That means a work place that everyone enjoys, but it also means predictability. We need to make sure that we, as a group, provide enough structure so that things operate in a predictable manner.

Further reflection suggests that such structure can't be imposed, only developed. Meaning that it is a cultural mechanism and the company culture has to be gentle nudged in a way that makes things more Agile while still feeling comfortable.

Well, the thoughts are jumping around a little more now, so I'll take that as an indication that I should stop. If it were possible to type coherently while I slept, I suppose I'd have more blog entries but thousands of commas don't really make for good content. Plus drool is hard to remove from keyboards.

The Agile

Yet again Kimota94 describes something that I feel I must comment on at length, rather than pollute his nice comments with my long-winded explanations. In this post, he comments on something that happens around forecasting.

I feel that there are two areas where the company has had trouble changing - the first being the idea of top-down description of what exactly needs to be done and second the issue described, forecasting versus planning.

Really these two items are closely related. Anthropomorphizing for a moment, the company is like a person who make rigid plans for going to see a movie: leave home 5:45, dinner start at 6, theater by 7, see {insert favorite current flick here} and home by 10. The more Agile company would be more like the person who says "let's see a movie tonight and catch a bite to eat." They meat (meet?) at the restaurant, go to theater and find the film is sold out, etc etc.

The second person is one that starts with general plans, but only fixes the items that absolutely require it. More easy-going, shall we say. The first person is one that must have every detail planned out, but suffers if circumstances change.

The "old way" was one where the implementors where told about the plan and told when things needed to be done, and what those things were. This is both the "top-down" sense and the "detailed planning" sense. I believe that the Agile ideal would be that vague ideas of what and when would be handed down, but then refinements would occur driven by the implementors. New circumstances would be presented from the "top" (customers, managers, etc) but details are established through dialog. The forecasting would be similarly vague because the implementors haven't driven through the details yet.

Note that these are the sleepy ideas of one who is typing mainly because stopping would be sleeping, so if it doesn't always make sense, it's not you.

The summary: everyone has to start finding out more answers rather than wait for them to be delivered (minimize the top-down-ness) and our partners (customers) need to be okay with vague plans a few months out.

The Agile Report

I'm not part of the group that has been steering our workplace towards an Agile methodology. I haven't participated in the training nor read any of the books. I don't advocate one methodology over another. I have read the Agile Manifesto and read the Principles of Agile Software. I have been experiencing the changes and observing the happenings. And so I feel that I can provide a different insight or set of observations. I've posted before on these observations, but I figure it is time to devote another post just to some current observations.

Things seemed to have moved quite well since the feature teams were formed. At the beginning, the biggest worry seemed to be that there may be resistance to the inter-disciplinary nature of the teams and I think a lot of effort went towards avoiding that. Turned out to be a non-issue - I can't think of any team that has any real issues with that. I don't know all the teams, but I think real problems would have stood out by now.

Communication has been a bigger issue. Communication between teams and between the teams and other entities (customers or product owners). Problem is pretty vague - specifically it seems to be something that has been overlooked, not done "wrong". Up to now, the teams have been working on getting the process right, being comfortable with themselves and trying to produce. Now, as some hard deadlines loom, short-comings are revealing themselves.

I think that the pressure and immediacy of certain milestones is pushing our Agile environment to the next level (whatever that is). I think that communication is the problem and that getting through this next little while will improve communication. It will also be a lesson in what and when to communicate, so that the right information is exchanged sooner.

This additional pressure also highlights communication issues between teams - people are feeling the heat and may cast aspersions too easily. Again everything I have observed has been cleared up with a few quick conversations.

So the bottom line: pressure seems to be rising, but the teams are rising with it. The result will be improved teams.

Retro... Specting...

Don't want to catch that 90's reference (Rico Suave)? I won't hold it against you. Had our latest retrospective for our Agile team today. It was pretty cool because some of us complained about the retrospectives and our feature lead (this guy) looked into changing the way it was run.

First off, when I say "complained" I mean that a few team members (me being the first - can't keep that mouth shut) started discussing some concerns with the way previous retrospectives were held. I felt that we spent a lot of time engaging in hand-waving exercises that distract and relax the participants and help generate data. I also thought the results of those exercises were not as good as they could be. Others voiced similar concerns - mainly that it was a lot of time invested for little return. As a group, we were generating action items from these retrospectives, but they seemed small return for 3 hours. Things like "we need to move our information radiators out of that meeting room and into the open". It was a good idea, a useful action item, but not worth a 2 or 3 hour investment to discover.

Another thing to note is that everyone who brought up issues with the current retrospective did not want to simply scrap it, at least when the possibility of change arose. After consulting the Agile Manager (he knows who he is) our fearless leader proposed a simpler format.

Let me just say now that it was the best retrospective I've attended.

The new format was simply this:

1. Present the results of the last iteration
2. Open the floor to ideas for improvement

There was an hour booked for this meeting, the first good thing. The presentation consisted of a simple chart that showed what items were finished and not finished over the last few iterations. Then the floor was opened up and the first item was brought up by someone who had just moved to a new team. He said "don't ever change" - meaning he felt our team worked together well, much better than his new team. Particularly how members will help out however they could, working in new areas and pitching in wherever effort was needed. Great feedback and something that helps us to know what we are doing right. Some of the items that caused problems, due to process or lack of information, were brought up and generated action items.

Eventually we got around to talk about retrospectives - how and why this format was better than previous ones. After discussion, I hit upon something. This meeting felt natural and flowed well - many items were discussed and it felt like things were identified that will lead to improvements. Everyone had their say and it happened quickly. The older retrospectives, with the hand-wavy exercises and their data gathering, felt awkward. Our team feels comfortable with itself and that sense of unease was an indication that we needed to change how things were done. We got more done in less time with this format, and I think that is also an indication that this was the right thing to do.

I have heard the comment from some other people that they spaced out their retrospectives more, that they weren't getting anything out of them. Perhaps they had the same sense of aimless unease in their retrospectives and chose to avoid them rather than change them. Maybe that is a lesson to be promulgated - it is better to tinker than to shirk. Or ah-voision is a sign that something should change. (ah-voision is a word - look it up. I don't say evasion, I say ah-voision!) Feel free to create a down-homey saying to help spread this insight. Rhyming probably will work better, jingles better still. Original music only please.

I've been learning many things about how our company is handling Agile methodologies over the last few weeks. The first was communication - do more of it. The next is adaptation - teams are self directing so if something doesn't seem right, change may be in order. Finally negotiation - more communication will not solve everything nor change dates, but may yield movement through negotiation. Surely this is not the end of Agile Insights (tm) but it is for this evening. I have gym to attend to on the morrow and the sweet surcease of wakefulness beckons...

Agile thoughts

Went through the day and had a few ideas about Agile. Nothing big or revolutionary - more like coalesced observations. The first was that our company is moving into another phase with Agile, the phase were we realize that we aren't communicating very well. With each other, up, down sideways and so on.

This comes from conversations I've had with various people and within our Software Craftsmanship group. The most pressing issues seem to arise from lack of communication, particularly from the feature teams back "up". That is "up" to the Product Owner, to the customer or whoever. One of the reasons for the move to Agile methodologies was to allow the people doing the work (the feature teams) more say. That meant more control and more responsibility. I think that the teams are working together well - there seems to be a lot of work happening and everyone cooperating. What the teams need to do, now that they seem to have an idea of themselves, is to communicate as an entity. Ask questions, get clarification and so on.

I think that's all I can wrap my head around tonight... I know there were some other things swimming around there, but nothing with the clarity of those observations. Tomorrow is another day - I'm sure it will reveal new things.

The Road to Self-Improvement...

... will lead me to drive over my own foot. Or something like that. I'm trying to work on some strategies to help get things done better at work, concentrating on me and moving on to others when I'm perfect. I think I have a lot of work to do, so don't hold your breath if you need fixin' - better start yourself now.

Kimota94 posted the other day musing on problem solving and how he had an epiphany about how to work better in a group. It got me thinking about how I approach working in a group and what I should do to improve. Now before everyone jumps on the comment-wagon to point out that I may verbalize a bit on the high side, particularly in group settings and particularly when someone has something that leads to this great episode of Futurama where the Professor says "You'll cease to exist!" and Fry says "but existing is basically all I do!" - I know. Really. I know that I do that, I understand that not everyone wants to follow me down that garden path (I imagine the path is in a garden owned by someone of phenomenal wealth and power and so it is a very large garden). That's why I've continued to blog - so that stuff has some place to go that isn't necessarily at work. If you made it this far, I have some fictional works with run on sentences that go on so far that you forget that the period had ever been invented - you'll think that an 'i' had an accident and was cut down in the prime of life, it's head lolling on the line there wasted and spent; nothing to show for it's former position of dignity and importance on that high pedestal of language which supported it - halted in a full, dead stop.

I think I've noted before that I shouldn't be doing this late at night because it only exacerbates the problem. That's why it is good work is early in the day and after sleep.

I noted in an earlier posting that I have found myself presenting ideas that other people don't follow. That used to happen a lot, but probably because I was in elementary school. As a result, I decided that it would be better if I explicated my ideas to make sure everyone was together. That lead to consensus building, but these take too much time. I find myself explaining things, possibly on the wrong level, to make little headway. So lately I've been changing my strategy to provide a solution and see if there is disagreement.

This works well because people will agree because they have no alternatives, disagree because they have a different idea or disagree because it made them realize something. The first group simply agree, but the second two groups begin to collaborate on a solution. Most importantly it keeps things moving towards any solution. Even if the situation doesn't necessarily call for a solution at the beginning, I think this would be a useful strategy.

There are flaws - some might feel intimidated and never say anything. This means that it has to be done in an inclusive, open manner - care taken in how information is presented. That's no different than any group situation anyway.

If this strategy is not appropriate, the other one I'm trying to employ is the 2-2-6 method. If I recall correctly, this is where a presentation for a group is divided into three sections: presentation, clarification and discussion with length two, two and 6 minutes (respectively). Forces the information presented to be brief and clear, otherwise there will be too many clarification questions. If the second section fails (too long, doesn't seem to be agreement on what the topic is) then you have to try again another time. Otherwise the open discussion proceeds and is limited as well. This technique is used to help keep meetings productive and bounded. Otherwise they become like posts - rambling barely coherent romps through a nether-world of infinite distraction. But I digress...

In another posting I made, I indicated that simple questions were a good thing. It goes beyond programming or development or meetings to more a general heuristic: minimalism. Simpler programs are easier to maintain and modify. Breaking down work into smaller, simpler chunks yields more accurate estimates and easier effort on each piece. Brevity during meetings means shorter meetings - and isn't that the goal of every young boy and girl? Minimalism is perhaps the ultimate engineering expression - doing the most with the least. A classic quote on design and engineering is how I'll end this:

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint Exupery

More Questions and Some Answers (but mostly questions)

Turns out I missed the point of Kimota94's post yesterday. Which, as I commented there, was actually great. I pulled out something different from that first post, which in turn triggered another post from Kimota94.

One thing Kimota94 mentioned was that he "... rarely even remember yesterday's problems today,..." At first this may seem odd that a person would forget such a thing, but I believe it has a simple explanation. As he explained, he dealt with the problem and it was done. There was no need to remember it because it is finished. The issue may never make it to long term memory simply because he generated the response so quickly. It is easier to regenerate the solution than recording the answer - an evaluation that is conducted subconsciously. I think that the reasoning provided as to why he can perform the evaluation so quickly is also very astute. I have known others who operate in a similar vein.

This of course leads me to reflect on my own strategies. I know I often engage in several of the other listed strategies. Particularly, I find that if don't explain myself enough, no one knows what I'm getting at (or what my decision was), so I try to explain myself, think out loud. I would like mutual agreement, so I try to seek consensus (approval) first.

People that know me realize that I like to talk (and write and type and blog....), but some have seen times were I present ideas or solutions in a group and no one gets it. That is a reason why I talk so much - I spend many words establishing a context or common basis for others to follow.

Today was actually an interesting example of not being understood. I was part of a group trying to get a system working on the Violet feed. The problem centered around the server portion of the system, so I didn't have much to do except test the changes as they were made. Nothing came of it. As the day wore on, I summarized the situation to the others by stating the problem and then that the solution must be in one of two areas. I explained it to someone else who looked at me for a second, turned to tap away for and then said "try it now." Of course whatever he did worked!

The problem I had was two fold: I do not know how the server side works in great detail and I couldn't describe my suspicion well enough. It was extremely frustrating to be in a situation where I could see the problem but lacked the knowledge to test the possible solutions. I correctly understood the problem and had the answer, but I had to explain that to someone who knew the server system well enough to translate my idea into a solution. When I looked back, I stated the problem out loud several times.

This is why it is great to work with other people, people who will listen and try to understand what you say and help. That's why it is important to remember that there are several approaches and sometimes one approach is better suited to the current problem.

Funny, this post didn't lead exactly where I expected. My fault for doing this too late again. I've got to try and do this earlier when my thoughts are still coherent! Anyway, I hope this is illuminating, but not in a self-aggrandizing manner. Definitely not my intention. Tomorrow will hopefully bring more thoughts with higher coherence.

The Question

Just reading Kimota94's new agile-related posting and somethings jumped out at me. The first is that I spend too much time writing up blog entries. The second is that I'm glad I don't read very many blogs or I wouldn't sleep at all. The third (and, let's face it, the real reason you've kept reading thus far), is the importance of the question.

Seems like such a simple thing and it is something I know I've even uttered aloud before, but it probably good to write about it here. Especially in the context of Agile development. Kimota94 stated that he has often found that when a question pops into his head, the answer is right there with it. I characterized that as "formulating the correct question". Once the correct question is posed, the answer is obvious. Hard problems are difficult because they haven't been broken down into simple questions. Which is where Agile comes in - it is a means to generate simple questions.

Kimota94 indicated that he discovered a new method of attacking a problem by deciding what the end-state should be and then working from there. I think of that as a "top-down" approach, where his first method is "bottom-up". Which is also why I say to people that I do top-down and bottom-up simultaneously sometimes - I constantly and instantly switch between the two approaches when I hit a roadblock. This leads to a lot of context switching overhead, but I think it leads to a better understanding. What does this have to do with Agile and questions?

Both approaches described here lead to or stem from simple questions. Some people will look at a problem and it will quickly because a question that has a simple answer, a bottom-up approach. Others will struggle because they cannot seem to make any headway at all. By creating groups of people to attack problems, hopefully someone in the group will turn a problem into a series of simple questions. These may be tasks or feature cards, user stories or test cases. Gathering these simple questions together makes it possible to answer more difficult questions like "how long with the problem take to solve?", which a product owner may ask a feature team. The customer may ask the product owners "how long to add this feature?". They use a top-down approach to break the problem into a series of smaller problems, which they send to the feature teams, and so on.

The vision that Kimota94 talks about, the top-down, end-state-first thinking is needed in some situations. That visionary thinking helps to formulate simple questions as well - things like "how should feature X interact with component Y?"

It's funny - the more I think about programming - the actual act of writing software - the more I realize that the "series of simple questions" is how I must go about it. I don't like writing code until I have a certain level of understanding as to what needs to be done. When I reach that, I write a set of simple comments (answers to simple questions, or questions themselves) and then fill in code between the comments. Each comment breaks down into high-level code ideas, which in turn break down (read: recurse) into smaller pieces.

Each player in the Agile organization should be able to conduct most of their work by posing or answering simple questions. Discovering the questions is the hard part - answering them is easy.