Agile: Organizational Balance

In my previous post Agile: Specialist vs Generalist, I recast the specialist vs generalist debate into broader understandings, considering that specialists and generalists naturally arise out of monolithic (waterfall-like) and startup-like (agile) organizations. One could say these are the default states, but these defaults do not preclude the other. The question that arose for me is this: What is the best balance between purely self-organizing (agile) and purely leader-based (waterfall) in larger organizations?

When I consider the experiences I've had with Agile, I found a very strong personal association with the Agile team and easily identify with roles, responsibilities and activities within an Agile team. I think it is an excellent way to organize a small group of 5-9 people. However that experience was embedded within an organization that had many groups of 5-9 people, all with varying ideas of what a team was responsible for and they should operate. I can't say that this is necessarily a problem, however I do not know what the best strategy the "layer above" could use to enable success for the entire group. Hence my question above - should the organization been structured as a recursive or self-similar entity where each of the "teams" were considered members of a meta-team, each group behaving as an individual developer in a simple "agile team"? Or should there be set of managers that treat the agile teams as members of their group and assign them tasks and mete out punishments?

Perhaps "punishment" is incorrect - consequences is probably better, but it is still something missing in the reading I've done on Agile. How does one handle a poorly performing team? How does one measure that output? From a team perspective, the team measures itself, but the team could be deluded or mistaken in its assumptions. I think I can see the comment postings now: "guide the team back to the right path by assigning a mentor or leader that can steer the team in the right direction." So maybe my initial recollections are not correct with respect to Agile and consequences, but it leads back to the idea leadership/management of many Agile teams: How does one make sure the projects stay on track?

One of the key items of Agile is communication between people. Probably the most critical communication junction is that between the implementors and the consumers. In a purely top-down model, the managers fill the role of "consumer" - the implementors talk to the people above themselves in the hierarchy and that's it. That abstraction can hurt the final product, but that abstraction (like any abstraction) provides flexibility: the manager can make absolute decisions and keep the project on track. I believe there must be a way to bring that flexibility into a large Agile organization without the loss of communication.

A hierarchical organization provides benefits - each layer can be thought of a leader with several "subordinates" that fulfill requests. This hierarchy does not preclude Agile - after all the best teams anticipate problems and act independently as much as possible. So I believe there must be a way to fit aspects of the hierarchy with Agile teams.

Why even bring this up? Consider a purely Agile environment with several (>= 10) teams. How would they organize the work? Each team would have to act as an individual member of a larger team. This larger team would define the overall backlog and each team member would take on work for the next iteration. Another way would be to designate one team to define the backlog and the rest to pull from the master backlog. What happens when disagreements occur? Disagreements can be resolved via communication between the effected teams, but what happens if one team is obstinate? More meetings and communication. The point is, to work in a consensus manner takes time, time which may not be available. Having a someone to that holds a "buck-stops-here" authority cuts down on this time. A decision has to be made and the quicker it is made, the better. Not everyone is going to agree with it, so just move on.

In fact, the Agile methodology is eminently suited for this quick-decision management - the organization must be able to take a new decision later if an old one proved incorrect. The benefit of this flexibility is lost if the initial decisions take too long. Have a single decision maker will speed the initial decision making process.

So what is the appropriate balance? My initial idea is that it requires someone who has authority within the company (as in the teams must heed the decisions and requests from this person), but who is not member of any team. This person would not define what needs to be done but would instead be responsible for the output of several teams. They could act on behalf of the teams in some instances, but would have to listen to the teams in others.

Yes very sketchy, but that is how many Agile things were described to me initially. The balance between authority and flexibility needs to be discovered within an organization. I believe this is a vital piece - one that was missing from the organization where I was first exposed to Agile. Moving to Agile, especially within a larger non-Agile organization, is a very tricky business. I'm just hoping to provoke some thought and comment.

Agile: Specialist vs Generalist

In his recent blog post, Peter Scheyen began a multi-part response to this comment. The blog post contrasted developer specialization and developer generalization in an agile context. The comment suggested that "... Agile robs us of specialization", and Peter's response described how specialization operates within an Agile team. I think the comment and the response come from different perspectives - one where Agile is not suitable (comment) and one where Agile is suitable (response). So allow me to step a little bit.

I wish to recast "specialist vs. generalist" to "Cog-in-the-machine vs. Member-of-Startup". Many large organizations have a decidedly non-Agile project management approach, and for many it works well. I think that specialization in a non-Agile environment is a side-effect of longevity or survival. If I were hired to be part of a large development organization, I would start off as a faceless cog - something that is interchangeable with any of the other cogs. The longer I stay in the organization, the specialized I will become, whether by accident (I worked on system X since day 1!) or design (I became the go-to guy for System X). Cast in pure evolutionary terms, this is a survival tactic, so the mere fact that I am still there (with the corresponding higher costs and "slowness") is because of something that makes me stand out from the faceless cogs. Specialization becomes inevitable.

The other extreme occurs if I were to be hired to a startup. In a startup, I may have some special skills, but everyone must work to pick up any work that needs to be done. In the startup, there is a chronic lack of staffing because the organization is short on resources. If things get left behind, the whole organization fails, so a weak team member is harder to accommodate. Generalization becomes inevitable.

This brings me back to the post: the ideal Agile team described is closer to a startup than anything else. The ideal team works like a startup, identifying and filling any gaps, regardless of specialty, but still leverages the team members with the greatest knowledge in any area.

I wrote a comment to summarize the above and it got me thinking about Agile organizations and more traditional organizations. The Agile team described in Peter Scheyen's post is ideal - the team has no distinct leader, just a set of developers with various skills and specialties that are engaged as needed. This works fantastically well when the entire development organization is "small" and the developers buy into the model. In the more traditional environments, there were team leaders, managers (at several levels), directors, etcetera, that had two roles: assignment of tasks and responsibility for the project. This is a very top-down model - the "top" decides to do something and the requests filter down to the ultimate workers who make it happen. If something goes wrong, there is a chain of responsibility up to the top that can easily be identified. Contrast this with the Agile team described - it is bottom up, so the team as a whole is responsible for getting a request done, but it has always been a bit murky to me on how a large Agile organization with many teams assigns tasks.

I don't want to make this post too long, so I will save this for my next post(Agile: Organizational Balance): How does an organization find the balance between purely distributed (self-organizing Agile teams) and purely leadership (top-down, waterfall) of large projects?

Excellent!

This posting was so good, I had to copy it here.

Scooping the Loop Snooper

an elementary proof of the undecidability of the halting problem

Geoffrey K. Pullum, University of Edinburgh

No program can say what another will do.
Now, I won’t just assert that, I’ll prove it to you:
I will prove that although you might work til you drop,
you can’t predict whether a program will stop.

Imagine we have a procedure called P
that will snoop in the source code of programs to see
there aren’t infinite loops that go round and around;
and P prints the word “Fine!” if no looping is found.

You feed in your code, and the input it needs,
and then P takes them both and it studies and reads
and computes whether things will all end as they should
(as opposed to going loopy the way that they could).

Well, the truth is that P cannot possibly be,
because if you wrote it and gave it to me,
I could use it to set up a logical bind
that would shatter your reason and scramble your mind.

Here’s the trick I would use – and it’s simple to do.
I’d define a procedure – we’ll name the thing Q -
that would take any program and call P (of course!)
to tell if it looped, by reading the source;

And if so, Q would simply print “Loop!” and then stop;
but if no, Q would go right back to the top,
and start off again, looping endlessly back,
til the universe dies and is frozen and black.

And this program called Q wouldn’t stay on the shelf;
I would run it, and (fiendishly) feed it itself.
What behaviour results when I do this with Q?
When it reads its own source, just what will it do?

If P warns of loops, Q will print “Loop!” and quit;
yet P is supposed to speak truly of it.
So if Q’s going to quit, then P should say, “Fine!” -
which will make Q go back to its very first line!

No matter what P would have done, Q will scoop it:
Q uses P’s output to make P look stupid.
If P gets things right then it lies in its tooth;
and if it speaks falsely, it’s telling the truth!

I’ve created a paradox, neat as can be -
and simply by using your putative P.
When you assumed P you stepped into a snare;
Your assumptions have led you right into my lair.

So, how to escape from this logical mess?
I don’t have to tell you; I’m sure you can guess.
By reductio, there cannot possibly be
a procedure that acts like the mythical P.

You can never discover mechanical means
for predicting the acts of computing machines.
It’s something that cannot be done. So we users
must find our own bugs; our computers are losers!

It's been...

... a wide spectrum of experiences. I didn't want to just say "fun" because it hasn't always been fun. But then again, it is work, so fun is an added bonus. The company I have been with for the last 8 years (plus a few more as a co-op student) is closing down. I've been trying to think of some words of wisdom, summaries, histories etc. but I don't think the time is right for that just yet. Instead, I found something in this article that nicely illustrated what I enjoyed about "the old days" at work.

The article is an ACM Queue interview with one of the original designers of the ARM chip, Steve Furber. During a description about why the original ARM chip was so small, power efficient and cheap, Mr. Furber commented that:

This is good management retrospective: by depriving us of resources of any sort, they forced us to make decisions in favor of simplicity. -- Steve Furber

There were many parallels to BBC Micro and my old company - both were small, had limited resources and was an outside player in a potentially large market. In those earlier days, trying to do the impossible seemed like the only way to survive, so we went for it. They were exciting times and it worked - for a while. The later down hill slide is where my old organization and BBC Micro differed. I believe the small size and lack of resources resulted in steps forward that could not be duplicated by the company we became.

Untitled

Wow - I was reflecting on the day trying to come up with something interesting, and I really don't have much that is worth discussing. Even vastly one-sided discussion generated entirely by me. I had to start writing though, so I tried to focus on this, but I'm tired. I'm really glad I didn't try and start playing on the PS3 'cause that would have been a giant time-suck.

I really thought I'd be able to get some stuff done on the wired home network, but no dice. I enjoyed hanging out too much. Then I was tired. And here I am. Guess this means I should pack things up and go to sleep, so I will.

Weird...

I'm really wondering what I'm doing in an "online" sense now. I have a blog, a twitter account, email, web and *nix powers, but what I have I done with them. More importantly what should I do with them? Rhetorical for the blog of course because here I am...

The most prolific blogger I know (in real life) tends to write about what he was doing in a day, things he's done or will do, thoughts on this or that. Same with most Twitter postings I've bothered to look at. First I'll admit that I haven't tried to read many tweets, but I use Slashdot like Twitter in a sense. But what I am doing? Other than practicing my rhetorical questions? When I try and start a rhythm going with daily posts I tend to write about meta issues, a sort of Sienfeld of blog posts, a blog about thoughts about blogs, endless self referential loops that eventually just peter out. Kind of like modern culture. Huh - talked myself into the mainstream. I'm the everyman, talking about talking about nothing, endless repeats and references. Yes I do get the irony that I am the one seemingly decrying pop culture references. I am the one with a Simpson's or Futurama quote at the ready. I'm not applying morality to the our-culture-is-a-self-referencing-house-of-mirrors, but I can't see it as being entirely wholesome either.

This is the sort of thing that will cause people to rise up and cast of the shackles of their oppressive machines, machines made in their own image, in a kind of Butlerian Jihad... And yet I can't keep from referencing something somewhere. It must be a crutch, a bad habit that I've developed over the years, that if I can't tie something I say to something that's come before it can't be good. That kind of ideology doesn't stand up - it doesn't make logical sense. I would hope that I could one day make something that isn't embedded in what came before - a wholly new thing, a creation I can point to and say "I thought of that. I was the first one and I can prove it." Yes it's wishful thinking, but apparently that helps you live longer. Not sure about stream-of-consciousness writing though. Not many studies there. Don't know too many people that do it, although my sister's boyfriend really does an excellent job with the ol' Facebook status updates. I'm more of a "open the faucet and let's see what comes out" rather than a couple of words slapped together, held only by a tenuous 'now' association... Even my gibberish is getting sloppy now - better move on.

Let's pretend that last paragraph didn't happen (if you didn't skip to here, I can't really take back that last paragraph. You've read it, you can't unread it... And don't bother trying it's too messy). Many have predicted "have and have-nots" future, with some sort of technology separating the two groups. Money used to be the easiest and most visible, but communication and computers has been a popular sci-fi theme in the past. It really seems like it is approaching, although there may be some different ways to classify people. Right now, the vast interconnectedness - searchable interconnectedness - provides tremendous benefits. The "always-on" society isn't really a burden, it's an analogy for humans that can communicate in parallel with what they're doing. Think of Twitter - you shout to the ether every time you start a task. Almost like you're a lawyer making notes in your log book for billing purposes. The act of communicating isn't really an act or effort - it's a side effect. But there are tremendous potential problems. The biggest is with power.

Electricity is the one thing I've been worried about being without for any length of time. It's the fungible energy and it makes all the fun things work. For me those things don't have to be connected to other people but whatever. People (and their homes) are starting to get to the point where they don't work right if there is no power. The techno-haves can't really operate without electricity. Many are helpless when the machines don't work. The article I read about trivia games and the internet become more clear - people don't have to remember things so they won't. If they can look it up online, it's good enough. I know I've said similar things in the past, but the problem is how to evaluate the information. It makes the most sense to put this in a social context.

Let's pretend that you weren't using a computer to find out the answer to a question, you could only talk to people. People who weren't using computers... *sigh* Anyway, you ask the same question, possibly in slightly different ways, and then you gather the responses and come up with an answer. If you are wise, you solicit several answers, hopefully from people you trust, and look for the common answers. The value of the response is based on your trust of the source and your knowledge about them. You can also evaluate the response to the particular question by watching and listening carefully to the answers given. Similarly, if you don't have an internal knowledge core, how can you possibly evaluate answers discovered on the Internet. Simply looking for the most common answer is a perversion of the "wisdom of crowds" idea. Playing games with internet search engine results started shortly after money was mentioned in the context of the internet. The "wisdom of crowds" idea works, much like free market capitalism, in a largely bias free environment.

So? Well, you've read this far (I'm impressed - I only read this far because I'm writing it. I didn't really edit it), so I should tie these threads together. The bias toward technology and always-on-communication can go to far, creating an insulated self-referential world that becomes your entire culture. This would be a "bad thing". Learn to operate without. That is a sentence and I believe it, rotund technoman that I am. Balance is elusive be one should seek it. Balance between operating with and without, connected and not, electricity or no.

Maybe tomorrow the diatribe will be better. Coherent. Brief. Illuminating. Less self-referential (doubtful).

Contractual obligation?

I guess this is like a contractual obligation post - if I'm going to write every day, I've got to do it before midnight :) Ah well - I guess that's just how the holidays go; I don't really find myself with any kind of schedule. Once the major holiday gatherings are done, then it's chores and errands that don't normally get done. Plus lots of gaming. Got the ol' skates sharpened - 5 pairs this time. New record. My sister and nephew went skating with us before Christmas and I'm shocked she could even stand up properly. There was a huge bit of corrosion or something smack dab in the middle of the blade. Plus I can't imagine they were very sharp. So those are done.

Managed to get something else done I've been thinking about for awhile, which is to get some old stuff off of floppies and on to a CD or something. Specifically my old Turbo Pascal 6 kit. It's held up well so far and I was able to pull everything off the floppies. I'm not exactly sure how I'd install it, but I thought qemu or dosbox or something like that would work, if nothing else. Maybe wine on Linux? The next step would be to find some of the old source code... That may still be around somewhere as well... At any rate, maybe one of my boys will be interested in learning pascal, although my older son has used and enjoyed Scratch a fair bit. Getting the algorithmic mindset in place is more important than the specific language.

2010: The next... thing

Creating a blog post for Jan 1 has to be one of those common activities that everyone does these days, akin to the crush at the gym for the month of January. Naturally I have to add to the rush, the useless torrent of nonsensical jabbering simply to add another notch to the old, um, blogpost. Tis no matter - not a huge thing, but it is a thing. Some kind of vague philosophical commentary that. If I was really going with the flow and not merely being dragged under by it I'd have created a todo list of sorts on Dec 31st and attempted to follow through with it. Well I forgot so I'm going to do it now, now being the "now that I remembered" now.

One thing I attempted last year and I desperately need to do this year is to keep that regular physical activity scale going. That or cut down on the ice cream consumption, but I'm not sure I can do two things. Well three because the other thing I attempted last year was to keep a regular writing pace. I mean this space has the capacity to contain my limitless dribblings and I think it's easier when no one is really paying attention to me. The opposite of many people I suspect. I mean if someone reads something I have written and gets a chuckle or provokes a thought or bridges a synapse, I'll be satisfied. Myself I've found this sort of writing closer to that of the classroom note taking exercise. I always enjoyed attending class and taking notes, but I rarely read them later. The act of writing was the path to deeper understanding and my penmanship was most horrifying. My mother always thought I'd be a doctor after seeing my scrawl. And I think that is what I have here.

My aunt lent me the memoirs of Albert Speer, one of the top ranking officials in the Third Reich government. He was initially Hitler's architect, but later an important minister. He was one of the few who admitted guilt at Nuremburg and served 20 years for it. He spent his time reflecting on what happened and why and produced a book. My aunt's comment was that it is slanted to show him in a good light, but that's to be expected to some extent. You can't be purely objective about things personally done or participated in. However that first hand account is always interesting. This is the another reason for writing, in a somewhat personal way, on a regular basis - to reveal one self to yourself. Sounds facile, but is a difficult thing to do. Those who are best at it are probably the people that lead the most peaceful existence.

So, perhaps I'll be able to write a little every day. Maybe this time I'll look for different times of the day to do it. Maybe I'll even factor in some pen-and-paper time. Don't want to lose the old skills just yet.