Does “Process” Work in Software Development?

UPDATE: There’s a Part 2 now.

When I started in the web development business about a decade ago, I worked at a small agency with a few smart people, and we were basically winging it. As the dotcom bubble expanded at a frenzied pace, we grew along with it, and inevitably we had a couple of projects take sharp turns south. People suffered on both sides, and it became clear that winging it was not scalable.

We needed a process – one we could articulate to clients, propagate through the organization and of course put into action in a repeatable way.

This is what “process” is supposed to do. It exists to mitigate unpleasant surprises by telling stakeholders what they can expect from a project. It’s also supposed to save teams from continually reinventing the wheel.

I worked with my bosses at that first agency job to define such a process, and we were very proud when it was complete. Some months later I left that job for an agency on the West coast. As it happened, I arrived just as this company too was in the midst of a campaign to define a process. People from every part of the agency worked on it for weeks, and the final product was a lovely thing to behold. We made posters for the office, and glossy booklets to give to would-be clients.

My next company – a large telecom – used a version of RUP that was the most over-engineered process I’ve ever experienced. When they laid me off, I returned to the agency world. I worked for two of the biggest interactive agencies, and both had processes that were beautifully-rendered in company collateral.

It might not surprise you to learn that all of these processes looked pretty much the same on paper – characterized by big phases with catchy labels. In fact it was only in these labels where the various processes really differed – one company’s “explore” is another’s “discover.” Even the language the agencies used to sell their respective processes was the same, with a lot of emphasis on words like flexible, adaptable and proven.

Finally, it’s worth adding that all of my agency work was in the service of clients who had their own processes, so at one time or another I’ve had to use variations of waterfall, RUP, agile, spiral and triple lindy. Kidding about that last one.

They all share similarities, but the really striking thing these processes have had in common in my experience is how quickly they broke down and became meaningless when put into practice – even by solid teams of nothing but smart people.

I had been a big believer in process, but because of repeated failures I eventually came to see the whole idea of “process” as a fantasy designed to woo clients and placate bosses. I began to believe that assembling a small team of really smart people and simply allowing them to wing it is the right approach after all.

At the very least, this approach doesn’t raise false hopes. If projects are destined for chaos or collapse, isn’t better not to expect them to go smoothly in the first place?

I’m joking of course. There are many reasons projects go south, and there are many ways to help make sure they don’t. Working within the framework of a heavy duty process is not the solution, but having no process at all amounts to blind luck, because successful projects depend on the right mix of a whole range of ingredients that aren’t likely to converge if you’re just winging it.

This is starting to run a bit long, so I’m going to push the rest to a sequel. In Part 2 of this series on process, I’ll talk about the ingredients that need to converge to make sure projects go well.