Estimating Might Be Broken, But It’s Not Evil

Ron Jeffries's essay Estimation is Evil talks about how absurd estimating can be on a software project, and the nightmare scenarios that teams can end up in:

…Then we demand that the developers “estimate” when they’ll be done with all this stuff. They, too, know less about this product than they ever will again, and they don’t understand most of these requirements very well. But they do their best and they come up with a date. Does the business accept this date? Of course not! First of all, it’s only an estimate. Second, clearly the developers would leave themselves plenty of slack—they always do. Third, that’s not when we want it. We want it sooner.



So we push back on the developers’ estimate, leaning harder and harder until we’re sure we’ve squeezed out all the fat they left in there. Sometimes we even just tell them when they have to be done.



Either way, the developers leave the room, heads down, quite sure that they have again been asked to do the impossible. And the business people write down the date: “Development swears they’ll be done November 13th at 12:25PM.”

Software Estimation is Broken

Software Estimation – the way that most of us do it – is broken. As an industry we’re bad at estimating, we've been bad at it for a long time, and there’s no evidence that we’re getting much better at it.

Developers know this. The business knows this – so they don’t trust what the development team comes up with, and try to make their own plans. Management knows this too, so they work around estimates (I’ll take everything and double it), or worse they abuse estimates, cut them to the bone, and then use them as a lever to drive the team towards an unachievable goal.

Jeffries says that even teams who are trying to estimate properly are excessively concerned with predictability (and all of the overheads and navel gazing that come with trying to be predictable), when they really should be working on getting the right things done as soon as possible, which is all that the business actually cares about.

So because it’s hard, and because we’re not good at it, and because some people ignore estimates or abuse them, we should stop estimating at all.

As developers what we need to do is make sure that we understand what the most important thing is to the business, break the problem down into the smallest pieces possible and start iterating right away, deliver something and then go onto the next most important thing, and keep going until the business gets what they really need. If you can’t convince “them” (the sponsors) that this is the right way to work, then go through the theatre of estimating to get the project approved (knowing that whatever you come up with is going to be wrong and anyways management and the business are going to ignore it or use it against you), and then get to real work: understand what the most important thing is to the business, break the problem down into the smallest pieces possible and start iterating right away. In other words

“Stop estimating. Start Shipping”.

Martin Fowler wrote a recent post PurposeofEstimation where he says estimates are needed only if they help you make “significant decisions”. His examples are getting resources allocated (portfolio management go/no go – the game Jeffries describes above), and coordination, where your team's work needs to fit in with other teams (although he talks only about coordinating with other developers, ignoring the need to coordinate with customers, partners and other projects that have nothing to do with software development). There are many other times when estimates are needed: delivering to fixed-price contracts, budgeting for programs, when you need to hit a drop dead date (for example, when an industry-wide change is going to happen whether you are done or not).

The rest of the time, if your team is small enough and they know what they’re doing and they’re working closely with the business and delivering working software often, then nobody cares all that much about estimating – which to be honest is how my team does a lot of our work.

But this is a problem-solving approach, not a project management approach.

If you don’t know what you are building, why estimate?

It can work for online startups and other exploratory development: you have an idea that you think is good, but you’re not sure of the details, what people are going to like and what will catch on. If you don’t really know what you are building, then there’s no point trying to estimate how long it is going to take. Somebody will decide how much money you can spend, you start simple, deliver something useful and important (“minimum viable product”) as soon as you can so you can get feedback and keep iterating until hopefully enough people are using it and are telling you what you really need to build, or you run out of money.

We’re back to Evolutionary Prototyping, but with a strict focus on minimal features and prioritization (do only what’s important, don’t even try to consider the entire project because you may not have to deliver it all anyways), and fast feedback loops. Now it’s called “The Lean Startup” method.

If you are going to change it again tomorrow, why estimate?

Working this way also makes sense for a lot of online businesses: Facebook, Twitter, Linkedin, Etsy Netflix all work this way today. They are constantly delivering small features, or maybe breaking bigger changes into small steps and pushing them out incomplete but “dark” as soon as they can (often several times a day), constantly fiddling with the UI, adding personalization features and new channels and integrating with new partners and continuously capturing behavioural data so that they can tell what changes users like or at least what they are willing to put up with, trying out new ideas and running A/B tests knowing that some or most of these ideas will fail.

This work can be done by small teams working independently, so the size and cost of each “project” is small. Marketing wants to add a new interface or run a small experiment of some kind (the details are fuzzy, we’re going to have to iterate through it), it will probably only take a few weeks or maybe a month or two, just get a few smart people together and you see how fast you can get something working. If it doesn't work, or it’s taking too long, cancel it and go on to the next idea. It’s an attention-deficit way of working, but if you are chasing after new customers or new revenue sources and your customers will put up with you experimenting with them, it can work.

Don’t bother estimating, just make it work

And routine maintenance (anything that doesn't have a fixed/drop-dead end date) can be done this way too. David Anderson’s most persuasive arguments in favor of Kanban (working without estimates and continuously pushing out individual pieces of work) are in streamlining maintenance and operations work.

The business doesn't care that much about how this kind of work is managed – they just want important things fixed ASAP, and at the least possible cost. Most of this work is done by individuals or small teams, so again the cost of any piece of work is small. Instead of wasting time trying to estimate each change or fix upfront, you assume everything takes 3 days or whatever to do, and if you aren't finished at 3 days, then stop and escalate it to management, let them review and decide whether the work needs more scoping or should be done at all. Focus on getting things done and everybody is happy.

Why bother estimating – it’s just another mobile app

And it works for mobile app development, again where most work is done by small teams and most of the focus is on the user experience, where the uncertainties are more around what customers are going to like (the product concept, the look-and-feel – which means lots of iterative design work, prototyping and usability testing... and this is going to take how long?) and not on technical risks or project risks.

But you can’t run a project without estimating

Yes a lot of work is done and can be done in small projects by small teams and if the project is small enough and short enough then you may not need to bother much or at all with estimating – because you’re not really running a project, you’re problem solving.

But this way of working doesn't scale up to large organizations running large projects and large programs with significant business and technical risks which need to be managed throughout, and work that needs to be coordinated between different teams doing lots of different things in different places at different times, with lots of handoffs and integration points and dependencies. This is where predictability – knowing where you are and seeing ahead to where you are going to be (and where everybody else is going to be) with confidence – is more important than minimizing cycle time and rapid feedback and improvisation.

It comes down to whether you need to deliver something as a big project or you can get away with solving many smaller problems instead. While there is evidence that software development projects are getting shorter on average (because people have learned that smaller projects fail less often or at least fail faster), some problems are too big to be solved piecemeal. So estimating isn’t going to go away – most of us have to understand estimating better and get better at doing it.

#NoEstimates – building software without estimating – is like Agile development in the early days. Then it was all about small teams of crackerjack programmers delivering small projects quickly (or not delivering them, but still doing it quickly) and going against the grain of accepted development methods. Agile didn’t scale, but it got a lot of buzz and achieved enough success that eventually many of the ideas crossed into the mainstream and we found ways to balance agility and discipline, so that now Agile development methods are being used successfully in even large scale programs. I don’t see how anyone can successfully manage a big project without estimating, but that doesn't mean that some people aren't going to try – I just wouldn't want to be around when they try it.

Book Review: The Phoenix Project

Everyone who attended the “Rugged Devops” panel at RSA this year received a free copy of The Phoenix Project (by Gene Kim, Kevin Behr and George Spafford – the authors of Visible Ops), the fictional story of the education and transformation of an IT manager, an IT organization, and eventually of an entire company.

I'm not sure why they wrote the Phoenix Project as a novel. But they did. So I’ll review it first as a piece of story telling, and then look at the messaging inside.

The reason that I don’t like didactic fiction is that so much of it is so poorly written – generally forced and artificial. I was pleasantly surprised by the Phoenix Project. The first half of the novel tells a story about an IT manager forced into taking on responsibility for saving his company. Our well-meaning hero, an ex-marine sergeant (for some reason unclear to me, the hero, the CEO, and even the mysterious guru all have a military background) with an MBA but without any ambition except to quietly provide for his family. He has been successfully running his own little part of the IT group, so successfully that he is dramatically promoted to take over all of IT operations (and so successfully that his own group is never mentioned again in the story – it seems to run on auto-pilot without him). For a successful manager, our hero knows alarmingly little about how the rest of the IT organization works, or about the business, and so is unprepared for his new responsibility. He reluctantly accepts the big job, and then regrets it immediately as he realizes what a shit storm he has walked into.

It’s a compelling narrative that draws you in and is seems realistic even with the stock characters: the sociopathic SOB CEO, the unpopular everything-by-the-book CISO, the Software Development Manager who only cares about hitting deadlines (but not about delivering software that works), the Machiavellian Marketing executive, and the autistic genius that the entire IT organization of several hundred people all depend on to get all the important stuff done or fixed.

As a pure piece of story telling, things start to unravel in the second half with the emergence of the IT / Lean Manufacturing guru – when the story ends and the devops fable begins. From this point on, the plot depends on several miraculous conversions: everyone except the marketing exec sees the light and literally transform overnight. They even start to groom themselves nicely and dress better. Lifetime friendships are made over a few drinks, and everyone learns to trust and share: there’s a particularly cringe-inducing management meeting in which people bare their souls and weep. Conflicts and problems disappear magically usually because of the guru’s intervention – including an unnecessary scene where the guru shows up at a crucial audit meeting and helps convince the partner of the audit firm (an old buddy of his) that there aren't any real problems with the company’s messed up IT controls (“these aren't the droids you’re looking for”).

But the real heroes are the people running the manufacturing group: the one part of this spectacularly mismanaged company that somehow functions perfectly is a manufacturing plant where everyone can all go to learn about Kanban and Lean process management and so on. Without the help of the smug and smart-alecky guru - who apparently helped create this manufacturing success - and his tiresome Zen master teaching approach (and sometimes even with his help), our hero is unable to understand what’s wrong or what to do about it. He doesn't understand anything about demand management, how to schedule and prioritize project work, that firefighting is actual work, how to get out of firefighting mode, how to recognize and manage bottlenecks in workflow, or even how important it is to align IT with business priorities (where did he get that MBA any ways?). Luckily, the factory is right there for him learn from, if he only opens his eyes and mind.

What will you learn from The Phoenix Project?

The other problem with this story telling approach is that it takes so damn long to get to the point – it’s a 338 page book with about 50 pages of meat. Like Goldratt's The Goal (which is referenced a couple of times in this book), The Phoenix Project leads the reader to understanding through a kind of detective story. You have to be patient and watch as the hero stumbles and goes down blind alleys and ignores obvious clues and only with help eventually figures out the answers. Unfortunately, I'm not a patient reader.

This is a gentle introduction to Lean IT and devops. If you've read anything on Kanban and devops you won’t find anything surprising, although the authors do a good job of explaining how Lean manufacturing concepts can be applied to managing IT work. The ideas covered in the book are standard Lean and Theory of Constraints stuff, with a little of David Anderson’s Kanban and some devops – especially Continuous Deployment as originally described by John Allspaw and Continuous Delivery.

The guru’s lessons are mostly about visualizing and understanding and limiting demand – that you should stop taking on more work until you can get things actually working so that you’re not spending all of your time task-switching and firefighting; identifying workflow bottlenecks and protecting or optimizing around them; how reducing batch size in development will improve control and to get faster feedback; that in order to do this you have to work on simplifying and standardizing deployment; and how valuable it is to get developers and operations to work together.

My complaints aren't with the ideas – I buy into a lot of Devops and agree that Kanban and Lean have a lot to offer to IT ops and support teams (although I'm not sold on Kanban by itself for development, certainly not at scale). But I was disappointed with the unrealistic turnaround in the second half of the book. It’s all rainbows and unicorns at the end. IT, the business, development and security all start working together seamlessly. Management is completely aligned. Performance problems? No problem – just go the Cloud. And they even bring in the famous Chaos Monkey in the last couple of pages just because.

Spoiler Art: Everything goes so well in a few months that the company is back on track, plans to outsource IT and to break up the company are cancelled, the selfish head of marketing is canned, and our hero is promoted to CIO and put on the fast track to corporate second in command. Sorry: nothing this bad gets that good that easily. It is a fable after all, and too hard to swallow.

The Phoenix Project is a unique book – when was the last time that you read an actual novel about IT management?! It was worth reading, and if it introduces devops and Lean ideas to more people in IT, The Phoenix Project will have accomplished something useful. But it’s not a book that you can use to get things done. There are lessons and recipes and patterns but they take work to pull out of the story. There’s no index, no good way to go back to find things that you thought were useful or interesting. So I am looking forward to the Devops Cookbook: something practical that hopefully explains how these ideas can work in businesses that don’t look all like Facebook and Twitter.