Looking for Answers at Agile 2013

I went to Agile 2013 last week in Nashville to look for answers on how Agile development ideas and practices could help more with high integrity, high assurance development; scale to handle large projects and programs; and improve the working environment for mature, high performance teams.

The Sessions

There was a lot going on, with over 200 sessions, informal workshops, lightning talks, Open Jams and other presentations.

A bunch of talks and workshops on soft people skills: team building, leadership issues, managing conflict, coaching, brainstorming games, personal assessments and personal development.
Several sessions on scaling issues and approaches for managing large projects and programs, UX design, design modeling, how to get testing done in Agile teams, product requirements definition and backlog management, and a handful on code quality (technical debt and refactoring and coding challenges), one on Kanban and one on application security.

There was also a full track on devops: most of this was introductory stuff on improving packaging and build and implementing continuous delivery, and basics on getting developers and ops working together and why. Devops is a natural next step for many organizations looking to be more agile – what’s the point of being faster and more responsive in product design and development, if you crash into a brick wall at operations? But devops has to move out of the friendly homogeneous chef/puppet/noSQL Webops niche and shed the hype about how deploying 10 or 100 times per day is the way to be cool, in order to be relevant to the rest of the world. If not, “devops” is going to end up meaning buying some expensive configuration management and deployment tools from IBM or HP, which won’t make the world a better place and will be a disappointment to everyone except the vendors.

Highlights

Many of the sessions early in the week were introductory level overviews and experience reports. A couple of points of interest included an intelligent and entertaining session on managing requirements and backlogs by Jeff Patton and some real-life experience reports on getting quality and testing under control in large Agile shops.

Most of the highlights for me came on Thursday, when I attended some especially good workshops on dealing with organizational and political issues in program management (how to build broad-based trust across the organization, identifying and dealing with hidden dependencies); getting the team together to craft Abuser Stories, looking at features from an attacker’s perspective to help build application security into Agile development; and a technical session on practical refactoring.

There continues to be a lot of confusion about what refactoring is and what it isn't, as demonstrated by two very different sessions at the conference on refactoring and code quality. The first described well-intentioned but misguided efforts by the Egyptian government to encourage large-scale “refactoring” of legacy code at some Egyptian technology companies, all but one of which failed – the one project succeeded because they severely limited the amount of code cleanup work to a small percentage of time and restricted developers to making only simple refactoring changes to start, until the developers and managers understood what they were doing.

This isn't refactoring: using refactoring techniques and tools is not refactoring. The real advance in refactoring isn't in cataloguing obvious techniques for cleaning up code. It is in defining a disciplined, iterative approach for cleaning up the structure of code immediately before and after a developer makes changes, as an integral part of development and maintenance. Restructuring is supposed to be part of development, not something that needs to be or should be done in a project on its own.

"You should not need a budget to refactor. You should refactor every chance you get."

Llewellyn Falco and Woody Zuill showed how this can be done properly, in a ballsy, well run session where they did live refactoring of code, removing clutter, complexity and cleverness, and then eliminating duplication, all in small steps that took a few minutes at most so that the code could be left at any point in better shape than when they started. They showed what techniques and tools could be used safely, how simple refactoring could be done on code that you don’t understand well (or at all) in order to make it more understandable, and what the payback was for making different kinds of changes.

Themes and Memes

There were a few ideas that came up frequently in the sessions and talking to people at the conference.

Product Ownership is a serious problem, especially in big companies. The simplified Product Owner (“the one throat to choke”) approach doesn't work, even if it is still the official dogma for Scrum. Most Product Owners don’t understand what they are supposed to do, or have the time to do it even if they do know what to do. And of course the single Product Owner idea doesn't scale either – most larger organizations are trying hybrid approaches with product managers, business analysts and other experts filling in at different levels.

Like any good conference, you get as much or more out the other people that you meet there, as you do out of the speakers and vendors.

Many of the people I talked to were working at big (sometimes really big) companies, trying to introduce Agile into big (often really big) projects and programs, and trying to make or remake their careers in an Agile world. What became clear is that Agile doesn't solve all of the problems that come up in large organizations. There are no good answers for executive demands for predictability (because software development is only a part of what needs to be delivered and coordinated in most organizations), or how to communicate and collaborate in massive programs with hundreds or thousands of people that run for years across departments and countries, or how to deal with power politics and organizational inertia, or how to satisfy governance and compliance requirements.

Some people are trying to adapt and integrate Agile practices using SAFe or Disciplined Agile Delivery, but most are putting together something on their own, with Scrum at the team level and PMO practices at higher levels. A lot of organizations are still trying it out before rolling it out, but there are few success stories so far.

Most people seem to accept that spending some time upfront to get the team together and understand what they really need to build is a necessity – you can’t just start off incrementally and iteratively building something. But most people wish it wasn't called “Sprint 0”, because it's not a sprint, it's a separate but important upfront step focused on discovery and simplifying things as much as possible from the beginning.

Getting the requirements right was another important focus area for everyone. The Lean concept of shaving requirements down to come up with a Minimum Viable Product is recognized as the only way to assure success, especially for people running big projects and programs – if you can’t give the business or customer everything that they want, make sure you at least give them the minimum that they need.

Most speakers (but not all) agreed that requirements should be defined in whatever format that makes sense, whatever is needed to get the point across: big stories or small stories, sketches or pictures, detailed specs if you have them, tests if you can get them defined upfront. Jeff Patton emphasized that the simplified story template

as a [type of user] I want [to do something] so that [reason]

is useful as a starting point, a learning tool, but it isn't necessary or even practical to try to fit every requirement into a standardized format.

There was also a strong emphasis on the need for coaching and training (maybe because so many of the speakers were coaches or trainers?), that you shouldn't (or at least don’t have to) try to do all of this without help.

Exhibitors and other things seen and heard

The exhibition floor was surprisingly small. Training firms and firms offering coaching, Agile project management / ALM tools which all looked pretty much the same to me, and a couple of automated testing tools suppliers desperate to convince you that what they are selling is so much better than those tools like QTP that you wasted so much time and money on in the past.

How you can tell IBM doesn't really "get" Agile or Lean:

URL for Scrum.Org: www.scrum.org


URL for IBM’s Agile/Lean practice: www-01.ibm.com/software/rational/agile/…

One Kanban training and certification vendor insisted that “Kanban is not an Agile practice” (although they were at an Agile conference). Another vendor cleared this up by explaining that there were “10 reasons that Kanban is like bacon”.

Other things seen.

There was a cool lighting talk by a scary dude wearing Google Glass on D-Wave’s quantum computer. It had nothing to do with Agile development but it was indeed cool.

Some speakers were riding on their reputation, and put little or no effort into their presentations: any session that had a guru sitting on a red couch on the stage was pretty much a waste of time. I don’t care if somebody worked on a project with Kent Beck 13 years ago, or if they were invited to Snowbird in 2001 but couldn't make it, or they showed up at that meeting long enough to agree on the manifesto before hitting the slopes. What matters is whether what they know and what they have to say today is relevant, do they understand and can they help solve the problems that people are facing now.

Other things heard:

“TDD is like teenage sex: everybody is talking about it, but not that many people are doing it.”

“Don’t estimate or worry about story points. Just make all your stories the same size and count them.”

Umm…How do you get them all the same size without estimating them???

“Team building by itself is useless. You have to follow it up and get people working together on something important for a team to form.”

“If you look at any product that you love using, I can guarantee that the reason that you love it is not because it was shipped on time.”

Keep Looking

I didn't get all of the answers that I was looking for, and I don’t think that anyone I met at the conference did either. But it was a good opportunity to understand the challenges that everyone is facing, and it was good to see so many smart and serious people working hard on solving them. It gives you hope that things are going to get better.

Architecture-Breaking Bugs – when a Dreamliner becomes a Nightmare

The history of computer systems is also the history of bugs, including epic, disastrous bugs that have caused millions of $ in damage and destruction and even death, as well as many other less spectacular but expensive system and project failures. Some of these appear to be small and stupid mistakes, like the infamous Ariane 5 rocket crash, caused by a one-line programming error. But a one-line programming error, or any other isolated mistake or failure, cannot cause serious damage to a large system, without fundamental failures in architecture and design, and failures in management.

Boeing's 787 Dreamliner Going Nowhere


The Economist, Feb 26 2013

These kinds of problems are what Barry Boehm calls “Architecture Breakers”: where a system’s design doesn't hold up in the real world, when you run face-first into a fundamental weakness or a hard limit on what is possible with the approach that you took or the technology platform that you selected.

Architecture Breakers happen at the edges – or beyond the edges – of the design, off of the normal, nominal, happy paths. The system works, except for a “one in a million” exceptional error, which nobody takes seriously until a “once in a million” problem starts happening every few days. Or the system crumples under an unexpected surge in demand, demand that isn't going to go away unless you can’t find a way to quickly scale the system to keep up – and if you can’t, you won’t have a demand problem any more because those customers won’t be coming back. Or what looks like a minor operational problem turns out to be the first sign of a fundamental reliability or safety problem in the system.

Dreamliner is Troubled by Questions about Safety


NY Times, Jan 10, 2013

Finding Architecture Breakers

It starts off with a nasty bug or an isolated operational issue or a security incident. As you investigate and start to look deeper you find more cases, gaping holes in the design, hard limits to what the system can do, or failures that can’t be explained and can’t be stopped. The design starts to unravel as each problem opens up to another problem. Fixing it right is going to take time and money, maybe even going back to the drawing board and revisiting foundational architectural decisions and technology choices. What looked like a random failure or an ugly bug just turned into something much uglier, and much much more expensive.

Deepening Crisis for the Boeing 787


NY Times, Jan 17 2013

What makes these problems especially bad is that they are found late, way past design and even past acceptance testing, usually when the system is already in production and you have a lot of real customers using it to get real work done. This is when you can least afford to encounter a serious problem. When something does go wrong, it can be difficult to recognize how serious it is right away. It can take two or three or more failures before you realize – and accept – how bad things really are and before you see enough of a pattern to understand where the problem might be.

Boeing Batteries Said to Fail 10 Times Before Incident


Bloomberg, Jan 30 2013

By then you may be losing customers and losing money and you’re under extreme pressure to come up with a fix, and nobody wants to hear that you have to stop and go back and rewrite a piece of the system, or re-architect it and start again – or that you need more time to think and test and understand what’s wrong and what your options are before you can even tell them how long it might take and how much it could cost to fix things.

Regulators Around the Globe Ground Boeing 787s


NY Times, Jan 18 2013

What can Break your Architecture?

Most Architecture Breakers are fundamental problems in important non-functional aspects of a system:

• 
  Stability and data integrity: some piece of the system won’t stay up under load or fails intermittently after the system has been running for hours or days or weeks, or you lost critical customer data or you can’t recover and restore service fast enough after an operational failure.
  
• 
  Scalability and throughput: the platform (language or container or communications fabric or database – or all of them) are beautiful to work with, but can’t keep up as more customers come in, even if you throw more hardware at it. Ask Twitter about trying to scale-out Ruby or Facebook about scaling PHP or anyone who has ever tried to scale-out Oracle RAC.
  
• 
  Latency – requirements for real-time response-time/deadline satisfaction escalate, or you run into unacceptable jitter and variability (you chose Java as your run-time platform, what happens when GC kicks in?).
  
• 
  Security: you just got hacked and you find out that the one bug that an attacker exploited is only the first of hundreds or thousands of bugs that will need to be found and fixed, because your design or the language and the framework that you picked (or the way that you used it) is as full of security holes as Swiss cheese.
  

These problems can come from misunderstanding what an underlying platform technology or framework can actually do – what the design tolerances for that architecture or technology are. Or from completely missing, overlooking, ignoring or misunderstanding an important aspect of the design.

These aren’t problems that you can code your way out of, at least not easily. Sometimes the problem isn't in your code any ways: it’s in a third party platform technology that can’t keep up or won’t stay up. The language itself, or an important part of the stack like the container, database, or communications fabric, or whatever you are depending on for clustering and failover or to do some other magic. At high scale in the real world, almost any piece of software that somebody else wrote can and will fall short of what you really need, or what the vendor promised.

Boeing, 787 Battery Supplier at Odds over Fixes


Wall Street Journal, Feb 27 2013

You’ll have to spend time working with a vendor (or sometimes with more than one vendor) and help them understand your problem, and get them to agree that it’s really their problem, and that they have to fix it, and if they can’t fix it, or can’t fix it quickly enough, you’ll need to come up with a Plan B quickly, and hope that your new choice won’t run into other problems that may be just as bad or even worse.

How to Avoid Architecture Breakers

Architecture Breakers are caused by decisions that you made early and got wrong – or that you didn't make early enough, or didn't make at all. Boehm talks about Architecture Breakers as part of an argument against Simple Design – that many teams, especially Agile teams, spend too much time focused on the happy path, building new features to make the customer happy, and not enough time on upfront architecture and thinking about what could go wrong. But Architecture Breakers have been around a lot longer than Agile and simple design: in Making Software (Chapter 10 Architecting: How Much and When), Boehm goes back to the 1980s when he first recognized these kinds of problems, when Structured Programming and later Waterfall were the “right way” to do things.

Boehm’s solution is more and better architecture definition and technical risk management through Spiral software development: a lifecycle with architecture upfront to identify risk areas, which are then explored through iterative, risk-driven design, prototyping and development in multiple stages. Spiral development is like today’s iterative, incremental development methods on steroids, using risk-based architectural spikes, but with much longer iterative development and technical prototyping cycles, more formal risk management, more planning, more paperwork, and much higher costs.

Bugs like these can’t all be solved by spending more time on architecture and technical risk management upfront – whether through Spiral development or a beefed up, disciplined Agile development
approach. More time spent upfront won`t help if you make naïve assumptions about scalability, responsiveness or reliability or security; or if you don’t understand these problems well enough to identify the risks. Architecture Breakers won’t be found in design reviews – because you won’t be looking for something that you don’t know could a problem – unless maybe you are running through structured failure modelling exercises like FMEA (Failure mode and effect analysis) or FMECA (Failure mode, effects and criticality analysis), which force you to ask hard questions, but which few people outside of regulated industries have even heard about.

And Architecture Breakers can't all be caught in testing, even extended longevity/soak testing and extensive fuzzing and simulated failures and fault injection and destructive testing and stress testing – even if all the bugs that are found this way are taken seriously (because these kinds of extreme tests are often considered unrealistic).

You have to be prepared to deal with Architecture Breakers. Anticipating problems and partitioning your architecture using something like the Stability Patterns in Michael Nygard’s excellent book Release It! will at least keep serious run-time errors from spreading and taking an entire system out (these strategies will also help with scaling and with containing security attacks). And if and when you do see a “once in a million” error in reviews or testing or production, understand how serious it can be, and act right away – before a Dreamliner turns into a nightmare.