Looking for Answers at Agile 2013

I went to Agile 2013 last week in Nashville to look for answers on how Agile development ideas and practices could help more with high integrity, high assurance development; scale to handle large projects and programs; and improve the working environment for mature, high performance teams.

The Sessions

There was a lot going on, with over 200 sessions, informal workshops, lightning talks, Open Jams and other presentations.

A bunch of talks and workshops on soft people skills: team building, leadership issues, managing conflict, coaching, brainstorming games, personal assessments and personal development.
Several sessions on scaling issues and approaches for managing large projects and programs, UX design, design modeling, how to get testing done in Agile teams, product requirements definition and backlog management, and a handful on code quality (technical debt and refactoring and coding challenges), one on Kanban and one on application security.

There was also a full track on devops: most of this was introductory stuff on improving packaging and build and implementing continuous delivery, and basics on getting developers and ops working together and why. Devops is a natural next step for many organizations looking to be more agile – what’s the point of being faster and more responsive in product design and development, if you crash into a brick wall at operations? But devops has to move out of the friendly homogeneous chef/puppet/noSQL Webops niche and shed the hype about how deploying 10 or 100 times per day is the way to be cool, in order to be relevant to the rest of the world. If not, “devops” is going to end up meaning buying some expensive configuration management and deployment tools from IBM or HP, which won’t make the world a better place and will be a disappointment to everyone except the vendors.

Highlights

Many of the sessions early in the week were introductory level overviews and experience reports. A couple of points of interest included an intelligent and entertaining session on managing requirements and backlogs by Jeff Patton and some real-life experience reports on getting quality and testing under control in large Agile shops.

Most of the highlights for me came on Thursday, when I attended some especially good workshops on dealing with organizational and political issues in program management (how to build broad-based trust across the organization, identifying and dealing with hidden dependencies); getting the team together to craft Abuser Stories, looking at features from an attacker’s perspective to help build application security into Agile development; and a technical session on practical refactoring.

There continues to be a lot of confusion about what refactoring is and what it isn't, as demonstrated by two very different sessions at the conference on refactoring and code quality. The first described well-intentioned but misguided efforts by the Egyptian government to encourage large-scale “refactoring” of legacy code at some Egyptian technology companies, all but one of which failed – the one project succeeded because they severely limited the amount of code cleanup work to a small percentage of time and restricted developers to making only simple refactoring changes to start, until the developers and managers understood what they were doing.

This isn't refactoring: using refactoring techniques and tools is not refactoring. The real advance in refactoring isn't in cataloguing obvious techniques for cleaning up code. It is in defining a disciplined, iterative approach for cleaning up the structure of code immediately before and after a developer makes changes, as an integral part of development and maintenance. Restructuring is supposed to be part of development, not something that needs to be or should be done in a project on its own.

"You should not need a budget to refactor. You should refactor every chance you get."

Llewellyn Falco and Woody Zuill showed how this can be done properly, in a ballsy, well run session where they did live refactoring of code, removing clutter, complexity and cleverness, and then eliminating duplication, all in small steps that took a few minutes at most so that the code could be left at any point in better shape than when they started. They showed what techniques and tools could be used safely, how simple refactoring could be done on code that you don’t understand well (or at all) in order to make it more understandable, and what the payback was for making different kinds of changes.

Themes and Memes

There were a few ideas that came up frequently in the sessions and talking to people at the conference.

Product Ownership is a serious problem, especially in big companies. The simplified Product Owner (“the one throat to choke”) approach doesn't work, even if it is still the official dogma for Scrum. Most Product Owners don’t understand what they are supposed to do, or have the time to do it even if they do know what to do. And of course the single Product Owner idea doesn't scale either – most larger organizations are trying hybrid approaches with product managers, business analysts and other experts filling in at different levels.

Like any good conference, you get as much or more out the other people that you meet there, as you do out of the speakers and vendors.

Many of the people I talked to were working at big (sometimes really big) companies, trying to introduce Agile into big (often really big) projects and programs, and trying to make or remake their careers in an Agile world. What became clear is that Agile doesn't solve all of the problems that come up in large organizations. There are no good answers for executive demands for predictability (because software development is only a part of what needs to be delivered and coordinated in most organizations), or how to communicate and collaborate in massive programs with hundreds or thousands of people that run for years across departments and countries, or how to deal with power politics and organizational inertia, or how to satisfy governance and compliance requirements.

Some people are trying to adapt and integrate Agile practices using SAFe or Disciplined Agile Delivery, but most are putting together something on their own, with Scrum at the team level and PMO practices at higher levels. A lot of organizations are still trying it out before rolling it out, but there are few success stories so far.

Most people seem to accept that spending some time upfront to get the team together and understand what they really need to build is a necessity – you can’t just start off incrementally and iteratively building something. But most people wish it wasn't called “Sprint 0”, because it's not a sprint, it's a separate but important upfront step focused on discovery and simplifying things as much as possible from the beginning.

Getting the requirements right was another important focus area for everyone. The Lean concept of shaving requirements down to come up with a Minimum Viable Product is recognized as the only way to assure success, especially for people running big projects and programs – if you can’t give the business or customer everything that they want, make sure you at least give them the minimum that they need.

Most speakers (but not all) agreed that requirements should be defined in whatever format that makes sense, whatever is needed to get the point across: big stories or small stories, sketches or pictures, detailed specs if you have them, tests if you can get them defined upfront. Jeff Patton emphasized that the simplified story template

as a [type of user] I want [to do something] so that [reason]

is useful as a starting point, a learning tool, but it isn't necessary or even practical to try to fit every requirement into a standardized format.

There was also a strong emphasis on the need for coaching and training (maybe because so many of the speakers were coaches or trainers?), that you shouldn't (or at least don’t have to) try to do all of this without help.

Exhibitors and other things seen and heard

The exhibition floor was surprisingly small. Training firms and firms offering coaching, Agile project management / ALM tools which all looked pretty much the same to me, and a couple of automated testing tools suppliers desperate to convince you that what they are selling is so much better than those tools like QTP that you wasted so much time and money on in the past.

How you can tell IBM doesn't really "get" Agile or Lean:

URL for Scrum.Org: www.scrum.org


URL for IBM’s Agile/Lean practice: www-01.ibm.com/software/rational/agile/…

One Kanban training and certification vendor insisted that “Kanban is not an Agile practice” (although they were at an Agile conference). Another vendor cleared this up by explaining that there were “10 reasons that Kanban is like bacon”.

Other things seen.

There was a cool lighting talk by a scary dude wearing Google Glass on D-Wave’s quantum computer. It had nothing to do with Agile development but it was indeed cool.

Some speakers were riding on their reputation, and put little or no effort into their presentations: any session that had a guru sitting on a red couch on the stage was pretty much a waste of time. I don’t care if somebody worked on a project with Kent Beck 13 years ago, or if they were invited to Snowbird in 2001 but couldn't make it, or they showed up at that meeting long enough to agree on the manifesto before hitting the slopes. What matters is whether what they know and what they have to say today is relevant, do they understand and can they help solve the problems that people are facing now.

Other things heard:

“TDD is like teenage sex: everybody is talking about it, but not that many people are doing it.”

“Don’t estimate or worry about story points. Just make all your stories the same size and count them.”

Umm…How do you get them all the same size without estimating them???

“Team building by itself is useless. You have to follow it up and get people working together on something important for a team to form.”

“If you look at any product that you love using, I can guarantee that the reason that you love it is not because it was shipped on time.”

Keep Looking

I didn't get all of the answers that I was looking for, and I don’t think that anyone I met at the conference did either. But it was a good opportunity to understand the challenges that everyone is facing, and it was good to see so many smart and serious people working hard on solving them. It gives you hope that things are going to get better.

Agile Development leads to Alzheimer's

Iterative development and design helps you to reach your way towards understanding what the customer really needs, to try out new ideas, evaluate designs, experiment, respond to feedback and react to changing circumstances. Everything gets better as you learn more about the domain and about the customer and about the language and technologies that you are using. This is important early in development, and just as important later as the product matures and in maintenance where you are constantly tuning and fixing things and dealing with exceptions.

But there are downsides as well. Iterative development erodes code structure and quality. Michael Feathers, who has been studying different code bases over time, has found that changes made iteratively to code tend to bias more towards the existing structure of the code, that developers make more compromises working this way. Code modules that are changed often get bigger, fatter and harder to understand.

Working iteratively you will end up going over the same ground, constantly revisiting earlier decisions and designs, changing the same code over and over. You’re making progress – if change is progress – but it’s not linear and it’s not clean. You’re not proceeding in a clear direction to a “right answer” because there isn’t always a right answer. Sometimes you go backwards or in circles, trying out variants of old ideas and then rejecting them again, or just wandering around in a problem space trying stuff until something sticks. And then somebody new comes in who doesn’t understand or doesn’t like the design, tries something else, and leaves it for the next guy to pick up. Changes in design, false starts, dead ends and flip flops leave behind traces in the code. Even with constant and disciplined refactoring, the design won’t be as clean or as simple as it would be if you “got it right the first time”.

It doesn’t just wear down the code, it wears down the team too

Iterative development also has an erosive effect on an organization’s memory – on everyone’s understanding of the design and how the system works. For people who have been through too many changes in direction, shifts in priorities and back tracking, it’s difficult to remember what changed and when, what was decided and why, what design options where considered and why they were rejected before, what exceptions and edge cases came up that needed to be solved later, and what you need to know when you’re trying to troubleshoot a problem, fixing a bug or making another design change.

Over the course of the last 6 or more years we've changed some ideas and some parts of the code a dozen times, or even dozens of times, sometimes in small, subtle but important ways, and sometimes in fundamental ways. Names stay the same, but they don’t mean what they used to.

The accumulation of all of these decisions and changes in design and direction muddies things. Most people can keep track of the main stories, the well-used main paths through the system. But it’s easy for smart people who know the design and code well, to lose track of details, the exception paths and dependencies, the not-always-logical things that were done for one important customer just because 25 or 50 or 110 releases ago. It gets even more confusing when changes are rolled out incrementally, or turned on and off in A/B testing, so that the system behaves differently for different customers at different times.

People forget or misremember things, make wrong assumptions. It’s hard to troubleshoot the system, to understand when a problem was introduced and why, especially when you need to go back and recreate a problem that happened in the past. Or when you’re doing trend analysis and trying to understand why user behaviour changed over time – how exactly did the system work then? Testers miss bugs because they aren't clear about the impact of a change, and people report bugs – and sometimes even fix bugs – that aren't bugs, they've just forgotten what is supposed to happen in a specific case.

When making changes iteratively and incrementally, people focus mostly on the change that they are working on now, and they forget or don’t bother to consider the changes that have already been made. A developer thinks they know how things work because they’ve worked on this code before, but they forget or don’t know about an exception that was added in the past. A tester understands what needs to be tested based on what has just been changed, but can’t keep track of all of the compatibility and regression details that also need to be checked.

You end up depending a lot on your regression test suite to capture the correct understanding of how the system really works including the edge cases, and to catch oversights and regression mistakes when somebody makes a fix or a change. But this means that you have to depend on the people who wrote and maintained the tests and their understanding and their memory of how things work and what impact each change has had.

Iterative development comes with costs

It’s not just the constant pace, the feeling of being always-on, always facing a deadline that wears people down over time. It’s also the speed of change, the constant accumulation of small decisions, and reversing or altering those decisions over and over that wears down people’s understanding, that wears down the mental model that everyone holds of how the system works and how the details tie together. All of this affects people’s accuracy and efficiency, and their confidence.

I am not sure that there is a way to avoid this. Systems, teams, people all age, and like in real life, it’s natural that people will forget things. The more changes that you make, the more chances there are for you to forget something.

Writing things down isn't much of a help here. The details can all be found somewhere if you look: in revision history, in documentation, in the test suite and in the code. The problem is more with how people think the system works than it is with how the system actually works; with how much change people can keep up with, can hold in their heads, and how this affects the way they think and the way they work.

When you see people losing track of things, getting confused or making mistakes, you need to slow down, review and reset. Make sure that before people try to fix or change something they have a solid understanding of the larger design – that they are not just focusing on the specific problem they are trying to solve. Two heads are better than one in these cases. Pair people up: especially developers and testers together, to make sure that they have a consistent understanding of what a change involves. Design and code reviews too, to make sure that you’re not relying too much on one person’s memory and understanding. Just like in real life, as we get older, we need to lean on each other a bit more.

Design Patterns after Design is Done

Design Patterns are a useful tool when you are designing a system, an effective shorthand for communicating and sharing design ideas and a way to build consistency into the code – if people understand them and follow patterns properly.

I'm not interested in arguments over whether design patterns are good or not, or which patterns are good and which ones aren't - although these are all important questions.

What I want to understand is how useful design patterns are over time. Do they still matter after the initial design is done?

Looking for Patterns in Code

The first thing to ask is whether developers working on the code can recognize the design patterns that were used, and how useful is it for them when they do.

In Design Patterns for Maintenance, Joshua Engel makes a case that when someone maintaining code recognizes a pattern, they instantly get more context, which means they can move faster and with more confidence:

“When the maintainer recognizes a pattern in a piece of code being maintained, the maintainer grasps a bit more about the code. The maintainer taps into a wealth of background material: personal experience as well as what's read in textbooks like Design Patterns. That background can clue you in to potential pitfalls, limitations, and the intended way for the code to evolve. Or, if the code fails to completely match the patterns, you have a guide to what needs to be added to gain the benefits of that pattern.”

This is backed up by research. In Making Software, chapter 22 “The Evidence for Design Patterns” reviewes two studies by Prof. Walter Tichy showing that design patterns can be useful to developers maintaining code, provided that the people maintaining the code recognize and understand the patterns. One study of computer science students who had some training in design patterns found that students made fewer mistakes and were faster at changing code if it followed well-known and easily-understood design patterns, and if the design patterns being used in the code were clearly documented in the comments. In another study, experienced programmers were also able to make changes quicker and with fewer bugs if the code followed design patterns that they were familiar with.

But another study on design patterns in legacy code explains how difficult it is to “mine for design patterns” in code:

• 
  recognizing design patterns in code requires a good understanding of the code as well as common patterns;
  


• 
  some patterns are easier to recognize than others – and some patterns probably won’t be recognized at all.
  

Understanding and Recognizing Patterns in Code

Patterns are only valuable if they are immediately recognizable by whoever is working on the code and can be easily followed. Which means that they have to be implemented properly in the first place, and sustained over time.

Besides the canonical GoF design pattern catalog, which most developers have at least heard of, and maybe Martin Fowler’s Patterns of Enterprise Application Architecture, there are lots of other less well-known pattern collections, never mind proprietary patterns that were invented by the team that wrote the software.

You can’t expect a developer to recognize these patterns, never mind understand them, from looking at code, unless they are otherwise made explicit in the code through naming conventions and comments (including, for more obscure patterns, live links to the pattern definition). The studies above prove thjat this kind of documentation is especially important for less experienced developers.

But just because something has “Factory” or “Strategy” in the name, or comments explaining that the code is following a pattern, doesn’t mean that it actually follows that pattern properly, at least not any more.

Refactoring to Patterns

Another place where patterns come into play is in refactoring. When cleaning up the structure of code, it’s natural (for some developers at least) to think about where patterns can be applied.

Refactoring to Patterns takes refactoring to a higher level, not just correcting obvious problems and inconsistencies in the code. It describes how to bring the design inline with common patterns (not all of them from the GoF book) using multiple refactoring steps.

Some patterns are simple to understand, simple to apply, don’t require a lot of changes to implement, and result in simpler code: Factories and Prototypes. Refactoring to other patterns requires a lot more work to understand and change the code, and may not be worth the effort: Strategies and State, Observer, Visitor.

What’s the real payback for refactoring or rewriting code to patterns for the sake of patterns? There often isn't one.

You don’t want to refactor to patterns unless:

1. 
   you have a good reason to refactor the code in the first place – the code is difficult to understand and change; and
   


2. 
   you know how to do refactoring properly and safely;
   


3. 
   you need the extra flexibility that most patterns offer; and
   


4. 
   you have the experience and judgement to know what patterns are needed and how to use them properly; and
   


5. 
   the people who you work with also understand patterns well enough to keep up with the changes that you want to make.
   

As it says in the GoF book:

Design patterns should not be applied indiscriminately. Often they achieve flexibility and variability by introducing additional levels of indirection, and that can complicate a design and/or cost you some performance. A design pattern should only be applied when the flexibility it affords is actually needed.

The Value of Patterns over Time

Refactoring to Patterns encourages more ambitious, larger-scale refactoring – which can be dangerous, because the more you refactor, the more chances there are of making mistakes and introducing bugs – and implementing patterns doesn't always make code more maintainable and easier to understand, which defeats the purpose of refactoring.

A study on design patterns and software quality at the University of Montreal (2008) found that design patterns in practice do not always improve code quality, reusability and expandability; and often makes code harder to understand. Some patterns are better than others: Composite makes code easier to follow and easier to change. Abstract Factory makes code more modular and reusable, but at the expense of understandability. Flyweight makes code less expandable and reusable, and much harder to follow. Most developers don’t recognize or understand the Visitor pattern. Observer can be difficult to understand as well, although it does make the code more flexible and extendible. Chain of Responsibility makes code harder to follow, and harder to change or fix safely. And Singleton, of course, while simple to recognize and understand, can make code much harder to change.

For maintainability and understandability, it’s more important to recognize and sustain coding conventions so that the code base is consistent than it is to implement patterns. And to understand common refactorings and how to use your IDE’s refactoring tools, as well as Michael Feathers’ patterns for cleaning up legacy code.

Whether you’re designing and writing new code, or changing code, or refactoring code, the best advice is:

• Don’t use patterns unless you need to.
  


• 
  Don’t use patterns that you don’t fully understand.
  
• 
  Don’t expect that whoever is going to work on the code in the future to recognize and understand the patterns that you used – stick to common patterns, and make them explicit in comments where you think it is important or necessary.
  
• 
  When you’re changing code, take some time to look for and understand the patterns that might be in place, and decide whether it is worth preserving (or restoring) them: whether doing this will really make the code better and more understandable.
  

Automated Tests as Documentation

One of the arguments for writing automated tests is that tests can act as useful documentation for a system. But what do tests document? And who will find this documentation useful?

Most developers don’t rely on system documentation because there isn't enough documentation to give them a complete idea of how the system works, or because there’s too much of it to read, or because it’s not well written, or because it’s not up to date, or because they don’t believe it’s up to date.

But a good set of automated tests can tell you how the system really works today – not how somebody thought it works or how they thought it was supposed to work or how it used to work, if anybody bothered to write any of this down.

“Tests are no more and no less than executable documentation. When writing new code, it makes sense that it should be documented by the person closest to it: the author. That’s my rationale for developer testing.



Comment “doc blocks” and wiki pages, are always found to be inaccurate to some extent. By contrast automated tests fail noisily whenever they go out of date. So with regard to documentation, automated tests are uniquely advantageous in that (given a CI server) you at least have the option of keeping your documentation up to date whenever it starts to drift away from reality.”


Noah Sussman

You might be able to use tests as documentation, if…

To be useful as documentation, tests have to be:

1. 
   comprehensive – they have to cover all of the important areas of the code and functions of the system;
   


2. 
   run often and work – run on every check-in, or least often enough to give everyone confidence that they are up to date; and the tests have to pass – you can’t leave tests broken or failing;
   


3. 
   written to be read – writing a test that works, and writing a test that can be used as documentation, are two different things;
   


4. 
   at the right level of abstraction – most people when they talk about automated tests mean unit tests. …
   

Unit tests constitute design documentation that evolves naturally with a system. Read that again. This is the Holy Grail of software development, documentation that evolves naturally with a system. What better way to document a class than to provide a coded set of use cases. That's what these unit tests are: a set of coded use cases that document what a class does, given a controlled set of inputs. As such, this design document is always up-to-date because the unit tests always have to pass.


Jeff Canna Testing, fun? Really?

Of course even tests that run frequently and pass in Continuous Integration or Continuous Delivery can still contain mistakes. There will be tests that pass but shouldn't, or tests that tell you what the system does, but what the system does is not what what it is supposed to do (one of the risks of having developers writing tests is that if they misunderstood the requirement and got the code wrong, they got the tests wrong too). But on the whole, tests that run often should be more accurate than some document that may or may not have been right in the first place and that probably hasn't been kept up to date since. And through code coverage analysis, you can at least understand what parts of the system are described by the tests, and what parts aren't.

Tests have to be written to be read

One problem with using tests as documentation, is that tests are only a means to an end – even tests written up front in TDD. Tests are just another part of the supporting infrastructure and tooling that developers rely on to write code (and it's all about the code). Tests are tools to help developers think about what code they need to write (if they are following TDD), to prove that the code does what it was supposed to do, and to catch mistakes when people make changes in the future.

As a result, developers don’t put the same amount of attention and discipline into designing and implementing tests as they do with the code itself. As long as the tests run quickly and they cover the code (or at least the important parts of the code), they’ve done the job. Nobody has to care that much about how the tests are named or what they look like inside. Few teams peer review unit tests, and even then most reviewers only check to see that somebody wrote a test, and not that every test is correct, or that each test has a nice meaningful and consistent name and is easy to understand. There aren't a lot of developers who understand xUnit Test Patterns or spend extra time – or can afford to spend extra time – refactoring tests. So, a lot of automated tests aren't clean, consistent or easy to follow.

Another thing that makes unit tests hard to follow as documentation, is the way that tests are usually organized. Common conventions for structuring and naming unit tests are that developers will write a test class/module for every code class/module in the system, with test methods to assert specific behaviour in that code module.

Assuming that whoever wrote the code wrote a comprehensive set of tests and followed a consistent, well-defined structure and naming approach, and that they came up with good, descriptive names for each test class and test method and that everyone who worked on the code over time understood all of this and took the trouble to keep all of it up as they changed and refactored the code and moved responsibilities around and wrote their own tests (which is assuming a lot), then you should be able to get a good idea of what’s happening inside each piece of code by following the tests.

Can you really understand a system from Unit Tests?

But you’re still not going to be able to read a nice story about how a system is designed or what the system does or how it does it by reading unit tests.

Even if unit tests are complete and up-to-date and well written and well organized and have good names (if, if, if, if, if), the accumulation of details built up from looking at all of these tests is overwhelming. Unit tests are too close to the metal, sometimes obscured by fixtures and other test plumbing, and too far removed from the important aspects of the business logic or the design.

UnitTests include a lot of testing of lower level processes that have no direct connection to the stories.


Steve Jorgensen, comment in Unit Test as Documentation

Tests follow the code. You can’t understand the tests without understanding the code. So why not read the code instead? You’ll have to do this eventually to make sure that you really know what’s going on, and because without reading the code, you can’t know if the tests are well-written in the first place.

One place where low-level developer tests may be useful as documentation is describing how to use an API – provided that tests are comprehensive, expressive and “named in such a way that the behavior they validate is evident”.

A good set of unit tests like this can act as a reference implementation, showing how the API is supposed to be used, documenting common usage details:

If you are looking for authoritative answers on how to use the Rails API, look no further than the Rails unit tests. Those tests provide excellent documentation on what's supported by the API and what isn't, how the API is intended to be used, the kind of use cases and domain specific problems that drove the API, and also what API usage is most likely to work in future versions of Rails.


Peter Marklund, Rails Tip: Use the Unit Tests as Documentation

But again, this can only work if you take the time to design, organize and write the tests to do “Double Duty” as tests and as documentation, (Double Duty: How to repurpose the unit tests you’re doing to help create the documentation you’re not, by Brian Button) and make your tests “as understandable as humanly possible for as many different readers as possible”.

Tests as documentation?

I like the idea that automated tests can serve as documentation – we’d all save time and money this way. But who is this documentation for, and how is it supposed to be used?

I don’t know any developers who would start by reading tests cases in order to understand the design of a system. A good developer knows that they can’t trust documents or pictures, or tests, or even what other programmers tell them, or comments in the code. The only thing that they can trust is the code.

Tests might be more useful as documentation to testers. After all, it’s a tester’s job to understand the tests in order to maintain them or add to them. But most testers aren't going to learn much from most tests. When it comes to unit tests, it is the same for testers as it is for developers: unit tests aren't useful unless you understand the code – and if you understand the code, then you should read it instead. Higher-level acceptance tests are easier to understand and more useful to look at especially for non-technical people. It should be easier to tell a story and to follow a story about what a system does through high-level functional and integration scenarios (big fat tests), or acceptance tests captured in a tool like Fitnesse. Rather than asserting detailed implementation-specific conditions, these tests describe technical scenarios, or business rules and business workflows and other requirements that somebody thought were important enough to test for.

But even if you can follow these tests, there’s no way to know how well they describe what the system does without spending time talking to people, learning about the domain, testing the system yourself, and… reading the code.

I asked Jonathan Kohl, one of the smartest testers I know, about his experience using automated tests as documentation:

Back in '03 and '04, Brian Marick and I were looking into this. We held an experimental workshop at XP/Agile Universe in 2004 with testers, developers and other project stakeholders to see what would happen if people who were unfamiliar with the program code (both the code and the automated unit tests) could get something useful from the automated tests as documentation. It was a complete and utter failure. No one really got any value whatsoever from the automated unit tests from a doc perspective. We had to explain what was going on afterwards…



Marick and I essentially tossed the idea aside after that experience. I gave up on the whole double duty thing. Documentation is really good at documenting and explaining, while code is good at program creation and executing. I go for the right tool for the job now, rather than try to overload concepts.



Bottom line, I do not find tests useful as documentation at all. When done well, I do find them useful as examples of implementation of interfaces, etc. when I am new to a system, but nothing replaces a good written doc, especially when coupled with some face-to-face brainstorming and explanations.

There are a lot of benefits to automated testing, especially once you have a good automated suite in place. But documentation is not one of them.

What does Code Ownership do to Code?

In my last post, I talked about Code Ownership models, and why you might want to choose one code ownership model (strong, weak/custodial or collective) over another. Most of the arguments over code ownership focus on managing people, team dynamics, and the effects on delivery. But what about the longer term effects on the shape, structure and quality of code – does the ownership model make a difference? What are the long-term effects of letting everyone working on the same code, or of having 1 or 2 people working on the same pieces of code for a long time?

Collective Code Ownership and Code Quality

Over time, changes tend to concentrate in certain areas of code: in core logic and in and behind interfaces (listen to Michael Feathers’ fascinating talk Discovering Startling Things from your Version Control System). This means that the longer a system has been running, the more chances there are for people to touch the same code.
Some interesting research work backs up what should be obvious: that the people who understand the code the best are the people who work on it the most, and the people who know the code the best make less mistakes when changing it.

In Don’t Touch my Code!, researchers at Microsoft (BTW, the lead author Christian Bird is not a relative of mine, at least not a relative who I know) found that as more people touch the same piece of code, it leads to more opportunities for misunderstandings and more mistakes. Not surprisingly, people who hadn't worked on a piece of code before made more mistakes, and as the number of developers working on the same module increased, so did the chance of introducing bugs.

Another study, Ownership and Experience in Fix-Inducing Code tries to answer which is more important in code quality: “too many cooks spoil the broth”, or “given enough eyeballs, all bugs are shallow”? Does more people working on the same code lead to more bugs, or does having more people working on the code mean that there are more chances to find bugs early? This research team found that a programmer’s specific experience with the code was the most important factor in determining code quality – code that is changed by the programmer who does most of the work on that code is of higher quality than code written by someone who doesn't normally work on the code, even if that someone is a senior developer who has worked on other parts of the code. And they found that the fewer the people working on a piece of code, the fewer the bugs that needed to be fixed.

And a study on contributions to Linux reinforces that as the number of developers working on the same piece of code increase, the chance of bugs and security problems increases significantly: code touched by more than 9 developers is 16x more likely to have security vulnerabilities, and more vulnerabilities are introduced by developers who are making changes across many different pieces of code.

Long-term Effects of Ownership Approach on Code Structure

I've worked at shops where the same programmers have owned the same code for 3 or 4 or 5 or even 10 years or sometimes even longer. Over that time, that programmer’s biases, strengths, weaknesses and idiosyncrasies are all amplified, wearing deep grooves in the code. This can be a good thing, and a bad thing.

The good thing is that with one person making most or all of the changes, internal consistency in any piece of code will be high – you can look at a piece of code written by that developer and once you understand their approach and way of thinking, the patterns and idioms that they prefer, everything should be familiar and easy to follow. Their style and approach might have changed over time as they learned and improved as a developer, but you can generally anticipate how the rest of the code will work, and you’ll recognize what they are good at and what their blind spots are, what kind of mistakes they are prone to: as I mentioned in the earlier post, this makes code easier to review and easier to test and so easier to find and fix bugs.

If a developer tends to write good, clean, tight code, and if they are diligent about refactoring and keeping the code clean and tight, then most of the code will be good, clean, tight and easy to follow. Of course it follows that if they tend to write sloppy, hard-to-understand, poorly structured code, then most of it will be sloppy, hard-to-understand and poorly-structured. Then again, even this can be a good thing – at least bad code is isolated, and you know what you have to rewrite, instead of someone spreading a little bid of badness everywhere.

When ownership changes – when the primary contributor leaves, and a new owner takes over, the structure and style of the code will change as well. Maybe not right away, because a new owner usually takes some time to get used to the code before they put their stamp on it, but at some point they’ll start adapting it – even unconsciously – to their own preferences and biases and ways of thinking, refactoring or rewriting it to suit them.

If a lot of developers have worked on the same piece of code, they will introduce different ideas, techniques and approaches over time as they each do their part, as they refactor and rewrite things according to their own ideas of what is easy to understand and what isn't, what’s right and wrong. They will each make different kinds of mistakes. Even with clear and consistent shared team conventions and standards, differences and inconsistencies can build up over time, as people leave and new people join the team, creating dissonance and making it harder to follow a thought through the code, harder to test and review, and harder to hold on to the design.

Ownership Models and Refactoring

But as Michael Feathers has found through mining version control history, there is also a positive Ownership Effect on code as more people work on the same code.

Over time, methods and classes tend to get bigger because it’s easier to add code to an existing method than to write a new method, and easier to add another method to an existing class than create a new class. By correlating the number of developers who have touched a piece of code with method size, Feathers research shows that as the number of developers working on a piece of code increases, the average method size tends to get smaller. In other words, having multiple people working on a code base encourages refactoring and simpler code, because people who aren't familiar with the code have to simplify it first in order to understand it.

Feathers has also found that code behind APIs tends to be especially messy – because some interfaces are too hard to change, programmers are forced to come up with their own workarounds behind the scenes. Martin Fowler explains how this problem is made worse by strong code ownership, which inhibits refactoring and makes the code more internally rigid:

In strong code ownership, there's my code and your code. I can't change your code. If I want to change the name of one of my methods, and it's called by your code, I've got to get you to change the call into me before I can change my name. Or I've got to go through the whole deprecation business. Essentially any of my interfaces that you use become published in that situation, because I can't touch your code for any reason at all.



There's an intermediate ground that I call weak code ownership. With weak code ownership, there's my code and your code, but it is accepted that I could go in and change your code. There's a sense that you're still responsible for the overall quality of your code. If I were just going to change a method name in my code, I'd just do it. But on the other hand, if I were going to move some responsibilities between classes, I should at least let you know what I'm going to do before I do it, because it's your code. That's different than the collective code ownership model.



Weak code ownership and refactoring are OK. Collective code ownership and refactoring are OK. But strong code ownership and refactoring are a right pain in the butt, because a lot of the refactorings you want to make you can't make. You can't make the refactorings, because you can't go into the calling code and make the necessary updates there. That's why strong code ownership doesn't go well with refactoring, but weak code ownership works fine with refactoring.


(Design Principles and Code Ownership)

Ownership, Technical Debt or Deepening Insight

An individual owner has a higher tolerance for complexity, because after all it’s their code and they know how it works and it’s not really that hard to understand (not for them at least) so they don’t need to constantly simplify it just to make a change or fix something. It's also easy for them to take short cuts, and even short cuts on short cuts. This can build up over time until you end up with a serious technical debt problem – one person is always working on that code, not because the problem is highly specialized, but because the code has reached a point where nobody else but Scotty can understand it and make it work.

There’s a flip side to spending more time on code too. The more time that you spend on the same problem, the deeper you can see into it. As you return to the same code again and again you can recognize patterns, and areas that you can improve, and compromises that you aren't willing to accept any more. As you learn more about the language and the frameworks, you can go back and put in simpler and safer ways of doing things. You can see what the design really should be, where the code needs to go, and take it there.

There's also opportunity cost of not sticking to certain areas. Focusing on a problem allows you to create better solutions. Specifically, it allows you to create a vision of what needs to be done, work towards that vision and constantly revise where necessary... If you're jumping from problem to problem, you're more likely to create an inferior solution. You'll solve problems, but you'll be creating higher maintenance costs for the project in the long term.

Jay Fields Taking a Second Look at Collective Code Ownership

So far I've found that the only way for a team to take on really big problems is by breaking the problems up and letting different people own different parts of the solution. This means taking on problems and costs in the short term and the long term, trading off quality and productivity against flexibility and consistency – not only flexibility and consistency in how the team works, but in the code itself.

What I've also learned is that whether you have a team of people who each own a piece of the system, or a more open custodian environment, or even if everyone is working everywhere all of the time, you can’t let people do this work completely on their own. It’s critical to have people working together, whether you are pairing in XP or doing regular egoless code reviews. To help people work on code that they’ve never seen before – or to help long-time owners recognize their blind spots. To mentor and to share new ideas and techniques. To keep people from falling into bad habits. To keep control over complexity. To reinforce consistency – across the code base or inside a piece of code.

Code Ownership – Who Should Own the Code?

A key decision in building and managing any development team is agreeing on how ownership of the code will be divided up: who is going to work on what code; how much work can be, and should be, shared across the team; and who will be responsible for code quality. The approach that you take has immediate impact on the team’s performance and success, and a long-term impact on the shape and quality of the code.

Martin Fowler describes three different models for code ownership on a team:

1. Strong code ownership – every module is owned exclusively by someone, developers can only change the code that they own, and if they need to change somebody else’s code, they need to talk to that owner and get the owner’s agreement first – except maybe in emergencies.
   
   
   
   
   
2. Weak code ownership – where modules are still assigned to owners, but developers are allowed to change code owned by other people. Owners are expected to keep an eye on any changes that other people make, and developers are expected to ask for permission first before making changes to somebody else’s code.
   
   
   
   
   
   This can be thought of as a shared custody model, where an individual is forced to share ownership of their code with others; or Code Stewardship, where the team owns all of the code, but one person is held responsible for the quality of specific code, and for helping other people make changes to it, reviewing and approving all major changes, or pairing up with other developers as necessary. Brad Appleton says the job of a code steward is not to make all of the changes to a piece of code, but to
   “safeguard the integrity + consistency of that code (both conceptually and structurally) and to widely disseminate knowledge and expertise about it to others”.
   
   
   
   
   
3. Collective Code Ownership – the code base is owned or shared by the entire team, and everyone is free to make whatever changes they need – or want – to make, including refactoring or rewriting code that somebody else originally wrote. This is a model that came out of Extreme Programming, where the Whole Team is responsible together for the quality and integrity of the code and for understanding and keeping the design.
   

Arguments against Strong/Individual Code Ownership

Fowler and other XP advocates such as Kent Beck don’t like strong individual code ownership, because it creates artificial barriers and dependencies inside the team. Work will stall and pause if you need to wait for somebody to make or even approve a change, and one owner can often become the critical path for the entire team. This could encourage developers to come up with their own workarounds and compromises. For example, instead of changing an API properly (which would involve a change to somebody else’s code), they might shoe horn in a change, like stuffing something into an existing field. Or they might take a copy of somebody’s code and add whatever they need to it, making maintenance harder in the future.

Other arguments against strong ownership are that it can lead to defensiveness and protectionism on the part of some developers (“hey, don’t touch my code!”), where they take any criticism of the code as a personal attack, creating tension on the team and discouraging reviewers from offering feedback and discouraging refactoring efforts; and local over-optimization, if developers are given too much time to spend to polish and perfect their precious code without thinking of the bigger picture.

And of course there is the “hit by a truck factor” to consider – the impact that a person leaving the team will have on productivity if they’re the only one who works on a piece of code.

Ward Cunningham. one of the original XPers, also believes that there is more pride of ownership when code is shared, because everyone’s work is always on display to everyone else on the team.

Arguments against Collective Code Ownership

But there are also arguments against Collective Code Ownership. A post by Mike Spille lists some problems that he has seen when teams try to “over-share” code:

• 
  Inconsistency. No overriding architecture is discernible, just individual solutions to individual problems. Lots of duplication of effort results, often leading to inconsistent behavior
  
• 
  Bugs. People "refactoring" code they don't really understand break something subtle in the original code.
  
• 
  Constant rounds of "The Blame Game". People have a knee jerk reaction to bugs, saying "It worked when I wrote it, but since Joe refactored it....well, that's his problem now.".
  
• 
  Slow delivery. Nobody has any expertise in any given domain, so people are spending more time trying to understand other people's code, less time writing new code.
  

Matthias Friedrich, in Thoughts on Collective Code Ownership believes that Collective Code Ownership can only work if you have the right conditions in place:

• 
  Team members are all on a similar skill level
  
• 
  Programmers work carefully and trust each other
  
• 
  The code base is in a good state
  
• 
  Unit tests are in place to detect problematic changes (although unit tests only go so far)
  

Remember that Collective Code Ownership came out of Extreme Programming. Successful team ownership depends on everyone sharing an understanding of the domain and the design, and maintaining a high-level of technical discipline: not only writing really good automated tests as a safety net, but everyone following consistent code conventions and standards across the code base, and working in pairs because hopefully one of you knows the code, or at least with two heads you can try to help each other understand it and make fewer mistakes.

Another problem with Collective Code Ownership is that ownership is spread so thin. Justin Hewlett talks about the Tragedy of the Commons problem: people will take care of their own yard, but how many people will pick up somebody else’s litter in the park, or on a street - even if they walk in that park or down that street everyday? If the code belongs to everyone, then there is always “someone else” who can take care of it – whoever that “someone else” may be. As a developer, you’re under pressure, and you may never touch this piece of code again, so why not get whatever you need to do as quickly as possible and get on to the next thing on your list, and let "somebody else" worry about refactoring or writing that extra unit test or...?

Code Ownership in the Real World

I've always worked on or with teams that follow individual (strong or weak) code ownership, except for an experiment in pure XP and Collective Code Ownership on one team over 10 years ago. One (or maybe two) people own different pieces of the code and do all or most of the heavy lifting work on that code. Because it only makes sense to have the people who understand the code best do most of the work, or the most important work. It’s not just because you want the work “done right” – sometimes you don’t really have a choice over who is going to do the work.

As Ralf Sudelbucher points out, Collective Code ownership assumes that all coding work is interchangeable within a team, which is not always true.

Some work isn't interchangeable because of technology: different parts of a system can be written in different languages, with different architectures. You have to learn the language and the framework before you can start to understand the other problems that need to be solved.

Or it might be because of the problem space. Sure, there is always coding on any project that is “just typing”: journeyman work that is well understood, like scaffolding work or writing another web form or another CRUD screen or fixing up a report or converting a file format, work that has to be done and can be taken on by anyone who has been on the team for a while and who understands where to find stuff and how things are done – or who pairs up with somebody who knows this.

But other software development involves solving hard domain problems and technical problems that require a lot of time to understand properly – where it can take days, weeks, months or sometimes even years to immerse yourself in the problem space well enough to know what to do, where anyone can’t just jump in and start coding, or even be of much help in a pair programming situation.

The worst disasters occur when you turn loose sorcerers' apprentices on code they don't understand. In a typical project, not everyone can know everything - except in some mature domains where there have been few business paradigm shifts in the past decade or two.


Jim Coplien, Code Ownership

I met someone who manages software development for a major computer animation studio. His team has a couple of expert developers who did their PHDs and post grad work in animating hair – that’s all that they do, and even if you are really smart you’ll need years of study and experience just to understand how they do what they do.

Lots of scientific and technical engineering domains are also like this – maybe not so deeply specialized, but they involve non-trivial work that can’t be easily or competently done by generalists, even competent generalists. Programming medical devices or avionics or robotics or weapons control; or any business domain where you are working at the leading edge of problem solving, applying advanced statistical models to big data analysis or financial trading algorithms or risk-management models; or supercomputing and high-scale computing and parallel programming,
or writing an operating system kernel or solving cryptography problems or doing a really good job of User Experience (UX) design. Not everyone understands the problems that need to be solved, not everyone cares about the problems and not everyone can do a good job of solving them.

Ownership and Doing it Right

If you want the work done right, or need it to be done right the first time, it should be done by someone who has worked on the code before, who knows it and who has proven that they can get the job done. Not somebody who has only a superficial familiarity with the code. Research work by Microsoft and others have shown that as more people touch the same piece of code, there is more chance of misunderstandings and mistakes – and that the people who have done the most work on a piece of code are the ones who make the fewest mistakes.

Fowler comes back to this in a later post about “Shifting to Code Ownership” where he shares a story from a colleague who shifted a team from collective code ownership to weak individual code ownership because weaker or less experienced programmers were making mistakes in core parts of the code and impacting quality, velocity and the team’s morale. They changed their ownership model so anyone could work around the code base, but if they needed to change core code, they had to do this with the help of someone who knew that part of the code well.

In deciding on an an ownership approach, you have to make a trade-off between flexibility and quality, team ownership and individual ownership. With individual ownership you can have siloing problems and dependencies on critical people, and you’ll have to watch out for trucks. But you can get more done, faster, better and by fewer people.