Why Agile Software Development Methods Don't Scale

I recently wrote a book called A New Chore for Sisyphus that addressed a number of software engineering issues (I'm still looking for a publisher by the way). Here and in the book I use "software engineering" in the classical sense of how to build better software. As an example, this is the way the Association for Computing Machinery (ACM) uses the term for their special interest group on software engineering called SIGSOFT. I do not consider coding to be software engineering any more than I consider pouring concrete to be civil engineering.

First off, if you don't know what Agile software development methodologies are, I suggest you check out some web sites such as http://agilemanifesto.org/. The idea behind agile methodologies is to improve software productivity by eliminating as much of the ceremonial reviews, endless but hopelessly out-of-date documentation and other activities that distract the developers from actually doing software development. The idea is to move from a document driven development effort to a human needs driven effort that focuses on the needs of the users. Perhaps best known among agile methods is eXtreme Programming or XP but a couple of other examples are Dynamic System Development Method (DSDM) and scrum.

Agile methodologies attempt to make up for their lack of formal documentation through collocation of both the developers and the user or user representative. Ideally, everyone involved with the project including developers, users, testers, technical publications folks, etc. should all be collocated in a single "bull pen." The goal is to eliminate the need for formal communication by facilitating informal communications. This approach to communication obviously puts a limit on the size of the team since only so many people can really be said to be collocated plus the noise level of larger groups tends to put a limit on group communication since individuals will eventually resort to music and other ploys to drown out the chatter that distracts them from their work.

But even before some projects reach this threshold of not being able to find a big enough "bull pen," difficulties are encountered. My contention is that these difficulties arise from two sources. One source is external to the narrow view of a development project as being just about cranking out code and the other is internal and fundamental to the project itself. The external source arises from the simple fact that big software projects typically have a big impact on the organizations sponsoring them. Proponents of agile methods may actually tout this as a feature but the fact remains that software is not created in a contextual vacuum and the bigger the software project, the more likely it is that there will be impacts to the organization beyond just some different screens. DSDM addresses this problem by emphasizing cross-functional teams but an appropriately sized DSDM team only has so much ability to address issues that may affect how the overall business does business.

This isn't to say that agile methodologies can't be used on larger projects. Only that the higher productivity of agile methodologies probably won't allow a significantly shorter schedule to be achieved. The schedule driver shifts from software development to determining what the system should do and how it should affect existing business processes, facilities and people. That is, the larger the project, the less likely it is that the project is just about "the code." Big projects tend to have a big impact and big impacts tend to bring out the people who are concerned about how that impact will affect them.

The internal project factor that prevents agile methodologies from scaling to larger projects is problem complexity. Here I am using problem complexity in a very specific way. I have long argued that a problem's design complexity imposes a "conservation of difficulty" effect on the project that seeks to solve the problem. That is, the problem's design complexity can be moved around, collected or distributed but a certain level of complexity remains regardless of the approach taken. If this complexity is sufficiently difficult, agile methodologies are not suitable for the problem.

A small complex problem may be solvable by assigning a very capable developer. A slightly larger complex problem may be solvable by assigning a small, tight-knit team with a very capable technical lead. This approach tends to have a limit of approximately four to seven people on the team which again sets an upper bound on how large of a problem such a team can solve. Once the team size grows beyond about seven people the nature of the project changes from just solving the problem to managing the effort that solves the problem.

The transition that occurs at this point is that no one person on the project understands the complete problem. A project manager may understand what the overall project is supposed to do but only by abstracting the details. Likewise, a team lead may know all aspects of what their team is attempting to do but must abstract the efforts of the other teams. No one person understands all aspects of the solution.

It may still be possible to apply agile methodologies to such a problem if an existing design pattern is appropriate for the solution. This reduces the problem to a previously solved problem and many of the issues that would make the problem inappropriate for agile methodologies would be incorporated in the design pattern. If there isn't a suitable existing design pattern, the problem is unsuitable for agile methodologies.

Agile methodologies discarded both the documents and the ceremonies of predictive software development methodologies. Given the success of agile methodologies, it's obvious that neither the documents nor the ceremonies were necessary for successfully developing software. But agile methodologies seem to have problems with large and complex software development efforts. Even smaller complex efforts can prove thorny but agile methods can step up to the problem if the complexity is recognized at the project's inception and a suitable methodology is chosen (e.g., scrum or DSDM instead of XP). If not, the team will find themselves flailing against a problem that seems to continually morph such that repeated "refactorings" simply move the problem(s) from one area of the program to another.

I professionally "grew up" in the belly of the beast of predictive software development. My first job after college was at TRW Defense and Space Systems Group (the company reorganized several times while I worked there but I always worked for a group doing software development for the U.S. government). This was predictive software development in the large with some projects employing several hundred developers (not to mention schedulers, QA folks, business types, etc.) and spanning multiple years.

What wasn't obvious even to those of us in the middle of such development processes was the enormous amount of engineering that was going on as we scurried from design review panic to document creation panic. To be fair, the great bulk of current Information Technology software efforts aren't doing anything even close to developing systems such as the Global Positioning System (GPS), the Tracking and Data Relay Satellite System (TDRSS), etc. to name only two of the larger systems development efforts of the 1980s. To the extent that today's IT development effort is just another reimplementation of a previously developed design pattern then agile development methodologies will probably be more than sufficient. Move to a unique, innovative, large, complex development effort and the engineering processes that support the resolution of the problems inherent in such systems have been removed in order to make agile methods agile.

Now let's bring the two factors that limit agile methodologies to smaller projects back together. Large projects tend to hit both limitations. That is, larger projects tend to have significant impacts that extend beyond just the users of the software and they tend to be complex. There are reports of fairly large agile development efforts succeeding but a quick review of the projects shows that a common characteristic of the projects was a low level of coupling between project elements. That is, the projects really consisted of a collection of individual pieces that had only minimal interaction. In other words, the projects were large but had a low level of complexity. At worst, creating the separate pieces required ensuring a common "look and feel" but no significant level of technical interaction.

I'll be the first to admit that I have yet to see a well run agile software development effort. My knowledge of agile methods comes from attempting to understand why projects supposedly following such methods continued to fail. For the most part, these projects were really only thinly disguised death marches with part of the disguise being the use of agile-speak in an effort to hide from reality. I welcome input from anyone who can explain how any of the recognized agile methodologies can deal with large (say greater than fifty man-years of effort) and complex problems.

Cheers,
Dave