Agile is NOT a Panacea

Agile Is Not a Panacea - By Payson Hall [email protected]

The phrase “Agile Practices” is starting to seem a bit like the label “Quality Management” did thirty years ago. During the manufacturing revolution of the 70’s and 80’s, there was a significant body of Quality Management research that described effective quality improvement and statistical process control techniques. Unfortunately, the label escaped into the wild and started to be applied with broad abandon to all practices considered “good”. In the same way, it seems that “Agile” is being applied to a broad range of practices and several subsets of their implementation, some well proven through experience, while others seem to have had more narrow success, or no track record at all. 

For some, “Agile” has come to define a dogma. Agile evangelists relish the opportunity to school us regarding which practices adhere to the dogma and which are blasphemy:

·     “If your team is not co-located, THAT’S NOT AGILE”

·     “If testing is not part of your sprint, THAT’S NOT AGILE”

·     “If you spend time up front doing analysis and design, THAT’S NOT AGILE”

·     “If you don’t do pair programming, THAT’S NOT AGILE”

·     “If your sprints are longer than 2 weeks, THAT’S NOT AGILE”

As someone with extensive software engineering, software development, systems integration, and project management experience; it is clear to me that some of the software development practices associated with the Agile brand are a significant improvement over practices in use over the past thirty years in many project contexts. 

Lately I’ve been beset by zealots possessing moderate levels of experience in a narrow set of circumstances (or managers with no experience applying agile practices who have “seen the light” after listening to zealots) making impossibly broad claims about the wonders of Agility.

When I raise questions about the applicability of Agile in some contexts, or more precisely, when I express concerns that Agile software development practices may be useful but not wholly sufficient to address some specific aspects of system or organizational complexity at hand, I’m treated to gently patronizing reassurance that “everything is going to be ok” and that “the age of ‘waterfall’ has passed” and that I just don’t understand the emergent beauty of agility because, presumably, I’ve had too many birthdays to be relevant at this point. This makes me feel particularly special when I discover my lecturer is a liberal arts major[1] with three years of web site development experience and a two days of scrum master training under his or her belt.

Let’s explore what is good about Agile, and then let’s discuss some things about software development that are really difficult – Agile or not – and perhaps we can agree that Agility does not miraculously solve all software engineering and project management problems.

History

In 2001 a group of software development professionals met at a conference to discuss lessons they had learned about software development practices. They published the “Manifesto for Agile Software Development” (see box). 

================================================================

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Kent Beck  James Grenning  Robert C. Martin  Mike Beedle  Jim Highsmith  Steve Mellor

Arie van Bennekum  Andrew Hunt  Ken Schwaber  Alistair Cockburn  Ron Jeffries

Jeff Sutherland  Ward Cunningham  Jon Kern  Dave Thomas  Martin Fowler  Brian Marick

© 2001, the above authors this declaration may be freely copied in any form, but only in its entirety through this notice.

================================================================

Since that time, a number of books have been written and software development practices have been described as being “Agile”, although the Manifesto itself was not prescriptive about practices. A review of the Wikipedia page for Agile will identify several detailed development practices associated with the term.

Generally speaking, Agile software development teams:

·     Are collocated

·     Have ongoing active participation from the user community to answer questions and review work products

·     Engage as teams in development “sprints” - chunks of effort that are time bound, usually 2-4 weeks long

·     Test as they go

·     Engage in pair programming

·     Have frequent short status meetings called “stand ups” once per day or twice per week

·     Perform retrospectives at the end of each sprint to look for process improvement opportunities

·     Seek to develop and deliver functioning systems that deliver user value at the end of each sprint

·     Describe requirements in terms of “stories” where a story is a system requirement as told from the perspective of a system user

·     Plan work in terms of the stories to be implemented in a given sprint

The Good

Sadly, the strongest aspect of Agile is the branding. Slapping the “Agile” label on common sense software development practices will likely go down as the single best marketing move in human history. It’s better than Coca-Cola, because the opposite of Coca-Cola isn’t “clumsy”. To be clear, the opposite of Agile practices is not “Waterfall”, that is a pejorative oversimplification. A better antonym might be the phrase “traditional development practices”, by which I mean more traditional analysis and design approaches, including fleshing out an architecture for more complex systems. In some contexts, these are necessary practices – in some cases exploration of the problem with prototyping or using Agile practices to build bits of the system would expedite insights that might not come from “imagining the system” without building some of it first.

Agile rebrands “iterative development” and “rapid application development” (which existed long before the Agile Manifesto) in a new and exciting way that doesn’t sound stodgy - and people have embraced it. Prototyping and quickly building minimum viable product existed before Agile, but the Agile movement has brought it mainstream and this is a good and healthy development. Fail early, fail fast is a good approach to exploring a new problem space and encourages experimentation.

Agile practices encourage greater user involvement and user intimacy than more traditional approaches by insisting that users be heavily involved with the development process from the beginning. This is a wonderful improvement that is clearly superior on its face in most contexts, although sadly this is one of the “Agile rules” that often gets bent or broken as part of customizing Agile to local practice, generally to the disadvantage of the project.

Agile introduces empirical process control to provide better information about what is happening – a consequence of short sprints and continuous integration, delivery, and testing of functional software. The standup meeting and retrospective – run well – provide solid information about what’s being built and the pace at which it’s being built which supports more informed decision-making. The better you can see, the better you can steer.

Agile encourages team colocation. Software development is a complex task and barriers such as geography and time zones are impediments to essential communication necessary for team to build complex software systems. While not always practical, co-location is desirable and usually reduces the friction on a complex team effort.

Agile normally entails regular, brief face-to-face communication among the project team members – the notion of a frequent stand up meeting, whether daily or twice per week to discuss impediments encourages timely communication is clearly a good idea.

Good coding skills are rare and hard to develop. Pair programming (when actually utilized) is probably one of the best ways to teach the arcane spells and incantations of really solid coding practice. Software inspections (when well implemented) have a similar effect. The advantage of pair programming is that questions can be asked in real time. The advantage of inspections is that code can be reviewed at the pace of the reviewer which encourages better understanding.

Agile encourages a conversation about “minimum viable product”, driving teams to deliver functional code quickly and get end user feedback rather than long development cycles which may be building the wrong product or developing features that are misunderstood or not as valuable as analysts and designers might imagine.

The Agile approach encourages executives and product owners to think of software development projects in terms of what traditional project management calls “phase limited commitment” - each iteration is an experiment and an investment. Rather than asking for $2M to build a system over a year, a team might ask for $50K to build a small bit of functionality that can be demonstrated in a few weeks. This reduces the stress of the team from trying to understand and estimate the whole effort at the start and reduces the stress of executives who can quietly euthanize a project if it is not delivering value quickly.

The Bad

“We are Agile” is often used as a justification (or excuse) for avoiding or minimizing documentation. To be fair, the Manifesto didn’t say documentation was evil – only that functioning code was generally preferable to documentation. Dogmatic advocates sometimes take this disdain for documentation to extremes. Most people agree that shelves full of arcane documentation that is never read is indicative of wasted effort, however there are consumers of documentation beyond just the software developers who build or maintain software systems. For example, complex systems often have “configuration” information that must be provided to establish the working environment for a system, perhaps to describe user roles and capabilities, or to describe the flow of work through a system. These configurations are often maintained by specialized administrators who are not software developers and don’t have access to the code (and wouldn’t want to have to read all the code in a system in order to configure a new user anyway). User’s guides, explanations of a systems architecture and interfaces – these are not “shelf-ware” but rather necessary artifacts for operating and maintaining a complex system. Several large Agile projects in my experience have downplayed or delayed creation of these work products, arguing that they were subject to change and documenting them would be a waste of time. When these documents were finally delivered, opportunities for knowledge transfer had been lost and the authors were not familiar with the design decisions and implications of aspects of the configuration. The resulting documents were incomplete, inconsistent, contained errors, and had minimal value.

Agile proponents often speak of time “wasted” trying to develop a comprehensive architecture too early in a project. The language has a seductive lure – no one wants to waste time developing a detailed architecture before most of the system details are known – however the argument that a comprehensive architecture for a complex system is some magical “emergent property” of Agile development is wishful thinking. When civil engineers are going to put up an office building, they don’t need to know the location of every interior wall and window before they can set the foundation – but they DO need to know about how tall the building will be and how much it will weigh to lay the foundation and roughly how much water will be consumed and how much wastewater will be created to size the plumbing. To the extent that those assumptions are incorrect, it can require significant rework to fix mistakes (up to and including a wrecking ball). Complex computer system architecture is not strikingly different. If early versions of a system assume it will run on a single local computer system but later it is determined that it must migrate to a distributed network, the architectural changes necessary might require significant rework.

In my experience, many Agile proponents have a “green field” bias – assuming that development projects build new systems from scratch. Approaching software development unencumbered by limitations of present data structures, a body of current users, or existing interfaces isn’t naïve – but when incorrect it can lead to significant problems. A system that shares files or databases with other systems cannot capriciously change data formats or data validation rules. If a legacy system has hundreds or thousands (or millions) of existing users, then significant changes to the user experience can result in substantial organizational change costs. If a legacy system has dozens of complex interfaces with other systems that expect data in specific formats or have specific timing dependencies (transaction x must follow transaction y within 10 seconds or both transactions will be cancelled), changing those interfaces can be a monumental undertaking. Many legacy systems built in the last century are getting long in the tooth and need to be updated or replaced. Although Agile practices might help once a strategy for system enhancement or replacement has been identified, there may be few opportunities for “two weeks sprints that deliver customer value” and while user participation in prioritization might be helpful, there are structural and technical considerations that exceed most user knowledge and experience. Determining an approach to such legacy systems is a complex analysis and design task that must be undertaken with care.

Estimating software development projects is a notoriously imprecise and imperfect undertaking, fraught with opportunities for error. Some Agile proponents point to this difficulty and suggest that, because estimates are difficult and imprecise, they are a waste of time. Through Agile methods, they say, executives don’t have to make large commitments – instead they can ask the Agile team to deliver incremental value until it no longer makes business sense to continue. This argument appears to largely misunderstand the role of executives in an organization. Most organizations have a limited amount of resource available to pursue business objectives. Executives need a reasonable guess about cost to make a crude determination of return on investment and prioritize resource allocation.

An agile approach to software development is often at odds with organizational contracting and procurement practices, particularly in the public sector. Procuring software development services has always been culturally challenging for large organizations. Procurement models usually assume that what is being purchased can be clearly and unambiguously described – like a desk or a pallet of plywood, and then procured from the lowest cost, responsible bidder. Procuring software this way has not been particularly successful because of the difficulty (impossibility?) of creating complete and unambiguous specifications and because requirements are constantly evolving – voiding previous fixed price agreements and subjecting such projects to ongoing project change management challenges. The public sector is particularly suspicious of fraud, waste, and abuse and tends to seek clear and objective criteria for vendor selection that minimizes subjectivity and opportunities for favoritism. This isn’t how you or I would procure a vendor (if you have a good one, keep it, else get recommendations from people you trust and go with that one).

The Ugly

Well implemented Agile practices are, in many cases, demonstrably superior to the alternatives practiced in more traditional development shops - but they are NOT a panacea. Software development is a complex human endeavor and it is hard. Agile practices help lubricate the software development machine and make it more efficient, but they don’t make the complexity go away.

In the rush to embrace the good that Agile offers, evangelists sometimes over promise the benefits of Agility while downplaying some of its limitations. Experienced software development professionals can usually ignore the hype and embrace what fits in their context – but business executives may not have the experience or knowledge base to filter the hype. This does our profession a disservice – setting management’s expectations impossibly high. A system estimated to take 18 months and $20M to develop and implement using “traditional methods” is redefined as an Agile project with the revised expectation that it will happen vaguely better, cheaper, and faster. It might or might not – depending on context.

I see three significant problems that result from overzealous Agile evangelism:

1)   If Overhyped Agile can’t meet the impossibly high expectations being set, executives may throw out the baby with the bathwater. With or without Agile, software development is still hard. There might be cost and time savings, but they may not appear immediately, and they may not meet unrealistic expectations – disappointing the executive suite and discrediting what might be a noble effort to introduce or expand Agile practices that would be valuable.

2)   The assertion that “Project Management is a thing of the past” and has been displaced by superior practices. Much of what goes on in developing and implementing a data system has nothing to do with software development – procuring and setting up the test and production hardware environments, preparing for the organizational change of implementation (documentation, training, establishing administrative structures), coordinating development and testing of interfaces. In the rush toward Agility, I have observed many organizations discounting the significance of planning for these activities to the detriment of successful implementation.

3)   A diffusion and loss of software engineering/computer science knowledge hard won over the past 50 years. Early computer programs were simple. They had to be. Storage and computing power were limited, and languages were primitive. As Moore’s law delivered faster computers and higher capacity storage, software complexity exploded. The best professionals in our industry struggled to keep up with the ever-increasing complexity brought about by distributed systems and layers of communications and security. We refined our skills at functional decomposition, diagnostics, testing, and performance tuning. Some of these lessons are still valuable but seem anathema to the young Turks advocating vague “Agile Practices” as a cure-all. I recently gave a guest lecture at the local college to senior Computer Science students and was told that design was no longer part of the curriculum… students were expected to learn that “on the job”. Computer Science grads who have never had a course in design – that seems like a bad thing.

A Case For Moderation

I think as software development professionals, we need to stop and take a breath. We need to discuss the merits (and business value) of Agile practices like co-location, greater product owner access and involvement, sprints that deliver tested code, and better methods of automated testing and regression testing. In most contexts, these are no-brainer good ideas that may require some local customization but will result in better work products and less stress.

We also need to assure that we aren’t forgetting the lessons learned over the past 50+ years of software development about system architecture, design, partitioning functionality, building systems to provide robust diagnostic information to make them easier to test and maintain, test driven development, and how the parts of a system interact to determine performance.

We must remember that Agile is not a replacement for some of the nuts and bolts of project management - the two disciplines can co-exist and complement one another.

The partisan politics and religious wars of “Agile” vs. “Waterfall” need to be set aside so that we can embrace what is good about the Agile innovations without disregarding the hard-won lessons of software engineering history. 

And finally, I encourage everyone to stop overselling the benefits of Agility to the executive suite. Yes, they are a helpful set of practices. Yes, they require some changes in the way we do business and there will be a learning curve. Yes, when successfully implemented they will sometimes enable us to deliver quality software a little faster. No, it does not make the complexity of software development go away – building systems is still a difficult, risky, and expensive undertaking that requires the focused attention of skilled professionals.

[1] I’m not slamming liberal arts majors here… the best system architect I’ve ever worked with was a liberal arts major – but he was also a genius who spent years developing his craft and devoured uncounted arcane computer science tomes from my library, retaining more information from them than I ever could. In general, if you spent three years studying English Literature and then took a night class in web design - how about I don’t lecture YOU on the subtleties of Shakespeare and you don’t lecture me about software engineering and computer science, ok?