Agile Software Development

In the 1990s, as the large software firms were “maturing” along the CMM dimension, and coinciding with the internet boom and massive growth in the commercial software industry, a parallel movement was taking place: lightweight software development methods, so-called “Agile” methods were evolving, focusing on “soft” factors such as cross functional teams, customer involvement, and face-to-face communication, collaboration, and creativity. Agile approaches emphasize rapidly building working software, rather than spending a lot of time writing specifications up front. Agile also is geared towards incremental delivery and frequent iteration, with continuous customer input along the way.

One of the early examples of agile methodologies is eXtreme Programming (XP,) which appeared in the mid-nineties. Extreme Programming emphasizes teamwork. Managers, customers, and developers are all equal partners in a collaborative team. It implements a simple, yet effective environment enabling teams to become highly productive5. It preaches such practices as pair-programming, automated unit-testing and incremental development.

In the dot-com era, time-to-market became critical for web-based start-ups, as the business became an extremely high-velocity environment. The ecosystem of users, needs, and technologies were changing at a break-neck speed. One of the main realizations that came with this was that “change is inevitable, so we might as well embrace it.” This goes against the traditional Change-Management approach of locking down requirements and specifications at the outset of a project; however it was the reality for most software developers in the commercial realm.

In February 2001, a group of experienced software professionals, representing practices ranging from Extreme Programming to SCRUM, and others sympathetic to the need for an alternative to heavyweight software development processes, convened in Utah. They formed the Agile Alliance and what emerged was the “Agile Manifesto”6, which put forth a declaration of the following core values:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools


Working software over comprehensive documentation

Customer collaboration over contract negotiation


Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

In addition to these values, the “Twelve Principles of Agile Software” were also declared

Commercial software companies started adopting agile practices. Today there are dozens of purportedly agile methodologies. Yet it is hard to point to a single document, framework, or process description to find out exactly what defines Agile.

The Case for Agile in Government Software Development

A common attitude in the software industry has been that Agile only works for small-scale projects with small experienced teams, and that CMM/Waterfall is better suited projects of larger scale. However, the waterfall development life cycle, based on an assumption of a relatively stable business environment, becomes overwhelmed by high change (Highsmith 2002b). High change is exhibited in today’s environment: whether it is to support the war efforts in the 2000s, to address rapidly-evolving cyber-security threats, or to compete in an ever-more net-centric world: the demands of the software customer today are such that capabilities are needed on shorter time horizons, quality expected out of the gate, and the software producers are expected to be nimble enough to mirror the fast-paced changes in their customer’s needs and environment changes.

Taking inspiration from the world of biology, Charles Fine argues that each industry has its own evolutionary life cycle (or “clock speed”), measured by the rate at which it introduces new products, processes, and organizational structures (Fine 1996). The government systems software market had historically been a slow-to-medium-clockspeed environment, with software being only a component within large multi-year projects. It is now becoming a fast-clockspeed environment, with demand for quick software delivery to run on existing networks, infrastructure, and systems. Fine argues that firms must adapt to the changing environment to maintain any competitive advantage. Government software contractors must then adapt to survive.

In 2010, the Under Secretary of Defense for Acquisition, Technology and Logistics, Dr. Ashton Carter (now deputy Secretary of Defense) issued a memo entitled “Better Buying Power: Guidance for Obtaining Greater Efficiency and Productivity in Defense Spending “– He was on a mission to rein in cost overruns and spending as part of an overall plan by Defense Secretary Robert Gates to cut $100 billion from the Pentagon budget over the subsequent five years. Carter also directed acquisition managers to award more services contracts to small businesses because they provide Defense with “an important degree of agility and innovation,” with lower overhead costs (Berwin 2010).

The big government software contractors took note… The Pentagon budget is shrinking and they now will increasingly have to compete with small/nimble firms for contracts. This is perhaps the single driving force behind why these companies are taking a serious look at disrupting their mature product development practices by incorporating Agile software development as methods that are practiced in the commercial world, including in today’s dominant “born-on-the-web” software giants like Google, Facebook, and Amazon. A Forrester Research survey of 1000+ IT professionals in 2009, followed by one in 2010, shows a decline in the use of “Traditional” development methodologies, and a rise in adoption of Agile.

Defining the term “Agile”

So far in this paper we have been using the words “agile” and “Agile” without pausing to ponder the meaning of or explain the context within with they are being used. At the outset of this research, I had a very narrow view of what Agile meant. This was based on my experiences with XP and Scrum. Subsequently, through research and discussion with software professionals from various software industry domains at M.I.T., I quickly realized that there is no consensus on the definition of Agile and that different mental models exist pertaining to how agile practices fit within the software engineering discipline.

The American Heritage Dictionary defines “agile” as: Characterized by quickness, lightness, and ease of movement; nimble. These are the characteristics that we as software project managers desire to imbue into our projects, and that we as software developers want to see in our development practices. We want to be able to quickly produce high-quality software and respond in near-real time to changes in customer needs and requirements, while reaping benefits in terms of cost and schedule.

Another popular word in industry that has come to represent this concept is “lean.” The terms Lean Software Engineering, Lean Software Development, and other similar constructs are also being used in conjunction with or en lieu of Agile. The lean concept finds its roots in manufacturing and supply chain management, tracing back to the Toyota Production System. Lean relates to Agile in that the end goal is to produce product (software or other) faster, cheaper, and more efficiently. Lean principles revolve around process improvement, such as improving efficiency (eliminating waste), Just-In- Time (JIT) engineering, rapid delivery, and team empowerment. For our purposes, Lean is the set of "tools" that assist in the identification and steady elimination of “muda” (a Japanese word meaning an activity that is wasteful and doesn't add value or is unproductive). As waste is eliminated, quality improves while production time and cost are reduced (Gershon 2011). As will be presented later in this research, Agile employs Lean tools and principles to deliver value.

Hereafter, when we refer to “agile software development” and “agile development methods” we are referring to practices geared towards improving the engineering effort of software production. When we refer to “agile project management” we are referring to the management of agile software development projects, and not an agile approach to project management in general. Finally, there is no definitive definition for the capitalized term “Agile” as it is used in various contexts to mean various things. To avoid semantic complications, in this research we use Agile an umbrella term symbolizing the general movement towards agile methods in software engineering and its management.