Fundamentals of Agile Project Management

How and why agile come into being

To understand how and why agile came into being, we first need to understand what was it like to develop software in the past and what changed in the meantime.

First, we used to manage software development projects using traditional project management techniques because we inherited them from other industries (like construction). The traditional approach consists of dividing a project into phases, planning everything out on a timeline using WBS and Gantt charts, and then following the plan.

But there's a problem - the traditional approach works nicely when you don't expect any changes in the middle of the project. But in software development, things are always changing. If you plan what you'll develop for the next 5 years, as soon as there's some new technology (eg. new protocol, new service, new hardware) your 5-year plan will become obsolete.

Second, the software economics worked differently in the past. In the 1980s, the cost of owning and maintaining software was twice as expensive as developing it. Today, it's reverse: everyone can run software on their personal computer and developers can deploy their own apps on AWS or Heroku within minutes.

Third, internet changed how we develop and consume software. With the rise of internet, the number of people who use software on a daily basis exploded. As the market demand for software expanded (driven by consumers and small businesses), any developer could quit their corporate job and start their own company. Software development was no longer a market reserved only for big players.

This democratization of software had a profound effect on how we develop software. But that's only half of the story.

We also have to take into the consideration that 3/4 of all software products used to fail because they didn't meet requirements or weren't used. That's a lot of money down the drain.

So, one hand we have a rise of small software development companies, and on the other, we have market uncertainty and a lot of risk.

Small software development companies needed a simpler, faster, and less riskier way to develop software. They couldn't afford to spend several months planning only to end up having to rewrite the whole thing because there was some fundamental error in their logic. Plus, they needed to move fast so they could capture the market and start making money as soon as possible.

Several lightweight methodologies were developed that would let companies have prototypes faster:

• 1991 Rapid Application Development
• 1994 Unified Process and Dynamic Systems Development Method (DSDM)
• 1995 Scrum
• 1996 Crystal Clear and Extreme Programming (XP)
• 1997 Feature-Driven Development

Today, we refer to all those methodologies as agile methodologies (because they follow agile principles, as defined in the Agile Manifesto). But at the time, they all were just a bunch of disparate methodologies, living in their own universe.

The initial adopters of these early agile methods were small-to-medium-sized teams who worked on unprecedented systems where it was difficult to scope requirements. To design and achieve product/market fit of those systems, you needed exploration and constantly change thing until you get the system right.

Thanks to methodologies that favored rapid-prototyping, small product companies could create and release the main software faster and add new features as time goes on.

Rapid-prototyping gave smaller companies an advantage because they could react to market needs faster than an enterprise. While enterprises relied heavily on documentation and long lead time, startups could build a prototype, test it, and ship in the fraction of the usual time.

Heavy, document-driven processes (like TickIT, CMM, and ISO 9000) supported the huge organizational structure of enterprises, but it hindered their innovation and market responsiveness.

Product companies benefited the most from these early agile methodologies, but development shops discovered that, in some cases, they too could benefit from a more lightweight approach (although that depended on the nature of their clients).

All the disparate lightweight project management trends came together in 2001 when 17 software developers met in Utah to discuss their processes. Together, they defined the concept of agile software development in Agile Manifesto...