Agile & Scrum

Let’s discuss the process of developing software.

People often think that building software is like building anything else, e.g., a car. At first glance is seems reasonable: you determine some requirements for what you want to build, design it, and then build a prototype. You refine your production process, and eventually you can mass-produce as many of these cars as you want.

However, if we attempt to apply these concepts to software, they don’t work. Designing and building software is subtly different than building other products.

Software Development Lifecycle

The Software Development Lifecycle (SDLC) is a simple model that we use to describe the steps of a software project, from inception to a final working product.

The model has evolved over the last few decades, and is similar to other project tracking processes. It usually prescribes a series of activities like these:

• Planning - “What are high level goals?”, “What is our budget?”, “Who is working on the project?”
• Requirements definition - “Who are our users?”, “What problem are we solving?”
• Analysis & design - “What technical constraints exist?”
• Implementation - “How do we build it?”
• Testing - “Does it meet specifications?”
• Deployment - “How do we keep it running properly?”

Formally, we consider the SDLC to be a software process model: a collection of activities, actions and tasks that we perform when creating software.

This particular model is also referred to as a waterfall model, named for the way in which you are supposed to move through each activity in sequence.

  ---
title: Waterfall Model
config:
    theme: neutral
---
flowchart LR
    A[Planning] --> B[Requirements]
    B[Requirements] --> C[Analysis & Design]
    C[Analysis & Design] --> D[Implementation]
    D[Implementation] --> E[Testing]
    E[Testing] --> F[Deployment]

The implications of using a waterfall model:

• A project starts at the first activity and advances through stages. Each stage must be completed before the next stage begins.
• Stages are typically modelled after organizational units that are responsible for that particular stage (e.g. Product Management owns Requirements, Architects own Analysis & Design, QA owns Testing and so on).
• There are criteria that need to be met before the project can exit one stage and enter the subsequent stage. This can be informal (e.g. an email letting everyone know that the design is “finished”), to a more formal handoff that includes artifacts (e.g. Product Requirements documents, Design documents, Test Plans and so on that must be formally approved).

This linear approach strongly suggests that you can and should define a project up-front (i.e. determine cost, time and so on before starting the project).

What’s wrong with this model?

If you’re a project manager, or a business owner, the Waterfall model is very appealing: it suggests that you can define work up-front, and plan out time, costs and other factors. This model suggests that you can work out the parameters of a project, then execute on that plan. Any failure to meet a deadline would be a failure of people to complete their stages ‘on-time’.

However, this isn’t realistic, at least not for software projects.

1. Requirements can and will change as project development is underway. Often the context in which the product is being developed will change in ways we cannot anticipate (e.g., the market changes, so new features need to be added). Our understanding of the problem that we’re solving will also naturally evolve as we’re building something, which can result in unexpected requirements changes as well.
2. Design and implementation activities are not that predicable! We are often building new features or combinations of features. Managing novelty is always going to be challenging and include some level of uncertainty (i.e., how long it will take, and what the solution will look like).

We knew in the 1970s that the Waterfall model didn’t work, and spent 20 years trying to come up with a better solution. By the mid-1990s, there was a widespread recognition that this way of building software just didn’t work – either for developers or for customers.

• Developers were frustrated by rigid processes and changing requirements.
• Business owners were frustrated by the inability to make changes to projects in-flight.
• Projects were being delivered late and/or over-budget. This frustrated everyone!

There are a large number of software process models that were developed at this time, including Extreme Programming (XP) (Beck 1999), Scrum (Schwaber & Sutherland 1995), Lean ( Poppendieck & Poppendieck 2003). No single model dominated.

Instead, the frustrating state of software development led to a a group of software developers, writers, and consultants meeting and delivering the Manifesto for Agile Software Development in 2001.

Agile was an attempt to find some common-ground principles across competing process models. It isn’t a process model by itself, but a set of principles that influenced the development of other models.

The Agile Manifesto

The Agile Manifesto (agilemanifesto.org)

What does it mean?

1. Individuals and interactions (over processes and tools): Emphasis on communication with the user and other stakeholders.
2. Working software (over comprehensive documentation): Deliver small working iterations of functionality, get feedback and revise based on feedback. You will NOT get it right the first time.
3. Customer collaboration (over contract negotiation): Software is a collaboration between you and your stakeholders. Plan on meeting and reviewing progress frequently. This allows you to be responsive and correct your course early.
4. Responding to change (over following a plan): Software systems live past the point where you think you’re finished. Customer requirements will change as the business changes.

Agile Software Development is used to describe any process that encompasses this philosophy. Agile encourages team structures and attitudes that make communication easier (among team members, business people, and between software engineers and their managers). It emphasizes rapid delivery of operational software, but also recognizes that planning has its limits and that a project plan must be flexible.

Iterative processes

This idea of an iterative, evolutionary development model remains central to all Agile processes (although they may present it differently). Instead of building a “complete” system and then asking for feedback, we instead attempt to deliver features in small increments, in a way that we can solicit feedback continuously though the process. Over time, we will add more features, until we reach a point where we have delivered sufficient functionality and value for the customer.

Agility forces us to continually reassess the state of the project, and supports the idea that decisions can unfold over time. The most significant project benefit of Agile is that it reduces (and tries to eliminate) breaking late-project changes.

Being Agile

Although we often refer to Agile as a singular model, it is actually an umbrella: a philosophy that drives a number of other models, such as Kanban, Lean and Scrum. In practice, we often find that software teams adopt practices from different methodologies, changing them where it makes sense.

Scrum

Scrum is an Agile process model focused primarily on development i.e. implementation.

Scrum breaks down projects into iterations called sprints, a time-boxing technique. Each sprint is typically 2-4 weeks in length.The team agrees on what will be done during the sprint, iterates, and delivers a fully tested and shippable product containing those features at the end of each sprint.

A project will normally require many sprints before it’s complete. Having a buildable and potential “shippable” product at the end of each cycle is incredibly valuable for testing, and demonstrating functionality to customers. It provides flexibility in how you deploy since you can choose to “stop” at any time that the stakeholders agree that enough functionality has been delivered.

One sprint finishes, and the next one ramps up right away.

Sprint activities

Each sprint has four main components:

1. Sprint planning

The first day of the sprint is a planning meeting, where your team collectively decides what you want to include in this release.

You start with a product backlog, which is a list of incomplete features. Your goal is to discuss the remaining issues, and decide which ones you want to include in the upcoming sprint. You to this by assigning them to the sprint and to individual team members in GitLab.

Here are some suggestions on how to approach this:

• Address feedback from the previous sprint (i.e. comments from the TA or instructor).
• If you had anything outstanding from the previous sprint, discuss it and determine how you should proceed (do you need to change your approach, or do you just need more time?)
• Address high-risk items early. This gives you time to pivot if needed, and also helps prevents you from investing too much time in a path that ultimately won’t work.
• Look for blocking issues: items that are critical for other systems. Examples of this might be the class structure for business objects (e.g. data classes) that are used by other features.
• Do NOT assign more work than you think you can do.

Outcomes from this meeting:

• You should have issues logged and assigned to the team, representing the work planned for this sprint.
• You should create meeting minutes documenting any major decisions made by the team during the meeting.

During the sprint, you and your team iterate on the features in the sprint backlog.

2. Daily standup

A standup is a short meeting where the teams meets to identify any blocking issues i.e., issues where someone needs help. Typically each person identifies (a) what they did since the last meeting, (b) what they are planning on doing today, and (c) any blocking issues or concerns that they have. These meetings should be very short.

Outcomes from this meeting:

• Issues may be updated, or postponed.
• Informal: team mates help one another, make suggestions etc.

We will not be doing daily standups in this course! We don’t meet regularly enough for this to work.

3. Sprint demo

A sprint demo takes place at the end of the sprint. It is an informal review of your progress with the customer, or user of your product. This serves as your primary mechanism for getting feedback and results should feed directly into your next sprint objectives.

4. Retrospective

After the sprint review, your team should spend time reviewing how they think they did during the sprint, with the goal of finding issues and making process improvements.