Development Lifecycle

Our ultimate goals as software developers is to build effective, high-quality software systems. This requires discipline:

Software Engineering: The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software.

A software process is the collection of activities, actions, and tasks that are performed when to create software. An activity is used to achieve a broad objective (e.g., communication with stakeholders) and ios applied during the process. Actions (e.g., architectural design) encompasses a specific set of tasks that may be used for a particular activity (e.g., an architectural model).

Process Activities

A process framework establishes the foundation for a complete software engineering process by identifying a small number of framework activities that are applicable to all software projects, regardless of their size or complexity. In addition, the process framework encompasses a set of umbrella activities that are applicable across the entire software process. A generic process framework for software engineering encompasses five activities:

• Communication: We need to define the problem that we’re trying to solve, and discuss the goals and requirements with the customer, potential users, and other stakeholders. This is critical to ensure that we’re building the “right” product or solving the main problem that needs to be addressed.

• Planning: This involves defining the tasks that need to be completed, and high level milestones that you need to achieve. It also includes identify people and resources, potential risks to the project, and developing a plan to mitigate these risks. These are what we would typically call “project management” activities.

• Modeling: It’s always cheaper to think ahead prior to actually building something. This step includes design efforts to create abstract models or representations of the product that you need to build. By iterating on a design, you refine your understanding of the problem, and move towards a more-correct solution. This is crucial before you actually start building it, both to improve the accuracy and usefulness of what you build, but to minimize costs.

• Construction: The process of building your product i.e. realizing your design. This may require successive iterations to complete all of the required features, and should also include some level of testing and validation.

• Deployment: Tasks required to produce a working product, and deliver it to the customer. This includes collecting their ongoing feedback and making revisions as needed.

Let’s review these in more detail.

Communication

Products, in general, are designed and built to solve particular problems for users. The first, and most important step in any project is ensuring that you understand both the users, and the problem that you are attempting to address. Once that’s done, you need to ensure that there is agreement by everyone on both the problem and proposed solution.

Users

The people impacted by your product are called stakeholders. This includes the people who will use your solution, but can also include anyone else that is affected by it. For example, in a software project, stakeholders can include:

• users, the people who will directly utilize your software;
• the Information Technology (IT) department who will have to install and maintain your software;
• people who do not directly use your product, but who may need to provide input to your system, or work with it’s output;
• your company, and you personally since you presumably need to maintain the software over time.

As software designer and developers, we tend to focus on the actual users of our software and services, but we need to consider the needs of all of these stakeholders.

Requirements

Requirements analysis is the set of activities designed to identify problems in sufficient detail to determine a solution. Requirement specification is the identification and documentation of the capabilities or requirements that are needed to address a particular problem for stakeholders. Types of requirements include:

• Architectural requirements: Related to the system architecture of a system. e.g. how it will integrate with an existing system, or how it must be deployed to be successful.
• Business requirements: High-level organizational goals that your product or solution will help address, directly or indirectly.
• User requirements: Statements of the needs of a particular set of stakeholders (namely, those that will use your product or software).
• Implementation requirements: Changes that are required to facilitate adoption of your solution. This can include education and training, data migration, and any other work that is triggered by the adoption of your system.
• Quality of service requirements: Detailed statements the system’s qualities. Examples include: reliability, testability, maintainability, availability.
• Regulatory requirements: Laws or policies, imposed by a third-party. e.g. privacy rules related to health data that your product might collect and use.

In software development, we are mostly focused on the design and implementation of a system that meets user requirements. In other words, we focus on the end-users of our product and attempt to design a product that is useful and usable to meet their particular needs.

We can also think about these requirements as being about system capabilities versus system qualities:

• The capabilities of a system refers to the functionality that we will design and implement. Capabilities are also known as functional requirements, and include user requirements from the list above, plus any other requirements that directly result in a product feature. These are the requirements that we will focus on in this phase.

• The qualities that the solution should have, constraints under which it might operate, are called non-functional requirements. These include quality of service from the list above. For software, this includes capabilities like accuracy, speed, quality of service. We will focus on non-fnctional requirements in the Analysis & Design phase.

Our goal as designers is to design and build a system that meets both capabilities and qualities.

Planning

Planning activities determine what actions are required to meet requirement and project goals, and establish a plan to deliver the project on-time, on-budget, with the requirements met.

Project planning has to consider:

• Project goals: There may be additional goals outside of the product itself. e.g. Integrate a remote testing team into our development process; deliver X in revenue during delivery and so on.
• Resources: Who and what we have available to dedicate to the project. Typically this means allocating staff, budget, necessary resources. If you and the customer are both committing resources, this is the time to identify what those might be.
• Constraints: Other factors that we need to consider e.g. we need a demo for a tradeshow in Spring 2022.

The triple constraint model (“project management triangle”) is a model of the constraints of project management. While its origins are unclear, it has been used since at least the 1950s. It contends that: The quality of work is constrained by the project’s budget, deadlines and scope. In other words, quality depends on the relationship between project costs, what is being done (scope) and time required. The relationship between these constraints, and what tradeoffs work is rarely very straightforward, but the constraints themselves are real. Projects do have time and budget constraints that need to be respected, and we need reasonable confidence that we can deliver our scope within those constraints.

This model is pervasive, but also has it’s detractors who point out that there are many times when this model does not work. e.g. late projects are often also over-budget.

Modeling

Before actually constructing anything, we want to ensure that we have met both functional requirements (gathered from users), and non-functional requirements (qualities that we want our system to have). It’s useful to think of non-functional requirements as the constraints or conditions on how our solution works:

• Technical constraints: requirements made for technical reasons (e.g. must be implemented in C++ for compatibility with our existing libraries).
• Business constraints: requirements made for business reasons (e.g. must run on Windows 11 because that’s what we have deployed at customer sites, or must use Java because that’s where we have expertise as a development team).
• Quality attributes: scalability, security, performance, maintainability, evolvability, reliability, deployability (e.g. must complete a core task in less than 5 seconds; must demonstrate 99.999% uptime; must support 1000 concurrent users).

Ideally, before we actually attempt to build something, we want to think through and plan the work, confirm our designs and assumptions, and fine-tune our understanding of the problem. The exact nature of modeling will vary based on the type of project, but can include building prototypes (to show a customer and confirm our understanding), or mockups of screens (to verify that the interface is clear to users).

Typically modeling is done in iterations, where we design something, use that to confirm our understanding with users, make corrections and modifications to our design, and continue iterating until we feel that we can proceed with construction. Taking time to design first saves considerable time and cost in the construction phase.

Construction

This is the step of actually manufacturing a product, or a software system, based on your earlier designs. This is typically the most expensive and time-consuming step of the project, and consumes most of our time and resources. Although we prefer to have a near-perfect design by the time we arrive at this stage, it’s common to have to iterate on design elements prior to realizing the completed product. Details of this step are highly dependent on the type of product that you’re building; we won’t focus too much on it at this time.

Deployment

Similarly, deployment may include packaging and distribution of a physical product. In some cases it many even include installation and validation on behalf of a customer, or on a customer site. We’ll discuss this later in the context of software delivery. For now, just appreciate that this can be a very costly step; correcting mistakes at this point is nearly impossible without going through the entire process again.

Umbrella Activities

Software engineering process framework activities are complemented by a number of umbrella activities. In general, umbrella activities are applied throughout a software project and help a software team manage and control progress, quality, change, and risk. Typical umbrella activities include:

• Software project tracking and control. Allows the software team to assess progress against the project plan and take any necessary action to maintain the schedule.
• Risk management. Assesses risks that may affect the outcome of the project or the quality of the product.
• Software quality assurance. Defines and conducts the activities required to ensure software quality.
• Technical reviews. Assess software engineering work products in an effort to uncover and remove errors before they are propagated to the next activity.
• Measurement. Defines and collects process, project, and product measures that assist the team in delivering software that meets stakeholders’ needs; can be used in conjunction with all other framework and umbrella activities.
• Software configuration management. Manages the effects of change throughout the software process.
• Reusability management. Defines criteria for work product reuse (including software components) and establishes mechanisms to achieve reusable components.
• Work product preparation and production. Encompasses the activities required to create work products such as models, documents, logs[…]

Software engineering process is not a rigid prescription that must be followed dogmatically by a software team. Rather, it should be agile and adaptable (to the problem, to the project, to the team, and to the organizational culture). Therefore, a process adopted for one project might be significantly different than a process adopted for another project.

Process Models

We use the term process model to describe the structure that is given to these activities. That is, it defines the complete set of activities that are required to specify, design, develop, test and deploy a system, and describes how they fit together. A software process model is a type of process model adapted to describe for software systems.

Software activities tend to be named slightly differently than the generic activity names that we’ve been using:

Compared to the standard model, we’ve split Planning into separate Planning and Requirements definition. These activities are often performed by different departments or individuals, so traditionally they’re split apart. Analysis & Design corresponds to Modeling, and Implementation and Testing together correspond to Construction.

This is a simplified conceptual understanding of the different “pieces” of a software development project. People generally accept that we have planning, formal requirements, modeling and design, implementation and testing - but there’s lots of disagreement on how these pieces “fit together”. Let’s continue talking about different forms of process models that we could use:

Waterfall Model

In the 1970s, there was a concerted effort to formalize ‘known-good’ methods of project management. Software projects were seen as expensive and time-consuming, and there was considerable pressure to improve how they were managed. In a 1970 paper, Winston Royce laid out a mechnism for formalizing the large-scale management of software projects [Royce 1970], dubbed the Waterfall model. This envisions software production as a series of steps, each cascading into the next one, much like a waterfall. In this model, requirements are defined first, a design is created and then implemented, then tested and so on. Software development is treated as a set of linear steps that are followed strictly in-order¹.

The Waterfall Model, as understood and practiced for a long time, most closely resembles a linear project model, and is similar to how other construction or manufacturing projects are organized.

The Waterfall model the following characteristics:

• A project starts at the top and advances through stages. Each stage must be completed before the next stage begins.
• The stages are modeled after organizational units that are responsible for that particular stage (e.g. Product Management owns Requirements, Architects own Analysis & Design, QA owns Testing and so on).
• There are criteria that need to be met before the project can exit one stage and enter the subsequent stage. This can be informal (e.g. an email letting everyone know that the design is “finished”), to a more formal handoff that includes artifacts (e.g. Product Requirements documents, Design documents, Test Plans and so on).

This linear approach strongly suggests that you can and should define a project up-front (i.e. determine cost, time and so on). This can be a very appealing proposition to risk-adverse businesses, but as we’ll see, this may not be realistic: requirements change, often as the project is underway, which makes this style of project structure difficult.

V-Model

The V-Model is an alternative that attempts to line-up the testing phase with the area of responsibilityt that is being tested. It’s a useful conceptual model, but it’s unclear how this is supposed to address the criticisms of a straight linear model.

Spiral Model

A spiral model acknowledges that iteration is useful, and suggests iterating from a high-level of abstraction through to lower-levels of detail. The product manager in this case is supposed to define the levels of detail (soooo close, but still problematic).

The Agility Movement

The late 90s were a particularly active period in terms of advancing software process. There was a widespread recognition that old, manufacturing-based ways of building software just didn’t work - either for developers or for customers.

There are a large number of software process models that were developed at this time, including Extreme Programming (XP) [Beck 1999] , Scrum [Schwaber & Sutherland 1995], Lean [ Poppendieck & Poppendieck 2003]. Collectively, these are called “Agile Processes”.

This culminated in In 2001, when a group of software developers, writers, and consultants signed and published the Manifesto for Agile Software Development.

“Agile Software Development” isn’t a single process, but rather an approach to software development that encompasses this philosophy. It encourages team structures and attitudes that make communication easier (among team members, business people, and between software engineers and their managers). It emphasizes rapid delivery of operational software, but also recognizes that planning has its limits and that a project plan must be flexible ².

What does this mean?

1. Individuals and interactions (over process and tools): Emphasis on communication with the user and other stakeholders.
2. Working software (over comprehensive documentation): Deliver small working iterations of functionality, get feedback and revise based on feedback. You will NOT get it right the first time.
3. Customer collaboration (over contract negotiation): Software is a collaboration between you and your stakeholders. Plan on meeting and reviewing progress frequently. This allows you to be responsive and correct your course early.
4. Responding to change (over following a plan): Software systems live past the point where you think you’re finished. Customer requirements will change as the business changes.

What is the benefit?

Agility means recognizing that requirements and plans will change over time.

1. Software is too complex to design and build all at once. It’s more manageable to add features and test incrementally.
2. Software is in a constant state of change, and requirements will change during the development cycle³.

The conventional wisdom in software development is that the cost of change increases nonlinearly as a project progresses. Agility is often characterized as “embracing change” since it expects project and requirements changes, and is constantly reassessing the state of the project. The benefit of Agile is that it reduces (and tries to eliminate) breaking late-project changes.

Agile assumptions

Any agile software process is characterized in a manner that addresses a number of key assumptions about the majority of software projects:

1. It is difficult to predict in advance which software requirements will persist and which will change. It is equally difficult to predict how customer priorities will change as the project proceeds.
2. For many types of software, design and construction are interleaved. That is, both activities should be performed in tandem so that design models are proven as they are created. It is difficult to predict how much design is necessary before construction is used to prove the design.
3. Analysis, design, construction, and testing are not as predictable (from a planning point of view) as we might like.

Given these three assumptions, how do we create a process that can manage unpredictability?

Central to Agile processes is that any propcess must be adapatable to rapidly changing project and technical conditions. It must also be incremental and incorporate customer feedback so that the appropriate adaptations can be made.

This idea of an iterative, evolutionary development model remains central to all Agile processes (although they may present it differently). Instead of building a “complete” system and then asking for feedback, we instead attempt to deliver features in small increments, ina. way that we can solicit feedback continuously though the process. Over time, we will add more features, until the we reach a point where we have delivered sufficient functionality and value for the customer.

Note that in this process model, we still do some initial project planning and requirements definition, but the majority of our time is spent iterating over features. Every time we implement and validate some new functionality, we have the opportunity to deploy it (either for further customer testing, or as a release).

Let’s take some time and talk about the two most influential process models: Scrum and Extreme Programming (XP).

Scrum & Sprints

If you adopt only one agile practice, let it be retrospectives. Everything else will follow.

– Woody Zuill

The important thing is not your process. The important thing is your process for improving your process.

– Henrik Kniberg.

Scrum is the defacto process model for managing scope during a project iterations i.e. it’s focused on the overall project structure. Scrum breaks down a project into fixed-length iterations called sprints (typically 2-4 weeks in length for each sprint). Sprints are defined so that you iterate on prioritized features in that time, and produce a fully-tested and shippable product at the end of each sprint.

Typically a project will consist of many sprints, and you will iterate until you and the customer decide that you’re done (i.e. the only remaining requirements are deemed low enough priority that you decide to defer them or not complete them). Practically, having a buildable and potentially “shippable” product at the end of each cycle is incredibly valuable for testing, demonstrating functionality to customers, and it provides flexbibility in how you deply.

In Scrum, everything is structured around sprints:

Key Concepts

• Product Owner: the person responsible for gathering requirements and making them available in the product backlog. They are not considered part of the project team, but represent both the external business and customer. At the start of each sprint, they work with the team to prioritize features and decide what will be assigned to a sprint.
• Product Backlog: a list of all possible features and changes that the Product Owner thinks should be considered. There is no guarantee that these will all be developed! The team must agree to bring features forward into a sprint before they are developed.
• Sprint Backlog is the set of features that are assigned to a specific sprint. This is the “scope” for that sprint.
• The Scrum Master is the person that helps facilitate work during the sprint. They are not in charge (!) but track progress and help identify blocking issues that might prevent the team from meeting their deliverables.
• The Daily Scrum is a standup meeting where you discuss (a) what you’ve done since the last meeting, (b) what you intend to do today, and (c) any obstacles that might prevent you from accomplishing b. The Scrum Master runs this meeting, and the entire team attends.

Sprint Breakdown

The following steps are followed in each sprint:

1. The project team and Product Owner collectively decide what requirements to address, and they are moved from the Product Backlog to the Sprint backlog. Once features have been decided, you do not allow any further scope changes (i.e. you cannot add anything to the sprint once its started). Work is actually assigned to team members (and you collctively agree that you believe it can be completed in the sprint).
2. During the sprint, you iterate on the features in the Sprint Backlog. This includes design, development, testing etc. until you complete the feature or the sprint is finished. The Scrum Master facilitates aily meetings to make sure that nobody is “stuck” on their feature.
3. At the end of the sprint, have a review with the team to see what was accomplished. Demo for the Product Owner (and sometimes the actual customer). Reflect on your progress, and be critical of how you might improve process the next sprint. (e.g. could we have communicated better? should we have done more testing during the sprint? did we take on too many features?)

Extreme Programming (XP)

The most important thing to know about Agile methods or processes is that there is no such thing. There are only Agile teams. The processes we describe as Agile are environments for a team to learn how to be Agile.

– Don Wells

Extreme Programming (XP) is an Agile methodology focused on best-practices for programmers. It was based on a large-scale project that Kent Beck managed at Chrysler in the late 90s, and attempted to capture what was working for them at that time. It aims to produce higher-quality software and a higher quality-of-life for the development team.

The five core values of XP are communication, simplicity, feedback, courage, and respect.

• Communication: The key to a successful project. It includes both communication within the team, and with the customer. XP empasizes face to face discussion with a white board (figurtively).
• Simplicity. Build the “simplest thing that will work”. Follow YAGNI (You Ain’t Gonna Need It) and DRY (Don’t Repeat Yourself ).
• Feedback. Team members solicit and react to feedback right away to improve their practices and their product.
• Courage: The courage to insist on doing the “right thing”. The course to be honest with yourselves if something isn’t working, and fix it.
• Respect: Respect your team members, development is a collaborative exercise.

XP launched with 12 best practices of software development [Beck 2004]. Some of these (e.g. The Planning Game, 40-Hour Week, Coding Standard) have fallen out of disuse. Others have been added or changed over time, so it is difficult to find a “definitive” list of commonly used XP practices ⁴.

Although some XP practices never really worked very well, many have been adopted as “best practices”. We’ll revisit these in the next secion.

Being Agile

We have generations of people claiming to have the “correct” solution, or the best process model. The reality is, every process model is a reflection of the organization that developed it. There are many process and all are equally valid in their domain.

In this section, we identify commonly used Agile principles and practices that we will use in this course. Although there will always be some contention on what constitutes “best practices”, this is a very common subset of approaches that in practice will work very well.

Principles

From these different Agile models, we can extract a set of useful guiding principles [Pressman 2018]. This is what we aspire to do with our practices.

• Principle 1. Be agile. The basic tenets of agile development are to be flexible and adaptable in your approach, so that you can adjust if needed between iterations. Keep your technical approach as simple as possible, keep the work products you produce as concise as possible, and make decisions locally whenever possible.
• Principle 2. Focus on quality at every step. The focus of every process activity and action should be the quality of the work produced.
• Principle 3. Be ready to adapt. When necessary, adapt your approach to constraints imposed by the problem, the people, and the project itself.
• Principle 4. Manage change. The approach may be either formal or informal, but mechanisms must be established to manage the way changes are requested, assessed, approved, and implemented.
• Principle 5. Build an effective team. Software engineering process and practice are important, but the bottom line is people. Build a self-organizing team that has mutual trust and respect.
• Principle 6. Establish mechanisms for communication and coordination. Projects fail because important information falls into the cracks and/or stakeholders fail to coordinate their efforts to create a successful end product. Keep lines of communication open. When in doubt, ask questions!

Process

Using the SDLC

We can describe our process as the Software Development Lifecycle (SDLC). This illustrates the process that we will follow for this project. Each block in the diagram represents a stage, containing related activities, that are performed in order.

To complete a project, you start with Planning activities, and move through Requirements, Analysis & Design and so on. The project is complete when you finish Evaluation and the team collectively decides that they are “done” (or you run our of time/resources!).

Note that the preliminary activities (Planning, Requirements, Analysis & Design) are only performed once.

Implementation and related activities are grouped together, since they are performed in-order, but we perform multiple passses over all of them. This iteration is called a sprint (taken from Scrum). In a typical development project, sprints should be relatively short, from two to four weeks in length, and the team typically works through multiple sprints until the project is completed.

Each sprint includes the following activities:

1. Feature Selection: On the first day of the Sprint, the team meets and decides what features to add (and what bugs to fix) during that iteration.
2. Implementation/Testing. During most of the sprint, the team iterates on their features. As each feature is completed, it is tested.
3. Evaluation. At the end of the Sprint, the team meets with the Product Owner to demo what they have completed, and get feedback. The team also has a Retrospective, where they reflect on how to improve.

The cycle repeats for however many Sprints the team has available (or until they decide they are “done”). The product should be usable and potentially shippable to a customer at the end of each Sprint (though obviously features may not be complete, but the ones that exist should be bug-free).

Best Practices

Along with the SDLC, we also have a set of best practices that we will use. The SDLC provides the overall organization (what we should do), and these practices provide more strict guidelines (how we should do it).

These will appear in later chapters related to specific activities.

• User stories: describe features in a way that makes sense to customers.

• Pair programming: critical code is written by two people working as a team; one codes while the other one watches, plans and makes suggestions. This results in demonstrably better code and is much more productive than working alone [Böckeler & Siessegger 2020] ⁵.

• Test-driven development: tests are written before the code. This helps to enforce contracts/interfaces as a primary focus of your design. We will discuss this in the Implementation section.

• Code reviews: before code changes are committed to the repository, they need to be reviewed by one or more other developers on the team, ideally more senior members. The claim is that (a) the developer gets feedback to help identify bugs, and improve their design, and (b) participating in code reviews helps spread knowledge about that code around the team. Research suggests that code reviews most often result in design recommendations, and aren’t particularly effective at finding bugs [Czerwonka et al. 2015].

https://xkcd.com/844/