Introduction to full-stack application design and development. Students will work in project teams to design and build complete, working applications and services using standard tools. Topics include best-practices in design, development, testing, and deployment.
Registration is controlled to ensure that we can form adequately sized project teams.
To join, you will need to be registered in corresponding LEC and LAB sections (i.e. both morning LEC/LAB sections, or both afternoon LEC/LAB sections).
Project teams must be registered in the same sections!
Subsections of CS 346 W23
Chapter 1
Syllabus
An outline of the basic course elements, including learning objectives, a weekly schedule of topics, and details on the course project and assessment.
Introduction to full-stack application design and development. Students will work in project teams to design and build complete, working applications and services using standard tools. Topics include best-practices in design, development, testing, and deployment.
Prereq: CS 246; Computer Science students only
Course Requirements
This course is restricted to Computer Science students. You must have completed CS 246 prior to taking this course, and you should be able to:
Design, code and debug small C++ programs using standard tools. e.g. GCC on Unix.
Write effective unit tests for these programs. e.g. informal I/O tests, unit tests.
Demonstrate programming proficiency in C++, which includes: understanding of fundamental OO concepts. e.g. abstraction, encapsulation; knowing how to use classes, objects, method overloading, and single inheritance; polymorphism; understanding how to use assertions, and how to manage exceptions.
Learning Objectives
On successful completion of the course, students will be able to:
Work effectively as a member of a software development team.
Use an interactive process to manage the design, development and testing of software projects.
Design and develop different styles of application software in Kotlin, using appropriate architectural and design patterns.
Design services that can provide remote capabilities to your application.
Apply debugging and profiling techniques to address design and performance issues.
Produce unit and integration tests as part of the development process.
Required Resources
There are no required textbooks for this course. This website is the primary source for reading material, recorded videos and other supporting materials. For those that want to pursue topics in greater detail, there are additional resources listed in each chapter. None of these are required reading unless specifically indicated.
You will need access to a computer to work on the project, which should be capable of running the toolchain described here. Please check your system requirements early in the course, and ask the instructor if you have any concerns.
Title / Name
Notes / Comments
Required
Computer/laptop
Windows, Linux or macOS with at least 8GB of RAM.
Yes
Slides and lecture videos
Will be made available on course website.
Yes
Supporting slides/videos
Will be made available on course website or via links to external resources.
Yes
Course Staff
Course staff are here to help! You can find us in-class, or you can post questions on MS Teams channels1.
Faculty & Staff
Dr. Jeff Avery (j2avery@). Course Instructor, responsible for course design and instruction.
Caroline Kierstead (ctkierst@). Instructional Support Coordinator (ISC), handles academic integrity and course accomodations.
Instructional Apprentice (IA)
Xiaoyan Xu (x439xu@uwaterloo.ca). Senior TA, who assists the instructors in running the course.
We will try and keep up with questions asked during normal business hours (9 AM - 5 PM) but there is no guarantee of a prompt reply after-hours or on weekends. ↩︎
Sprint demos are your chance to show a TA what you’ve been working on! We just expect you to describe and demonstrate your features so that we can provide feedback.
See sprint demo in the project activities section.
Quizzes are completed on your own time, and must be submitted by 11:59 Fri evening. ↩︎
Classes are cancelled on Fri Apr 7 due to Good Friday, so the final demo is moved to Mon Apr 10. The final submission must be handed in by 11:59 PM that evening. There are no extensions past the end of the term. ↩︎
Subsections of Schedule
Assessment
This course is designed around a single project that you will design and develop with your project team. Most course components are tied to the course project in some way.
Team Grade (65%)
You will have three different types of deliverables this term.
You are expected to meet regularly with your team. This includes participating in sprint kickoffs, sprint demos, and in-class working sessions. Your participation grade will be calculated at the end of the term, and will be based on project data (e.g. your relative contribution) and attendance (e.g. did you attend and participate through the term).
Quizzes are a review of the lecture material and need to be done individually. See the quizzes pages for details.
Failing to attend a demo will result in a grade of zero for that component IN ADDITION to any penalty to your participation grade. Under exceptional circumstances e.g. coop interview, documented illness, you may be excused from a demo without penalty, as long as you coordinate with your team and the instructor ahead of time. ↩ ↩︎
Quizzes
You have 5 quizzes to complete during the term. Quizzes are comprehensive, and can include any content from the indicated lectures (though they will tend to favour newer material).
#
Lectured Covered
Quiz Availability
Q1
Weeks 1-2
Week 3 : Mon Jan 23 - Fri Jan 27
Q2
Weeks 1-4
Week 5 : Mon Feb 6 - Fri Feb 10
Q3
Weeks 1-7
Week 8 : Mon Feb 27 - Fri Mar 3
Q4
Weeks 1-9
Week 10 : Mon Mar 13 - Fri Mar 17
Q5
Weeks 1-11
Week 12 : Mon Mar 27 - Fri Mar 31
Policies
All quizzes are available in LEARN during the dates listed above. They open 10:00 AM on the Monday morning, and close at 11:59 PM on the Friday of each week.
Quizzes must all be complete and submitted by Fri 11:59 PM of the given week. Late submissions will NOT be accepted.
Once you start the quiz, you must complete and submit it within 30 minutes. Failure to submit results in a grade of zero for that quiz. There are no extensions on Quizzes once they have been started.
Quizzes must represent your individual work. You may not communicate with anyone else about quiz contents, or disclose information about the quizzes to anyone else. You agree to not record or disclose quiz contents to any third-party (including but not limited to Chegg.com and similar sites).
Course Policies
Team Formation
This is a team-based project course, and students are expected to work together all-term. The following guidelines apply to team formation:
Students are responsible for matching up and forming teams. Assistance will be given in the first two weeks.
Course enrolment will be managed by the instructor so that we are able to form teams of four. If required, the instructor may authorize larger or smaller teams to accomodate everyone that is enrolled.
Teams must be formed by the end of the second week of the term (i.e. the add-course deadline). If you fail to find a team, you must inform the instructor by the end of the second week.
We do not guarantee you a team after the end of the second week. If you have not joined a team, and have not made some arrangement with the instructor, you may be required to withdraw from the course.
Absences
Students are expected to attend classes with their team. Failing to attend a demo of any kind (e.g. kickoff, demo) will normally result in a grade of zero for that course component1.
However, we recognize that circumstances may require you to miss a demo due to coop interviews, testing positive for COVID, short-term absences, or other unusual circumstances. In all cases where you need to miss a component, the following guidelines apply:
You must contact the instructor and your teammates ahead of the due date. If missing a demo, you must coordinate with your team since they will be expected to proceed without you.
You must provide the instructor with a reason for missing the component (e.g. illness/COVID, STA).
You are still expected to complete your work leading up to that deadline i.e. you are excused from presenting but not automatically excused for completing your work for the sprint!
The instructor will consider your request, and either (a) grant you an exemption from this component and calculate your grade across other components, (b) grant you an extension, where you are expected to meet the requirement at a later date, or (c) not grant your request, in which case you have to either attend/contribute or accept a grade of zero. Granting an exemption is ultimately at the discretion of the instructor.
Normally you can miss at-most one demo or other deliverable during the term without penalty.
Info
If you test positive for COVID or have COVID-symptoms, you should use the COVID self-declaration form. If you are ill but do not meet the COVID self-declaration guidelines, you should instead follow the guidelines and steps outlined in the Math Accommodations page. In both cases, you must also follow the instructions listed above to determine if you are eligible for an accommodation and what form it will take.
Inclusiveness
It is our intent that students from all diverse backgrounds and perspectives be well-served by this course, and that student’s learning needs be addressed both in and out of class. We recognize the immense value of the diversity in identities, perspectives, and contributions that students bring, and the benefit it has on our educational environment. Your suggestions are encouraged and appreciated. Please let us know ways to improve the effectiveness of the course for you personally or for other students or student groups. In particular:
We will gladly honour your request to address you by an alternate/preferred name or gender pronoun. Please advise us of this preference early in the term so we may make appropriate changes to our records.
We will honour your religious holidays and celebrations. Please inform us of these at the start of the course.
We will follow AccessAbility Services guidelines and protocols on how to best support students with different learning needs.
Academic Integrity
In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility. Contact the Office of Academic Integrity for more information. You are expected to follow the policies outlined above for quiz and project submissions.
To ensure academic integrity, MOSS (Measure of Software Similarities) is used in this course as a means of comparing student projects. We will report suspicious activity, and penalties for plagiarism/cheating are severe. Please read the available information about academic integrity very carefully.
Ethical Behaviour
Students are expected to act professionally, and engage one another in a respectful manner at all times. This expectation extends to working together in project teams. Harassment or other forms of personal attack will not be tolerated. Course staff will not referee interpersonal disputes on a project team; incidents will be dealt with according to Policy 33.
Plagiarism
Students are expected to either work on their own (in the case of quizzes), or work within a project team (for the remaining deliverables in the course). All work submitted should either be their own or created by the team for use in their project. However, we realize that it is common practice to use third-party libraries and sources found online to solve programming problems. For this reason, the team is allowed to use third-party source or libraries for their project provided that (a) they document the source of this contribution in source code, typically as a comment, and in their README file, and (b) no single source constitutes more than 10% of their project. Failure to acknowledge a source will result in a significant penalty (10% or more) of your final project grade, depending on the severity of the infraction. Note that MOSS will be used to compare student assignments, and that this rule also applies to copying from other student projects.
Student Discipline
A student is expected to know what constitutes academic integrity to avoid committing an academic offence, and to take responsibility for his/her actions. A student who is unsure whether an action constitutes an offence, or who needs help in learning how to avoid offences (e.g., plagiarism, cheating) or about ‘rules’ for group work/collaboration should seek guidance from the course instructor, academic advisor, or the undergraduate Associate Dean. For information on categories of offences and types of penalties, students should refer to Policy 71, Student Discipline. For typical penalties check Guidelines for the Assessment of Penalties.
Intellectual Property
Students should be aware that this course contains the intellectual property of their instructor, TA, and/or the University of Waterloo. Intellectual property includes items such as:
Lecture content, spoken and written (and any audio/video recording thereof)
Lecture handouts, presentations, and other materials prepared for the course (e.g., PowerPoint slides)
Questions or solution sets from various types of assessments (e.g., assignments, quizzes, tests, final exams), and
Work protected by copyright (e.g., any work authored by the instructor or TA or used by the instructor or TA with permission of the copyright owner).
Course materials and the intellectual property contained therein, are used to enhance a student’s educational experience. However, sharing this intellectual property without the intellectual property owner’s permission is a violation of intellectual property rights. For this reason, it is necessary to ask the instructor, TA and/or the University of Waterloo for permission before uploading and sharing the intellectual property of others online (e.g., to an online repository). Permission from an instructor, TA or the University is also necessary before sharing the intellectual property of others from completed courses with students taking the same/similar courses in subsequent terms/years. In many cases, instructors might be happy to allow distribution of certain materials. However, doing so without expressed permission is considered a violation of intellectual property rights.
Continuity Plan
As part of the University’s Continuity of Education Plan, every course should be designed with a plan that considers alternate arrangements for cancellations of classes and/or exams.
Here is how we will handle cancellations in this course, if they occur.
In the case of minor disruptions (e.g. one lecture), the lecture content will be reorganized to fit the remaining time. This should not have any impact on demos or deliverables.
Cancellation of in-person classes, may result in a reduction in the number of sprints and associated deliverables to fit the remaining time. If this happens, lecture content will also be pruned to fit available time. Assessment weights will be redistributed evenly over the remaining content if required to align with the material.
Cancellation of in-person (midterm or final) examinations has no effect on this course, since we do not have scheduled exams. Quizzes will continue to be written, but will be adjusted to the modified schedule if necessary.
See the assessment section. Part of your grade is based on participation and attendance. ↩︎
Supports
Students with Disabilities
AccessAbility Services collaborates with all academic departments to arrange appropriate accommodations for students with temporary or permanent disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations, please register with the AccessAbility Services at the beginning of each academic term. They will in-turn contact your instructors and arrange accomodations if necessary.
Mental Health Resources
If you or anyone you know experiences any academic stress, difficult life events, or feelings like anxiety or depression, we strongly encourage you to seek support.
MATES: one-to-one peer support program offered by Waterloo Undergraduate Student Association (WUSA) and Counselling Services: mates@wusa.ca
Health Services: located across the creek from the Student Life Centre. 519-888-4096.
Off-campus
Good2Talk (24/7): Free confidential help line for post-secondary students. Phone: 1-866-925-5454 (Ontario and Nova Scotia only)
Here 24/7: Mental Health and Crisis Service Team. Phone: 1-844-437-3247 (Waterloo Region only)
OK2BME: set of support services for lesbian, gay, bisexual, transgender, or questioning teens. Phone: 519-884-0000 extension 213 (Waterloo Region only)
Details of the course project, including the primary activities that you will undertake this term.
Subsections of Course Project
Introduction
This is a project-based course, where teams collectively scope, define, implement and deliver a full-stack application. Over the course of the project, you and your team will:
Determine what to build (requirements analysis).
Determine what features to implement and how they interact (design).
Write automated tests to ensure quality (testing).
Figure out how to package and present your software (deployment).
What are we building?
The specification page details a number of different projects, and provides some guidance on the basic requirements for each project. You and your team must pick one, and then determine (a) how to design and build it, using the specification, (b) who you target audience is, and what other features might be useful, and (c) decide what additional features to add over the term.
What will we be doing?
Review the project activities page for details on how the course is structured. The schedule also provides a weekly guide to what to expect.
Forming teams
Software development is not a solitary activity! You need a variety of people with different skills on your team. Teams should consist of four people, all enrolled in the same section of the course.
You are expected to find teammates and form project teams in the first week of the course 1. Here’s some ideas on how to find team members:
Join friends who are taking the course! If you are in different sections, ask the instructor, and they may be able to move you all into the same section.
Post in the course forums: we will have a forum thread where you can introduce yourself.
If you’re in-class, introduce yourself to people sitting near you.
Info
Look for team members that have similar goals and work habits as you.
Do you have the same work schedule? Are you available and willing to work at the same times (e.g. mornings? evenings?)
Are you all willing to make the same time and effort commitment to the course? If most of the team wants to put in extra time to get an A+, then you need to make sure that everyone is on-board to do that.
Look for complementary skills! Not everyone needs to be a (fill-in-the-blank) programmer. There’s room for a lot of different skills to be applied to your project.
Consider how team members may be able to specialize in different areas that match their skills, interests or both!
We will attempt to enrol the correct number of students to meet this requirement. If smaller or larger teams are required, you will to need to get permission and coordinate with the instructor. We will NOT normally authorize team changes past the end of the second week, so if you do not have a team at that point, you may be required to withdraw from the course. ↩︎
Specification
Choice of Project
You and your team will need to choose from one of projects listed below
It is equally possible to get a good grade with any of these projects. However, features vary in complexity and effort required, so your choice will obviously have some impact on how your term will proceed (e.g. you may need to spend more time tinkering with graphics for the UML editor compared with other projects).
The lectures tend to deal with broad problems that you will encounter in any project (e.g. “how do I use a local database”, or “how do I build a GUI”), and aren’t tied to any specific project.
Requirements
The specification documents below describe required features for each project. Implementing these features is sufficient to pass the project and the course.
However, for a higher grade, you may wish to consider adding extra features. From the Assessment page:
Each project specification includes both Required Features and Additional Features. You are expected to deliver ALL of the Required Features, and doing so will earn you up to a B on the Final Submission. If you wish to earn a higher grade on the Final Submission, you need to implement one or more Additional Features. Each specification has some ideas, but you are welcome to come up with your own as well - just check with the TA and instructor to make sure they are reasonable additions.
Technical Stack
Regardless of the project you choose, you need to use the following technical stack:
If you wish to use anything different, you must gain instructor approval1.
Info
You will need a personal computer that is capable of running these tools. Any relatively recent computer with at least 8 GB of RAM should be sufficient. These tools all run on macOS, Windows and Linux equally well.
Email is fine, but the request and approval must be in writing, and included in your project documentation (e.g. README). ↩︎
Subsections of Specification
Notes Application
A note-taking application is a standalone application designed for recording and searching short, ad-hoc notes. Typically they are used by people as an informal way of organizing information for their own consumption e.g. code snippets, recipes, daily journal notes.
Although there are many different designs, the focus of every note-taking application is storing and managing text notes. Common features include adding, editing, deleting notes, as well as ways to organize or search them.
Example: Apple Notes.
There are even command-line applications which attempt to solve the same issue. For example, here’s Terminal Velocity on Linux, optimized for very quick note entry, for users who are accustomed to working from a terminal.
Required Features
There are four specific deliverables for this project:
Core infrastructure
Console application
GUI application
Web service
Core Infrastructure
Your source code should be stored in your team’s GitLab repository.
You must use issues in GitLab to track your work.
You must use a Gradle project, with the project structure provided in-class.
You are expected to write unit tests for all of your code, and add appropriate code comments.
You are also expected to produce some form of installer, so that a user (or TA) could install and run your application without needing to build it from source!
We will examine your source code! You should strive for well-structured and readable code.
Console Application
Your first application should be a console application that launches the notes application. You can add support for additional arguments if you wish.
Usage: notes
GUI Application
Your main application is a desktop application. You are expected to implement the following standard desktop functionality:
A top-level menu bar that lists major functions, and indicates the hotkeys for each feature (e.g. File, Edit, View menu and submenus).
Toolbars that let the user control settings and modes that apply to the application e.g. a Bold button that can be used to embolden text, and reflects the state of selected text.
Window resizing, so that the user can resize and reposition the application. You should save the window size and position on exit, and restore that size and position when they relaunch it.
Minimize/mazimize buttons function as expected.
Undo-redo support for actions in the user interface.
Cut-copy-paste text.
In addition to the platform requirements, you should support the following features:
Create, edit, delete a note (using both keyboard and point-click).
Group related notes together e.g. folders or tags. Support add/delete and rename groups, and move notes between groups.
Display a list of notes by title, and allow sorting the list by title, modified date or created date.
Search notes, either by title or body contents, and display the matching notes.
Rich text support for the note body, including bold, italics, underlining and changing colour of text.
Support for bulleted lists, including the ability to apply or remove bullets from a block of selected text.
Save local application settings (e.g. window size and position) on the local system where the application is installed.
Note data should be stored remotely and shared across application instances.
Your application should work equally well on Windows, Linux or macOS.
Service
You will need to build a web service so your main data can be served remotely.
A user should be able to launch multiple instances of the application, and have the instances load and use this remote data.
Your application should communicate with the web service using HTTP protocol.
Data should be saved so that if the web service is stopped and restarted, the data will persist.
Additional Features
The requirements above represent the minimal set of requirements for this project. Here we’ve listed some other ideas that might be interesting to implement! Pick-and-choose, or brainstorm your own.
Image support, so that users can drag-drop images into a note, and resize them.
Diagramming, so that the user can drag-drop and manipulate simple shapes within a note e.g. box and arrow diagrams.
Support light and dark themes. The user should be able to set their preference in a Settings dialog and this setting should persist.
Examples
Here’s an incomplete list of similar applications. Use these to get a sense of what features you might want to consider, or some design approaches.
You will create a Markdown editor: a text editor with features that support that markup language, including syntax highlighting, headings, and bulleted lists.
Here’s a screenshot of Typora, my preferred editor.
Required Features
There are four specific deliverables for this project:
Core infrastructure
Console application
GUI application
Web service
Core Infrastructure
Your source code should be stored in your team’s GitLab repository.
You must use issues in GitLab to track your work.
You must use a Gradle project, with the project structure provided in-class.
You are expected to write unit tests for all of your code, and add appropriate code comments.
You are also expected to produce some form of installer, so that a user (or TA) could install and run your application without needing to build it from source!
We will examine your source code! You should strive for well-structured and readable code.
Console Launcher
Your first application should be a console application that can launch the markdown editor with the document indicated on the command line. For example:
Usage: mdown [filename]
GUI Application
Your main application is a desktop application. You are expected to implement the following standard desktop functionality:
A top-level menu bar that lists major functions, and indicates the hotkeys for each feature (e.g. File, Edit, View menu and submenus).
Hotkeys/keyboard shortcuts for major functionality.
Window resizing, so that the user can resize and reposition the application. You should save the window size and position on exit, and restore that size and position when they relaunch it.
Minimize/mazimize buttons function as expected.
Undo-redo support for actions in the user interface.
Cut-copy-paste text.
In addition to the platform requirements, you should support the following features:
You should support basic Markdown syntax, including headings, bold, italics, and so on. This includes syntax highlighting for these elements, and hotkeys to apply formatting (e.g. CMD-B to bold selected text).
Users should be able to open an existing markdown file (extensions .md or .markdown), close the current file, save or save-as with appropriate dialogs.
It should be possible to have multiple markdown files open simultaneously (in different windows or different tabs).
You should support light and dark themes, and the user should be able to set their preference in a Settings dialog. These settings should persist across sections and files.
Provide two view modes: raw markdown syntax (as above), or formatted text. You could do this as two panes that exist side-by-side (e.g. VS Code, or MarkdownPad) or as a mode so that the user can switch between the two different views (Typora).
Save application settings in the cloud so that they are preserved across sessions. See below.
Your application should work equally well on Windows, Linux or macOS.
Service
You will need to build a web service so your data can be shared across multiple systems.
A user should be able to launch multiple instances of the application, and have the instances load and use this remote data. You should determine what data is appropriate to save in this fashion.
Your application should communicate with the web service using HTTP protocol.
Data should be saved so that if the web service is stopped and restarted, the data will persist.
Additional Features
The requirements above represent the minimal set of requirements for this project. Here we’ve listed some other ideas that might be interesting to implement! Pick-and-choose, or brainstorm your own.
Add themes! Dark, Light, Solarized etc.
Stylize the toolbars e.g. a fancy VS Code style, expanding toolbar.
Support editing both in raw markdown or formatted text (similar to Typora, where you can mode-switch but edit in either mode).
Save files in the cloud, instead of just settings. [ed. this would likely mean the ability to import files, browse, and delete from the cloud as well].
You will build a collaborative diagram editor, that allows people to build different types of diagrams together.
Here’s an example of draw.io, a drawing application that be used directly from their website, or as an installed application [ed. Note that draw.io supports more shapes than you would be expected to support in your project].
Required Features
There are four specific deliverables for this project:
Core infrastructure
Console application
GUI application
Web service
Core Infrastructure
Your source code should be stored in your team’s GitLab repository.
You must use issues in GitLab to track your work.
You must use a Gradle project, with the project structure provided in-class.
You are expected to write unit tests for all of your code, and add appropriate code comments.
You are also expected to produce some form of installer, so that a user (or TA) could install and run your application without needing to build it from source!
We will examine your source code! You should strive for well-structured and readable code.
Console Launcher
Your first application should be a console application which can launch the diagram editor with the document indicated on the command line.
Usage: diagram [filename]
GUI Application
Your main application is a desktop application. You are expected to implement the following standard desktop functionality:
A top-level menu bar that lists major functions, and indicates the hotkeys for each feature (e.g. File, Edit, View menu and submenus).
Hotkeys/keyboard shortcuts for major functionality.
Window resizing, so that the user can resize and reposition the application. You should save the window size and position on exit, and restore that size and position when they relaunch it.
Minimize/mazimize buttons function as expected.
Undo-redo support for actions in the user interface.
Cut-copy-paste text.
In addition to the platform requirements, you should support the following features:
Allow creation of different types of diagrams, including simple UML diagrams (e.g. class diagrams), flowcharts or other box diagrams.
Users can add/remove/edit shapes; shapes should be selectable, movable and resizable.
Visual properties of shapes can be modified (e.g. border width, colour).
Intuitive hotkeys that handle mode-switching well.
Support saving, loading the diagram from a file (using a format that you create).
Additionally, save local application data (for window size and position).
Save application settings in the cloud so that they are preserved across sessions. See below.
Your application should work equally well on Windows, Linux or macOS.
Service
You will need to build a web service so that shared data can be served remotely.
A user should be able to launch multiple instances of the application, and have the instances load and use remote data.
Your application should communicate with the web service using HTTP protocol.
Data should be saved so that if the web service is stopped and restarted, the data will persist.
Additional Features
The requirements above represent the minimal set of requirements for this project. Here we’ve listed some other ideas that might be interesting to implement! Pick-and-choose, or brainstorm your own.
Connectors snap-to the shapes, so that you can connect them, move them and the connections follow.
Add different types of diagrams e.g. floorplans.
Allow users to build their own templates and populate these with shapes.
Support full collaborative editing, so that users can edit a diagram together.
Save files in the cloud, instead of just settings. [ed. this would likely mean the ability to import files, browse, and delete from the cloud as well].
You will design and build a digital whiteboard - similar to something that we would have in a classroom, but meant to be used to collaborate online! The whiteboard is initially a large blank canvas; users can write on it, sketch simple diagrams, or even insert images. Multiple people can connect to the whiteboard from different systems and collaborate together in real-time!
There are four specific deliverables for this project:
Core infrastructure
GUI application
Web service
Core Infrastructure
Your source code should be stored in your team’s GitLab repository.
You must use issues in GitLab to track your work.
You must use a Gradle project, with the project structure provided in-class.
You are expected to write unit tests for all of your code, and add appropriate code comments.
You are also expected to produce some form of installer, so that a user (or TA) could install and run your application without needing to build it from source!
We will examine your source code! You should strive for well-structured and readable code.
Client Application
Your main application is a desktop application. You are expected to implement the following standard desktop functionality:
A top-level menu bar that lists major functions, and indicates the hotkeys for each feature (e.g. File, Edit, View menu and submenus).
Window resizing, so that the user can resize and reposition the application. You should save the window size and position on exit, and restore that size and position when they relaunch it.
Minimize/mazimize buttons function as expected.
In addition to the platform requirements, you should support the following features:
Multi-user login. You should prompt the user for username/password and use these to authenticate against your service.
Changes made by one person should be shown in real-time on other user’s applications. (No “manual sync” required).
Once logged in, the whiteboard will be loaded automatically.
There are tools that support:
drawing (multiple pen styles and colours),
inserting text (multiple fonts, sizes and colours).
inserting shapes (rectangles, plus one or more additional shapes)
editing existing content
selecting and moving content around the whiteboard
erasing content
Your application will need to save some local data (for window size and position). Other data should be saved on your remote service. See below.
Your client application should work equally well on Windows, Linux or macOS.
Service
You will need to build a web service so your data can be shared across multiple systems.
A user should be able to launch multiple instances of the application, and have the instances load and use this remote data. You should determine what data is appropriate to save in this fashion.
Your application should communicate with the web service using HTTP protocol.
Data should be saved so that if the web service is stopped and restarted, the data will persist.
Additional Features
The requirements above represent the minimal set of requirements for this project. Here we’ve listed some other ideas that might be interesting to implement! Pick-and-choose, or brainstorm your own.
Add themes! Dark, Light, Solarized etc.
Stylize the toolbars e.g. a fancy VS Code style, expanding toolbar.
Allow users to save/restore different whiteboards so that they can save their progress. Data would need to be saved in the cloud.
Add specialised tools e.g. class diagram templates, or arrows/boxes that people can use to draw more professional looking diagrams.
Flag the annotations on the board with a username. Establish permissions based on username and limits editing each others content (e.g. a user could set a flag to make her image “read-only” and restrict others from erasing it).
This section describes the project activities that you will perform.
1. Project Review
In the first few weeks, you and your team will choose a project, define requirements, and discuss design issues.
Around the end of week 4, you will have a project review meeting with a TA to discuss your project.
You should have early project decisions made by that time, and requirements should be logged in GitLab.
You do not need to submit anything! This is an informal presentation and discussion of your project to ensure that you have done the required preliminary work. You should be able to answer all of the points in the project review checklist.
See the project review checklist (pdf, docx) for marking guidelines.
2. Sprints
Most of the term will be spent iterating on your project. You will be working in two-week iterations, called sprints, that have the structure outlined below. Over a two-week period, you will have a sprint-kickoff, do some work, and finally demo that work to the TA.
Your first sprint will start the week after your project review. The last sprint will complete in the last full week of the term. See the schedule for exact dates of each sprint.
Day 1: Kickoff
Although not formally part of the sprint kickoff, you can expect the first day of the sprint to open with a 1-hour lecture by the instructor. This lecture will cover core technical topics needed for that sprint, and aim to provide some guidance on how to structure your work.
The remainder of the first day is the official sprint kickoff, where your team collectively decides what you want to include in the sprint. Your primary task in the kickoff is to choose items from the Product Backlog and assign them.
Here’s some suggestions to help you determine what to do during the sprint.
The sprint guidelines page has suggestions for each sprint. You are not required to follow this path, but use it as a guideline.
Address feedback from the previous sprint’s demo. You may have received feedback from the previous sprint. Treat suggestions from course staff as important - they represent your customers!
Address high-risk items early. This gives you time to pivot if needed, and also helps prevents you from investing too much time in a path that ultimately won’t work.
Look for blocking issues: items that are critical for other systems. Examples of this might be the class structure for business objects (e.g. data classes) that are used by other features.
Do NOT assign more work than you think you can do.
Outcomes from this meeting:
You should have all issues logged and assigned to the team, representing the work for this sprint.
Meeting minutes should be recorded using the meeting minutes template (pdf, docx) and stored in the /meeting-minutes directory in your repository.
Day 2/3: Standup
These are “working days” where your team gets together and does the actual work towards the sprint’s goals. You are expected to work in-class together; the instructor and TAs will be available to help you out.
Outcome from these meeting:
Meeting minutes recorded using the daily standup template (pdf, docx). Store them in the /meeting-minutes directory in your repository.
Day 4: Demo
The last day of a sprint is demo to the course staff of what you’ve accomplished during the sprint. This is a checkpoint of sorts, where we can provide feedback and make suggestions.
This is what you should have completed before the demo:
You should have recorded meeting minutes for every team meeting prior to the demo.
The core requirements for the sprint should be completed - see sprint guidelines for details.
Completed issues should be closed in GitLab with details on what was done.
Completed work needs to be committed and merged back to the main branch.
You should have unit tests that cover the features that you have completed.
You should have readied a software release - see project artifacts for details.
You should have release-notes for this release in your GitLab project, which documents the release.
You should have an installer for your application, which you use for the demo. You should NOT demo from IntelliJ.
What does the demo look like?
Your demo will be informal and last about 15 minutes.
Using an INSTALLED version of the application, demo each completed feature, and answer any questions. (Your issues list in GitLab is a great way to drive this disccussion).
The person who developed the feature should demo it.
All demos need to be run from the same machine, using the build from the software release.
After the demo, your team should discuss how things went, and what areas could be improved. Reflect on ways to improve your development process so that you can be even more effective in the next sprint.
Outcomes from this meeting:
Open issues from this sprint should be moved to the product backlog (i.e. unassigned from the sprint). They can be discussed again in the next sprint kickoff.
Reflections from the team should be recorded in your meeting minutes using the meeting minutes template (pdf, docx) and stored in the /meeting-minutes directory in your repository.
The TA will assign a mark and your grade will be returned the week after the demo.
See the sprint demo checklist (pdf, docx) for marking guidelines.
3. Final Submission
There is also a final submission due at the end-of-term. This will be an offline evaluation, where we consider all completed features, how well they work together, application design, source code structure and so on. Everyone on the team will receive the same grade for this component.
We will grade this after the end of the term (and after the final sprint/demo has been presented).
See project artifacts for more details on what you should have completed.
See final checklist (pdf, docx) for marking guidelines.
Info
You will be provided with a project specification that includes both Required Features and Additional Features. You are expected to deliver ALL of the Required Features, and doing so will earn you up to a B on the Final Submission.
If you wish to earn a higher grade on the Final Submission, you need to implement one or more extra features - either from the Additional Features in the specification, or ideas of your own. Check with the instructor if you are uncertain what to add!
Project Artifacts
This section describes the project artifacts that you will create and use through the term. In general, each of these represents something that you will iterate over through the course of the term.
It also represents what we will grade at the end of the term, for your final submission grade.
Give it a meaningful name; a “project slug” is just a version of the name with no spaces (and typically lowercase).
Make the repository private so that it’s not visible to the rest of the class.
Add your Team Members and course staff as Developers on the project.
Initialize with a README.md file, and see below for suggestions on formatting and contents.
Finally, email the instructor with a list of team members and your repository URL.
Info
This is where you should keep track of everything that you do in this course! This includes meeting minutes, source code, scripts etc. Your completed projects with all of the contents below constitute your final submission (see assessment for details).
The following sections of the project should be developed for the first sprint and updated during successive sprints.
Issue tracking, where each issue represents a requirement that you wish to implement or a task/issue that needs to be resolved. You should document your progress in the issue, and close it when the work is complete.
Milestones, where each Sprint is tracked as a milestone with a due date. This helps you keep track of what you have assigned in each sprint.
Continuous integration (CI/CD) to create builds and run tests automatically from your repository.
2. Source Code Repository
All assets (source code, documents, diagrams, images, scripts) should be stored in your project’s source code repository. The structure should look something like this by the end of the term.
Your repository should contain the following contents:
README.md
You should have a markdown file in the root of your source code repository named README.md. This will serve as a landing-page for your project (i.e. its the first thing that users will see when visiting your project page!). It should contain the following sections.
# PROJECT NAME
## Goal
A brief description of your product. What is it? What does it do?
## Team members
List each person's name and email address.
## Quick-start
How to install and launch your application.
## Screenshots/videos
Optional, but often helpful to have a screenshot or demo-video for new users.
## Releases
Each sprint should produce a software release. Provide a list of releases,
including links to release notes and installers. For example:
version 0.1: released Feb 17, 2023
* release-notes (txt)
* installers (linux, macos, windows)
version 0.2: released Mar 10, 2023
* release-notes (txt)
* installers (linux, macos, windows)
Source code
Most of your repository contains the source code and related documents for your project. It is recommended that you follow the structure provided above, which will build properly in GitLab CI/CD.
Your entire project, and all related code and resources, must reside in this repository.
Software releases
Each sprint should produce an installable and runnable version of your project, which should contain the features that were completed during the sprint. The output from a sprint should be a release-quality build of your product, which you will store in your project repository.
Your README.md file (from above) should include links to the components of each software release. Your software release should include:
Release notes: a markdown or text file, containing at least:
the release date (which should be the sprint demo date)
the version of this release (which you increment for each release)
a list of changes that were included.
Installers: this is a packaged version of your application that a user could use to install your software. You should support installation on at least 2 platforms (pick two from Linux, Windows, macOS - whatever you have available).
Acceptable forms of packaging include:
a zip or tar file that, when extracted, contains bin/ and lib/ directories, and a script that launches the application (produced from distZip or distTar tasks in Gradle).
an exe, dmg or pkg file that, when executed, installs the application (produced from JPackage task in Gradle ).
Sprint Guidelines
Here’s how we would recommend that you structure your sprints.
Tip
Lectures will be presented in this order, so approaching your project this way may be the easiest in terms of staying on top of the course material. You are welcome to approach topics in a different order if you feel that is more appropriate for your project.
Sprint 1: Infrastructure & GUI
Complete all of the Core Infrastructure requirements. These are critical and must be completed before you can start adding functionality.
Complete your domain objects e.g. data class, specialized container classes. Making early design decisions on your data model will make other features much, much easier to implement.
Complete the Console Application project. This could be a simple application, or a launcher for the GUI, depending on the project that you choose.
Complete as many of the GUI Application features as you can schedule. Prioritize basic functionality that will help you test other functions (e.g. create a note, or create a document).
Sprint 2: Data Persistance
Continue working on GUI features. You should be progressing past simple features into more complex ones. Undo/Redo, and Cut/Copy/Paste are good features to consider at this point because they rely on a completed data model.
Implement your persistance layer in your shared project. You need to persist local data e.g. window size and position, as may need to save remote domain data e.g. notes, or shared settings. Decide if you are saving data in JSON files, XML, or a database.
Make sure that you are saving all UI state that you want to restore as part of your local data.
Sprint 3: Web Service
Finalize your GUI features. You should implement the more advanced features that build on the core functionality e.g. rich-text support, syntax highlighting for programming languages and so on.
Implement a web service. Create a new project for this. Your web service should be able to handle HTTP requests and use HTTP methods to fetch/update data from your remote service.
Expand your service to store data. This is required to support concurrent access, scalability etc.
Sprint 4: Wrapup
Move your service to a cloud platform. This will likely mean containering it (Docker) and installing the service to run remotely (Microsoft Azure).
Address synchronization. You need some way for multiple connected client applications to keep their data synchronized. e.g. how does client1 know that client2 has updated some data? Use web sockets to have your service notify connected clients when data is updated to avoid stale data.
Build a better installer! Use JPackage to build an installation wizard.
Complete any other remaining features. Everything should be done at this end of this sprint!
Chapter 3
Software Process
An outline of the basic course elements, including learning objectives, a weekly schedule of topics, and details on the course project and assessment.
Subsections of Software Process
Introduction
Software as a product is quite unique:
It’s intangible. There is no physical product.
Software is highly customizable, more than than practically any other product.
Software is initially expensive to produce, but relatively cheap to distribute since distribution can often be done digitally i.e. with no physical copies required. Once “built”, we can distribute as many copies as we wish at little to no additional cost.
Software doesn’t “wear out” like a physical product but it requires ongoing maintenance as new features are introduced, or the operating environment changes e.g. new OS versions being released, driver updates.
Although software development can be seen as a creative endeavor, the process of building software shares a lot of similarities with other forms of manufacturing. To build reliable, safe, high-quality systems, we lean on the discipline of software engineering:
Software Engineering: The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software.
The foundation for software engineering is the process layer. The software engineering process is the glue that holds the technology layers together and enables rational and timely development of computer software. Process defines a framework that must be established for effective delivery of software engineering technology.
A software process is a collection of activities, actions, and tasks that are performed when to create software. An activity is used to achieve a broad objective (e.g., communication with stakeholders) and ios applied during the process. Actions (e.g., architectural design) encompasses a specific set of tasks that may be used for a particular activity (e.g., an architectural model).
Process Activities
A process framework establishes the foundation for a complete software engineering process by identifying a small number of framework activities that are applicable to all software projects, regardless of their size or complexity. In addition, the process framework encompasses a set of umbrella activities that are applicable across the entire software process. A generic process framework for software engineering encompasses five activities:
Communication: We need to define the problem that we’re trying to solve, and discuss the goals and requirements with the customer, potential users, and other stakeholders. This is critical to ensure that we’re building the “right” product or solving the main problem that needs to be addressed.
Planning: This involves defining the tasks that need to be completed, and high level milestones that you need to achieve. It also includes identify people and resources, potential risks to the project, and developing a plan to mitigate these risks. These are what we would typically call “project management” activities.
Modeling: It’s always cheaper to think ahead prior to actually building something. This step includes design efforts to create abstract models or representations of the product that you need to build. By iterating on a design, you refine your understanding of the problem, and move towards a more-correct solution. This is crucial before you actually start building it, both to improve the accuracy and usefulness of what you build, but to minimize costs.
Construction: The process of building your product i.e. realizing your design. This may require successive iterations to complete all of the required features, and should also include some level of testing and validation.
Deployment: Tasks required to produce a working product, and deliver it to the customer. This includes collecting their ongoing feedback and making revisions as needed.
Let’s review these in more detail.
Communication
Products, in general, are designed and built to solve particular problems for users. The first, and most important step in any project is ensuring that you understand both the users, and the problem that you are attempting to address. Once that’s done, you need to ensure that there is agreement by everyone on both the problem and proposed solution.
Users
The people impacted by your product are called stakeholders. This includes the people who will use your solution, but can also include anyone else that is affected by it. For example, in a software project, stakeholders can include:
users, the people who will directly utilize your software;
the Information Technology (IT) department who will have to install and maintain your software;
people who do not directly use your product, but who may need to provide input to your system, or work with it’s output;
your company, and you personally since you presumably need to maintain the software over time.
As software designer and developers, we tend to focus on the actual users of our software and services, but we need to consider the needs of all of these stakeholders.
Requirements
Requirements analysis is the set of activities designed to identify problems in sufficient detail to determine a solution. Requirement specification is the identification and documentation of the capabilities or requirements that are needed to address a particular problem for stakeholders. Types of requirements include:
Architectural requirements: Related to the system architecture of a system. e.g. how it will integrate with an existing system, or how it must be deployed to be successful.
Business requirements: High-level organizational goals that your product or solution will help address, directly or indirectly.
User requirements: Statements of the needs of a particular set of stakeholders (namely, those that will use your product or software).
Implementation requirements: Changes that are required to facilitate adoption of your solution. This can include education and training, data migration, and any other work that is triggered by the adoption of your system.
Quality of service requirements: Detailed statements the system’s qualities. Examples include: reliability, testability, maintainability, availability.
Regulatory requirements: Laws or policies, imposed by a third-party. e.g. privacy rules related to health data that your product might collect and use.
In software development, we are mostly focused on the design and implementation of a system that meets user requirements. In other words, we focus on the end-users of our product and attempt to design a product that is useful and usable to meet their particular needs.
We can also think about these requirements as being about system capabilities versus system qualities:
The capabilities of a system refers to the functionality that we will design and implement. Capabilities are also known as functional requirements, and include user requirements from the list above, plus any other requirements that directly result in a product feature. These are the requirements that we will focus on in this phase.
The qualities that the solution should have, constraints under which it might operate, are called non-functional requirements. These include quality of service from the list above. For software, this includes capabilities like accuracy, speed, quality of service. We will focus on non-fnctional requirements in the Analysis & Design phase.
Our goal as designers is to design and build a system that meets both capabilities and qualities.
Planning
Planning activities determine what actions are required to meet requirement and project goals, and establish a plan to deliver the project on-time, on-budget, with the requirements met.
Project planning has to consider:
Project goals: There may be additional goals outside of the product itself. e.g. Integrate a remote testing team into our development process; deliver X in revenue during delivery and so on.
Resources: Who and what we have available to dedicate to the project. Typically this means allocating staff, budget, necessary resources. If you and the customer are both committing resources, this is the time to identify what those might be.
Constraints: Other factors that we need to consider e.g. we need a demo for a tradeshow in Spring 2022.
The triple constraint model (“project management triangle”) is a model of the constraints of project management. While its origins are unclear, it has been used since at least the 1950s. It contends that: The quality of work is constrained by the project’s budget, deadlines and scope. In other words, quality depends on the relationship between project costs, what is being done (scope) and time required. The relationship between these constraints, and what tradeoffs work is rarely very straightforward, but the constraints themselves are real. Projects do have time and budget constraints that need to be respected, and we need reasonable confidence that we can deliver our scope within those constraints.
This model is pervasive, but also has it’s detractors who point out that there are many times when this model does not work. e.g. late projects are often also over-budget.
Modeling
Before actually constructing anything, we want to ensure that we have met both functional requirements (gathered from users), and non-functional requirements (qualities that we want our system to have). It’s useful to think of non-functional requirements as the constraints or conditions on how our solution works:
Technical constraints: requirements made for technical reasons (e.g. must be implemented in C++ for compatibility with our existing libraries).
Business constraints: requirements made for business reasons (e.g. must run on Windows 11 because that’s what we have deployed at customer sites, or must use Java because that’s where we have expertise as a development team).
Quality attributes: scalability, security, performance, maintainability, evolvability, reliability, deployability (e.g. must complete a core task in less than 5 seconds; must demonstrate 99.999% uptime; must support 1000 concurrent users).
Ideally, before we actually attempt to build something, we want to think through and plan the work, confirm our designs and assumptions, and fine-tune our understanding of the problem. The exact nature of modeling will vary based on the type of project, but can include building prototypes (to show a customer and confirm our understanding), or mockups of screens (to verify that the interface is clear to users).
Typically modeling is done in iterations, where we design something, use that to confirm our understanding with users, make corrections and modifications to our design, and continue iterating until we feel that we can proceed with construction. Taking time to design first saves considerable time and cost in the construction phase.
Construction
This is the step of actually manufacturing a product, or a software system, based on your earlier designs. This is typically the most expensive and time-consuming step of the project, and consumes most of our time and resources. Although we prefer to have a near-perfect design by the time we arrive at this stage, it’s common to have to iterate on design elements prior to realizing the completed product. Details of this step are highly dependent on the type of product that you’re building; we won’t focus too much on it at this time.
Deployment
Similarly, deployment may include packaging and distribution of a physical product. In some cases it many even include installation and validation on behalf of a customer, or on a customer site. We’ll discuss this later in the context of software delivery. For now, just appreciate that this can be a very costly step; correcting mistakes at this point is nearly impossible without going through the entire process again.
Umbrella Activities
Software engineering process framework activities are complemented by a number of umbrella activities. In general, umbrella activities are applied throughout a software project and help a software team manage and control progress, quality, change, and risk. Typical umbrella activities include:
Software project tracking and control. Allows the software team to assess progress against the project plan and take any necessary action to maintain the schedule.
Risk management. Assesses risks that may affect the outcome of the project or the quality of the product.
Software quality assurance. Defines and conducts the activities required to ensure software quality.
Technical reviews. Assess software engineering work products in an effort to uncover and remove errors before they are propagated to the next activity.
Measurement. Defines and collects process, project, and product measures that assist the team in delivering software that meets stakeholders’ needs; can be used in conjunction with all other framework and umbrella activities.
Software configuration management. Manages the effects of change throughout the software process.
Reusability management. Defines criteria for work product reuse (including software components) and establishes mechanisms to achieve reusable components.
Work product preparation and production. Encompasses the activities required to create work products such as models, documents, logs[…]
Software engineering process is not a rigid prescription that must be followed dogmatically by a software team. Rather, it should be agile and adaptable (to the problem, to the project, to the team, and to the organizational culture). Therefore, a process adopted for one project might be significantly different than a process adopted for another project.
Process Models
We use the term process model to describe the structure that is given to these activities. That is, it defines the complete set of activities that are required to specify, design, develop, test and deploy a system, and describes how they fit together. A software process model is a type of process model adapted to describe for software systems.
Software activities tend to be named slightly differently than the generic activity names that we’ve been using:
Compared to the standard model, we’ve split Planning into separate Planning and Requirements definition. These activities are often performed by different departments or individuals, so traditionally they’re split apart. Analysis & Design corresponds to Modeling, and Implementation and Testing together correspond to Construction.
This is a simplified conceptual understanding of the different “pieces” of a software development project. People generally accept that we have planning, formal requirements, modeling and design, implementation and testing - but there’s lots of disagreement on how these pieces “fit together”. Let’s continue talking about different forms of process models that we could use:
Waterfall Model
In the 1970s, there was a concerted effort to formalize ‘known-good’ methods of project management. Software projects were seen as expensive and time-consuming, and there was considerable pressure to improve how they were managed. In a 1970 paper, Winston Royce laid out a mechnism for formalizing the large-scale management of software projects [Royce 1970], dubbed the Waterfall model. This envisions software production as a series of steps, each cascading into the next one, much like a waterfall. In this model, requirements are defined first, a design is created and then implemented, then tested and so on. Software development is treated as a set of linear steps that are followed strictly in-order1.
The Waterfall Model, as understood and practiced for a long time, most closely resembles a linear project model, and is similar to how other construction or manufacturing projects are organized.
The Waterfall model the following characteristics:
A project starts at the top and advances through stages. Each stage must be completed before the next stage begins.
The stages are modeled after organizational units that are responsible for that particular stage (e.g. Product Management owns Requirements, Architects own Analysis & Design, QA owns Testing and so on).
There are criteria that need to be met before the project can exit one stage and enter the subsequent stage. This can be informal (e.g. an email letting everyone know that the design is “finished”), to a more formal handoff that includes artifacts (e.g. Product Requirements documents, Design documents, Test Plans and so on).
This linear approach strongly suggests that you can and should define a project up-front (i.e. determine cost, time and so on). This can be a very appealing proposition to risk-adverse businesses, but as we’ll see, this may not be realistic: requirements change, often as the project is underway, which makes this style of project structure difficult.
V-Model
The V-Model is an alternative that attempts to line-up the testing phase with the area of responsibilityt that is being tested. It’s a useful conceptual model, but it’s unclear how this is supposed to address the criticisms of a straight linear model.
Spiral Model
A spiral model acknowledges that iteration is useful, and suggests iterating from a high-level of abstraction through to lower-levels of detail. The product manager in this case is supposed to define the levels of detail (soooo close, but still problematic).
The Agility Movement
The late 90s were a particularly active period in terms of advancing software process. There was a widespread recognition that old, manufacturing-based ways of building software just didn’t work - either for developers or for customers.
There are a large number of software process models that were developed at this time, including Extreme Programming (XP) [Beck 1999] , Scrum [Schwaber & Sutherland 1995], Lean [ Poppendieck & Poppendieck 2003]. Collectively, these are called “Agile Processes”.
“Agile Software Development” isn’t a single process, but rather an approach to software development that encompasses this philosophy. It encourages team structures and attitudes that make communication easier (among team members, business people, and between software engineers and their managers). It emphasizes rapid delivery of operational software, but also recognizes that planning has its limits and that a project plan must be flexible 2.
What does this mean?
Individuals and interactions (over process and tools): Emphasis on communication with the user and other stakeholders.
Working software (over comprehensive documentation): Deliver small working iterations of functionality, get feedback and revise based on feedback. You will NOT get it right the first time.
Customer collaboration (over contract negotiation): Software is a collaboration between you and your stakeholders. Plan on meeting and reviewing progress frequently. This allows you to be responsive and correct your course early.
Responding to change (over following a plan): Software systems live past the point where you think you’re finished. Customer requirements will change as the business changes.
Info
Agile is also implicitly about shifting power and decision making from product managers and other business leaders to the development team, the ones actually building software. At least part of the failure of previous models is the failure to understand that development is not usually predictable. We’re often building something for the first time, or solving a unique problem, so it’s extremely difficult to predict the outcome far in advance.
What is the benefit?
Agility means recognizing that requirements and plans will change over time.
Software is too complex to design and build all at once. It’s more manageable to add features and test incrementally.
Software is in a constant state of change, and requirements will change during the development cycle3.
The conventional wisdom in software development is that the cost of change increases nonlinearly as a project progresses. Agility is often characterized as “embracing change” since it expects project and requirements changes, and is constantly reassessing the state of the project. The benefit of Agile is that it reduces (and tries to eliminate) breaking late-project changes.
Agile assumptions
Any agile software process is characterized in a manner that addresses a number of key assumptions about the majority of software projects:
It is difficult to predict in advance which software requirements will persist and which will change. It is equally difficult to predict how customer priorities will change as the project proceeds.
For many types of software, design and construction are interleaved. That is, both activities should be performed in tandem so that design models are proven as they are created. It is difficult to predict how much design is necessary before construction is used to prove the design.
Analysis, design, construction, and testing are not as predictable (from a planning point of view) as we might like.
Given these three assumptions, how do we create a process that can manage unpredictability?
Central to Agile processes is that any propcess must be adapatable to rapidly changing project and technical conditions. It must also be incremental and incorporate customer feedback so that the appropriate adaptations can be made.
This idea of an iterative, evolutionary development model remains central to all Agile processes (although they may present it differently). Instead of building a “complete” system and then asking for feedback, we instead attempt to deliver features in small increments, ina. way that we can solicit feedback continuously though the process. Over time, we will add more features, until the we reach a point where we have delivered sufficient functionality and value for the customer.
Note that in this process model, we still do some initial project planning and requirements definition, but the majority of our time is spent iterating over features. Every time we implement and validate some new functionality, we have the opportunity to deploy it (either for further customer testing, or as a release).
Let’s take some time and talk about the two most influential process models: Scrum and Extreme Programming (XP).
Scrum & Sprints
If you adopt only one agile practice, let it be retrospectives. Everything else will follow.
– Woody Zuill
The important thing is not your process. The important thing is your process for improving your process.
– Henrik Kniberg.
Scrum is the defacto process model for managing scope during a project iterations i.e. it’s focused on the overall project structure. Scrum breaks down a project into fixed-length iterations called sprints (typically 2-4 weeks in length for each sprint). Sprints are defined so that you iterate on prioritized features in that time, and produce a fully-tested and shippable product at the end of each sprint.
Typically a project will consist of many sprints, and you will iterate until you and the customer decide that you’re done (i.e. the only remaining requirements are deemed low enough priority that you decide to defer them or not complete them). Practically, having a buildable and potentially “shippable” product at the end of each cycle is incredibly valuable for testing, demonstrating functionality to customers, and it provides flexbibility in how you deply.
In Scrum, everything is structured around sprints:
Key Concepts
Product Owner: the person responsible for gathering requirements and making them available in the product backlog. They are not considered part of the project team, but represent both the external business and customer. At the start of each sprint, they work with the team to prioritize features and decide what will be assigned to a sprint.
Product Backlog: a list of all possible features and changes that the Product Owner thinks should be considered. There is no guarantee that these will all be developed! The team must agree to bring features forward into a sprint before they are developed.
Sprint Backlog is the set of features that are assigned to a specific sprint. This is the “scope” for that sprint.
The Scrum Master is the person that helps facilitate work during the sprint. They are not in charge (!) but track progress and help identify blocking issues that might prevent the team from meeting their deliverables.
The Daily Scrum is a standup meeting where you discuss (a) what you’ve done since the last meeting, (b) what you intend to do today, and (c) any obstacles that might prevent you from accomplishing b. The Scrum Master runs this meeting, and the entire team attends.
Sprint Breakdown
The following steps are followed in each sprint:
The project team and Product Owner collectively decide what requirements to address, and they are moved from the Product Backlog to the Sprint backlog. Once features have been decided, you do not allow any further scope changes (i.e. you cannot add anything to the sprint once its started). Work is actually assigned to team members (and you collctively agree that you believe it can be completed in the sprint).
During the sprint, you iterate on the features in the Sprint Backlog. This includes design, development, testing etc. until you complete the feature or the sprint is finished. The Scrum Master facilitates aily meetings to make sure that nobody is “stuck” on their feature.
At the end of the sprint, have a review with the team to see what was accomplished. Demo for the Product Owner (and sometimes the actual customer). Reflect on your progress, and be critical of how you might improve process the next sprint. (e.g. could we have communicated better? should we have done more testing during the sprint? did we take on too many features?)
Extreme Programming (XP)
The most important thing to know about Agile methods or processes is that there is no such thing. There are only Agile teams. The processes we describe as Agile are environments for a team to learn how to be Agile.
– Don Wells
Extreme Programming (XP) is an Agile methodology focused on best-practices for programmers. It was based on a large-scale project that Kent Beck managed at Chrysler in the late 90s, and attempted to capture what was working for them at that time. It aims to produce higher-quality software and a higher quality-of-life for the development team.
The five core values of XP are communication, simplicity, feedback, courage, and respect.
Communication: The key to a successful project. It includes both communication within the team, and with the customer. XP empasizes face to face discussion with a white board (figurtively).
Simplicity. Build the “simplest thing that will work”. Follow YAGNI (You Ain’t Gonna Need It) and DRY (Don’t Repeat Yourself ).
Feedback. Team members solicit and react to feedback right away to improve their practices and their product.
Courage: The courage to insist on doing the “right thing”. The course to be honest with yourselves if something isn’t working, and fix it.
Respect: Respect your team members, development is a collaborative exercise.
XP launched with 12 best practices of software development [Beck 2004]. Some of these (e.g. The Planning Game, 40-Hour Week, Coding Standard) have fallen out of disuse. Others have been added or changed over time, so it is difficult to find a “definitive” list of commonly used XP practices 4.
Although some XP practices never really worked very well, many have been adopted as “best practices”. We’ll revisit these in the next secion.
Info
XP is rarely used as-is. It’s common for development teams to adopt one or more of these ideas based on what suits them, and their environment. For example, daily standups are very common, but very few places will implement pair programming.
Being Agile
We have generations of people claiming to have the “correct” solution, or the best process model. The reality is, every process model is a reflection of the organization that developed it. There are many process and all are equally valid in their domain.
In this section, we identify commonly used Agile principles and practices that we will use in this course. Although there will always be some contention on what constitutes “best practices”, this is a very common subset of approaches that in practice will work very well.
Info
Teams SHOULD be adopting their own best practices and process according to their specific domain. It’s reasonable to assume that healthcare, game development, telecom, compiler development all have unique work environments and constraints, that make it reasonable to customize how they work for that environment.
Principles
From these different Agile models, we can extract a set of useful guiding principles [Pressman 2018]. This is what we aspire to do with our practices.
Principle 1. Be agile. The basic tenets of agile development are to be flexible and adaptable in your approach, so that you can adjust if needed between iterations. Keep your technical approach as simple as possible, keep the work products you produce as concise as possible, and make decisions locally whenever possible.
Principle 2. Focus on quality at every step. The focus of every process activity and action should be the quality of the work produced.
Principle 3. Be ready to adapt. When necessary, adapt your approach to constraints imposed by the problem, the people, and the project itself.
Principle 4. Manage change. The approach may be either formal or informal, but mechanisms must be established to manage the way changes are requested, assessed, approved, and implemented.
Principle 5. Build an effective team. Software engineering process and practice are important, but the bottom line is people. Build a self-organizing team that has mutual trust and respect.
Principle 6. Establish mechanisms for communication and coordination. Projects fail because important information falls into the cracks and/or stakeholders fail to coordinate their efforts to create a successful end product. Keep lines of communication open. When in doubt, ask questions!
Process
Using the SDLC
We can describe our process as the Software Development Lifecycle (SDLC). This illustrates the process that we will follow for this project. Each block in the diagram represents a stage, containing related activities, that are performed in order.
To complete a project, you start with Planning activities, and move through Requirements, Analysis & Design and so on. The project is complete when you finish Evaluation and the team collectively decides that they are “done” (or you run our of time/resources!).
Note that the preliminary activities (Planning, Requirements, Analysis & Design) are only performed once.
Implementation and related activities are grouped together, since they are performed in-order, but we perform multiple passses over all of them. This iteration is called a sprint (taken from Scrum). In a typical development project, sprints should be relatively short, from two to four weeks in length, and the team typically works through multiple sprints until the project is completed.
Info
In our course, Sprints are two-weeks long, and we will have four sprints in total (i.e. 4 x 2-week sprints).
Each sprint includes the following activities:
Feature Selection: On the first day of the Sprint, the team meets and decides what features to add (and what bugs to fix) during that iteration.
Implementation/Testing. During most of the sprint, the team iterates on their features. As each feature is completed, it is tested.
Evaluation. At the end of the Sprint, the team meets with the Product Owner to demo what they have completed, and get feedback. The team also has a Retrospective, where they reflect on how to improve.
The cycle repeats for however many Sprints the team has available (or until they decide they are “done”). The product should be usable and potentially shippable to a customer at the end of each Sprint (though obviously features may not be complete, but the ones that exist should be bug-free).
Along with the SDLC, we also have a set of best practices that we will use. The SDLC provides the overall organization (what we should do), and these practices provide more strict guidelines (how we should do it).
These will appear in later chapters related to specific activities.
User stories: describe features in a way that makes sense to customers.
Pair programming: critical code is written by two people working as a team; one codes while the other one watches, plans and makes suggestions. This results in demonstrably better code and is much more productive than working alone [Böckeler & Siessegger 2020] 5.
Test-driven development: tests are written before the code. This helps to enforce contracts/interfaces as a primary focus of your design. We will discuss this in the Implementation section.
Code reviews: before code changes are committed to the repository, they need to be reviewed by one or more other developers on the team, ideally more senior members. The claim is that (a) the developer gets feedback to help identify bugs, and improve their design, and (b) participating in code reviews helps spread knowledge about that code around the team. Research suggests that code reviews most often result in design recommendations, and aren’t particularly effective at finding bugs [Czerwonka et al. 2015].
To be fair to Royce, he warns against treating development as a purely linear model, but most people didn’t read past the second page of his paper. ↩︎
Much of this section is based on a more thorough discussion of Agility by [Pressman & Maxim 2020]. ↩︎
There’s a common misperception that software can be “finished”. That’s not really true. By the time software reaches the market, the conditions will have changed (e.g. a competitor has introduced a feature, or you need to support a new type of device…). Software also needs to be constantly updated to keep up with changing environments (e.g. OS versions). Simple programs may last years without requiring updates, but as a program becomes more complex, its more likely to need ongoing maintenance. ↩︎
Code reviews serve a similar purpose, in that they ask developers to review one another’s code after it is written. Although that’s proven to be a useful practice, it really only helps to locate and address mistakes after they’ve been made. Pair programming results in better up-front designs. ↩︎
Planning
Planning is the set of activities related to the structure and management of a software project. In a large project, this will likely be handled by a project manager (or even a project management team). However, even on small projects, there are still planning activities that should be undertaken.
Objective
Identify high-level project goals, get commitment from stakeholders
Activities
Identify team members; Determine high-level goals and milestones; Determine project constraints; Identify risks and plans to mitigate them.
Duration
Varies (for our course, 1 week)
Outcome
Project plan
Project planning is concerned with:
Identifying project goals, which can include direct goals (e.g. build product x), and indirect goals (e.g. provide Jerry with the opportunity to grow design skills)1. Goals are typically product driven, but may also be organizational.
Identifying resources (e.g. hardware, software, staff that are available) and constraints (e.g. must run in Windows).
Identifying risks to your project, and devising a plan to adjust and accomodate these risks. e.g. your Open Source database might not be fast enough for what you’re suggesting, so as a backup, you might switch to a faster commercial database; you would need to budget for the cost and effort required to make that change if it was required. This could be listed as a contengency in the project plan.
Determining an overall schedule, including milestones and time constraints. e.g. must be able to demo at a conference in October; must be delivered in time for the holiday buying season.
Creating a Project Plan
You should aim to capture this information in a project plan:
Goal: You typically want a succinct project statement that captures the overall purpose of this project. You also want to identify key high-level requirements that must be included.
Resources: Identify resources required and available for the project. Typically this includes an estimate of staff required, plus any additional hardware or software systems that are needed. If you believe that you are missing something that you need, identify it!
Risks. What risks can you identify at this stage? These can be technical or non-technical. For every risk, you should identify a mitigation strategy e.g. We have identified Windows 11 as a target but it is not releasing until midway through the project. There is a risk of technical compabitility issues. If this is a problem, we have permission to drop Windows 11 support and revert to Windows 10 for testing.
Schedule: Finally, you need a list of critical deadlines. e.g. must be able to demo feature X at a conference in October.
Here’s an example:
Project Tracking
The most common means of tracking project schedules is called a Gantt Chart, which is a high-level schedule showing all tasks and their dependencies.
Key concepts:
Task: Some “piece of work” to perform. Usually an independent task, though it may be have predecessors or successors.
Milestones: Important deadlines that you want to highlight. Typically you’re ordering tasks in a way that ensures that you meet milestones.
Tasks and dependencies: Work that needs to be complete (tasks) and order that needs to be enforced (dependencies).
Critical path: The longest path in your project; the path that determines the overall duration of the project (i.e. the thing that “everyone else is waiting for”). Often there is enough “slack” in a project that tasks can go over scheduled time without harming your ability to hit milestone dates; the critical path is the one set of tasks which, if delayed, will delay the project.
Since tasks are assigned to people, it’s also a way of identifying resources as well, since most of the resources in a software project are people.
Info
Hang on, how do you have a schedule if you’re Agile?! Simple, you schedule the iterations and focus on highlighting constraints – dates the team needs to hit. This makes the Gantt chart more of a useful tool for the team, instead of just a way of tracking their progress.
With the rise of Agile, Burndown Charts have become popular. They show the amount of work remaining in a Sprint at any given time, and can be often generated on-the-fly from an issue tracking system. They are significantly more useful for Agile projects, since they focus less on the details of what is being built, and more on the team’s velocity.
Estimating Time & Effort
Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.
– Douglas Hofstadter. Gödel, Escher, Bach (1979)
It is extremely difficult to determine a realistic, detailed schedule at the beginning of a project. However, for planning purposes, we need some structure in place that identifies critical deadlines or milestones, and clearly defines targets that we need to hit.
The recommended process for determining how long a project will take to complete is:
Look at similar projects that you have delivered, and use those as a guideline for the resources that you will need, timelines and so on. Different types of software have different project characteristics, so you should model your expectations on your team’s capabilities, and compare to projects that are similar (e.g. building a new compiler will likely take significant up-front research; other projects may not require this).
Identify major milestones: these are targets that you know you need to meet for the project to be considered successful e.g. final shipment date. Work backwards as much as you can from the final date, and fill in intermediate milestones 2.
Fill in project phases based on the work that needs to be done. In the diagram above, you can see that the team has considered research, design and setup phases before actual implementation work. These phases may vary based on your style of project.
You want to provide general guidance, without actually fleshing out requirements. You should expect to refine your understanding of the problem and potential solutions in the Requirements phase. ↩︎
Part of the reason why short iterations work so well in software development is that they force stopping points; everyone has to wrap up what they’re doing, and get it stable enough to demo to a customer. The amount of lost time to a bad decisions is never more than the length of a sprint. ↩︎
Requirements
The requirements stage is where we explore the problem that we wish to address, and determine what to build. This is the time for market research, talking with customers, and determining the details of how we want our feature to work (note: these are the “what to build” decisions, not the “how to build them” decisions). At the end of this phase, you should have a detailed list of product features that you will consider implementing.
Objective
Determine the problems that we’re solving; Identify and prioritize user requirements
Activities
Interview users to understand the problem, their needs and any constraints in how they work.
Duration
Varies (for our course, 1 week)
Outcome
Personas, User Stories, Use Cases, Requirements (represented by the Product Backlog)
Traditionally, this would be done by a product manager (business or marketing specialist) based on what they perceived as the market need for that feature. In other words, they would determine what to build based on what they thought would sell.
However, we want to ensure that we design and build a product that is compelling for our users, so we want to make sure that we include our users in this process and get their input as a primary means of determining what to design. This focus on users-first is a critical tenet of User-Experience Design (or UXD).
UXD can be traced back to the mid-1980s, when Don Norman suggested that design should focus making products more usable for the people that use them. His book, The Design of Everyday Things became the foundation for the UX movement that followed. Through the 1990s and early 2000s, significant research in HCI and related disciplines contributed to a whole-scale shift contributed to the widespread acceptance of human-centric product design. Today, it is widely accepted that our best practices in requirements and design are human-oriented1.
Our first step is to gather information that will help us understand our users and their problems, and determine what features we need to design and implement for them.
Characterizing Users
You want to clearly understand:
“Who are my users?”
“What are they trying to accomplish?”
“How do they relate to the problem that I’m trying to solve?”
A key objective at the start of the project is to determine who your users are. In some situations, you may be told what users to target, and your early efforts would be to interview people and refine your understand of user roles (e.g. imagine you’re told to create a new system for airline booking agents; you would probably start by interviewing everyone that relates to that role, who in turn may help you identify other potential users).
In other situations, your users may not be obvious: you may need to interview people to determine your potential user base.
Info
You may need to interview people just to figure out how they relate to your product idea. Start by interviewing potential users, and then look at your results - you can always narrow down with successive interviews.
Interview 5-6 users [Nielsen 2000] and ask open-ended questions about the problem. Use followup questions to to make sure that you understand their goals and motivations. The outcome from interviews should be detailed interview notes.
Good interview questions:
Describe the problem that you have.
How would you accomplish X?
What do you like about this system? What do you dislike?
How would you expect (some feature) to work?"
Another useful technique for gather information is to survey users. Surveys are relatively inexpensive, and it is fairly easy to collect information from a large group of users quickly. However, response rates for surveys tend to be low, and you are limited to fairly simple questions (plus, unlike interviews, you cannot ask follow-up questons!).
From your interview notes, it can helpful to characterize your target users, and the tasks that they want to perform. This involves generation of Personas and User Stories.
Personas
A persona is an archetypal description of a user of your application or product. They are often based on “real people” but are generalized descriptions that represent a category of people that share similar characteristics. We use personas a stand-ins when discussing product goals and features.
Each role or “type” of user that you identifed should generate a unique persona. You typically have 2 or 3 personas for even a small solution.
To build a persona, you interview one or more users representing that role in an organization. You then write a fictional portrait of a user, which represents as accurately as possible the consensus among your users in that role.
There are some basic elements that your user persona should include:
Personal info: Present personal information first, including name, job title, company, job description, family status, and more. Include any details that can help your team understand the persona better.
Demographic info: Other demographic information, such as age, gender, income, education, location, and other background information can help you create a more authentic character.
Image: Images speak louder than words. A vivid image, sketch, or cartoon can help your team picture your users clearly, and help you establish a consistent understanding of the target users. Use a photo that reminds you of this user.
Goals & motivations: You should also make it clear what your audience wants to get or to do with your product. List all possible goals and motivations that you’ve found from your user research.
Here’s an example:
Actors
In UML notation, a user would be an Actor. Actors aren’t nearly as well documented as personas, but are still generalized standins, representing a particular user role. One peculiarity of actors is that they can also represent other systems that engage with your system.
In this course, we’ll use personas to characterize our users. Actors will only be used if we need to show an interaction on a UML diagram.
Determining Requirements
Affinity Diagrams
The outcome of your interviews will likely be a lot of qualitative data – opinions, suggestions and ideas that the user expressed. The challenge is extracting ideas, themes from this narrative that you can use in your design.
We use thematic analysis to do this, and the mechanism that we will use is the creation of an affinity diagram.
To create an affinity diagram:
Record all notes or observations on individual cards or sticky notes
Look for patterns in notes or observations that are related and group them
Create a group for each pattern or theme
Give each theme or group a name
Create a statement of what you learned about each group (provide your analysis or key insight)
You should end up with clusters of related ideas, all around a common theme. You can them examine these to expose critical ideas or themes from the user’s ideas. (You can even do this across multiple users and looks for which themes occur more frequently!).
From your user interviews, you should have identified some basic tasks that you will need to address. Often, these will come in the form of stories or narratives from your users, which you’ve recorded. Your job in this step is to extract these common stories or tasks from your interview notes.
User-Stories are a description of activities that are performed with a system to perform a particular task. A user story is often simple text, and is intentionally describing the problem at a high level:
“As the HR manager, I want to create a screening quiz so that I can understand whether I want to send possible recruits to the functional manager.”
“As a user, I can indicate folders not to backup so that my backup drive isn’t filled up with things I don’t need saved”.
User stories are useful because they help identify users and their high-level goals, and they serve as an excellent starting point for discussions.
User Story Maps
Once you’ve collected a number of user stories, you might arrange them into User-Story Maps to show their larger relationships. They can also be plain-text, but often include diagrams, sketches and other visual elements that help describe a user’s interaction with the system.
Info
Story maps are intended to spark collaboration and conversation among Agile team members, while providing them with the bigger picture of how the digital product flows and fits together. – Nielsen Norman Group
User Stories Maps often include three levels:
Activities represent the high-level tasks that users aim to complete in the digital product. e.g. deposit a cheque.
Steps sit directly underneath activities and also display in sequential order. They represent the specific subtasks that users will go through in the product to complete the activity above. e.g. enter pin, select deposit from the menu, and so on.
Details are the third level of the story map and describe the lowest-granularity interactions that the team anticipates users will experience to complete the step above. e.g. enter pin by pressing the numbers in order, and ok when complete.
These activities should be logged as requirements in your tracking system, along with the steps and details that have been captured. Each activity is a feature, and the steps and details provide information helpful in the implementation of that feature.
Use Cases (Optional)
You may choose to convert these user stories into more detailed Use Cases and Requirements.
A use case is similar to a user story, in that it describes a particular action or set of actions by a user. However, where a user story is high-level and leaves out a lot of details, a use case is much lower level and attempts to describe all of the required details.
Use cases were first introduced by Ivar Jacobsen while working at Objectory, as a way of specifying how actors (users) interact with functional requirements. Here’s an example, using simple UML notation showing a series of related use cases.
Use cases in this example include “Search for items”, “Place order” and “Obtain help”. You can see which Actors (or Users) interact with each use case. This particular example also shows which use cases will be addressed in which software release.
Although user stories often sufficient, the extra level of detail that use cases provides can be helpful in determining exactly how functionality should work. A full realized use case might provide this level of detail:
Finally, you want to realize these use cases as features in your product. Use cases should be much more specific than user stories and should be realizable. In other words, you should attempt to provide enough detail in the task description that the Design and Implementation phases can act on this requirement.
Your final step should be to convert these requirements - derived from your affinity diagram, and use cases - into actionable items.
Log each item into your GitLab project as an Issue (better to have a custom type?)
Don’t assign anything yet!
Assignee: Unassigned
Milestone: No milestone
Due date: Unselected
These unassigned issues constitute your Product Backlog. You only schedule this during Sprint planning, and only if the team agrees to take it on.
The term “design” is overloaded in Computer Science. UX overlaps with traditional engineering-focused approaches to both requirements analysis & design, but both are fundamentally about understanding your users and designing products focused on meeting their requirements. ↩︎
Analysis & Design
Outline
The Analysis & Design phase refers to a set of activities that we perform ahead of writing code. We want to examine our high-level technical requirements and make sure that we’ve addressed the business needs, user goals and technical requirements prior to implementation. We also want to determine the technical viability of our functional requirements, and any technical constraints that they will place on our design.
Objective
Define non-functional requirements; Model high-level system (physical/logical models)
Activities
Determine technical impact and constraints, non-functional requirements; diagram high-level relationships
Duration
Varies (for our course, 2 weeks)
Outcome
Refined requirements that include technical detail; NFRs in GitLab; Architectural diagrams
In the following sections, we will consider:
Technical assessment - determine technical impact of functional requirements
Software architecture - determine high level structure
Software design - determine low-level structure and design
Technical Assessment
When we create functional requirements, we try and focus on determing our user’s needs without any consideration for the technical impact that they will have. We need to clearly separate functional from technical concerns so that we can collect information without prejudicing our requirements gathering process.
However, at some point, we need to determine the technical impact of each requirement:
Will we be able to to implement this feature?
What other technologies do we require for this to be feasible? (e.g. frameworks, libraries, toolkits).
Does this put a constraint on our overall design? e.g. are we forced to use a particular OS, or deployment target?
Is this feature risky; does it require significant novel research, or is it otherwise difficult to meet?
Each functional requirement from the requirements phase should be examined, and considered in this light. For many projects, the majority of requirements should be low-risk/feasible; introducing too many high-risk requirements could put the project itself at risk!
Your goal should be to amend the functional requirements with additional detail that will be useful in the Planning stage. Is this requirement risky? Add a statement to that effect and attempt to clarify the degree of risk! Does implementing one feature mean that you need to implement some other dependent functionality? Document that in the requirement as well! Your goal is to add technical detail to the existing requirements.
Info
We should NOT attempt to disqualify or invalidate feature requests at this stage. Analysis & design is about gathering information and refining our understanding of our requirements. We do not make a determination of what to include or exlude until the Sprint planning stage.
Software Architecture & Design
Software Architecture and Design are related disciplines that are both concerned with the structure of a system. However, they represent different levels of abstraction: software architecture examines characteristics at a relatively high level of abstraction, but rarely goes down to the level of actual code; design is concerned with lower levels of abstraction – class composition and so on.
While a team might have a dedicated software architect, it’s very common for a senior developer or team lead to take on the “role” of the architect for the team. In other words, these don’t have to be different people, they simply represent different concerns that need to be addressed.
In the next two sections, we will address software architecture and design in greater detail. We cover architecture first, then move down into the details of software design.
As with many of the topics that we discuss, architecture & design are large and complex areas of study, and at best, we’re introducing and summarizing theses topics. Consider this a “highlight reel” of what we consider to be key considerations for analysis & design.
Formally, Software architecture is the “fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution” [IEEE 1471-200].
Architecture can also be seen as a shared understanding how the system is structured. Martin Fowler (2003) attempts to pin down the term in a couple of different ways:
Definition 1: “Expert developers working on a project have a shared understanding of the system design. This shared understanding is called ‘architecture’ [and] includes how the system is divided into components and how the components interact through interfaces.”
Definition 2: “Architecture is the set of design decisions that must be made early in a project [and that we would like to get right]”.
Architecture is the also holistic analysis of a system, and how it’s parts relate to one another. Instead of examining requirements in isolation, we instead want to look at the consequences of the structure itself, including the qualities that emerge from this structure.
Architecture can be said to address the intersection of business goals, user goals and technical (system) qualities. The architect needs to determine how to deliver the functional requirements in a way that also addresses these qualities, and other potential business needs (e.g. cost). This may very well include making tradeoff decisions ahead of time. e.g. a user may want a system to return the results of a query in less than 5 seconds, but the cost of doing this might be prohibitively expensive!
The benefit to a careful architecture is that we have a more stable initial design that reflects our project concerns, while still allowing for adaptability, flexibility and other desireable qualities. We’ll discuss different qualities of a system below.
Info
Diagrams and portions of the following sections have been taken from: Mark Richards & Neal Ford. 2020. Fundamentals of Software Architecture: An Engineering Approach. O’Reilly. ISBN 978-1492043454.
Imposing Structure
Architects need to be concerned with both the logical structure of systems, and the physical realization of that structure.
Modularity (Logical)
Modularity refers to the logical grouping of source code into related groups. This can be realized as namespaces (C++), packages (Java or Kotlin). Modularity is important because it helps reinforce a separation of concerns, and also encourages reuse of source code through modules.
When discussing modularity, we can identify two related concepts: cohesion, coupling.
Cohesion is a measure of how related the parts of a module are to one another. A good indication that the classes or other components belong in the same module is that there are few, if any, calls to source code outside of the module (and removing anything from the module would necessitate calls to it outside of the module).
Coupling refers to the calls that are made between components; they are said to be tightly coupled based on their dependency on one another for their functionality. Loosely coupled means that there is little coupling, or it could be avoided in practice; tight coupling suggests that the components probably belong in the same module, or perhaps even as part of a larger component.
When designing modules, we want high cohesion (where components in a module belong together) and low coupling between modules (meaning fewer dependencies). This increases the flexibility of our software, and makes it easier to achieve desireable characteristics e.g. scalability.
Info
In Kotlin, modules can be created by assigning classes to the same package (using the package keyword at the top of a class). If you do this, you also need to place your files in a directory with the same name as the namespace e.g. classes in the package graphics would need to be located in a common directory named /graphics.
Components (Physical)
Modules are logical collections of related code. Components are the physical manifestation of a module2. Components can represent a number of different abstractions, from a simple wrapper of related classes, to an entire layer of software that runs independently and communicates with external systems.
Library. A simple wrapper is often called a library, which tends to run in the same memory address as the calling code and communicate via language function call mechanisms. Libraries are usually compile-time dependencies.
Layers or subsystems. Groups of related code deployed together that may communicate with one another directly.
Distributed service. A service tends to run in its own address space and communicates via low-level networking protocols like TCP/IP or higher-level formats like REST or message queues, forming stand-alone, deployable units. These are useful in in architectures like microservices.
Info
In Kotlin, a jar file is the component that we most often create to represent a module. Jar files are designed to be distributed much like a library in other languages.
Top-Level Partitioning
Partitioning is the decision on how we organize and group functionality (we use the term top-level partitioning, because this is the highest level of organization).
There are multiple approaches to how we group functionality.
Technical partitioning: we group functionality according to technical capabilities. e.g. presentation or UI layer, business rules or domain layer and so on.
Domain partitioning: we group functionality according to the domain area or area of interest. e.g. a payment processing module, a shopping cart module, a reporting module and so on.
So which is correct? Good question.
Technical partitioning tends to be used more often. If we’re concerned about reusability, it’s much easier to design a third party library that can be injected into an application if you provide technical capabilities, as compared to designing a domain-specific library.
For example, we have lots of UI frameworks that sit at the presentation layer, that can be used to build any sort of application regardless of domain. There are very few CatalogCheckout libraries, since any code produced to address that functionality is likely designed around the assumptions of that specific instance of that domain - and is unlikely to be reusable.
From here, developers subdivide components into classes, functions, or subcomponents. In general, class and function design is the shared responsibility of architects, tech leads, and developers.
Determining Components
So we know what components are, but how do we determine what components we need to create? Here’s a couple of common approaches, both of which assume that you’ve identified Use Cases (in your Requirements phase).
Actor/Actions: From your Use Cases, identify actors who perform activities, and the actions that they may perform. This is simply a technique for discovering the typical users of the system and what kinds of things they might do with the system. These actions represent activities that can be mapped directly to a corresponding software component.
Workflow: This approach looks at the key activities being performed, determines workflows and attempts to build components to address those specific activities.
Architectural Characteristics
Where the requirements phase is concerned about functional requirements, this phase is concerned about the non-functional requirements or “architectural characteristics” [Richards & Ford 2020]. These are all of the things that software must do that aren’t directly related to the functional requirements. This includes considerations of the technical environment, choice of programming languages, technical constraints and so on.
An architectural characteristic meets three criteria:
specifies a non-domain design consideration,
influences some structural aspect of design,
is critical or important to the success of the application.
Examples of an architectural characteristic include software performance, and data security. These tend to not be identified as functional requirements, but should be considered.
It’s important to identify only those concerns that are relevant to your project! Addressing one of these characteristics adds cost and complexity to your project:
“Applications could support a huge number of architecture characteristics…but shouldn’t. Support for each architecture characteristic adds complexity to the design. Thus, a critical job for architects lies in choosing the fewest architecture characteristics rather than the most possible”
– Richard & Ford. 2020. Fundamentals of Software Architecture
Here are some potential characteristics that are often considered [Richards & Ford 2020]:
Operational characteristics
These are often relevant for large-scale software systems with dedicated uptime, and typically a large number of concurrent users. e.g. Gmail, Facebook.
Term
Definition
Availability
How long the system will need to be available (if 24/7, steps need to be in place to allow the system to be up and running quickly in case of any failure).
Continuity
Disaster recovery capability.
Performance
Includes stress testing, peak analysis, analysis of the frequency of functions used, capacity required, and response times. Performance acceptance sometimes requires an exercise of its own, taking months to complete.
Recoverability
Business continuity requirements (e.g., in case of a disaster, how quickly is the system required to be on-line again?). This will affect the backup strategy and requirements for duplicated hardware.
Reliability
Assess if the system needs to be fail-safe, or if it is mission critical in a way that affects lives. If it fails, will it cost the company large sums of money?
Robustness
Ability to handle error and boundary conditions while running if the internet connection goes down or if there’s a power outage or hardware failure.
Scalability
Ability for the system to perform and operate as the number of users or requests increases.
Structural characteristics
These include characteristics related to code structure and code quality and production concerns.
Term
Definition
Configurability
Ability for the end users to easily change aspects of the software’s configuration (through usable interfaces).
Extensibility
How important it is to plug new pieces of functionality in.
Installability
Ease of system installation on all necessary platforms.
Reuse
Ability to leverage common components across multiple products.
Localization
Support for multiple languages on entry/query screens in data fields; on reports, multibyte character requirements and units of measure or currencies.
Maintainability
How easy it is to apply changes and enhance the system?
Portability
Does the system need to run on more than one platform? (For example, does the frontend need to run against Oracle as well as SAP DB?
Supportability
What level of technical support is needed by the application? What level of logging and other facilities are required to debug errors in the system?
Upgradeability
Ability to easily/quickly upgrade from a previous version of this application/solution to a newer version on servers and clients.
Cross-cutting characteristics
These don’t fit neatly into a category but are worth considering:
Term
Defintion
Accessibility
Access to all your users, including those with disabilities like colorblindness or hearing loss.
Archivability
Will the data need to be archived or deleted after a period of time? (For example, customer accounts are to be deleted after three months).
Authentication
Security requirements to ensure users are who they say they are.
Authorization
Security requirements to limit access to prescribed functionality based on assigned user roles or capabilities.
Legal
What legislative constraints is the system operating in (data protection, Sarbanes Oxley, GDPR, etc.)? What reservation rights does the company require? Any regulations regarding the way the application is to be built or deployed?
Privacy
Ability to hide transactions from internal company employees (encrypted transactions so even DBAs and network architects cannot see them).
Security
Does the data need to be encrypted in the database? Encrypted for network communication between internal systems? What type of authentication needs to be in place for remote user access?
Usability
Level of training required for users to achieve their goals with the application/solution.
Info
Another reason to limit yourself to “critical” characteristics for your project is that are in often competition with one another. e.g. making a project highly secure will negatively impact performance, since you will likely need to encrypt/decrypt data throughout your application. This isn’t something to undertake lightly.
Identifying Characteristics
So how do you identify which characteristics are relevant? You should look at the functional requirement (and project goals if they apply) and determine if there are any characteristics that help address those needs.
For example:
Domain Concern
What do we look for?
Relevant Characteristics
Time to market
Characteristics that ensure that we can execute quickly and reliably.
Agility; Testability; Deployability
User satisfaction
Does your user work directly with this product? These are characteristics that directly affect user’s perception, and availability of your product.
Some types of data are sensitive (e.g. working in Healthcare). Anything that would protect the integrity and access to data.
Authorization; Authentication; Security
Competitive advantage
In a competitive market, we want to be able to introduce and modify features quickly, even after we’ve released to the market.
Testability; Deployability; Supportability
Non-Functional Requirements
Once you’ve identified a characteristic that you should address, you need to document it as a non-functional requirement (NFR).
Include details of how you will measure it (all NFTs need to be measurable)
Document how you will verify that you have met the requirement.
Include consideration of this NFT in your topology and design (below).
Clearly communicate this as an undercutting requirement to your team.
Example of non-functional requirements that would be included in our product backlog:
“The application should launch and restore all state in less than 5 seconds on our target platform. We will measure this by manually timing execution during integration testing”.
“When the Save button is pressed, the entire state should be saved, and control returned to the user in less than 10 seconds. We will write unit tests to check the amount of time consumed by this action and ensure that it never exceeds this threadhold”.
“User records in all cases should consume less than 100 Kb each on disk. We will design our data classes to address this, and write integration tests to verify that this is the case”.
Architectural Styles
An architectural style (or architectural pattern) is the overall structure that we create to represent our software. In describes how our components are organized and structured. Similar to design patterns, an architectural style is a general solution that has been found to work well at solving specific types of problems. The key to using these is to understand the problem well enough that you can determine if a pattern is applicable, and useful to your particular situation.
An architectural pattern describes both the topology (organization of components) and the associated architectural characteristics.
Fundamental Patterns
There are some fundamental patterns that have appeared through history.
Big Ball of Mud
Architects refer to the absence of any discernible architecture structure as a Big Ball of Mud.
A Big Ball of Mud is a haphazardly structured, sprawling, sloppy, duct-tape-and-baling-wire, spaghetti-code jungle. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. – Foote& Yoder 1997.
A Big Ball of Mud isn’t intentional - it’s what happens when you fail to consider architecture in a software project. Treat this as an anti-pattern.
Unitary (Monolithic)
A monolithic structure simply means an application that is designed to run on a single system, and not communicate with any other systems. Source code had very little structure, these systems generally worked in isolation, on data that was carefully fed to them.
However, one inviolatable rule is that systems increase in complexity and capabilities over time. As systems grow, software has to be more carefully structures and managed to continue to meet these requirements.
Client-Server
Client-server architectures were the first major break away from a monolithic architecture, and split processing into front-end and back-end pieces. This is also called a two-tier architecture. There are different ways to divide up the system into front-end and back-end. Examples include splitting between desktop application (front-end) and shared relational database (back-end), or web browser (front-end) and web server (back-end).
Three-tier architectures were also popular in the 1990s and 2000s, which would also include a middle business-logic tier:
In this particular example, the presentation tier handled the UI, the logic tier handled business logic or applicaiton logic, and the data tier managed persistance.
These tiers are commonly used in other architectures, and we’ll revisit them shortly.
Monolithic Architectures
Monolithic architectures consist of a single deployment unit. i.e. the application is self-contained and deployed to a single system. (It may still communicate with external entities, but these are separate systems).
Layered
A layered or n-tier architecture is a very common architectural style that organizes software into horizontal layers, where each layer represents some logical functionality.
There is some similarlty to client-server, though we don’t assume that these layers are split across physically distinct systems (which is why we describe them as logical layers and not physical tiers).
Standard layers in this style of architecture include:
Presentation: UI layer that the user interacts with.
Business Layer: the application logic, or “business rules”.
Persistence Layer: describes how to manage and save application data.
Database Layer: the underlying data store that actually stores the data.
The major characteristic of a layered architecture is that it enforces a clear separation of concerns between layers: the Presentation layer doesn’t know anything about the application state or logic, it just displays the information; the Business layer knows how to manipulate the data, but not how it is stored and so on. Each layer is considered to be closed to all of the other layers, and can only be communicated with through a specific interface.
The layered architecture makes an excellent starting point for many simple applications that have few external interactions. However, be careful to ensure that your layers are actually adding functionality to a request, otherwise they are just added overhead with no added value. Layered is well-suited for small simple applications, but may not scale well if you need to expand your application’s functionality across more than a single tier.
Pipeline
A pipeline (or pipes and filters) architecture is appropriate when we want to transform data in a sequential manner. It consists of pipes and filters, linked together in a specific fashion:
Pipes form the communication channel between filters. Each pipe is unidirectional, accepting input on one end, and producing output at the other end.
Filters are entities that perform operation on data that they are fed. Each filter performs a single operation, and they are stateless. There are different types of filters:
Producer: The outbound starting point (also called a source).
Transformer: Accepts input, optionally transforms it, and then forwards to a filter (this resembles a map operation).
Tester: Accepts input, optionally transforms it based on the results of a test, and then forwards to a filter (this resembles a reduce operation).
Consumer: The termination point, where the data can be saved, displayed etc.
These abstractions may appear familiar, as they are used in shell programming. It’s broadly applicable anytime you want to process data sequentially according to fixed rules.
A microkernel architecture (also called plugin architecture) is a popular pattern that provides the ability to easily extend application logic to external, pluggable components. e.g. IntelliJ IDEA which uses application plugins to add functionality for new programming languages.
This architecture works by focusing the primary functionality into the core system, and providing extensibility through the plugin system. This allows the developer, for instance, to invoke functionality in a plugin when the plugin is present, using a defined interface that describes how to invoke it (without need to understand the underlying code).
An example would be a payment processing system, where the core system handles shopping and payment calculations, and behaviour specific to a payment vendor could be contained within a plugin (e.g. Visa plugin, AMEX plugin and so on).
One final note: interaction between other system components and plugins is done through the core system as a mediator. This reduces coupling of components and the plugins, and retains the flexibility of this architecture.
Examples of this architecture include web browsers (which support extensions), and IDEA (which support plugins for various programming languages).
Distributed Architectures
Distributed architectures assume multiple deployments across different systems. These deployments communicate over a network, or similar medium using a defined protocol.
This overhead leads to some unique challenges that are referred to collectively as the fallacies of distributed computing. This includes concerns with network reliability, latency, bandwith, security and so on - things that are non-issues with monolithic architectures3.
Services-Based
A services-based architecture splits functionality into small “portions of an application” (also called domain services) that are independent and separately deployed. This is demonstrated below with a separately deployed user interface, a separately deployed series of coarse-grained services, and a monolithic database. Each service is a separate monolithic application that provides services to the application, and they share a single monolithic database.
Each service provides coarse-grained domain functionality (i.e. operating at a relatively high level of abstraction) and addresses a particular business-need. e.g. a service might handle a customer checkout request to process an order.
Working at a coarse-grained level of abstraction like this means that these types of services can rely on regular ACID (atomicity, consistency, isolation, durability) database transactions to ensure data integrity. In other words, since the service is handling the logic of the entire operation, it can consolidate all of the steps in a single database transaction. If there is a failure of any kind, it can report the status to the customer and rollback the transaction.
e.g. a customer purchasing an items from your online storefront: the same service can handle updating the order details, adjusting available inventory and processing the payment.
Microservices
A microservices architecture arranges an application as a collection of loosely coupled services, using a lightweight protocol.
Some of the defining characteristics of microservices:
Services are usually processes that communicate over a network.
Services are organized around business capabilities i.e. they provide specialized, domain-specific services to applications (or other services).
Service are not tied to any one programming language, platform or set of technologies.
Services are small, decentralized, and independently deployable.
Each microservice is expected to operate independently, and contain all of the logic that it requires to perform its specialized task. Microservices are distributed, so that each can be deployed to a separate system or tier.
The advantage of microservices over services is that we have prioritized decoupling of components and maximized cohesion - each microservice has a specific role and no dependencies. This makes extending and scaling out new microservices trivial. However, the cost of this is performance – communication over the network is relatively slow compared to inter-process communication on the same system.
The driving philosophy of microservices is the notion of bounded context: each service models a domain or workflow. Thus, each service includes everything necessary to operate within the application, including classes, other subcomponents, and database schemas. – Mark Richards
Although the services themselves are independent, they need to be able to call one another to fulfil business requirements. e.g. a customer attempting to checkout online may have thier order sent to a Shipping service to organize the details of the shipment, but then a request would need to be sent from the Shipping service to the Payment service to actually process the payment.
This suggests that communication between microservices is a key requirement. The architect utilizing this architecture would typically define a standard communication protocol e.g. message queues, or REST.
Coordinating a multi-step process like this involves either cooperation between services (as described above), or a third coordinating service.
Physical in the sense of where it is installed. This matters in systems where software can consist of components installed on different hardware, communicating and exchanging information. ↩︎
If you find this interesting, CS 454 Distributed Systems is highly recommended! This is far too complex a topic to cover in a few paragraphs. ↩︎
Software Design
The term “software design” is heavily overloaded, with many different interpretations of what it entails.
A UX designer will treat design as the process of working with users to identify requirements, and iterating on the interaction and experience design with them to fine tune how they want the experience to work.
A software engineer will want to consider ways of designing modules and source code that emphasize desireable characteristics like scalability, reliability and performance.
A software developer may want to consider readability of the code, and compatibility with existing code bases (among other things).
In this course, we’ll treat design as the complete set of low-level implementation decisions that are made prior to coding a system. We’ll discuss some different approaches to design that have been impactful and useful.
Features of Good Design
It doesn’t take a huge amount of knowledge and skill to get a program working. Kids in high school do it all the time… The code they produce may not be pretty; but it works. It works because getting something to work once just isn’t that hard.
Getting software right is hard. When software is done right, it requires a fraction of the human resources to create and maintain. Changes are simple and rapid. Defects are few and far between. Effort is minimized, and functionality and flexibility are maximized.
– Bob Martin, Clean Architecture (2016).
One recurring theme keep cropping up: the notion that software should be enduring. Software that you produce should be able to function for a long period of time, in a changing environment, where adjustments will need to be made over time; defects will be found and fixed; new features will be introduced and old features phased out.
As “Uncle Bob” points out, It’s relatively easy to get something to compile and work once, in a restricted environment; it’s much more difficult to build something that can be extended and modified over time. If you want software that can be useful for a long time, you need to design for that as well.
Of course, first and foremost, we want to design software that performs its intended function, but we also want robust software that is a joy to extend and maintain. Let’s talk about the characteristics of “good” software that support this approach.
Code Reuse
Software is expensive and time-consuming to produce, so anything that reduces cost or time is welcome. Reusability, or code reuse is often positioned as the easiest way to accomplish this. It also reduces defects, since you’re presumably reusing code that is tested and known-good.
“I see three levels of reuse.
At the lowest level, you reuse classes: class libraries, containers, maybe some class “teams” like container/iterator.
Frameworks are at the highest level. They really try to distill your design decisions. They identify the key abstractions for solving a problem, represent them by classes and define relationships between them. JUnit is a small framework, for example. It is the “Hello, world” of frameworks. It has Test, TestCase, TestSuite and relationships defined.
A framework is typically larger-grained than just a single class. Also, you hook into frameworks by subclassing somewhere. They use the so-called Hollywood principle of “don’t call us, we’ll call you.” The framework lets you define your custom behavior, and it will call you when it’s your turn to do something. Same with JUnit, right? It calls you when it wants to execute a test for you, but the rest happens in the framework.
There also is a middle level. This is where I see patterns. Design patterns are both smaller and more abstract than frameworks. They’re really a description about how a couple of classes can relate to and interact with ”
One of the reasons that we like design patterns is that they’re a different type of reuse: instead of reusing the software directly, we’re reusing designs in a way that results in better code. We’ll discuss these in detail below.
Extensibility
Extensibility implies the ability to modify your code, to expand existing features or add new features. e.g. an image editor adding support for a new image type; a plain text editor adding support for code fences and syntax highlighting. Conditions will change over the lifetime of your software, and you need to design in a way that allows you to respond to changes.
In the sections below, we will discuss different approaches to handling these challenges.
Readability
We’re programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.
There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:
It’s very likely that the software that you write will need to be read by someone else: your teammates, the people that follow you on a project, maybe even hundreds or thousands of other developers if you relase it publically.
For that reason, it’s not enough to have code that works; it should work, and be clear and understandable to other people that will need to read it. Keep in mind that the “other people” may include future-you. Will your code still make sense if you have to come back to it a year from now? Five years from now? Code comments (that describe why you made your design decisions), and consistent code structure go a long way to making code readable.
Design Principles
What is good software design? How would you measure it? What practices would you need to follow to achieve it? How can you make your architecture flexible, stable and easy to understand?
These are the great questions; but, unfortunately, the answers are different depending on the type of application you’re building.
– Shvets, Dive Into Design Patterns (2020).
We do have some universal principles that we can apply to any situation.
Encapsulate What Varies
Info
Identify the aspects of your application that vary and separate them from what stays the same.
The main goal of this principle is to minimize the effect caused by changes.
You can do this by encapsulating classes, or functions. In both cases, your goal is separate and isolate the code that is likely to change from the rest of your code. This minimizes what you need to change over time.
The following example is taken from Shvets (and rewritten in non-idiomatic Kotlin).
fungetOrderTotal(order){
total=0for (iteminorder.lineItems)
total+=item.price*item.quantity
if(order.country=="US")
total+=total*0.07// US sales tax
elseif(order.country=="EU"):
total+=total*0.20// European VAT
returntotal
}
Given that the tax rates will likely vary, we should isolate them into a separate function. This way, when the rates change, we have much less code to modify.
Similarly, we can split up classes into smaller independent units.
Restructured classes.
Program to an Interface, Not an Implementation
Info
Program to an interface, not an implementation. Depend on abstractions, not on concrete classes.
When classes rely on one another, you want to minimize the dependency - we say that you want loose coupling between the classes. This allows for maximum flexibility.
Do do this, you extract an abstract interface, and use that to describe the desired behaviour between the classes.
For example, in the diagram below, our cat on the left can eat sausage, but only sausage. The cat on the right can eat anything that provides nutrition, including sausage. The introduction of the food interface complicates the model, but provides much more flexibility to our classes.
Favor Composition over Inheritance
Inheritance is a useful tool for reusing code. In principle, it sounds great - derive from a base class, and you get all of it’s behaviour for free!
Unfortunately it’s rarely that simply. There are sometimes negative side effects of inheritance.
A subclass cannot reduce the interface of the base class. You have to implement all abstract methods, even if you don’t need them.
When overriding methods, you need to make sure that your new behaviour is compatible with the old behaviour. In other words, the derived class needs to act like the base class.
Inheritance breaks encapsulation, because the details of the parent class are potentially exposed to the derived class.
Subclasses are tightly coupled to superclasses. A change in the superclass can break subclasses.
Reusing code through inheritance can lead to parallel inheritance hierarchies, and an explosion of classes. See below for an example.
A useful alternative to inheritance is composition.
Where inheritance represents an is-a relationship (a car is a vehicle), composition represents a has-a relationship (a car has an engine).
Imagine a catalog application for cars and trucks.
Here’s the same application using composition. We can create unique classes representing each vehicle which just implement the appropriate interfaces, without the intervening abstract classes.
SOLID Principles
SOLID was introduced by Robert (“Uncle Bob”) Martin around 2002. Some ideas were inspired by other practitioners, but he was the first to codify them.
The SOLID Principles tell us how to arrange our functions and data structures into classes, and how those classes should be interconnected [ed. The use of the word “class” does not imply that these principles are applicable only to object-oriented software. A class is simply a coupled grouping of functions and data. Every software system has such groupings, whether they are called classes or not].
The goal of the principles is the creation of mid-level software structures that:
Tolerate change (extensibility),
Are easy to understand (readability), and
Are the basis of components that can be used in many software systems (reusability).
Info
If there is one principle that Martin emphasizes, it’s the notion that software is ever-changing. There are always bugs to fix, features to add. In his approach, a well-architected system has features that facilitate rapid but reliable change.
“A module should be responsible to one, and only one, user or stakeholder.” – Robert C. Martin (2002)
The Single Responsibility Principle (SRP) claims to result in code that is easier to maintain. The principle states that we want classes to do a single thing. This is meant to ensure that are classes are focused, but also to reduce pressure to expand or change that class. In other words.
A class has responsibility over a single functionality.
There is only one single reason for a class to change.
There should only be one “driver” of change for a module.
2. Open-Closed Principle
“A software artifact should be open for extension but closed for modification. In other words, the behavior of a software artifact ought to be extendible, without having to modify that artifact. “ – Bertrand Meyers (1988)
This principle champions subclassing as the primary form of code reuse.
A particular module (or class) should be reusable without needing to change its implementation.
Often used to justify class hierarchies (i.e. derive to specialize).
This principle is also warning against changing classes in a hierarchy, since any behaviour changes will also be inherited by that classes children! In other words, if you need different behaviour, create a new subclass and leave the existing classes alone.
Info
The Design Pattern principle of Composition over inheritance runs counter to this, suggesting instead that code reuse through composition is often more suitable.
3. Liskov-Substitution Principle
“If for each object o1 of type S there is an object o2 of type T such that for all programs P defined in terms of T, the behaviour of P is unchanged when o1 is substituted for o2, then S is a subtype of T”. -– Barbara Liskov (1988)
It should be possible to substitute a derived class for a base class, since the derived class should still be capable of all of the base class functionality. In other words, a child should always be able to substitute for its parent.
In the example below, if Sam can make coffee but Eden cannot, then you’ve modelled the wrong relationship between them.
4. Interface Substitution
It should be possible to change classes independently from the classes on which they depend.
Also described as “program to an interface, not an implementation”. This means focusing your design on what the code is doing, not how it does it. Never make assumptions about what the underlying code is doing – if you code to the interface, it allows flexibility, and the ability to substitute other valid implementations that meet the functional needs.
5. Dependency Inversion Principle
The most flexible systems are those in which source code dependencies refer to abstractions (interfaces) rather than concretions (implementations). This reduces the dependency between these two classes.
High-level modules should not import anything from low-level modules. Both should depend on abstractions (e.g., interfaces).
Abstractions should not depend on details. Details (concrete implementations) should depend on abstractions.
This is valuable because it reduces coupling between two classes, and allows for more effective reuse.
In the example below, the PizzaCutterBot should be able to cut with any tool (class) provided to it that meets the requirements of able-to-cut-pizza. This might include a pizza cutter, knife or any other sharp implement. PizzaCutterBot should be designed to worwk with the able-to-cut interface, not a specific tool.
UX & UCD
User Experience Design (UX)
User experience design (UX) is about designing for people first: building a well-organized, cohesive and compelling user experience. It focuses the design lens on the user-interface itself, and the role of user interaction in satisfying a customer’s expectations.
Building a consistent visual and behavioural experience provides a number of benefits1.
Users will learn your application faster if the user interface is familiar.
User can accomplish their tasks more quickly, since the interface is consistent with other applications they’ve used.
Users with special needs will find your product more accessible.
Your application will be easier to document and explain.
Apple’s Human Interface Guidelines
Apple has published a series of Human Interface Design principles which characterize a compelling, useful and efficient user interface. First published with the MacIntosh in 1983, these have been refined over the years, but still reflect underlying design principles:
Metaphor. Take advantage of people’s knowledge of the world by using metaphors to convey concepts and features of your application. Use metaphors that represent concrete, familiar ideas, and make the metaphors obvious, so that users can apply a set of expectations to the computer environment. For example, graphicsl operating systems often use the metaphor of file folders for storing documents.
Reflect the User’s Mental Model. The user already has a mental model that describes the task your software is enabling. This model arises from a combination of real-world experiences, experience with other software, and with computers in general. For example, users have real-world experience writing and mailing letters and most users have used email applications to write and send email. An email application that ignores the user’s mental model and does not meet at least some of the user’s expectations would be difficult and even unpleasant to use.
Explicit and Implied Actions. Most operations involve the manipulation of an object using an action. In the first step of this manipulation, the user sees the desired object onscreen. In the second step, the user selects or designates that object. In the final step, the user performs an action, either using a menu command or by direct manipulation of the object with the mouse or other device. Explicit actions are obvious and clear from user actions (e.g. a menu command). Implied actions arise from interaction with objects (e.g. dragging a graphic into a window to import it). The user’s mental model and expectations should drive your choice of action.
Direct Manipulation. Direct manipulation is an example of an implied action that allows users to feel that they are controlling the objects represented by the computer. According to this principle, an onscreen object should remain visible while a user performs an action on it, and the impact of the action should be immediately visible. e.g. selecting text with the mouse; dragging a graphic between windows to move it.
User Control. Allow the user to initiate and control actions. Provide users with capabilities, but let them remain in control.
Feedback and Communication. Keep users informed about what’s happening by providing appropriate feedback and enabling communication with your application. Look for ways to communicate state!
Consistency. Consistency in the interface allows users to transfer their knowledge and skills from one application to another. Use the standard elements of the interface to ensure consistency within your application and to benefit from consistency across applications. When uncertain, look at what previous applications have done.
Modelessness. As much as possible, allow users to do whatever they want at all times. Avoid using modes that lock them into one operation and prevent them from working on anything else until that operation is completed.
User-Centred Design (UCD)
“I walk around the world and encounter new objects all the time. How do I know how to use them?”
— Don Norman
In the mid 1980s, while on sabbatical at the Applied Psychology Unit at Cambridge University, Don Norman found himself puzzled by the light switches he encountered. Why light switches? As the story goes, the switches moved in the opposite direction to what he expected (“up” denoting “on” by North American convention, while at Cambridge they worked in the opposite direction ). This led him to consider the “perceived affordance” between user and item being used; the visual “hints” that suggest how something should be used. In other words, “how we [manage] in a world of tens of thousands of objects.”
These ponderings eventually led to the publication of his book, The Design of Everyday Things in 1988 [Norman 1988, 2013]. In this book, Norman presents case studies of objects we interact with everyday, from door handles to computers and phones. He shows what good and bad design look like across a wide spectrum of devices, inviting us to think about all the daily “user experiences” we take for granted. He wasn’t thinking about software explicitly, but his insights apply equally well to that domain.
For Norman, the two most important characteristics of good design are discoverability – the ability of a user to determine the purpose of an object – and understanding – figuring out how it works and how to use it. From Norman’s perspective, good design should make these characteristics obvious and intuitive.
Norman illustrates what he considers the fundamental principles of interaction:
Affordances: cues that allow a person to figure out what an object does without any instructions. e.g. a door with a “pull” handle (formally called “perceived affordance”, which positions then as properties of the perceiver, as much as the object itself).
Signifiers: visible signs or sounds that communicate meaning. They indicate what is happening, or what can be done. e.g. a door that says “push” on it.
Mapping: the relationship between two sets of things that conveys associated meaning. e.g. grouping controls and putting them next to the thing that you want to manipulate.
Feedback: the result of an action, communicated to a user. e.g. a button changing color when depressed; a walk indicator that lights up when the walk button is activated.
Conceptual models: simplifed explanations of how something works; related to mental models that people automatically form when working with an object to help them explain and predict its behaviour.
“Good” design is about getting these characteristics “right” to reduce the friction between the user’s intention and the outcome of interacting with an object. In order for something to be useful, it needs to serve a clear purpose for some user, and it should be intuitive to use (in other words, designed using the principles he specified).
So how do we know when we have things “right”? We talk to users! We iterate on our designs, collect feedback and made revisions until we get the best design that we can. Norman’s insights kick-started a shift towards designing for users, and directly led to the emergence of UCD and modern UX.
User-Centered Design (UCD) is design process that was developed in parallel to the Agile models that we have discussed. It focuses on addressing the needs of users first, to ensure that we’re building a high-quality solution and solving the correct problem for our users. UCD uses a mixture of investigative tools (e.g. interviewing) and generative methods (e.g. brainstorming) to design systems based on user needs, and collect feedback through the process2.
The first requirement for an exemplary user experience is to meet the exact needs of the customer, without fuss or bother.
Next comes simplicity and elegance that produce products that are a joy to own, a joy to use. True user experience goes far beyond giving customers what they say they want, or providing checklist features.
In order to achieve high-quality user experience in a company’s offerings there must be a seamless merging of the services of multiple disciplines, including engineering, marketing, graphical and industrial design, and interface design.
Understand and address the core problems. Ensuring that we solve the core, root issues, not just the problem as presented.
Be people-centred. We are designing software for people. A well-engineered solution that nobody uses is a failure.
Use an Activity-Centred Approach. Design must focus upon the entire activity under consideration, not just isolated components. Most complications result from the interdependencies of multiple parts.
Use Rapid Iterations of Prototyping and Testing. Continually test and refine our solution, and get feedback to ensure that it truly meets the needs of users.
UCD is frequently mistaken for UX (User Experience). They are related but not identical. User-Centred Design is the process of considering users through planning design and development of a product. It’s a set of steps and methods that we can use to ensure that we are designing for users. User Experience (UX) is the conceptualization of a user’s full experience with a product. It’s concerned with the users perceptions of the entire solution.
This course is primarily concerned with design and implementation, so we’ll utilize UCD in our design3.
Collecting Feedback with Prototypes
It’s possible that a large feature may take multiple iterations to complete. In cases like this, you may want to elicit feedback on the feature before completing it. We can do this by building a low-effort prototype which demonstrates some degree of functionality, without being complete.
A prototype is a mockup of a system, intended to demonstrate some functionality. A prototype is NOT an “early version” of release software, but an early demonstration of functionality intended to elicit feedback from users.
The value in a prototype is (a) that is represents a lower commitment (time, cost) than building the feature outright, and (b) because of this, can be discarded or modified as needed.
Prototypes make sense for high-effort features. For simple features that don’t require feedback, or ones that can be implemented in a single sprint, it may not be worth the effort.
Low-fidelity prototypes are simple, low-tech, and represent a minimal investment of time and effort. They are shown to users to elicit early feedback on layout, navigation and other high-level features. Low-fidelity prorotypes are often sketches on paper, or non-interactive wireframe diagrams.
High-fidelity prototypes represent a more significant effort and are intended to be close to the final product. They are suitable for demonstrating complex functionality. High-fidelity prototypes may include interactive screen mockups that demonstrate the expected interaction. Commercial products exist for this purpose e.g. balsamiq.
CS 449 covers UX/UCD in greater detail and is highly recommended if you have an interest in design. ↩︎
Design Patterns
A design pattern is a generalizable solution to a common problem that we’re attempting to address. Design patterns in software development are one case of a formalized best practice, specifically around how to structure your code to address specific, recurring design problems in software development.
We include design patterns in a discussion of software development because this is where they tend to be applied: they’re more detailed that architecture styles, but more abstract than source code. Often you will find that when you are working through high-level design, and describing the problem to address, you will recognize a problem as similar to something else that you’ve encoutered. A design pattern is a way to formalize that idea of a common, reusable solution, and give you a standard terminology to use when discussing this design with your peers.
Patterns originated with Christopher Alexander, an architect, in 19771. Design patterns in software gained popularity with the book Design Patterns: Elements of Reusable Object-Oriented Software, published in 1994 [Gamma 1994]. There have been many books and articles published since then, and during the early 2000s there was a strong push to expand Design Patterns and promote their use.
Design patterns have seen mixed-success. Some criticisms levelled:
They are not comprehensive, and do not reflect all styles of software or all problems encountered.
They are old-fashioned and do not reflect current software practices.
They add flexibility, at the cost of increased code complexity.
Broad criticisms are likely unfair. While it’s true that not all patterns are used, many of them are commonly used in professonal practice, and new patterns are being suggested. Design patterns certainly can add complexity to code, but they also encourage designs that help avoid subtle bugs later on.
In this section, we’ll outline the more common patterns, and indicate where they may be useful. The original set of patterns were subdivided based on the types of problems they addressed. We’ll examine a number of patterns in each category below: the original patterns are taken from Eric Gamma et al. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Examples and some explanations are from Alexander Shvets. 2019. Dive Into Design Patterns.
Specify the kinds of objects to create using a prototypical instance, and create new objects from the ‘skeleton’ of an existing object, thus boosting performance and keeping memory footprints to a minimum.
Ensure a class has only one instance, and provide a global point of access to it.
Example: Builder Pattern
Builder is a creational design pattern that lets you construct complex objects step by step. The pattern allows you to produce different types and representations of an object using the same construction code.
Imagine that you have a class with a large number of variables that need to be specified when it is created. e.g. a house class, where you might have 15-20 different parameters to take into account, like style, floors, rooms, and so on. How would you model this?
You could create a single class to do this, but you would then need a huge constructor to take into account all of the different parameters.
You would then need to either provide a long parameter list, or call other methods to help set it up after it was instantiated (in which case you have construction code scattered around).
You could create subclasses, but then you have a potentially huge number of subclasses, some of which you may not actually use.
The builder pattern suggests that you put the object construction code into separate objects called builders. The pattern organizes construction into a series of steps. After calling the constructor, you call methods to invoke the steps in the correct order (and the object prevents calls until it is constructed). You only call the steps that you require, which are relevant to what you are building.
Even if you never utilize the Builder pattern directly, it’s used in a lot of complex Kotlin and Android libraries. e.g. the Alert dialogs in Android.
val dialog = AlertDialog.Builder(this)
.setTitle("Title")
.setIcon(R.mipmap.ic_launcher)
.show()
Example: Singleton
Singleton is a creational design pattern that lets you ensure that a class has only one instance, while providing a global access point to this instance.
Why is this pattern useful?
Ensure that a class has just a single instance. The most common reason for this is to control access to some shared resource—for example, a database or a file.
Provide a global access point to that instance. Just like a global variable, the Singleton pattern lets you access some object from anywhere in the program. However, it also protects that instance from being overwritten by other code.
All implementations of the Singleton have these two steps in common:
Make the default constructor private, to prevent other objects from using the new operator with the Singleton class.
Create a static creation method that acts as a constructor.
In languages like Java, you would express the implementation in this way:
objectSingleton{
init {
println("Singleton class invoked.")
}
funprint(){
println("Print method called")
}
}
funmain(args: Array<String>) {
Singleton.print()
// echos "Print method called" to the screen
}
The object keyword in Kotlin creates an instance of a generic class. i.e. it’s instantiated automatically. Like any other class, you can add properties and methods if you wish.
Singletons are useful for times when you want a single, easily accessible instance of a class. e.g. Database object to access your database, Configuration object to store runtime parameters and so on. You should also consider it instead of extensively using global variables.
Convert the interface of a class into another interface clients expect. An adapter lets classes work together that could not otherwise because of incompatible interfaces.
Compose objects into tree structures to represent part-whole hierarchies. Composite lets clients treat individual objects and compositions of objects uniformly.
Attach additional responsibilities to an object dynamically keeping the same interface. Decorators provide a flexible alternative to subclassing for extending functionality.
Provide a surrogate or placeholder for another object to control access to it.
Example: Adapter
Adapter is a structural design pattern that allows objects with incompatible interfaces to collaborate.
Imagine that you have a data source that is in XML, but you want to use a charting library that only consumes JSON data. You could try and extend one of those libraries to work with a different type of data, but that’s risky and may not even be possible if it’s a third-party library.
An adapter is an intermediate component that converts from one interface to another. In this case, it could handle the complexities of converting data between formats. Here’s a great example from Shvets (2019):
The simplest way to implement this is using object composition: the adapter is a class that exposes an interface to the main application (client). The client makes calls using that interface, and the adapter performs necessary actions through the service (which is often a library, or something whose interface you cannot control).
The client is the class containing business logic (i.e. an application class that you control).
The client interface describes the interface that you have designed for your application to communicate with that class.
The service is some useful library or service (typically which is closed to you), which you want to leverage.
The adapter is the class that you create to serve as an intermediary between these interfaces.
The client application isn’t coupled to the adapter because it works through the client interface.
Behavioural Patterns
Behavioural Patterns are about identifying common communication patterns between objects.
Encapsulate a request as an object, thereby allowing for the parameterization of clients with different requests, and the queuing or logging of requests. It also allows for the support of undoable operations.
Define a one-to-many dependency between objects where a state change in one object results in all its dependents being notified and updated automatically.
Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it.
Represent an operation to be performed on the elements of an object structure. Visitor lets a new operation be defined without changing the classes of the elements on which it operates.
Example: Command
Command is a behavioural design pattern that turns a request into a stand-alone object that contains all information about the request (a command could also be thought of as an action to perform).
Imagine that you are writing a user interface, and you want to support a common action like Save. You might invoke Save from the menu, or a toolbar, or a button. Where do you put the code that actually handles saving the data?
If you attach it to the object that the user is interacting with, then you risk duplicating the code. e.g.
The Command pattern suggests that you encapsulate the details of the command that you want executed into a separate request, which is then sent to the business logic layer of the application to process.
The command class relationship to other classes:
Example: Observer (MVC)
Observer is a behavioral design pattern that lets you define a subscription mechanism to notify multiple objects about any events that happen to the object they’re observing. This is also called publish-subscribe.
The object that has some interesting state is often called subject, but since it’s also going to notify other objects about the changes to its state, we’ll call it publisher. All other objects that want to track changes to the publisher’s state are called subscribers, or observers of the state of the publisher.
Subscribers register their interest in the subject, who adds them to an internal subscriber list. When something interest happens, the publisher notifies the subscribers through a provided interface.
The subscribers can then react to the changes.
A modified version of Observer is the Model-View-Controller (MVC) pattern, which puts a third intermediate layer between the Publisher and Subscriber, which manages user input. That layer is not required for this pattern.
For more details on MVC, see the Building Applications section.
“The Pattern of Streets,” JOURNAL OF THE AIP, September, 1977, Vol. 32, No. 3, pp. 273–278 ↩︎
UML
Architecture and design are all about making important, critical decisions early in the process. It’s extremely valuable to have a standard way of documenting systems, components, and interactions to aid in visualizing and communicating our designs.
The Unified Modelling Language (aka UML) is a modeling language consisting of an integrated set of diagrams, useful for designers and developers to specify, visualize, construct and communicate a design. UML is a notation that resulted from the unification of three competing modelling techniques1:
Booch [Grady Booch 1994] - was excellent for design and implementation. Grady Booch had worked extensively with the Ada language, and had been a major player in the development of Object Oriented techniques for the language. Although the Booch method was strong, the notation was less popular.
OOSE (Object-Oriented Software Engineering [Ivar Jacobson 1992]) - featured a model known as Use Cases.
All three designers joined Rational Software in the mid 90s, with the goal of standardizing this new method. Along with many industry partners, they drafted an intial proposal which was submitted to the Object Management Group in 1997. This led to a series of UML standards driven through this standards body, with UML 2.5 being the current version.
The primary goals of UML include:
providing users with a common, expressive language that they can use to share models.
provide mechanism to extend the language if needed.
remain independent of any particular programming language or development process2
support higher-level organizational concepts like frameworks, patterns.
UML contains a large number of diagrams, intended to address the needs to a wide range of stakeholders. e.g. analysis, desigers, coders, testers, customers.
UML contains both structure and behaviour diagrams.
Structure diagrams show the structure of the system and its parts at different level of abstraction, and shows how they are related to one another. Behaviour diagrams show the changes in the system over time.
These diagrams are intended to cover the full range of possible scenarios that we want to model. It’s common (and completely reasonable!) to only use the diagrams that you actually need. You will find, for instance, that component and class diagrams are commonly used when discussing component-level behaviour; package and deployment diagrams are used when determining how to install and execute your deliverable and so on.
Below we’ll highlight the most commonly used UML diagrams3. For more comprehensive coverage, see Visual Paradigm or Martin Fowler’s UML Distilled [Fowler 2004].
Info
You should NOT create diagrams for every components, interaction or state in your system. That’s overkill for most projects. Instead, focus on building a high-level component diagram that shows the basic component interactions, which you can use to plan your system. Secondly, use diagrams if you have a particular component or sequence that is exceptionally complex, or important to get “right”.
Structure Diagrams
These document the static components in a system.
Class Diagram
The class diagram is a central modeling technique that runs through nearly all object-oriented methods. This diagram describes the types of objects in the system and various kinds of static relationships which exist between them.
There are three principal kinds of relationships which are important to model:
Association - represent relationships between instances of types (a person works for a company, a company has a number of offices.
Inheritance - the most obvious addition to ER diagrams for use in OO. It has an immediate correspondence to inheritance in OO design.
Aggregation - Aggregation, a form of object composition in object-oriented design.
Component Diagram
A component diagram depicts how components are wired together to form larger components or software systems. It illustrates the architectures of the software components and the dependencies between them. Those software components including run-time components, executable components also the source code components.
Deployment Diagram
The Deployment Diagram helps to model the physical aspect of an Object-Oriented software system. It is a structure diagram which shows architecture of the system as deployment (distribution) of software artifacts to deployment targets. Artifacts represent concrete elements in the physical world that are the result of a development process. It models the run-time configuration in a static view and visualizes the distribution of artifacts in an application. In most cases, it involves modeling the hardware configurations together with the software components that lived on.
Behaviour Diagrams
These document behaviours of the system over time.
Use Case Diagram
A use-case model describes a system’s functional requirements in terms of use cases. It is a model of the system’s intended functionality (use cases) and its environment (actors). Use cases enable you to relate what you need from a system to how the system delivers on those needs. The use-case model is generally used in all phases of the development cycle by all team members and is extremely popular for that reason.
Info
Keep in mind that UML was created before UX. Use Cases are very much like User Stories, but with more detail. Think of Use Cases == User Stories, and Actors == Personas (except that they can also model non-human actors like other systems).
Activity Diagram
Activity diagrams are graphical representations of workflows of stepwise activities and actions with support for choice, iteration and concurrency. It describes the flow of control of the target system. Activity diagrams are intended to model both computational and organizational processes (i.e. workflows).
Info
I use these a lot when designing interactive systems, and modelling transitions between screens or areas of functionality.
Interaction Overview Diagram
The Interaction Overview Diagram focuses on the overview of the flow of control of the interactions. It is a variant of the Activity Diagram where the nodes are the interactions or interaction occurrences. The Interaction Overview Diagram describes the interactions where messages and lifelines are hidden. You can link up the “real” diagrams and achieve high degree navigability between diagrams inside the Interaction Overview Diagram.
Info
I don’t know that I’ve ever seen these used, but I’m specifically calling them out b/c you might be tempted to use them for interactive applications. I would probably choose a more straightforward activity diagram, and invest my remaining design time in prototyping and iterating with users.
State Machine Diagram
A state diagram is a type of diagram used in UML to describe the behavior of systems which is based on the concept of state diagrams by David Harel. State diagrams depict the permitted states and transitions as well as the events that effect these transitions. It helps to visualize the entire lifecycle of objects and thus help to provide a better understanding of state-based systems.
Sequence Diagram
The Sequence Diagram models the collaboration of objects based on a time sequence. It shows how the objects interact with others in a particular scenario of a use case.
Info
When do you use a Sequence Diagram over a more simple Activity Diagram? When you have a more complex interaction between components. I’ve used these to model client-server authentication for instance, or passing data between systems (where it might need to be encrypted/decrypted at each side).
… but not programming paradigm. UML assumes OO, and specifically includes OO-specific diagrams, though it doesn’t require them to be used. ↩︎
The diagrams below are all taken from the Visual Paradigm UML site. Visual Paradigm is the name of the company, but also their main product, which is a UML modelling tool. They have a vested interest in getting the diagrams “right”. ↩︎
Implementation
The implementation phase is focused on developing features and/or making changes to software. At this stage, you should have an understanding of what your user needs from this feature (requirements) and high-level technical decisions have been made (analysis & design).
This step consists of four sets of related activities:
Objective
Implement your design
Activities
Document low-level/component design; Implement features; Produce unit tests
Outcome
Source code and unit tests
Implementation consists of all activities required to realize a feature. This includes any remaining design, coding, and writing appropriate unit tests. It may also include other supporting activities: writing documentation for other users, or scripts that may be needed to deploy it properly.
The Goals of Software Implementation
In A Philosophy of Software Design, John Ousterhout suggests that software design is all about:
Abstraction: picking the appropriate abstractions to represent the domain.
Problem decomposition: breaking a task into small, comprehensible problems.
Reducing complexity: simplifying the problem enough that you can build an appropriate mental model.
Good design focuses on reducing complexity in software; decomposing problems in a way that minimizes complexity. In other words, we’re fundamentally trying to break down a complex problem into smaller, manageable chunks. You need to be able to form a conceptual mental model of what you’re building (“holding the system in your head”).
Courtesy of Ousterhout1, here are some suggestions for avoiding complexity in your designs:
1. Classes Should be Deep
This is another way of talking about information hiding [Parnas 1972]. We hide information not due to distrust or security concerns, but as a way of reducing complexity for the programmer. We want to package up functionality in a way that reduces the programmer’s need to understand how it works.
The interface of a class is the complexity cost of the implementation (again, complexity refers to the programmer’s mental model - what they have to know to use the class properly). We want an interface that hides a LOT of complexity to make the cost of using the class worth the benefit.
Here’s an example of a shallow class that has a negative benefit: it costs more mental effort to determine what it does than it does to just write the underlying code yourself (and it would be fewer keystrokes!)
You cannot always eliminate shallow classes, but a shallow class doesn’t help you much in the fight against complexity.
“Classes and Methods Should Be Small” [Many CS textbooks].
I absolutely disagree. Classes and methods should be as large as they need to be to appropriately abstract their functionality.
Example: shallow functionality, complex interfaces
Look at the Java SDK for the consequences of class “explosion”, where each class adds an almost trivial amount of functionality. The class hierarchy is far too broad and deep, and requires a huge amount of effort to figure out.
e.g. Java File libraries. You need multiple classes to open a file with buffering, with serialization. The common case requires three classes (!). See Kotlin, which wraps all of this in a single class.
Example: deep functionality, simple interface
Unix File I/O handles the same problem much more simply and elegantly.
This abstracts: on-disk representation, disk block allocation, directory management, permissions, caching, device independence.
2. Define Errors Out Of Existence
Exceptions are a huge source of complexity.
Common wisdom: detect and throw as many errors as possible.
Better approach: define semantics to eliminate exceptions.
Goal: minimize the number of places where exceptions must be handled
Often we can redefine the semantics so that there is no error.
Examples of poor design choices
TCL unset a variable that doesn’t exist throws an exception
Windows cannot delete a file if it’s open.
Java substring class throws out-or-range exceptions.
The common case should be simple and just work. Save exceptions for runtime behaviour that you cannot otherwise manage.
3. Practice Strategic Programming
Most of us are driven to make tactical programming decisions:
Goal: get next feature working or bug fixed ASAP
A few shortcuts are taken, hacks are put in-place.
Result: bad design, high complexity.
Complexity is incremental and these kludges and hacks build up over time. Mistakes build up over time, until our code ends up in a poor state - highly complex, poorly designed.
In the long-term, thinking tactically harms code quality. We need to be disciplined and think strategically.
Strategic programming
Goal: produce a great design (while solving today’s problem)
Simplify future development
Minimize complexity
“Sweat the small stuff”
We need an investment mindset: extra time taken now will pay off in the long-term with higher-quality code. Yes it’s slower at first, but usually worth the investment2.
Examples
Most startups are purely tactical.
Pressure to get products out quickly/first.
“We’ll clean this up later”.
Code quickly turns to spaghetti.
Extremely difficult an expensive to repair later.
Facebook
Culture of “Move quickly and break things”.
They’ve since changed their motto to “Move quickly with solid infrastructure”.
Google and Apple
Both companies have a strong design culture, which attracts the best engineers.
Their products tend to work very well.
Continuous Integration (CI)
Test-Driven Development addresses the issue of doing local, small-scope testing as part of implementation. However, it doesn’t address issues related to the system as a whole, or that might only occur when components are integrated.
The fundamental benefit of continuous integration is that it removes sessions where people spend time hunting bugs where one person’s work has stepped on someone else’s work without either person realizing what happened. These bugs are hard to find because the problem isn’t in one person’s area, it is in the interaction between two pieces of work.
– Fowler, 2000.
A system that supports continuous integration needs, at a minimum, the following capabilities:
It requires a revision control system, with a centralized main revision that can be used.
The build process should be automated so that anyone can manually launch the process. [it should also support automated testing based on other events, like integrating a branch in the source tree].
Tests should be automated so that they can be launched manually as well.
The system should produce a final distribution.
CI Systems
Continuous Integration Systems are software systems that provide these capabilities. Early standalone systems include Jenkins (Open Source), and CircleCI. Many source control platforms also provide CI functionality, including Bitbucket, GitHub and GitLab.
For example, you can automate GitLab so that it will build and run your tests anytime a specific action is performed like committing to a branch, or merging a PR. This is managed through the CI/CD section of the project configuration.
The GitLab configuration and terminology is pretty standard:
A pipeline represents the work or the job that will be done.
A stage represents a set of jobs that need to be executed together.
Jobs are executed by runners, which define and where a job will be executed.
These all represent actions that will be taken against your source code repository at specific times. The examples that they provide include:
A build stage, with a job called compile.
A test stage, with two jobs called test1 and test2.
A staging stage, with a job called deploy-to-stage.
A production stage, with a job called deploy-to-prod.
In this way, you can setup your source code repository to build, test, stage and deploy your software automatically one or more times per day, as a result of some key event, or when manually executed.
Info
Although we have a GitLab instance running, we do not have access to a cluster that can run jobs for us. In other words, we cannot do this in production using our current setup – at least not without gaining access to a Kubernetes cluster somewhere.
John Ousterhout. 2018. A Philosophy of Software Design. ↩︎
This assumes that we will pass the point of intersection of these curves. MOST software will live long enough to justify the time investment. ↩︎
Subsections of Implementation
Pair Programming
Pair programming means that two people design and implement code together, on a single machine. This is a very collaborative way of working that involves a lot of communication and collaboration between them. While a pair of developers work on a task together, they do not only write code, they also plan and discuss their work. They clarify ideas on the way, discuss approaches and come to better solutions.
Originally an Extreme Programming practice, it is considered a best-practice today, used successfully on many programming teams.
Benefits
There are tangible benefits to pair programming - both to the organization and project team, and to the quality of the work produced.
For the organization:
remove knowledge silos to increase team resiliency
build collective code ownership
help with team building
accelerate on-boarding
serve as a short feedback loop, similar to a code review happening live
increase safety and trust of the developers pairing
increase developer confidence
Research supports the idea that a pair of programmers working together will produce higher quality software, in the same or less time as the developers working independently (Williams et al. 2000).
Surprising, research also suggests that there is no loss in productivity when pair programming. In other words, they produce at least the same amount of code as if they were working independntly, but it tends to be higher quality than if they worked alone.
The Driver is the person at the wheel, i.e. the keyboard. She is focussed on completing the tiny goal at hand, ignoring larger issues for the moment. A driver should always talk through what she is doing while doing it.
The Navigator is in the observer position, while the driver is typing. She reviews the code on-the-go, gives directions and shares thoughts. The navigator also has an eye on the larger issues, bugs, and makes notes of potential next steps or obstacles.
Remember that pair programming is a collaboration. A common flow goes like this:
Start with a reasonably well-defined task. Pick a requirement or user story for example.
Agree on one tiny goal at a time. This can be defined by a unit test, or by a commit message, or some goal that you’ve written down.
Switch keyboard and roles regularly. Alternating roles keeps things exciting and fresh.
As navigator, leave the details of the coding to the driver - your job is to take a step back and complement your pair’s more tactical mode with medium-term thinking. Park next steps, potential obstacles and ideas on sticky notes and discuss them after the tiny goal is done, so as not to interrupt the driver’s flow.
Ping-Pong
This technique is ideal when you have a clearly defined task that can be implemented in a test-driven way.
“Ping”: Developer A writes a failing test
“Pong”: Developer B writes the implementation to make it pass.
Developer B then starts the next “Ping”, i.e. the next failing test.
Each “Pong” can also be followed by refactoring the code together, before you move on to the next failing test.
Strong-Style Pairing
This is a technique particularly useful for knowledge transfer. In this style, the navigator is usually the person much more experienced with the setup or task at hand, while the driver is a novice (with the language, the tool, the codebase, …). The experienced person mostly stays in the navigator role and guides the novice.
An important aspect of this is the idea that the driver totally trusts the navigator and should be “comfortable with incomplete understanding”. Questions of “why”, and challenges to the solution should be discussed after the implementation session.
Switching roles while pairing is essential to the process—it’s also one of the trickiest things to do correctly. The navigator and driver have very different frames of reference.
The Wrong Way
Pairing is about working together. Anything that impedes one of the pairers from contributing or breaks their flow is bad. Two of the more obvious wrong ways are to “grab the keyboard” or “push the keyboard”.
Grabbing the keyboard: Sometimes when working as the navigator it’s tempting to take the keyboard control away to quickly do something. This puts the current driver in a bad position. Not only are they now not contributing, but such a forceful role change is likely to lead to conflict.
Pushing the keyboard: Other times, the driver feels a strong need to direct the strategy. It’s very tempting to just “push” the keyboard to the navigator, forcing them to take the driver’s seat, and start telling them what to do. This sudden context switch can be jarring and confusing to the unsuspecting navigator. It can lead to resentment and conflict as the navigator feels invalidated or ignored.
Finally, even a consensual role switch can be jarring and confusing if done too quickly and without structure.
The Right Way
The first step to switching roles is always to ask. The navigator needs to ask if they can grab the keyboard before doing so. The driver needs to ask if the navigator is willing to drive before starting to direct them. Sometimes, switching without asking works out but these situations are the exception.
It’s important to take some time when switching as well. Both pairers need to time to acclimatizing to their new roles. This time can be reduced somewhat by having a structure around switching (e.g. Ping-pong pairing) which allows the pairers to be mentally prepared for the switch to happen.
Pair Rotation
If a feature will take a long time, you might consider rotating people into and out of the pair over time (e.g. one person swaps out and a new person comes). You don’t want to do this frequently, no more than once per day over a full working day. It’s helpful to prevent the duo from become stale or frustrated with one another, and may help with knowledge transfer.
In a small team, with very short cycles, this may not be practical or necessary.
Setup for Pairing
Physical Setup
Ideally, you would work together in the same space. It is worth spending some time figuring out a comfortable setup for both of you.
Make sure both of you have enough space, and that both of you can sit facing the computer.
Agree on the computer setup (key bindings, IDE etc). Check if your partner has any particular preferences or needs (e.g. larger font size, higher contrast, …)
It’s common to have a single keyboard, mouse for the driver, but some pairs setup with two keyboard - that’s your choice.
Remote Setup
If you’re working remotely, you may not be able to physically sit together. Luckily, there are practical solutions to pairing remotely. For remote pairing, you need a screen-sharing solution that allows you to not only see, but also control the other person’s machine, so that you are able to switch the keyboard.
There are also development tools that are designed to specifically address this:
The biggest challenges with pair programming are not code related, but issues with communication and understanding one another. Don’t be afraid to switch roles, or take breaks when required, and be patient with one another!
Post-Mortem
If possible, spend some time reflecting on the pairing session when its over. Ask yourself what went well, and what you can improve. This can be help clear the air of any tension or issues that came up during the session, and help you improve as a team.
Info
There’s a number of solutions that help you pair remotely: JetBrains CodeWithMe or Microsoft’s VS Code Liveshare.
TDD & Unit Testing
Test-Driven Development (TDD) is a strategy introduced by Kent Beck, which suggests writing tests first, before you start coding. You write tests against expected behaviour, and then write code which works without breaking the tests. TDD suggests this process:
Think about the function (or class) that you need to create.
Write tests that describe the behaviour of that function or class. As above, start with valid and invalid input.
Your test will fail (since the implementation code doesn’t exist yet). Write just enough code to make the tests pass.
Repeat until you can’t think of any more tests to write.
Running out of tests means that there is no more behaviour to implement… and you’re done, with the benefit of fully testable code.
Early bug detection. You are building up a set of tests that can verify that your code works as expected.
Better designs. Making your code testable often means improving your interfaces, having clean separation of concerns, and cohesive classes. Testable code is by necessity better code.
Confidence to refactor. You can make changes to your code and be confident that the tests will tell you if you have made a mistake.
Simplicity. Code that is built up over time this way tends to be simpler to maintain and modify.
A unit test is a test that meets the following three requirements [Khorikov 2020]:
Verifies a single unit of behavior,
Does it quickly, and
Does it in isolation from other tests.
Unit tests are just Kotlin functions that execute and check the results returned from other functions. Ideally, you would produce one unit or more unit tests for each function. You would then have a set of unit tests to check all of the methods in a class, and multiple sets of units tests to cover all of your implementation classes.
Your goal should be to have unit tests for every critical class and function that you produce1.
Installing JUnit
We’re going to use JUnit, a popular testing framework2 to create and execute our unit tests. It can be installed in number of ways: directly from the JUnit home page, or one of the many package managers for your platform.
We will rely on Gradle to install it for us as project dependency. If you look at the section of a build.gradle file below, you can see that JUnit is included, which means that Gradle will download, install and run it as required. IntelliJ projects will typically include Gradle by default.
dependencies {// Use the Kotlin test library.
testImplementation org.jetbrains.kotlin:kotlin-test'
// Use the Kotlin JUnit integration.
testImplementation'org.jetbrains.kotlin:kotlin-test-junit'}
How to write tests
A unit test is just a Kotlin class, with annotated methods that tell the compiler to treat the code as a test. It’s best practice to have one test class for each implementation class that you want to test. e.g. class Main has a test class MainTest. This test class would contain multiple methods, each representing a single unit test.
Tests should be placed in a special test folder in the Gradle directory structure. When building, Gradle will automatically execute any tests that are placed in this directory structure3.
Your unit tests should be structured to create instances of the classes that they want to test, and check the results of each function to confirm that they meet expected behaviour.
For example, here’s a unit test that checks that the Model class can add an Observer properly. The addObserver() method is the test (you can tell because it’s annotated with @Test). The @Before method runs before the test, to setup the environment. We could also have an @After method if needed to tear down any test structures.
classObserverTests() {
lateinitvar model: Model
lateinitvar view: IView
classMockView() : IView {
overridefunupdate() {
}
}
@Beforefunsetup() {
model = Model()
view = MockView()
}
@TestfunaddObserver() {
val old = model.observers.count()
model.addObserver(view)
assertEquals(old+1, model.observers.count())
}
Gradle will automatically execute and display the test results when you build your project.
Tests shouldn’t verify units of code. Rather, they should verify units of behaviour: something that is meaningful for the problem domain and, ideally, something that a business person can recognize as useful. The number of classes it takes to implement such a unit of behaviour is irrelevant. — Khorikov (2020)
Code coverage is a metric comparing the number of lines of code with the number of lines of code that have unit tests covering their execution. In other words, what “percentage” of your code is tested?
This is a misleading statistic at the best of times (we can easily contrive cases where code will never be executed or tested).
TDD would suggest that you should have 100% unit test coverage but this is impractical and not that valuable. You should focus instead on covering key functionality. e.g. domain objects, critical paths of your source code.
One recommendation is to look at the coverage tools in IntelliJ, which will tell you how your code is being executed, as well as what parts of your code are covered by unit tests. Use this to determine which parts of the code should be tested more completely.
You will NOT likely get 100% code coverage - see the Testing chapter on dependencies and limits to what we can realistically test. ↩︎
Kent Beck and Eric Gamma invented xUnit, a Smalltalk unit testing framework, while on a flight to OOPSLA in 1997. Over time, it was adapted into nUnit for .NET, CPPUnit for C++ and JUnit for Java. ↩︎
Android has two test folders: src/test is used for tests that can run on your development machine, and src/androidTest is used for tests that need to run on an Android device (e.g. the AVD where you are testing). ↩︎
Code Reviews
What is the value of a code review?
When to perform them?
What feedback is useful?
What to do with the feedback you’re given?
Refactoring
We all want to write perfect code, but given , we often end-up with less-than-perfect solutions.
Rushed features: Sometimes we don’t have enough time, so we cut corners.
Lack of tests: We might think that the code is ready, but we haven’t tested it adequately.
Lack of communication: Perhaps we misunderstood requirements, or how a feature would integrate into the larger product.
Poor design: Possibly our design is rigid and makes adding or modifying features difficult.
These are all actions that may cost us time later. We may need to stop and redesign a rushed feature, or we may need to fix bugs later.
We refer to this as technical debt — the deferred cost of doing something poorly.
What is refactoring?
Martin Fowler (2000) also introduced the notion of refactoring: systematically transforming your working code base into a cleaner, improved version of itself that is easier to read, maintain and extend.
Refactoring is a controlled technique for improving the design of an existing code base. Its essence is applying a series of small behavior-preserving transformations, each of which “too small to be worth doing”. However the cumulative effect of each of these transformations is quite significant.
– Martin Fowler, 2018.
Refactoring suggests that code must be continually improved as we work with it. We need to be in a process of perpetual, small improvements.
The goal of refactoring is to reduce technical debt by making small continual improvements to our code. It doesn’t reduce the likelihood of technical debt, but it amortizes that debt over many small improvements.
Refactoring your code means doing things like:
Cleaning up class interfaces and relationships.
Fixing issues with class cohesion.
Reducing or removing unnecessary dependencies.
Simplifying code to reduce unnecessary complexity.
Making code more understandable and readable.
Adding more exhaustive tests.
In other words, refactoring involves code improvement not related to adding functionality.
TDD and Refactoring work together. You continually refactor as you expand your code, and you rely on the tests to guarantee that you aren’t making any breaking changes to your code.
How to refactor code?
The Rule of Three
When you’re doing something for the first time, just get it done.
When you’re doing something similar for the second time, do the same thing again.
When you’re doing something for the third time, start refactoring.
When adding a feature
Refactor existing code before you add a new feature, since it’s much easier to make changes to clean code. Also, you will improve it not only for yourself but also for those who use it after you.
When fixing a bug
If you find or suspect a bug, refactoring to simplify the existing code can often reveal logic errors.
During a code review
The code review may be the last chance to tidy up the code before it becomes available to the public.
Refactorings
Martin Fowler. 2018. Refactoring: Improving the Design of Existing Code. 2nd Edition. Addison-Wesley. ISBN 978-0134757599.
IntelliJ IDEA also makes this easy by providing automated ways of safely transforming your code. These refactorings often involve operations that would be tricky to do by-hand but easy for the tool do perform for you (e.g. renaming a method that is called from multiple locations).
To invoke refactorings, select an item in your source code (e.g. variable or function name) and press Ctrl-T to invoke the refactoring menu. You can also access from the application menu.
Modify class the become a singleton (returns a single instance).
Code Smells
A “code smell” is a sign that a chunk of code is badly designed or implemented. It’s a great indication that you may need to refactor the code.
Adjectives used to describe code:
“neat”, “clean”, “clear”, “beautiful”, “elegant” <— the reactions that we want
“messy”, “disorganized”, “ugly”, “awkward” <— the reactions we want to avoid
A negative emotional reaction is a flag that your brain doesn’t like something about the organization of the code - even you can’t immediately identify what that is.
Conversely, a positive reaction indicates that your brain can easily perceive and following the underlying structure.
The following categories and examples are taken from refactoring.guru.
Bloaters
Bloaters are code, methods and classes that have increased to such gargantuan proportions that they are hard to work with. These smells accumulate over time as the program evolves (and especially when nobody makes an effort to eradicate them).
Long method: A method contains too many lines of code. Generally, any method longer than ten lines should make you start asking questions.
Large class: A class contains many fields/methods/lines of code. This suggests that it may be doing too much. Consider breaking out a new class, or interface.
Primitive obsession: Use of related primitives instead of small objects for simple tasks (such as currency, ranges, special strings for phone numbers, etc.). Consider creating a data class, or small class instead.
Long parameters list: More than three or four parameters for a method. Consider passing an object that owns all of these. If many of them are optional, consider a builder pattern instead.
Object-Oriented Abusers
All these smells are incomplete or incorrect application of object-oriented programming principles.
Alternative Classes with Different Interfaces: Two classes perform identical functions but have different method names. Consolidate methods into a single class instead, with support for both interfaces.
Refused bequest: If a subclass uses only some of the methods and properties inherited from its parents, the hierarchy is off-kilter. The unneeded methods may simply go unused or be redefined and give off exceptions. This violates the Liskov-substitution principle! Add missing behaviour, or replace inheritance with delegation.
Switch Statement: You have a complex switch operator or sequence of if statements. This sometimes indicates that you are switching on type, something that should be handled by polymorphism instead. Consider whether a class structure and polymorphism makes more sense in this case.
Temporary Field: Temporary fields get their values (and thus are needed by objects) only under certain circumstances. Outside of these circumstances, they’re empty. This may be a place to introduce nullable types, to make it very clear what is actually happening (vs. constantly checking fields for the presence of data).
Dispensibles
A dispensable is something pointless and unneeded whose absence would make the code cleaner, more efficient and easier to understand.
Comments: A method is filled with explanatory comments. These are usually well-intentioned, but they’re not a substitute for well-structured code. Comments are a maintenance burden. Replace or remove excessive comments.
Duplicate Code: Two code fragments look almost identical. Typically, done accidentally by different programmers. Extract the methods into a single common method that is used instead. Alternately, if the methods solve the same problem in different ways, pick and keep the most efficient algorithm.
Dead Code: A variable, parameter, field, method or class is no longer used (usually because it’s obsolete). Delete unused code and unneeded files. You can always find it in Git history.
Lazy Class: Understanding and maintaining classes always costs time and money. So if a class doesn’t do enough to earn your attention, it should be deleted. This is tricky: sometimes a small data class is clearer than using primitives (e.g. a Point class, vs using x and y stored as doubles).
Couplers
All the smells in this group contribute to excessive coupling between classes or show what happens if coupling is replaced by excessive delegation.
Feature envy: A method accesses the data of another object more than its own data. This smell may occur after fields are moved to a data class. If this is the case, you may want to move the operations on data to this class as well.
Inappropriate intimacy: One class uses the internal fields and methods of another class. Either move those fields and methods to the second class, or extract a separate class that can handle that functionality.
Middle man: If a class performs only one action, delegating work to another class, why does it exist at all? It can be the result of the useful work of a class being gradually moved to other classes. The class remains as an empty shell that doesn’t do anything other than delegate. Remove it.
Test-Driven Development addresses the issue of doing local, small-scope testing as part of implementation. However, it doesn’t address issues related to the system as a whole, or that might only occur when components are integrated.
The fundamental benefit of continuous integration is that it removes sessions where people spend time hunting bugs where one person’s work has stepped on someone else’s work without either person realizing what happened. These bugs are hard to find because the problem isn’t in one person’s area, it is in the interaction between two pieces of work.
– Fowler, 2000.
A system that supports continuous integration needs, at a minimum, the following capabilities:
It requires a revision control system, with a centralized main revision that can be used.
The build process should be automated so that anyone can manually launch the process. [it should also support automated testing based on other events, like integrating a branch in the source tree].
Tests should be automated so that they can be launched manually as well.
The system should produce a final distribution.
CI Systems
Continuous Integration Systems are software systems that provide these capabilities. Early standalone systems include Jenkins (Open Source), and CircleCI. Many source control platforms also provide CI functionality, including Bitbucket, GitHub and GitLab.
For example, you can automate GitLab so that it will build and run your tests anytime a specific action is performed like committing to a branch, or merging a PR. This is managed through the CI/CD section of the project configuration.
The GitLab configuration and terminology is pretty standard:
A pipeline represents the work or the job that will be done.
A stage represents a set of jobs that need to be executed together.
Jobs are executed by runners, which define and where a job will be executed.
These all represent actions that will be taken against your source code repository at specific times. The examples that they provide include:
A build stage, with a job called compile.
A test stage, with two jobs called test1 and test2.
A staging stage, with a job called deploy-to-stage.
A production stage, with a job called deploy-to-prod.
In this way, you can setup your source code repository to build, test, stage and deploy your software automatically one or more times per day, as a result of some key event, or when manually executed.
Info
Although we have a GitLab instance running, we do not have access to a cluster that can run jobs for us. In other words, we cannot do this in production using our current setup – at least not without gaining access to a Kubernetes cluster somewhere.
Testing & Evaluation
The goal of testing is to ensure that the software that we produce meets our objectives when deployed into the environment in which it will be used, and when faced with real-world contraints.
Objective
Test components and systems together
Activities
Produce a testing strategy to ensure completeness; produce integration and system tests.
Outcome
Test strategy; integration tests and system tests.
The goal of testing is to ensure that the software that we produce meets our objectives when deployed into the environment in which it will be used, and when faced with real-world constraints.
Why do we test?
We consider testing to be an integral part of software production. Why do we test?
To gain confidence in the correctness of your results.
To gain confidence that you are handling edge cases and errors properly, which will result in a better user experience.
To produce an improved design, usually as a by-product of having written tests.
As a matter of course, it’s also concerned with identifying deficiencies and flaws in software, and helping to determine how they should be addressed. In an iterative model, testing impacts implementation (by providing iterative feedback during the implementation process), and by providing input into planning phases, where we may need to fix bugs or plan product changes.
How do we test?
We can identify three types of testing:
Unit testing: tests operating at the class level (or smallest functional unit), which are meant to check the validity of low-level interfaces and systems.
Integration testing: testing across multiple classes or functional units, to check interaction between objects.
Functional testing: this is testing functionality from the perspective of the user; end-to-end feature testing. Sometimes called System testing.
Unit tests are particularly important and are meant to exercise the interface of a single class or module.
Unit tests should be very quick to execute and report results.
They should be restricted in scope to a class or entity, and dependencies should be eliminated for testing.
They should be integrated into our development workflow, so that they are routinely executed (e.g. prior to commits).
Unit tests should always include “black box” tests. You should always test the interface by examining expected/actual results from a given input. NEVER write tests that rely on knowledge of the inner workings of a class. You should be able to interchange classes with identical interfaces and expect identical results.
Info
Software entropy is the tendency of software to become slower and more error prone over time as we continue to make changes. Unit testing helps overcome software entropy by ensuring that what worked before continues to work now.
We also need to want to confirm that the software works in the environment where it will be deployed. We may need to have multiple staged configurations where we will perform integration tests, to confirm that various systems work together, and functional tests to check user-level correctness.
When should we test?
Traditional views were that testing should be done after implementation. This is costly. Testing is more useful when done earlier in the process.
Different tests are suitable for different parts of the development process:
Unit Tests: done during implementation, when you are working on a class.
Integration Tests: done during implementation, when you want to ensure that classes work together.
System Tests: done when features are complete and merged, to ensure that the system continues working.
In the previous chapter, we discussed Unit Testing and the value of writing tests as you develop (TDD). We’ll continue talking about testing that is done post-implementation. Specifically, let’s discuss Integration testing.
Info
We used junit for unit testing and we’ll also use it for integration testing. Although it’s described as a unit-testing framework, JUnit is perfect suitable for doing broader scoped tests, as we’ll see.
Lisa Crispin, Janet Gregory. 2008. Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley Professional. ISBN 978-0321534460.
Vladimir Khorikov. 2020. Unit Testing Principles, Practices, and Patterns. Manning Publishing. ISBN 978-1617296227.
Glenford J. Myers, Corey Sandler, Tom Badgett. 2011. The Art of Software Testing. Wiley. ISBN 978-1118031964.
Subsections of Testing & Evaluation
Unit Testing
What is a unit test?
Let’s define it more formally. A unit test is a test that meets the following three requirements [Khorikov 2020]:
Verifies a single unit of behavior,
Does it quickly, and
Does it in isolation from other tests.
Unit tests should target classes or components in your program. i.e. they should exercise how a particular class works. They should be small, very focused, and quick to execute and return results.
Testing in isolation means removing the effects of any dependencies on other code, libraries or systems. This means that when you test the behaviour of your class, you are assured that nothing else is contributing to the results that you are observing. We’ll discuss strategies to accomplish this in the [Dependencies] section.
Info
This is also why designing cohesive, loosely coupled classes is critical: it makes testing them so much easier if they work independently!
Structuring a test
A typical unit test uses a very particular pattern, known as the arrange, act, assert pattern. This suggests that each unit test should consist of these three parts:
Arrange: bring the system under test (SUT) to the starting state.
Act: call the method or methods that you want to test.
Assert: verify the outcome of the above action. This can be based on return values, or some other conditions that you can check.
Note that your Act section should have a single action, reflecting that it’s testing a single behaviour. If you have multiple actions taking place, that’s a sign that this is probably an integration test (see below). As much as possible, you want to ensure that you are writing minimal-scope unit tests.
Info
Another anti-pattern is an if statement in a test. If you are branching, it means that you are testing multi-ple things, and you should really consider breaking that one test into multiple tests instead.
How to write tests
Practically, a unit test is just a Kotlin class with annotated methods that tell the compiler how to treat the code. It’s best practice to have one test class for each implementation class that you want to test. e.g. class Main with a test class MainTest. This test class would contain multiple methods, each representing a single unit test.
Tests should be placed in a special test folder in the Gradle directory structure. When building, Gradle will automatically execute any tests that are placed in this directory structure1.
Your unit tests should be structured to create instances of the classes that they want to test, and check the results of each function to confirm that they meet expected behaviour.
For example, here’s a unit test that checks that the Model class can add an Observer properly. The addObserver() method is the test (you can tell because it’s annotated with @Test). The @Before method runs before the test, to setup the environment. We could also have an @After method if needed to tear down any test structures.
classObserverTests() {
lateinitvar model: Model
lateinitvar view: IView
classMockView() : IView {
overridefunupdate() {
}
}
@Beforefunsetup() {
model = Model()
view = MockView()
}
@TestfunaddObserver() {
val old = model.observers.count()
model.addObserver(view)
assertEquals(old+1, model.observers.count())
}
}
Let’s walkthrough creating a test.
To do this in IntelliJ, select a class in the editor, press Alt-Enter, and select “Create Test” from the popup menu. This will create a unit test using JUnit, the default test framwork. There is a detailed walkthrough on the IntelliJ support site.
Info
You can also just create the test files by-hand. Just make sure to save them in the correct location!
The convention is to name unit tests after their class they’re testing, with “Test” added as a suffix. In this example, we’re creating a test for a Model class so the test is automatically named ModelTest. This is just a convention - you can name it anything that you want.
We’ve manually added the addition() function. We can add as many functions as we want within this class. By convention, they should do something useful with that particular class.
Below, class ModelTest serves as container for our test functions. In it, we have two unit tests that will be automatically executed when we built. NOTE that you would need more than these two tests to adequately test this class - this is just an example.
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Before
import org.junit.Test
classModelTests {
lateinitvar model: Model
@Beforefunsetup() {
model = Model()
model.counter = 10 }
@TestfuncheckAddition() {
val original = model.counter
model.counter++ assertEquals(original+1, model.counter)
}
@TestfuncheckSubtraction() {
val original = model.counter
model.counter-- assertEquals(original-1, model.counter)
}
@Afterfunteardown() {
}
}
The kotlin.test package provides annotations to mark test functions, and denote how they are managed:
Annotation
Purpose
@Test
Marks a function as a test to be executed
@BeforeTest
Marks a function to be invoked before each test
@AfterTest
Marks a function to be invoked after each test
@Ignore
Mark a function to be ignored
@Test
Marks a function as a test
In our test, we call utility functions to perform assertions of how the function should successfully perform.
Function
Purpose
assertEquals
Provided value matches the actual value
assertNotEquals
The provided and actual values do not match
assertFalse
The given block returns false
assertTrue
The given block returns true
How to run tests
Tests will be run automatically with gradle build or we can execute gradle test to just execute the tests.
$ gradle test
BUILD SUCCESSFUL in 760ms
3 actionable tasks: 3 up-to-date
Gradle will report the test results, including which tests - if any - have failed.
You can also click on the arrow beside the test class or name in the editor. For example, clicking on the arrow in the gutter would run this addObserver() test.
Android has two test folders: src/test is used for tests that can run on your development machine, and src/androidTest is used for tests that need to run on an Android device (e.g. the AVD where you are testing). ↩︎
Integration Testing
Unit tests are great at verifying business logic, but it’s not enough to check that logic in a vacuum. You have to validate how different parts of it integrate with each other and external systems: the database, the message bus, and so on." – Khorikov, 2020.
A unit test is a test that meets these criteria (from the previous chapter):
Verifies a single unit of behaviour (typically a class),
Does it quickly, and
Does this in isolation from other dependencies and other tests.
An integration test is a test that fails to meet one or more of these criteria. In other words, if you determine that you need to test something outside of the scope of a unit test, it’s considered an integration test (typically because it’s integrating behaviours from multiple components). Performance tests, system tests, etc. are all kinds of integration tests.
There are a lot of different nuanced tests that are used in computer science, but we’ll focus on building generic integration tests.
Typically an integration test is one where you leave in selected dependencies so that you can test the combination of classes together. Integration tests are also suitable in cases where it is difficult to completely remove a dependency. This can happen with some critical, external dependencies like an external database.
This diagram demonstrates how unit tests primarily test the domain model, or business logic classes. Integration tests focus on the point where these business logic classes interact with external systems or dependencies.
Note that in this diagram, we’re also identifying code that we shouldn’t bother testing. Trivial code is low complexity, and typically has no dependencies or external impact so it doesn’t require extensive testing. Overcomplicated code likely has so many dependencies that it’s nearly impossible to test - and it should likely be refactored into something similer and more manageable before you attempt to add tests to it.
How many tests?
When discussing unit tests, we suggested that you should focus on core classes and their behaviours. This is reasonable for unit tests.
We can expand this to suggest that you should “check as many of the business scenario’s edge cases as possible with unit tests; use integration tests to cover one happy path, as well as any edge cases that can’t be covered by unit tests.”
A “happy path” in testing is a successful execution of some functionality. In other words, once your unit tests are done, you should write an integration tests that exercises the functionality that a customer would likely exercise if they were using your software with common features and a common workflow. Focus on that first and only add more integration tests once you have the main execution path identified and tested.
The primary purpose of integration tests is to exercise dependencies, so that should be your main goal. Your main integration test (the “happy path test”) should exercise all external dependencies (libraries, database etc). If it cannot satisfy this requirement, add more integration tests to satisfy this constraint.
Guidelines
Here’s some guidelines for creating integration tests.
Make domain model boundaries explicit.
Try to always have an explicit, well-known place for the domain model in your code base. The domain model is the collection of domain knowledge about the problem your project is meant to solve. This can be a separate class, or even a package reserved for the model.
Reduce the number of layers of abstraction
“All problems in computer science can be solved by another layer of indirection, except for the problem of too many layers of indirection.” – David J. Wheeler
Try to have as few layers of indirection as possible. In most backend systems, you can get away with just three: the domain model, application services layer (controllers), and infrastructure layer. Having excessive layers of abstraction can lead to bloated and confusing code.
Eliminate circular dependencies
Try to keep dependencies flowing in a single direction. i.e. don’t have class A call into class B, and class B call back into class A with results. Circular dependencies like this create a huge cognitive load for the person reading the code, and make testing much more difficult.
Dependencies
The idea of a dependency is central to our understanding of how to isolate a particular class or set of classes for testing.
When you are examining a software component, we say that your component may be dependent on one or more other software entities to be able to run successfully. For example, you may need to link in another library, or import other classes, or connect to another software system (like a database). Each of these represents code that affects how the code being tested will execute.
We often call the external software component or class a dependency. That word describes the relationship (classes dependent on one another), and the type of component (a dependency with respect to the original class).
A key strategy when testing is to figure out how to control these dependencies, so that you’re exercising your class independently of the influence of other components.
We can further differentiate these dependencies:
Managed vs. Unmanaged dependencies. There is a difference between those that we control directly (managed), vs. those that may be shared with other software. A managed dependency suggests that we control the state of that system.
Examples: a library that is statically linked is managed; a dynamically linked library is not. A database could be single-file and used only for your application (managed) or shared among different applications (unmanaged).
Internal vs. External dependencies. Running in the context of our process (internal) or out-of-process (external). A library is internal, a database is typically external.
Examples: A library, regardless of whether it is managed, is internal. A database, as a separate process is always external.
It is important to think about these distinctions for our dependencies because they affect what we can and cannot control during testing. For example, an unmanaged dependency means that we do not control it’s state and we may not be able to test that specific component or dependency. The best we could do it test our interface against it, since we may not be able to trust the results of any actions that we take against it.
Generally speaking, we cannot test unmanaged dependencies since we cannot control them. We also tend to be limited in our ability to test external systems, since we do not manage their state.
Test doubles
To achieve isolation in testing, we often create test doubles — classes that are meant to look like a dependent class, but that don’t actually implement all of the underlying behaviour. This lets us swap in these “fake” classes for testing.
There are five principal kinds of test doubles: Dummies, Fakes, Stubs, Spies and Mocks.
Mocks are “objects pre-programmed with expectations which form a specification of the calls they are expected to receive.” – Martin Fowler, 2006.
A mock is a fake object that holds the expected behaviour of a real object but without any genuine implementation. For example, we can have a mocked File System that would report a file as saved, but would not actually modify the underlying file system.
You can fairly easily create these mock classes yourself for code domain objects.
Several libraries have also been established to help create mocks of objects. Mockito is one of the most famous, and it can be complemented with Mockito-Kotlin. Here’s an example of a mock File that reports a path, but doesn’t actually do anything else.
The value in mocks is that they break the dependency between your method-under-test (MUT) and any external classes, by replacing the external dependency with a “fake class” with predetermined behaviour that helps you test.
The value of interfaces
One important recommendation is that you introduce interfaces for out-of-class dependencies. For instance, you will often see code like this:
This is common when testing, even in cases when that class may represent the only realization of an interface. This allows you to easily write mocks against the interface, where it’s relatively easy to determine what expected behaviour should be.
Dependency injection
The second half of this is dependency injection. This is the practice of supplying dependencies to an object in it’s argument list instead of allowing the object to create them itself. If you design your classes this way, then it’s easy to create an instance of a mock and provide that to your function, instead of allowing the function to instantiate the object itself.
Dependency injection is basically providing the objects that an object needs (its dependencies) instead of having it construct them itself. It’s a very useful technique for testing, since it allows dependencies to be mocked or stubbed out.
For instance, we might have a class that uses a database for persistence:
classPersistence {
val repo = UserRepository() // dependency
funsaveUserProfile(val user: User) {
repo.save(user)
}
}
val persist = Persistence()
persist.saveUserProfile(user) // save using the real database
Imagine that we want to test our saveUserProfile method. This would be extremely difficult to do cleanly, since we have a dependency on the repo that our Persistance class creates.
We could instead change our Persistence class so that we pass in the dependency. This allows us to control how it is created, and even replace the UserRepository() with a mock.
classPersistence(val repo: IUserRepository) {
funsaveUserProfile(val user: User) {
repo.save(user)
}
}
classMockRepo : IUserRepository {
// body with functions that mirror how the repo would work
// but no real implementation
}
val mock = MockRepo()
val persist = Persistance(mock)
persist.saveUserProfile(user) // save using the mock database
Obviously you don’t want to mock everything - that would turn this into a unit test! However, you should use this technique to isolate the classes that you are using, and test just the intended path.
Evaluation
To this point, we’ve focused on testing the functional requirements and the correctness of our results. However, there’s other criteria: the non-functional requirements or properties of our system that we defined during the requirements gathering phase.
Non-functional requirements are defined as significant, measurable characteristics of a system. They can include things like:
Processing speed: how long particular computations or functions take to execute.
Volume of data processed: images processed, calculations performed and so on.
Network performance: how long transmitting, receiving data over a network takes.
Memory consumed: peak, and average memory consumption.
Startup time: how long it takes to launch.
Verifying these requirements is only possible if we can collect the appropriate information.
In this section, we’ll talk about approaches to measuring and capturing different forms of data. Doing this often involves instrumenting your application – adding functionality directly to your source code that helps us collect this information. Unlike writing tests, which aim to existing externally to your source code, instrumenting your code can mean making direct modifications.
Gathering information
The Kotlin and JDK libraries contains a number of useful functions that can help us collect system information.
Typically you want to capture data before and after a critical operation, and then compare them to determine the cost of that operation, whatever that might be (e.g. time that it took to execute, memory consumed).
If you are measuring time, the measureTimeMillis function from the kotlin-stdlib is particularly helpful since it takes a lambda argument which is the block of code to execute.
funperformLengthyComputation() {
// a lot of processing goes on here
}
// manually calculating how long a function takes to run
val start = getTimeMillis()
performLengthyComputation()
val end = getTimeMillis()
println("The elapsed time in milliseconds is ${end-start}")
// using kotlin-stdlib
val elapsed = measureTimeMillis { performLengthyComputation() }
println("The elapsed time in milliseconds is $elapsed")
You are going to want to measure everything that you listed in your non-functional requirements.
If the functionality that you want to measure spans many classes or functions, you may need to write a top-level function to encapsulate that behaviour so that it’s easier to measure.
Milliseconds is probably the most coarse-grained you want to measure; there are also functions to measure microseconds but your tests will not be accurate to that level.
Keep in mind that your system has other things running that will consume memory, drive space and so on. It is difficult/impossible to completely isolate your application.
To gather information, it’s important that you actually use the functions that you want to measure: run the application, and interact with the features that you care about testing. You should plan on doing this multiple times, and collect data from each attempt - average the results to get a more accurate estimate the value that you are collecting, which will reduce the impact of other applications that may be running on the system and consuming resources.
Logging your data
You will want to gather the information in a text file or database, so that you can examine and analyze it later. You should record each event as a single row in your log file. Each row should contain at least the following:
Timestamp: Measure events and record time in milliseconds.
Event type: this is more useful if you use logging for debugging purposes (where you can setup INFO, WARNING, EXCEPTION categories of messages). Use a default option here e.g. INFO.
Description: Include a description of what your are logging. e.g. “time to execute bigFunction()”
Value: Capture the numeric value in question. e.g. time in milliseconds.
Kotlin does not have a built-in logging class, but there are some options:
Android has a Log class which is extremely robust and integrates with the development environment.
Desktop/JDK users can use the Java Logging API. Although it’s a Java library, its easy to use and compatible with Kotlin.
There are also third-party solutions like Log4J that are popular for commercial environments.
Here’s an example of using the Java logging class to save log information.
import java.util.logging.*
objectLoggerExample {
privateval LOGGER = Logger.getLogger(LoggerExample::class.java.getName())
@JvmStaticfunmain(args: Array<String>) {
// Set handlers for both console and log file
val consoleHandler: Handler? = ConsoleHandler()
val fileHandler: Handler? = FileHandler("./example.log")
LOGGER.addHandler(consoleHandler)
LOGGER.addHandler(fileHandler)
// Set to filter what gets logged at each destination
consoleHandler?.setLevel(Level.ALL)
fileHandler?.setLevel(Level.ALL)
LOGGER.level = Level.ALL
// Log some data
LOGGER.info("Information")
LOGGER.config("Configuration")
LOGGER.warning("Warning")
// Console handler removed
LOGGER.removeHandler(consoleHandler)
// This should still go to the log file
LOGGER.log(Level.FINE, "Finer logged without console")
}
}
This class was designed for logging large amounts of data, so it provides a great deal of flexibility in determining how data gets saved, and even allows you to send different data to different destinations. By default, this library generates output in XML. Here’s one record that was generated from the example above.
<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE log SYSTEM "logger.dtd"><log><record><date>2022-03-08T22:13:55.289391Z</date><millis>1646777635289</millis><nanos>391000</nanos><sequence>0</sequence><logger>LoggerExample</logger><level>INFO</level><class>LoggerExample</class><method>main</method><thread>1</thread><message>Information</message></record></log>
Real-time Profiling
Finally, IntelliJ IDEA and Android Studio both include real-time profiling tools that can help you detect performance issues. These are not a replacement for instrumenting your application but are extremely helpful for debugging and finding issues.
To launch the profiler in IntelliJ IDEA:
Run - Run
Run - Attach Profiler to Process…
This will launch your application. IntelliJ IDEA will collect data from your application as it’s running. Once you feel you have enough data (i.e. you’ve run through the functionality one or more times), choose to stop collecting in the UI, and display the data. You will then be able to browse memory, CPU usage and so on over the time your application was being profiled.
Examining these results is beyond what we will attempt in this course.
Deployment
The final step to getting our software into the hands of our users is software distribution. This can take many forms, depending on the environment in which the software needs to operate.
Objective
Test components and systems together
Activities
Determine how our software will be installed. Create user documentation. Package applications.
Outcome
Installers for required platforms.
Deployment means consideration of how our software will be used.
We sometimes need to install the software into a client’s environment. With complex or specialized software, we may need to install and configure out software system for the customer - often with the expectation that we will configure it to integrate with other systems e.g. we might write medical software that needs to integrate with existing patient data or billing systems.
We might sell packaged software, for distribution through a retail chain. e.g. shrink-wrapped games, or firmware updates for the latest carOS. This used to be the most common way to distribute software, but it’s less common compared to other mechanisms due to the easy availability of internet downloads.
Sometimes software is distributed from a website. This could include distributing as an installer from your company’s website e.g. Visual Studio Code, or a third-party distributor e.g. SetApp. This is extremely common, especially for OSS, free software or restricted software (e.g. distributed as a demo or with a trial license).
Finally, distribution through an online application store is very common. This is typically an attempt by a vendor to provide software distribution as a service. e.g. Steam for distributing games; the Apple App Store or Google Play Store for mobile and desktop applications. Online app stores represent by-far the largest concentration of apps for distribution, but they are somewhat restricted by platform/genre.
Each of these deployment mechanisms will require software to be prepared in a different way. e.g. selling packaged software will require a DVD image that includes digital assets. Selling from an online application store required code signing, going through a code review process with the company that runs the store, and so on.
There are a number of issues to sort out that apply to each of these situations:
Legal: how do we protect ourselves from unauthorized use, or further redistribution of our software?
Usability: how do we ensure that users know how to use our software? do we provide documentation or training?
Logistical: how do we package our software so that we can distribute and install it successfully? how do we enforce licensing?
See the sections below for each step that you wish to consider.
Release Process
What do we need to do for a software release?
Update our internal project tracking systems
Close the sprint
Update and close issues
Update our internal documentation e.g. wiki
Prepare additional materials for the release
Release notes
User documentation
Update build scripts/configuration
Build installers
Versioning
You should version your software, so that every release has a release number and date associated with it.
The standard convention is a triple, separated by decimals, of the format: major.minor.build. For example, 1.2.3 would be major version 1, minor version 2, build 3.
Major signifies a major product release. This is somewhat arbitrary, but typically is released infrequently and includes major features changes or additions. If you charge by release, you would typically charge for every new major version. You might release a new major version as frequently as once per year, or as infrequently as once very few years.
Minor indicates a minor product release, typically a combination of new minor features, and bug or compatibility fixes. You might release a minor version a few times per year and users would not ordinarily expect to pay for these.
Build number is internal build number within a minor release. This is intended to reflect bug fixes only; you typically iterate over builds internally and release the final successful version publically.
Release Chart
Subsections of Deployment
Copyright & Licensing
Copyright
Before distributing your software, in any form, it’s critical to establish ownership and rights pertaining to that software.
A copyright is a type of intellectual property that gives its owner the exclusive right to copy and distribute a creative work, usually for a limited time. Copyright is intended to protect the original expression of an idea in the form of a creative work, but not the idea itself. A copyright is subject to limitations based on public interest considerations, such as the fair use doctrine.
Practically, this means that different companies can independently produce software that solves the same problem, and there is no law preventing that from occurring, provided that they do not reuse actual source or object code from their competitor.
How do I assert copyright?
In Canada, software is protected under the Copyright Act of Canada. Copyright is acquired automatically when an original work is generated; the creator is not required to register or mark the work with the copyright symbol in order to be protected. The rights holder is granted: the exclusive right of reproduction, the right to rent the software, the right to restrain others from renting the software and the right to assign or license the copyright to others.
It’s common practice to assert your copyright claim in the header of your software source files. Although is not required to assert copyright, it’s a flag for potential violators, and it might make it easier to defend in court.
Copyright (c) 2022. Jeff Avery.
Licensing
As the rights holder, you can grant others rights with respect to your software. A software license is a legal instrument that grants the licensee (i.e. an end-user) permission to use the software in a manner dictated by the license. These rights could include (but aren’t limited to) the right to install and use it, the right to modify the source code, or the right to redistribute the software with or without changes.
Authors of copyrighted software can also choose to donate their software to the public domain, in which case it is also not covered by copyright and, as a result, cannot be licensed.
A permissive software license, sometimes also called BSD-style is a free-software license which instead of copyleft protections, carries only minimal restrictions on how the software can be used, modified, and redistributed, usually including a warranty disclaimer. Examples include the GNU All-permissive License, MIT License, BSD licenses, Apple Public Source License and Apache license. As of 2016, the most popular free-software license is the permissive MIT license.
Copyleft is the practice of granting the right to freely distribute and modify intellectual property with the requirement that the same rights be preserved in derivative works created from that property.
Copyleft software licenses are considered protective in contrast with permissive free software licenses, and require that information necessary for reproducing and modifying the work must be made available to recipients of the software program. This information is most commonly in the form of source code files, which usually contain a copy of the license terms and acknowledge the authors of the code. Notable copyleft licenses include the GNU General Public License (GPL), originally written by Richard Stallman, and the Creative Commonsshare-alike license.
Non-commercial licenses are intended to be used only be entities with no profit motive, including charities and public institutions. This is not common, and is both difficult to interpret and enforce.
A proprietary license refers to any other license, but usually one which grants few to no rights to the end-user (e.g. most commercial software licenses, which do not grant you any rights apart from the ability to install and use on a single machine).
It’s standard practice to include your license agreement with your product, typically as a license.txt file or something similar in your distribution.
How do I apply a license?
Distribute the license with your program.
Include a license.txt file in your distribution.
If you provide source code to anyone, include a statement about how it is licensed in the header of each file. Check the license to see what is required.
Include a licensing statement on your website.
See terms of each license to check what’s suitable. e.g.
Unless explicitly stated otherwise all files in this repository are licensed under the Apache Software License 2.0 [insert boilerplate notice here]
Of course, you can always write your own software licensing agreement, but for it to be enforcable you will need a lot of assistance from some highly specialized lawyers, who tend to be very expensive, and who charge by the hour… ↩︎
Documentation
If your product is complex, or if you want a way to showcase your features, you might consider producing user documentation. This comes in many different forms:
Getting started guide (web page, PDF): the basics to use your software
Tutorials (web pages): walk through simple tasks with examples
User Guide: comprehensive documentation
Documentation can be printed and bundled (uncommon), distributed with your software (PDF, other formats, more common) or hosted on your website (very common).
Keep in mind that this all takes effort. Tailor your documentation to the complexity of your product (and the likelihood of users actually taking the time to read it!)
Release notes
Every public release should include release notes: a list of changes that you have made to your product for that particular release. Depending on the nature of your product, and your relationship with your users, the details can be fairly general (“added support for Windows 11”) or incredibly detailed (“fixed bug XXX”).
The release notes are also a good place to put things like
OS or hardware compatibility details
copyright and license information
information on how users can contact you or get help
Release notes are typically released in one of the following ways:
a readme.txt file included in your distribution.
a popup window in your application that includes these details.
a page on your website (if you do this, put a way to open that page from within your application).
Suggestions
A discussion of how to write effective documentation is beyond the scope of this course, but we make some broad suggestions:
Consider Markdown as an authoring format. You can then convert markdown to PDF, EPUB, or HTML for publishing using pandoc. This site, for example, is authored in Markdown and then converted to a website using Hugo.
If you wish to create developer documentation (including class diagrams and so on), there are specialized authoring tools. Dokka can generate Javadoc, HTML or Markdown documentation from Kotlin code (much as Javadoc does for Java code).
Finally, IntelliJ IDEA can generate class diagrams from source code. Right-click on the project file, then select Diagrams from the popup menu.
Release Process
What do we need to do for a software release?
Update our internal project tracking systems
Close the sprint
Update and close issues
Update our internal documentation e.g. wiki
Prepare additional materials for the release
Release notes
User documentation
Update build scripts/configuration
Build installers
Versioning
You should version your software, so that every release has a release number and date associated with it.
The standard convention is a triple, separated by decimals, of the format: major.minor.build. For example, 1.2.3 would be major version 1, minor version 2, build 3.
Major signifies a major product release. This is somewhat arbitrary, but typically is released infrequently and includes major features changes or additions. If you charge by release, you would typically charge for every new major version. You might release a new major version as frequently as once per year, or as infrequently as once very few years.
Minor indicates a minor product release, typically a combination of new minor features, and bug or compatibility fixes. You might release a minor version a few times per year and users would not ordinarily expect to pay for these.
Build number is internal build number within a minor release. This is intended to reflect bug fixes only; you typically iterate over builds internally and release the final successful version publically.
Release Chart
Chapter 4
Getting Started
How to get started with the tools and infrastructure that you will need in-place to work on your project.
Subsections of Getting Started
Introduction
The information below should assist in setting up your computer to run samples from class, and allow you to work on your project. Any modern computer should work reasonably well, as long as you have admin rights to install and configure software.
Please install these tools in the following order.
We use Git for version control, so you will need Git installed to perform operations on your source code.
Git is pre-installed on macOS and Linux; Windows users can install it from https://git-scm.org. Once it’s installed, you may need to update your path to include the Git installation directory. You can check your installation using the git version command.
❯ git version
git version 2.37.1 (Apple Git-137.1)
Java JDK
We will be building Kotlin applications and services, and using the Java JDK as our target. To ensure that you get the correct version, you should:
Download and install the version above from Azul or OpenJDK (again, make sure to match your system architecture).
Add JAVA_HOME to your system’s environment variables, pointing to the installation.
Update your path to include the directory containing the Java executables.
For example, I have the following lines in my .zshrc:
You can check the installed version by opening IntelliJ IDEA and looking at the IntelliJ IDEA - About dialog.
Kotlin
We will need a Kotlin compiler, and the Kotlin standard libraries.
IntelliJ IDEA includes a Kotlin plugin, so if you have installed IntelliJ IDEA and you are working in the IDE, then you do not need to install Kotlin.
However, if you wish to compile from the command-line, or use a different editor, then you will need to install Kotlin manually. It can be installed from https://www.kotlinlang.org or from most package managers (e.g. brew install kotlin if you are a Mac user with Homebrew installed).
If you install the command-line version, you can check your installation using the kotlin -version command.
Kotlin 1.8 only supports up to Gradle 7.3. Use this version of Gradle in your build files to ensure compatibility.
Using Git
A Version Control Systems (VCS) is a software system designed to track changes to source code. It is meant to provide a canonical version of your project’s code and and other assets, and ensure that only desireable (and tested) changes are pushed into production. Common VCS systems include Mercurial (hg), Subversion (SVN), Perforce, and Microsoft Team Foundation Server. We’ll be using Git, a very popular VCS, in this course.
All VCS’s, including Git, let you take a snapshot of your source code at any given point in time. This example shows a project that starts with a single index.html file, adds about.html at a later time, and then finally makes some edits. The VCS tracks these changes, and provides functionality that we’ll discuss below.
History: a VCS provides a long-term history of every file1. This includes tracking when files were added, or deleted, and every change that you’ve made. Changes are grouped together, so you can look at (for instance) the set changes that introduced a feature.
Versions: the ability to version your code, and compare different versions. Did you break something? You can always unwind back to the “last good” change that was saved, or ever compare your current code with the previously working version to identify an issue.
Collaboration: a VCS provides the necessary capabilities for multiple people to work on the same code simultaneously, while keeping their changes isolated. You can create branches where your changes are separate from other ongoing changes, and the VCS can help you merge changes together once they’re tested.
Installing Git
Git binaries can be installed from the Git home page or through a package manager (e.g. Homebrew on Mac). Although there are graphical clients that you can install, Git is primarily a command-line tool. Commands are of the form: git <command>.
You’ll also want to make sure that the git executable (git or git.exe) is in your path.
Concepts
Version control is modeled around the concept of a changeset: a grouping of files that together represent a change to the system (e.g. a feature that you’ve implemented may impact multple source files). A VCS is designed to track changes to sets of files.
Git is designed around these core concepts:
Repository: The location of the canonical version of your source code.
Working Directory: A copy of your repository, where you will make your changes before saving them in the repository.
Staging Area: A logical collection of changes from the working directory that you want to collect and work on together (e.g. it might be a feature that resulted in changes to multiple files that you want to save as a single change).
A repository can be local or remote:
A local repository is where you might store projects that you don’t need to share with anyone else (e.g. these notes are in a local git repository on my computer).
A remote repository is setup on a central server, where multiple users can access it (e.g. GitLab, GitHub effectively do this, by offering free hosting for remote repositories).
Git works by operating on a set of files (aka changeset): we git add files in the working directory to add them to the change set; we git commit to save the changeset to the local repository. We use git push and git pull to keep the local and remote repositories synchronized.
To create a local repository that will not need to be shared:
Create a repository. Create a directory, and then use the git init command to initialize it. This will create a hidden .git directory (where Git stores information about the repository).
$ mkdir repo
$ cd repo
$ git init
Initialized empty Git repository in ./repo/.git/
$ ls -a
. .. .git
Make any changes that you want to your repository. You can add or remove files, or make change to existing files.
$ vim file1.txt
ls -a
. .. .git file1.txt
Stage the changes that you want to keep. Use the git add command to indicate which files or changes you wish to keep. This adds them to the “staging area”. git status will show you what changes you have pending.
$ git add file1.txt
$ git status
On branch master
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)new file: file1.txt
Commit your staging area. git commit assigns a version number to these changes, and stores them in your local repository as a single changeset. The -m argument lets you specify a commit message. If you don’t provide one here, your editor will open so that you can type in a commit message. Commit messages are mandatory, and should describe the purpose of this change.
$ git commit -m "Added a new file"Remote Workflow
Remote Workflow
A remote workflow is almost the same, except that you start by making a local copy of a repository from a remote system.
Clone a remote repository. This creates a new local repository which is a copy of a remote repository. It also establishes a link between them so that you can manually push new changes to the remote repo, or pull new changes that someone else has placed there.
# create a copy of the CS 346 public repository$ git clone https://git.uwaterloo.ca/j2avery/cs346.git ./cs346
Making changes and saving/committing them is the same as the local workflow (above).
Push to a remote repository to save any local changes to the remote system.
$ git push
Pull from remote repository to get a copy of any changes that someone else may have saved remotely since you last checked.
$ git pull
Status will show you the status of your repository; log will show you a history of changes.
# status when local and remote repositories are in sync$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
# condensed history of a sample repository$ git log --oneline
b750c10 (HEAD -> master, origin/master, origin/HEAD) Update readme.md
fcc065c Deleted unused jar file
d12a838 Added readme
5106558 Added gitignore
Branching
The biggest challenge when working with multiple people on the same code is that you all may want to make changes to the code at the same time. Git is designed to simplify this process.
Git uses branches to isolate changes from one another. You think of your source code as a tree, with one main trunk. By default, everyone in git is working from the “trunk”, typically named master or main (you can see this when we used git status above).
A branch is a fork in the tree, where we “split off” work and diverge from one of the commits (typically we split from a point where everything is working as expected)! Once we have our feature implemented and tested, we can merge our changes back into the master branch.
Notice that there is nothing preventing multiple users from doing this. Because we only merge changes back into master when they’re tested, the trunk should be relatively stable code2.
We have a lot of branching commands:
$ git status // see the current branch
On branch master
$ git branch test // create a branch named test
Created branch test
$ git checkout test // switch to it
Switched to a new branch 'test'$ git checkout master //switch back to master
Switched to branch 'master'$ git branch -d test // delete branch
Deleted branch test (was 09e1947).
When you branch, you inherit changes from your starting branch. Any change that you make on that branch are isolated until you choose to merge them.
A typical workflow for adding a feature would be:
Create a feature branch for that feature.
Make changed on your branch only. Test everything.
Code review it with the team.
Switch back to master and git merge from your feature branch to the master branch. If there are no conflicts with other change on the master branch, your changes will be automatically merged by git. If your changed conflict (e.g multiple people changed the same file and are trying to merge all changed) then git may ask you to manually merge them.
$ git checkout -b test // create branch
Switched to a new branch 'test'$ vim file1.md // make some changes
$ git add file1.md
$ git commit -m "Committing changed to file1.md"$ git checkout master // switch to master
$ git merge test // merge changes from test
Updating 09e1947..ebb5838
Fast-forward
file1.md | 136 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 118 insertions(+), 18 deletions(-)$ git branch -d test // remove branch (optional)Deleted branch test (was ebb5838).
Merging Code
This is a trivial case, demonstrating a merge that happens very soon after the branch was created. However, it’s more likely that branches will be created and worked on for a long period of time before you merge back to main.
When you merge, Git examines your copy of each file, and attempts to apply any other changes that may have been committed to main since you created the branch. (If there’s multiple people working on the project, it’s not unusual for multiple changes to be made to the same file). In many cases, as long as there are no conflicts, Git will merge the changes together. However, if Git is unable to do so (e.g. you and a colleague both changed the same file and your changes overlap), then you will be prompted to manually merge the changes together.
When this happens, Git will apply both changes to the source file, and add inline comments. You have to manually fix the file, and then commit the change before attempting to merge again.
Pull Requests (PRs)
One way to avoid merge issues is to review changes before they are merged into main (this also lets you review the code, manually run tests etc). The standard mechanism for this is a Pull Request (PR). A PR is simply a request to another developer (possibly the person responsible for maintaining the main branch) to git pull your feature branch and review it before merging.
We will not force PRs in this course, but you might find them useful within your team.
These are suggestions for working with Git effectively.
Work iteratively. Learn to solve a problem in small steps: define the interface, write tests against that interface, and get the smallest functionality tested and working.
Commit often! Once you have something work (even partly working) commit it! This gives you the freedom to experiment and always revert back to a known-good version.
Branch as needed. Think of a branch as an efficient way to go down an alternate path with your code. Need to make a major change and not sure how it will work out? Branch and work on it without impacting your main branch.
Store your projects in private, online repositories. Keep them private so that you don’t share them unless it’s appropriate. Being online provides a remote backup and makes it easy to add someone to your project later.
Versioning is useful for more than just source code. These course notes, for instance, are in a git repo, along with source code, image and other assets. ↩︎
Unless your changes conflict, but that’s why we do integration testing! ↩︎
IntelliJ IDEA
An Integrated Development Environment (IDE) is custom software designed to support all aspects of software development. It typically includes a code editor, debugging, testing and profiling tools, and anything else that might be needed to support your workflow.
While you certainly can use command-line tools to build applications, it’s strongly encouraged to use an IDE which provides a large number of advanced features (e.g. debugging, profiling support, auto-completion in code, integration with other systems and so on).
In this course, we’re going to use IntelliJ IDEA, an IDE produces by JetBrains (the company that invented Kotlin), which provides all development functionality, and integrates with all of our tools.
Anything we can do to reduce the friction of common software activities is worth pursuing. This is why we like Integrated Development Environment (IDE)s like IntelliJ – they supports a range of common activities:
Producing new code: tools for navigating existing classes, maybe filling in code snippets for us.
Reading existing code: browsing classes, maybe diagramming the system to allow us to build a mental model of an existing system.
Refactoring: the ability to produce a series of possibly sweeping changes without breaking existing code. e.g. renaming a method, and everywhere it’s invoked; extracing an interface from an existing set of classes.
Debugging: visualizations and other tools designed to help diagnose.
Profiling: tools to help us understand performance and runtime behaviour.
The following section outlines some features that you should investigate. In the examples below, you can generally copy-paste the code into a main() method in IntelliJ and execute it.
There are two versions of IntelliJ IDEA: the Community Edition, which is open source, and the Ultimate Edition which is more capable. JetBrains offers free educational licenses to students, which includes a license for IntelliJ IDEA Ultimate if you wish. Either one will work for this course.
Make sure to install the version appropriate for your system architecture (e.g. x86 for an Intel processor, or ARM for Apple M1/M2 processors).
IntelliJ will attempt to use the Java JDK that you installed previously. If intelliJ is unable to locate a suitable JDK, it may complain that it cannot locate a Project JDK (i.e. Java JDK). To fix this, click on Setup SDK and enter details for the JDK that you installed (i.e. where it is installed).
Creating a Project
IntelliJ fully supports Gradle, and we can create a new project directly in the IDE.
From the Splash screen, select Create New Project.
You will need to supply project parameters.
Kotlin as your programming language.,
Gradle for your build system.
Gradle can use either Groovy or Kotlin as a DSL.
Either is fine, though course examples are mostly in Groovy (they were created before Kotlin was widely supported).
JDK should point to your installation.
If successful, IntelliJ IDEA will open into the main window to an empty project.
After the project has loaded, use View -> Tools -> Gradle to show the Gradle tasks window, and use build -> build and application -> run to Build and Run respectively.
Navigating Projects
IntelliJ has a number of windows that it will display by default. You can use Cmd-Hotkey (or Ctrl-Hotkey) to navigate these windows[^3].
Project: a list of all files (CMD-1).
Structure: methods and properties of the current open class/source file (CMD-7).
Source: the current source files (no hotkey).
Git: Git status and log (CMD-9) - not shown.
Gradle: tasks that are available to run (no hotkey) - not shown.
Helps you complete the names of classes, methods, fields, and keywords within the visibility scope.
⌃ Space
Running Sample Code
We maintain a public Git repository of the source code shown in lectures. To get a copy, git clone the repository URL. This command, for instance, would create a working copy of the sample code in a directory named cs346.
You can build and execute these projects directly in IntelliJ:
File -> Open and navigate to the top-level directory containing the build.gradle file. Do NOT open a specific file, just the directory. Click Ok.
After the project has loaded, use View -> Tools -> Gradle to show the Gradle tasks window, and use build -> build and application -> run to Build and Run respectively.
Reading and Understanding Code
IntelliJ can generate UML diagrams from existing code. These diagrams will reflect the structure of actual classes and methods in your application.
The documentation contains a section on source code navigation that is worth reading carefully.
To navigate backwards, press ⌘ [. To navigate forward, press ⌘ ].
To navigate to the last edited location, press ⇧ ⌘ ⌫.
To move caret between matching code block braces, press ⌃ M.
To move the caret to the next word or the previous word, press ⌥ → or ⌥ ←.
There are also built-in dialogs that help you navigate through existing code.
Method hierarchies show subclasses where the method overrides the selected one as well as superclasses or interfaces where the selected method gets overridden.
⇧ ⌘ H
Gradle Builds
Introduction
When writing complex applications, there is potentially a large list of steps that need to be completed before we can deploy our software. We might need to:
Download and import new versions of libraries that we’re using.
Run a code analysis tool against your source code to check for suspicious code, formatting etc.
Run a documentation tool to generate revised documentation.
Build a directory structure containing images, fonts and other resources for our executable to use.
Compile the code and run automated tests to ensure that its working correctly.
Create an installer that you can use to deploy everything.
Performing these steps manually is error prone, and very time-consuming. Instead of doing this by-hand, we tend to rely on build systems: software that is used to build other software. Build systems provide consistency in how software is built, and let you automate much of the process. They addresses issues like:
How do I make sure that all of my steps (above) are being handled properly?
How do I ensure that everyone is building software the same way i.e. compiling with the same options?
How do I know that I have the correct library versions?
How do I ensure that tests are being run before changes are committed?
There are a number of build systems on the market that attempt to address these problems. They are often programming-language or toolchain dependent.
C++: CMake, Scons, Premake
Java: Ant, Maven, Gradle
Info
Make is one of the most widely used build systems, which allows you to script your builds (by creating a makefile to describe how to build your project). Using make, you can ensure that the same steps are taken every time your software is built.
For small or relatively simple projects, make is a perfectly reasonable choice. It’s easy to setup, and is pre-installed on many systems. However, make has limitations and may not be the best choice for large or more complex projects.
Build dependencies must be explicitly defined. Libraries must be present on the build machine, manually maintained, and explicitly defined in your makefile.
Make is fragile and tied to the underlying environment of the build machine.
Performance is poor. Make doesn’t scale well to large projects.
Its language isn’t very expressive, and has a number of inconsistencies.
It’s very difficult to fully automate and integrate with other systems.
We’re going to use Gradle in this course:
It handles all of our requirements (which is frankly, pretty impressive).
It’s the official build tool for Android builds, so you will need it for Android applications.
It fits nicely into the Kotlin and JVM ecosystem.
It’s cross-platform and language agnostic.
You write Gradle build scripts in a DSL (Groovy or Kotlin). You describe tasks, and Gradle figures out how to perform them. Gradle handles dependency management and manages complex dependencies automatically!
Gradle Tasks
Gradle works by running tasks - some are built-in, and you can define your own. Gradle tasks can be executed from the command-line. e.g.
gradle help: shows available commands
gradle init: create a new project and dir structure.
gradle tasks: shows available tasks from build.gradle.
gradle build: build project into build/
gradle run: run from build/
$ gradle help
> Task :help
Welcome to Gradle 6.4.1.
To run a build, run gradle
$ gradle build
Starting a Gradle Daemon ...
BUILD PASSED in 2s
Creating a Project
A Gradle project is simply a set of source files, resources and configuration files structured so that Gradle can build it.
Gradle projects require a very specific directory structure. A typical Gradle project directory looks like this:
We could create this by hand, but for now let’s use Gradle to create a starting directory structure and build configuration file that we can modify.
gradle init will run the project wizard to create a new project in the current directory. Select application , Kotlin for a language, and one application project for this sample.
$ gradle init
Select type of project to generate:
1: basic
2: application
3: library
4: Gradle plugin
Enter selection (default: basic)[1..4]2Select implementation language:
1: C++
2: Groovy
3: Java
4: Kotlin
5: Scala
6: Swift
Enter selection (default: Java)[1..6]4Split functionality across multiple subprojects?:
1: no - only one application project
2: yes - application and library projects
Enter selection (default: no - only one application project)[1..2]1Select build script DSL:
1: Groovy
2: Kotlin
Enter selection (default: Kotlin)[1..2]1Generate build using new APIs and behavior (some features may change in the next minor release)? (default: no)[yes, no]Project name (default: single-project):
Source package (default: single.project):
> Task :init
Get more help with your project: https://docs.gradle.org/7.6/samples/sample_building_kotlin_applications.html
BUILD SUCCESSFUL in 16s
2 actionable tasks: 2 executed
app is the application source code folder for our project. app/src is source code, and app/test is for unit tests.
gradle is the gradle wrapper files, which allows gradle to bootstrap itself if required. gradlew, gradlew.bat are Gradle scripts that you should use to run commands.
settings.gradle and build.gradle are configuration files.
You can use gradle tasks to see all supported actions. The available tasks will vary based on the type of project you create.
$ gradle tasks
> Task :tasks
------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------
Application tasks
-----------------
run - Runs this project as a JVM application
Build tasks
-----------
assemble - Assembles the outputs of this project.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
classes - Assembles main classes.
clean - Deletes the build directory.
jar - Assembles a jar archive containing the main classes.
A “standard” Gradle project has about 30 tasks. Many of them are called infrequently, or called by other tasks (e.g. build calling buildNeeded). The most commonly used commands are build, run and clean.
The settings.gradle file contains basic project settings. It specifies the project name, and the directory containing our project source code.
rootProject.name ='single-project'include('app')
The build.gradle file contains our project configuration. Gradle supports either groovy or kotlin as build scripts.
#
```kotlin
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
kotlin("jvm") version "1.6.20"
application
}
group = "net.codebot"
version = "1.0.0"
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
repositories {
mavenCentral()
}
dependencies {
testImplementation(kotlin("test"))
}
tasks.test {
useJUnitPlatform()
}
tasks.withType {
kotlinOptions.jvmTarget = "1.8"
}
application {
mainClass.set("single.project.AppKt")
}
```
#
```groovy
plugins {
// Apply the org.jetbrains.kotlin.jvm Plugin to add support for Kotlin.
id 'org.jetbrains.kotlin.jvm' version '1.7.10'
// Apply the application plugin to add support for building a CLI application in Java.
id 'application'
}
repositories {
// Use Maven Central for resolving dependencies.
mavenCentral()
}
dependencies {
// Use the Kotlin JUnit 5 integration.
testImplementation 'org.jetbrains.kotlin:kotlin-test-junit5'
// Use the JUnit 5 integration.
testImplementation 'org.junit.jupiter:junit-jupiter-engine:5.9.1'
// This dependency is used by the application.
implementation 'com.google.guava:guava:31.1-jre'
}
application {
// Define the main class for the application.
mainClass = 'single.project.AppKt'
}
tasks.named('test') {
// Use JUnit Platform for unit tests.
useJUnitPlatform()
}
```
The build.gradle file contains information about your project, including the versions of all external libraries that you require. In this project file, you define how your project should be built:
You define the versions of each tool that Gradle will use e.g. compiler version. This ensures that your toolchain is consistent.
You define versions of each dependency e.g. library that your build requires. During the build, Gradle downloads and caches those libraries. This ensures that your dependencies remain consistent.
Gradle has a wrapper around itself: gradlew and gradlew.bat You define the version of the build tools that you want to use, and when you run Gradle commands using the wrapper script, it will download and use the correct version of Gradle. This ensures that your build tools are consistent.
Here’s how we run using the wrapper.
$ ./gradlew run
Downloading https://services.gradle.org/distributions/gradle-7.6-bin.zip
...........10%............20%...........30%............40%............50%...........60%............70%............80%...........90%............100%
> Task :app:run
Hello World!
BUILD SUCCESSFUL in 14s
2 actionable tasks: 2 executed
Example: Console
Let’s setup the build for a calculator application.
package calc
funmain(args: Array<String>) {
try {
println(Calc().calculate(args))
} catch (e: Exception ) {
print("Usage: number [+|-|*|/] number")
}
}
classCalc() {
funcalculate(args:Array<String>):Any {
if (args.size !=3) throw Exception("Invalid number of arguments")
val op1:String = args.get(0)
val operation:String = args.get(1)
val op2:String = args.get(2)
return(
when(operation) {
"+"-> op1.toInt() + op2.toInt()
"-"-> op1.toInt() - op2.toInt()
"*"-> op1.toInt() * op2.toInt()
"/"-> op1.toInt() / op2.toInt()
else->"Unknown operator" }
)
}
}
Let’s migrate this code into a Gradle project.
Use Gradle to create the directory structure. Select “application” as the project type, and “Kotlin” as the language.
$ gradle init
Select type of project to generate:
1: basic
2: application
Copy the calc.kt file into src/main, and modify the build.gradle file to point to that source file.
application {
// Main class for the application.
// Kotlin generates a wrapper class for our main method
mainClassName = 'calc.CalcKt'}
Use gradle to make sure that it builds.
$ gradle build
BUILD SUCCESSFUL in975ms
If you use gradle run, you will see some unhelpful output:
$ gradle run
> Task :run
Usage: number [+|-|*|/] number
We need to pass arguments to the executable, which we can do with –args.
$ gradle run --args="2 + 3"> Task :run
5
Multi-Project Setup
This configuration works well with a single program, but often you want to built related projects together.
e.g.
console
graphical client
shared components
service
Gradle supports multi-project configurations, so that you can track and manage sub-projects together.
You can add an extra project to the single-project above by adding a second project directory, and then modifying the settings.gradle to include the new project.
For example, here we have added a server project directory and then added it to the settings.gradle file.
This gives us the ability to build both client AND server from the same project.
If you’re creating a new project, you can instead choose to run gradle init and select multiple-projects from the wizard. This will generate a multi-project setup with a client, server and shared libraries.
Split functionality across multiple subprojects?:
1: no - only one application project
2: yes - application and library projects
Enter selection (default: no - only one application project)[1..2]2
Managing Dependencies
This works well if your projects are completely independent, but often you will have shared code that you want to share between projects. We call this relationship a project dependency. In this case, it’s an internal dependency, since we’re responsible for producing all relevant classes ourselves (within our organization).
Project dependencies
To add a shared library that can be used by both our client and server projects, you need to:
Create a new shared project.
Add it to the top-level settings.gradle file.
Add this shared project to the build.gradle.kts file for any project that requires it.
If we modify our project from above, we now have app/, server/ and shared/ projects:
To include the shared project in the client and server projects, we modify the app/build.gradle.kts and server/build.gradle.kts to include this dependency:
It would be unusual for us to write all of the code in our application. Typically, you’re leveraging software that was written by someone else e.g. OpenGL for graphics, Kotlin standard library for everything in this course. We refer to this relationship between our source code and these external libraries as an external dependency.
The challenge with using libraries like this is ensuring that you are building, testing and deploying against a consistent version of the same libraries.
Traditionally, software was distributed in release archives, or tarballs, by the people that maintained these libraries. Their users would then download this code and compile it into tools or add it as a library to their own applications. This is extremely error prone (“what version did I test with again?”). The modern way to manage dependencies is using a package manager: a system used to manage dependencies and required libraries e.g. npm,pip, go mod, maven, gradle,apt-get,brew, etc.
All package managers work roughly the same at a high level:
A user asks to install a package or set of packages (with specific versions of each one)
The package manager performs some basic dependency resolution
The package manager calculates the full set of transitive dependencies, including version conflict resolution
The package manager installs them, often from a remote repository.
In this course, we’ll use Gradle for both building software and managing dependencies. Gradle can download specific versions of libraries for us, from an online *repository: a location where libraries are stored and made available. Typically a repository will offer a large collection of libraries, and include many years of releases, so that a package manager is able to request through some public interface, a specific version of a library and all of its dependencies.
Repositories can be local (e.g. a large company can maintain its own repository of internal or external libraries), or external (e.g. a collection of public libraries). The most popular Java/Kotlin repository is mavenCentral, and we’ll use it with Gradle to import any external dependencies that we might require.
You can control the repository that Gradle uses by specifying its location in the build.gradle file.
repositories { jcenter() mavenCentral()}
You add a specific library or dependency by adding it into the dependencies section of the build.gradle file.
The details include how to import it into your project.
Managing Dependencies
One challenge with setting up dependencies in multi-project builds is that you will have multiple build.gradle.kts files, each with their own list of dependencies and versions.
It’s important to keep your versions consistent across projects. How do we do this?
Version Catalogs
A Version Catalog is a list of versions, plugins and dependencies that we are using in our application. We can extract them from our build.gradle.kts files, and store them in a single location to avoid duplication.
In our settings.gradle.kts, add a Version Catalog like this:
application, console and shared all represent projects, with their own configuration files.
Info
You can find the source code for this project in the GitLab Public Repository for this course, under sample-code/project-template.
Top-Level (Root)
The project root contains the settings.gradle.kts file, which defines the overall project structure.
// the name of the project
rootProject.name = "multi-project"// which projects to include
include("application", "console", "shared")
// a way of tracking version numbers globally
// this ensures that we use the same version of libraries in each project
// not every project will use all of these plugins or libraries
// see individual project configuration files for the actual usage
dependencyResolutionManagement {
versionCatalogs {
create("libs") {
// constants
version("jdk", "17")
version("javafx", "18.0.2")
// https://plugins.gradle.org/
plugin("kotlin-lang", "org.jetbrains.kotlin.jvm").version("1.8.10")
plugin("jlink", "org.beryx.jlink").version("2.26.0")
plugin("javafx", "org.openjfx.javafxplugin").version("0.0.13")
plugin("javamodularity", "org.javamodularity.moduleplugin").version("1.8.12")
// https://mvnrepository.com/
library("kotlin-coroutines", "org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4")
library("sqlite", "org.xerial:sqlite-jdbc:3.40.1.0")
library("exposed-core", "org.jetbrains.exposed:exposed-core:0.40.1")
library("exposed-dao", "org.jetbrains.exposed:exposed-dao:0.40.1")
library("exposed-jdbc", "org.jetbrains.exposed:exposed-jdbc:0.40.1")
library("junit-jupiter", "org.junit.jupiter:junit-jupiter:5.9.2")
library("sl4j-api", "org.slf4j:slf4j-api:2.0.6")
library("sl4j-simple", "org.slf4j:slf4j-simple:2.0.6")
}
}
}
It also contains gradle.properties, which includes project definitions.
kotlin.code.style=official
Gradle wrapper settings
There is a top-level gradle directory, containing the Gradle bootstrap files (the means by which Gradle downloads and installs itself when you run gradlew). These files are auto-generated by Gradle when you setup the project.
You might want to update the gradle-wrapper.properties to point to a recent version fo Gradle by changing the distributionURL line. In this example, we’re specifying Gradle 8.0.2.
The Application project contains a single src folder, containing the directory tree, and a single build.gradle.kts file which includes the configuration for this specific project. This is a JavaFX application, so we should expect to see application-style plugins and dependencies.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
// notice the syntax for the plugin section uses alias
// this is how we pull in the plugins from the Version Catalog (above)
// e.g. libs.plugins.kotlin.lang inserts the kotlin-lang plugin details.
@Suppress("DSL_SCOPE_VIOLATION")
plugins {
application
alias(libs.plugins.kotlin.lang)
alias(libs.plugins.javamodularity)
alias(libs.plugins.javafx)
alias(libs.plugins.jlink)
}
// used for packaging only
group = "net.codebot"version = "1.0-SNAPSHOT"// telling Gradle to put Java and Kotlin output in the same build structure
// required since we have Java files (module-info.java) and Kotlin source
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
// pull all dependencies from here
repositories {
mavenCentral()
}
// libraries that we need, using versions from Version Catalog
// we also want to use the shared/ library, so we need to include it here
dependencies {
implementation(project(":shared"))
implementation(libs.kotlin.coroutines)
testImplementation(libs.junit.jupiter)
}
// fancy way of saying "use JUnit 5"
tasks.test {
useJUnitPlatform()
}
// tell Gradle to use a specific version of the JDK when compiling
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(libs.versions.jdk.get()))
}
}
// application plugin settings
// module name that we've set in the module-info.java
// fully-qualified classname to excecute when you use "gradle run"
application {
mainModule.set("net.codebot.application")
mainClass.set("net.codebot.application.Main")
}
// JavaFX plugin settings
// tell the JavaFX plugin which modules to include
// you may need to add others e.g. javafx.web
javafx {
version = libs.versions.javafx.get()
modules = listOf("javafx.controls", "javafx.graphics")
}
// get around small output bug
// https://stackoverflow.com/questions/74453018/jlink-package-kotlin-in-both-merged-module-and-kotlin-stdlib
jlink {
forceMerge("kotlin")
}
Console Application
The Console application has a similar structure to the Application project, with a single src folder and a build.gradle.kts file for this project. This is a console application so it doesn’t need JavaFX support.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
// notice the syntax for the plugin section uses alias
// this is how we pull in the plugins from the Version Catalog (above)
// e.g. libs.plugins.kotlin.lang inserts the kotlin-lang plugin details.
@Suppress("DSL_SCOPE_VIOLATION")
plugins {
application
alias(libs.plugins.kotlin.lang)
alias(libs.plugins.javamodularity)
}
// used for packaging only
group = "net.codebot"version = "1.0.0"// telling Gradle to put Java and Kotlin output in the same build structure
// required since we have Java files (module-info.java) and Kotlin source
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
// pull all dependencies from here
repositories {
mavenCentral()
}
// libraries that we need, using versions from Version Catalog
// we also want to use the shared/ library, so we need to include it here
dependencies {
implementation(project(":shared"))
testImplementation(libs.junit.jupiter)
}
// fancy way of saying "use JUnit 5"
tasks.test {
useJUnitPlatform()
}
// tell Gradle to use a specific version of the JDK when compiling
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(libs.versions.jdk.get()))
}
}
// application plugin settings
// module name that we've set in the module-info.java
// fully-qualified classname to excecute when you use "gradle run"
application {
mainModule.set("net.codebot.console")
mainClass.set("net.codebot.console.MainKt")
}
Shared Project
Finally, the Shared project provides services to both the Application and Console projects. It’s basically a library - not something that we execute directly, but code that we need to pull into the other projects. We keep it in a shared project to avoid code duplication.
Here’s the relevant build.gradle.kts.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
// notice the syntax for the plugin section uses alias
// this is how we pull in the plugins from the Version Catalog (above)
// e.g. libs.plugins.kotlin.lang inserts the kotlin-lang plugin details.
@Suppress("DSL_SCOPE_VIOLATION")
plugins {
`java-library` alias(libs.plugins.kotlin.lang)
alias(libs.plugins.javamodularity)
}
// used for packaging only
group = "net.codebot"version = "1.0.0"// telling Gradle to put Java and Kotlin output in the same build structure
// required since we have Java files (module-info.java) and Kotlin source
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
// pull all dependencies from here
repositories {
mavenCentral()
}
// libraries that we need, using versions from Version Catalog
// notice that the shared project needs sqlite and exposed, since
// it manages a database that the other projects use (indirectly).
dependencies {
implementation(libs.sqlite)
implementation(libs.exposed.core)
implementation(libs.exposed.jdbc)
implementation(libs.sl4j.api)
implementation(libs.sl4j.simple)
testImplementation(libs.junit.jupiter)
}
// fancy way of saying "use JUnit 5"
tasks.test {
useJUnitPlatform()
}
// tell Gradle to use a specific version of the JDK when compiling
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(libs.versions.jdk.get()))
}
}
For details on module-setup for each of these projects, refer to packages and modules
GitLab Setup
GitLab is a source code hosting platform similar to GitHub or BitBucket. The University of Waterloo maintains a hosted instance of GitLab, and you can login at https://git.uwaterloo.ca using your UW credentials 1.
We need some way to track project information, in a way that is transparent and makes information available to everyone on the team.
There are many ways of accomplishing this. Some organizations track project artifacts on paper, with written SRS documents, requirements and estimates on spreadsheets, written test plans and so on2. However, this is very inefficient, requiring changes to be carefully coordinated across different documents and tracking systems. More recently, it has become common to use online project tracking. We’ll use GitLab to track all of our project artifacts.
Capabilities
GitLab offers the following functionality high-level functionality:
Feature
Description
Repository
Version control for source code, or other files.
Issues
Mechanism to track project tasks or issues. They can capture details, be assigned a due date, passed between people and have an associated status that changes as the project progresses (e.g. Open, Closed).
Wiki
Create online documents using Markdown. This is very useful for longer-documents (e.g. design documents).
CI/CD
Continuous Integration and Deployment. We can setup rules that will automatically test or deploy our source code when it’s committed (or other conditions are met). This includes powerful analytics and reporting capabilities (that are beyond what we will cover in this course).
Snippets
Share snippets of source code outside of the project.
Project Tracking
Here’s some suggestions on how we can use GitLab at each phase of the SDLC:
Phase
SDLC Activities
GitLab Feature
Planning
We need to document project goals and objectives; log and assign requirements (which we can track as issues).
Wiki, Issues
Requirements
Definition and elaboration of requirements that are identified during the Requirements stage. Update issues as required.
Wiki, Issues
Analysis & Design
Track design decisions that are made and documented during the Analysis and Design phase. Document these decisions and make information available.
Wiki
Development
Log and document issues3 that are discovered during the Development phase. Manage feature branches and store source code.
Issues
Testing
Log and document issues that are discovered during the Testing phase. Manage unit tests. Handle continuous integration.
Issues
Evaluation
Log issues that are found. Log change requests (CR)4, based on customer feedback.
Issues
*
Manage source code, tests and all related materials. Ensure that we are using consistent versions of everything.
Repository
Project Structure
Setup
To start using GitLab for this course, you should do the following:
When it opens, select Members from the left-hand menu. Invite each member of your team with the appropriate role: typically, Developer or Maintainer for full access.
Check to ensure that each member can login on their own machine using their own credentials.
Optional. Under Settings - General, add a project description. Create and upload an avatar!
Organization
Most project information can be tracked in the Issues menu (left-hand side when the project is open).
Each major deliverable should have a milestone attached to it. For this project, this means that each sprint should be a separate milestone.
Under Issues-Milestones, create suitable milestones (aka deadlines). Make sure to assign dates to these!
Under Issues-Labels, create keywords that will help you organize your issues.
Under Issues-List, create all of the issues that you wish to work towards. At the beginning of your project, none of your issues should be assigned to a person; they should be listed as “No milestone”, since you haven’t scheduled them yet.
Common Tasks
Writing Documents
GitLab projects have a Wiki attached to them, which you can use to create hyperlinked documents, with formatting, images and so on. It’s an ideal place to store any documentation that you might create, from your Project Plan, to an SRS, to Architecture digrams. A Wiki also has the advantage of staying up to date with your other project details. e.g. you can create a Project Plan that links to Issues in your GitLab project.
GitLab uses Markdown as its native format (specifically GitHub-Flavored Markdown), a common human-readable data format5.
Tracking Issues
GitLab has the ability to attach and track issues to a project, as shown below.
An issue is meant to represent some unit of work that can be appled to the project. Historically, they often referred to “bugs” or software defects. However, for planning purposes, there is little difference between a feature, a change request and a bug - they all represent work that needs to prioritized, assigned to someone and scheduled.
As suggested above, your defaults for a new issue should be Assignee: Unassigned, Milestone: No milestone. When you schedule it into a Sprint, then you change the Milestone to reflect that sprint and the assignee to the team member responsible for it.
Updating Issues
Issues should be considered living documents, that reflect the current state of your work.
Issues should be assigned as part of the sprint kickoff
when you do something significant, you should add a comment to the issue! this helps you recall, and helps your team mates if they need to help out.
when you complete it, mark it completed.
You can create and host your own projects here as long as you remain a student. Make sure to backup your projects before you graduate! ↩︎
You will find that some disciplines or industries require more scrutiny than others. If you work in Healthcare for instance, you will need to carefully track every requirement and change that is made to a system. ↩︎
An issue is considered a “Defect” or a feature that is not working as it was initially defined. ↩︎
A change request (CR) is a request to modify a feature from it’s original requirements and design. While a defect may be viewed as a “mistake” made by the team, a CR is viewed as a late-change to the requirements. This distinction often affects the priority of the issue or CR. ↩︎
Markdown is much nicer to read than many other semi-readable formats like HTML, and there’s a lot of tools to convert it into stylized HTML, PDF, EPUB and so on. As a matter of fact, this website is written in Markdown and converted to HTML. You should use it for README files and other bundled documentation. ↩︎
Chapter 5
Learning Kotlin
An introduction to the Kotlin programming language and the Java Virtual Machine (JVM) ecosystem. Object-Oriented, Functional and Idiomatic Kotlin.
Subsections of Learning Kotlin
Why Kotlin?
Kotlin is a modern language designed by JetBrains in 2011. Originally designed as a drop-in replacement for Java, Kotlin has a number of language features that make it desireable for building applications.
We’ll focus on Kotlin/JVM, which gives us the ability to build programs that will run anywhere where we have a JVM installed (including Windows, macOS, Linux).
It has a very clean syntax, and supports quality-of-life features like default arguments, variable argument lists and rich collection types. It’s syntax closely resembles modern languages like Swift or Scala.
It’s a hybrid language: it can be used for declarative programming or class-based object-oriented programming. It also supports a number of functional features, especially with the use of collection classes. This allows a programmer to use the best appropach for a particular task.
Kotlin is statically compiled, so it catches many potential errors during compilation (not just at runtime). It’s also strongly typed, with type inference.
Critically, it supports compilation to a number of targets: JVM for Windows/macOS/Linux Desktop, Android native1, or Web. It can also build to native macOS and Windows (with some restrictions).
It has outstanding tools support with IntelliJ IDEA.
Kotlin has been adopted as the “official” language for Android development! ↩︎
Getting Started
Installing Kotlin
You need the Kotlin compiler and runtime. We’ll run on the Java JVM. You can either install the command-line tools, or install IntelliJ IDEA and run everything from within the IDE (which is recommended).
Compiling Code
Compiled languages require an explicit step to compile code and generate native executables. This is done ahead of time, and the executables are distributed to users. e.g. C++
The compilation cost is incurred before the user runs the program, so we get optimal startup performance.
The target system architecture must be known ahead of time, since we’re distributing native binaries.
Interpreted languages allow developers to distribute the raw source code which can be interpreted when the user executes it.
This requires some ‘runtime engine‘ that can convert source code to machine code on-the-fly. Results of this operation can often be cached so that the compilation cost is only incurred when it first executes.
Some languages can be compiled to a secondary format (IR, ”intermediate representation”) and then interpreted. Languages running on the Java Virtual Machine (JVM) are compiled ahead of time to IR, and then interpreted at runtime.
Kotlin can be compiled or interpreted!
Kotlin/JVM compiles Kotlin code to JVM bytecode, which can run on any Java virtual machine.
Kotlin/Android compiles Kotlin code to native Android binaries, which leverage native versions of the Java Library and Kotlin standard libraries.
Kotlin/Native compiles Kotlin code to native binaries, which can run without a virtual machine. It is an LLVM based backend for the Kotlin compiler and native implementation of the Kotlin standard library.
Kotlin/JS transpiles (converts) Kotlin to JavaScript. The current implementation targets ECMAScript 5.1 (with plans to eventually target ECMAScript 2015).
There are three primary ways of executing Kotlin code:
Read-Evaluate-Print-Loop (REPL): Interact directly with the Kotlin runtime, one line at-a-time. In this environment, it acts like a dynamic language.
KotlinScript: Use Kotlin as a scripting language, by placing our code in a script and executing directly from our shell. The code is compiled automatically when we execute it, which eliminates the need to compile ahead-of-time.
Application: We can compile standalone applications, targetting native or JVM [ed. we will use JVM in this course].
REPL
REPL is a paradigm where you type and submit expressions to the compiler one line-at-a-time. It’s commonly used with dynamic languages for debugging, or checking short expressions. It’s not intended as a means of writing full applications!
> kotlin
Welcome to Kotlin version 1.6.10 (JRE 17.0.2+8-86)
Type :help for help, :quit for quit
>>> val message="Hello Kotlin!">>> println(message)
Hello Kotlin!
KotlinScript
KotlinScript is Kotlin code in a script file that we can execute from our shell. This makes Kotlin an interesting alternative to a language like Python for shell scripting.
Kotlin compiles scripts in the background before executing them, so there’s a delay before it executes [ed. I fully expect that later versions of Kotlin will allow caching the compilation results to speedup script execution time].
This is a great way to test functionality, but not a straight-up replacement for shell scripts, due to the runtime costs1.
Applications
Kotlin applications are fully-functional, and can be compiled to native code, or to the JVM. Kotlin application code looks a little like C, or Java. Here’s the world’s simplest Kotlin program, consisting of a single main method2.
funmain() {
val message="Hello Kotlin!" println(message)
}
To compile from the command-line, we can use the Kotlin compiler, kotlinc. By default, it takes Kotlin source files (.kt) and compiles them into corresponding class files (.class) that can be executed on the JVM.
Notice that the compiled class is named slightly differently than the source file. If your code isn’t contained in a class, Kotlin wraps it in an artificial class so that the JVM (which requires a class) can load it properly. Later when we use classes, this won’t be necessary.
This example compiles Hello.kt into Hello.jar and then executes it:
Classes in Kotlin can be grouped together into namespaces, or packages. You specify a namespace for a file using the package keyword at the top of the file. The naming convention for packages is to use a reverse-DNS name (since it’s a unique identifier for a company or organization), and then some logical grouping beneath that.
For example, we could use ca.uwaterloo as our top-level DNS name for this course. To make unique package names beneath that, we might use :
ca.uwaterloo
- ca.uwaterloo.cs346
-- ca.uwaterloo.cs346.graphics // graphics classes for this course
-- ca.uwaterloo.cs346.networking // networking classes for this course
Classes in the same namespace are visible to one another.
If you want to access a class that is in a different namespace, you need to import that class into your file. For example, if we have an ErrorMessage class that is in a different package, we need to import is so that we can use it.
// this is a logging class, so we'll place in the logging package
package ca.uwaterloo.cs346.logging
// we want to use this class from a different package
import ca.uwaterloo.cs346.errorhandling.ErrorMessage
// we could also import all graphics classes from this package
import ca.uwaterloo.cs346.graphics.*
classLogger {
val error = ErrorMessage()
error.printMessage()
// ....
}
Type Aliases
Type aliases provide alternative names for existing types. If the type name is too long you can introduce a different shorter name and use the new one instead. This can make your code much more readable!
Note that type aliases do not introduce new types. They are equivalent to the corresponding underlying types, so there is no runtime overhead.
Packages work well within an application’s source code, since they let you carefully partition your classes. However, they aren’t particularly helpful if we want to publish a library, or share our functionality with some other application. The fundamental problem with packages is that cannot discriminate internal and external access, so you might end up accidentally exposing internals of your package.
How do we get around this? Kotlin also supports modules, which allow us to have rules on how our packages will be exposed outside of our application.
The rule-of-thumb is that every project should have a single module that describes how it should be exported. To define a module, add a file named module-info.java under the src/main/java' folder in your project. You may need to create this folder and add the file manually to your project3.
Here’s the syntax, continuing the example from above:
module is the module name. The convention is to name this the same as your top-level package.
requires indicates any module that your project uses that we need to include. IntelliJ will prompt you in your source code if you forget to include something.
exports indicates that it is a package that you wish to be externally visible. Make sure to export each relevant namespace.
Using Libraries
Kotlin has full access to it’s own class libraries, plus any others that are imported and made available. Kotlin is 100% compatible with Java libraries, and makes extensive use of Java libraries when possible. For example, Kotlin collection classes actually use some of the underlying Java collection libraries!
In this section, we’ll discuss how to use existing libraries in your code. We need to talk about namespaces and qualifying classes before we can talk about libraries.
Kotlin Standard Library
The Kotlin Standard Library is included with the Kotlin language, and contained in the kotlin package. This is automatically imported and does not need to be specified in an import statement.
Some of the features that will be discussed below are actually part of the standard library (and not part of the core language). This includes essential classes, such as:
// import all classes in the java.io package
// this allows us to refer to any of those classes in the current namespace
import java.io.*
// we can also just import a single class
// this allows us to refer to just the ListView class in code
import javafx.scene.control.ListView
// Kotlin code calling Java IO libraries
import java.io.FileReader
import java.io.BufferedReader
import java.io.FileNotFoundException
import java.io.IOException
import java.io.FileWriter
import java.io.BufferedWriter
if (writer !=null) {
writer.write(
row.toString() + delimiter +
s + row + delimiter +
pi + endl
)
Info
Importing a class requires your compiler to locate the file containing these classes! The Kotlin Standard Library can always be referenced by the compiler, and as long as you’re compiling to the JVM, the Java class libraries will also be made available. However, to use any other Java or Kotlin library, you will need to take additional steps. We’ll discuss this when we cover build systems and Gradle.
Scripts will be compiled and cached locally, but there’s still some runtime performance issues. ↩︎
This chapter focuses mainly on the Kotlin language. In the next chapter, we’ll dive deeper into constructing applications. ↩︎
We already have src/main/kotlin for our source code. Why are we adding a Java file? Modularity is a Java/JVM feature that was introduced in Java 9, and we’re leveraging it for JVM builds. Your compiler will handle mixing Java and Kotlin files just fine. ↩︎
Types & Mutability
Programming languages can take different approaches to enforcing how types are managed.
Strong typing: The language has strict typing rules, which typically enforced at compile-time. The exact type of a variable must be declared or fixed before the variable is used. This has the advantage of catching many types of errors at compile-time (e.g. type-mismatch).
Weak typing: These languages have looser typing rules, and will often attempt to infer types based on runtime usage. This means that some categories of errors are only caught at runtime.
Kotlin is a strongly typed language, where variables need to be declared before they are used. Kotlin also supports type infererence. If a type isn’t provided, Kotlin will infer the type at compile time (similar to ‘auto‘ in C++). The compiler is strict about this: if the type cannot be inferred at compile-time, an error will be thrown.
Variables
Kotlin uses the var keyword to indicate a variable and Kotlin expects variables to be declared before use. Types are always placed to the right of the variable name. Types can be declared explicitly, but will be inferred if the type isn’t provided.
funmain() {
var a:Int = 10var b:String = "Jeff"var c:Boolean = falsevar d = "abc"// inferred as a String
var e = 5// inferred as Int
var f = 1.5// inferred as Float
}
All standard data-types are supported, and unlike Java, all types are objects with properties and behaviours. This means that your variables are objects with methods! e.g. "10".toInt() does what you would expect.
Integer Types
Type
Size (bits)
Min value
Max value
Byte
8
-128
127
Short
16
-32768
32767
Int
32
-2,147,483,648 (-2 31)
2,147,483,647 (2 31- 1)
Long
64
-9,223,372,036,854,775,808 (-2 63)
9,223,372,036,854,775,807 (2 63- 1)
Floating Point Types
Type
Size (bits)
Significant bits
Exponent bits
Decimal digits
Float
32
24
8
6-7
Double
64
53
11
15-16
Boolean
The type Boolean represents boolean objects that can have two values: true and false. Boolean has a nullable counterpart Boolean? that also has the null value.
Built-in operations on booleans include:
|| – disjunction (logical OR)
&& – conjunction (logical AND)
!- negation (logical NOT)
|| and && work lazily.
Strings
Strings are often a more complex data type to work with, and deserve a callout. In Kotlin, they are represented by the String type, and are immutable. Elements of a string are characters that can be accessed by the indexing operation: s[i], and you can iterate over a string with a for-loop:
funmain() {
val str = "Sam"for (c in str) {
println(c)
}
}
You can concatenate strings using the + operator. This also works for concatenating strings with values of other types, as long as the first element in the expression is a string (in which case the other element will be case to a String automatically):
funmain() {
val s = "abc" + 1 println(s + "def")
}
Kotlin supports the use of string templates, so we can perform variable substitution directly in strings. It’s a minor but incredibly useful feature that replaces the need to concatenate and build up strings to display them.
funmain() {
println("> Kotlin ${KotlinVersion.CURRENT}")
val str = "abc" println("$str.length is ${str.length}")
var n = 5 println("n is ${if(n > 0) "positive" else "negative"}")
}
is and !is operators
To perform a runtime check whether an object conforms to a given type, use the is operator or its negated form !is:
funmain() {
val obj = "abc"if (obj is String) {
print("String of length ${obj.length}")
} else {
print("Not a String")
}
}
In most cases, you don’t need to use explicit cast operators in Kotlin because the compiler tracks the is -checks and explicit casts for immutable values and inserts (safe) casts automatically when needed:
funmain() {
val x = "abc"if (x !is String) return println("x=${x.length}") // x is automatically cast to String
val y = "defghi"// y is automatically cast to string on the right-hand side of `||`
if (y !is String || y.length ==0) return println("y=${y.length}") // y must be a string with length > 0
}
Immutability
Kotlin supports the use of immutable variables and data structures [mutable means that it can be changed; immutable structures cannot be changed after they are initialized]. This follows best-practices in other languages (e.g. use of ‘final‘ in Java, ‘const‘ in C++), where we use immutable structures to avoid accidental mutation.
var - this is a standard mutable variable that can be changed or reassigned.
val - this is an immutable variable that cannot be changed once initialized.
var a = 0// type inferred as Int
a = 5// a is mutable, so reassignment is ok
val b = 1// type inferred as Int as well
b = 2// error because b is immutable
var c:Int = 10// explicit type provided in this case
Operators
Kotlin supports a wide range of operators. The full set can be found on the Kotlin Language Guide.
NULL is a special value that indicates that there is no data present (often indicated by the null keyword in other languages). NULL values can be difficult to work with in other programming languages, because once you accept that a value can be NULL, you need to check all uses of that variable against the possibility of it being NULL.
NULL values are incredibly difficult to manage, because to address them properly means doing constant checks against NULL in return values, data and so on1. They add inherent instability to any type system.
In Kotlin, every type is non-nullable by default. This means that if you attempt to assign a NULL to a normal data type, the compiler is able to check against this and report it as a compile-time error. If you need to work with NULL data, you can declare a nullable variable using the ? annotation [ed. a nullable version of a type is actually a completely different type]. Once you do this, you need to use specific ? methods. You may also need to take steps to handle NULL data when appropriate.
Conventions
By default, a variable cannot be assigned a NULL value.
? suffix on the type indicates that it’s NULL-able.
?. accesses properties/methods if the object is not NULL (“safe call operator”)
?:elvis operator is a ternary operator for NULL data
!! override operator (calls a method without checking for NULL, bad idea)
funmain() {
// name is nullable
var name:String? = null// only returns value if name is not null
var length = name?.length
println(length) // null
// elvis operator provides an `else` value
length = name?.length ?:0 println(length) // 0
}
Generics
Generics are extensions to the type system that allows us to parameterize classes or functions across different types. Generics expand the reusability of your class definitions, because they allow your definitions to work with many types.
We’ve already seen generics when dealing with collections:
val list: List<Int> = listOf(5, 10, 15, 20)
In this example, <Int> is specifying the type that is being stored in the list. Kotlin infers types where it can, so we typically write this as:
val list = listOf(5, 10, 15, 20)
We can use a generic type parameter in the place of a specific type in many places, which allows us to write code towards a generic type instead of a specific type. This prevents us from writing methods that might only differ by parameter or return type.
A generic type is a class that accepts an input of any type in its constructor. For instance, we can create a Table class that can hold a differing values.
You define the class and make it generic by specifying a generic type to use in that class, written in angle brackets < >. The convention is to use T as a placeholder for the actual type that will be used.
classTable<T>(t: T) {
var value = t
}
val table1: Table<Int> = Table<Int>(5)
val table2 = Table<Float>(3.14)
Tony Hoare invented the idea of a NULL reference. In 2009, he apologized for this, famously calling it his “billion-dollar mistake”. ↩︎
Control-Flow
Kotlin supports the style of control flow that you would expect in an imperative language, but it uses more modern forms of these constructs
if then else
if... then has both a statement form (no return value) and an expression form (return value).
funmain() {
val a=5val b=7// we don't return anything, so this is a statement
if (a > b) {
println("a is larger")
} else {
println("b is larger")
}
val number = 6// the value from each branch is considered a return value
// this is an expression that returns a result
val result =
if (number > 0)
"$number is positive"elseif (number < 0)
"$number is negative"else"$number is zero" println(result)
}
Info
This is why Kotlin doesn’t have a ternary operator: if used as an expression serves the same purpose.
for in
A for in loop steps through any collection that provides an iterator. This is equivalent to the for each loop in languages like C#.
funmain() {
val items = listOf("apple", "banana", "kiwifruit")
for (item in items) {
println(item)
}
for (index in items.indices) {
println("item $index is ${items[index]}")
}
for (c in"Kotlin") {
print("$c ")
}
}
Kotlin doesn’t support a C/Java style for loop. Instead we use a range collection.. that generates a sequence of values.
funmain() {
// invalid in Kotlin
// for (int i=0; i < 10; ++i)
// range provides the same funtionality
for (i in1..3) {
print(i)
}
println() // space out our answers
// descending through a range, with an optional step
for (i in6 downTo 0 step 2) {
print("$i ")
}
println()
// we can step through character ranges too
for (c in'A'..'E') {
print("$c ")
}
println()
// Check if a number is within range:
val x = 10val y = 9if (x in1..y+1) {
println("fits in range")
}
}
while
while and do... while exist and use familiar syntax.
funmain() {
var i = 1while ( i <=10) {
print("$i ")
i++ }
}
when
when replaces the switch operator of C-like languages:
funmain() {
val x = 2when (x) {
1-> print("x == 1")
2-> print("x == 2")
else-> print("x is neither 1 nor 2")
}
}
funmain() {
val x = 13val validNumbers = listOf(11,13,17,19)
when (x) {
0, 1-> print("x == 0 or x == 1")
in2..10-> print("x is in the range")
in validNumbers -> print("x is valid")
!in10..20-> print("x is outside the range")
else-> print("none of the above")
}
}
We can also return a value from when. Here’s a modified version of this example:
funmain() {
val x = 13val validNumbers = listOf(11,13,17,19)
val response = when (x) {
0, 1->"x == 0 or x == 1"in2..10->"x is in the range"in validNumbers ->"x is valid"!in10..20->"x is outside the range"else->"none of the above" }
println(response)
}
When is flexible. To evaluate any expression, you can move the comparison expressions into the when statement itself:
funmain() {
val x = 13val response = when {
x < 0->"negative" x >=0&& x <=9->"small" x >=10->"large"else->"how do we get here?" }
println(response)
}
return
Kotlin has three structural jump expressions:
return by default returns from the nearest enclosing function or anonymous function
break terminates the nearest enclosing loop
continue proceeds to the next step of the nearest enclosing loop
Functions
Functions are preceded with the fun keyword. Function parameters require types, and are immutable. Return types should be supplied after the function name, but in some cases may also be inferred by the compiler.
Named Functions
Named function have a name assigned to them that can be used to invoke them directly (this is the expected form of a “function” in most cases, and the form that you’re probably expecting).
// no parameters required
funmain() {
println(sum1(1, 2))
println(sum1(3,4))
}
// parameters which require type annotations
funsum1(a: Int, b: Int): Int {
return a + b
}
// return types can be inferred based on the value you return
// it's better form to explicitly include the return type in the signature
funsum2(a: Int, b: Int) {
a + b // Kotlin knows that (Int + Int) -> Int
}
Single-Expression Functions
Simple functions in Kotlin can sometimes be reduced to a single line aka a single-expression function.
// previous example
funsum(a: Int, b: Int) {
a + b // Kotlin knows that (Int + Int) -> Int
}
// this is equivilant
funsum(a: Int, b: Int) = a + b
// this works since we evaluate a single expression
funminOf(a: Int, b: Int) = if (a < b) a else b
Function Parameters
Default arguments
We can use default arguments for function parameters. When called, a parameter with a default value is optional; if the value is not provided by the caller, the default will be used.
// Second parameter has a default value, so it’s optional
funmult(a:Int, b:Int = 1): Int {
return a * b
}
funmain() {
mult(1) // 1
mult(5,2) // 10
// mult() will throw an error, `a` must be provided
}
Named parameters
You can (optionally) provide the parameter names when you call a function. If you do this, you can even change the calling order!
Finally, we can have a variable length list of arguments:
// Variable number of arguments can be passed!
// Arguments in the list need to have the same type
funsum(vararg numbers: Int): Int {
var sum: Int = 0for(number in numbers) {
sum += number
}
return sum
}
funmain() {
sum(1) // 1
sum(1,2,3) // 6
sum(1,2,3,4,5,6,7,8,9,10) // 55
}
Collections
A collection is a finite group of some variable number of items (possibly zero) of the same type. Objects in a collection are called elements.
These collection classes exists as generic containers for a group of elements of the same type e.g. List would be an ordered list of integers. Collections have a finite size, and are eagerly evaluated.
Kotlin offers functional processing operations (e.g. filter, map and so on) on each of these collections.
funmain() {
val list = (1..10).toList()
println( list.take(5).map{it* it} )
}
Under-the-hood, Kotlin uses Java collection classes, but provides mutable and immutable interfaces to these classes. Kotlin best-practice is to use immutable for read-only collections whenever possible (since mutating collections is often very costly in performance).
A Pair is a tuple of two values. Use var or val to indicate mutability. Theto keyword can be used to indicate a Pair.
funmain() {
// mutable
var nova_scotia = "Halifax Airport" to "YHZ"var newfoundland = Pair("Gander Airport", "YQX")
var ontario = Pair("Toronto Pearson", "YYZ")
ontario = Pair("Billy Bishop", "YTZ") // reassignment is ok
// immutable, mixed types
val canadian_exchange = Pair("CDN", 1.38)
// accessing elements
val characters = Pair("Tom", "Jerry")
println(characters.first)
println(characters.second)
// destructuring
val(first, second) = Pair("Calvin", "Hobbes") // split a Pair
println(first)
println(second)
}
Pairs are extremely useful when working with data that is logically grouped into tuples, but where you don’t need the overhead of a custom class. e.g. Pair for 2D points.
funmain() {
// define an immutable list
var fruits = listOf( "advocado", "banana")
println(fruits.get(0))
// advocado
// add elements
var mfruits = mutableListOf( "advocado", "banana")
mfruits.add("cantaloupe")
mfruits.forEach { println(it) }
// sorted/sortedBy returns ordered collection
val list = listOf(2,3,1,4).sorted() // [1, 2, 3, 4]
list.sortedBy { it % 2 } // [2, 4, 1, 3]
// groupBy groups elements on collection by key
list.groupBy { it % 2 } // Map: {1=[1, 3], 0=[2, 4]}
// distinct/distinctBy returns unique elements
listOf(1,1,2,2).distinct() // [1, 2]
}
Set
A Set is a generic unordered collection of unique elements (i.e. it does not support duplicates, unlike a List which does). Sets are commonly constructed with helper functions:
val numbersSet = setOf("one", "two", "three", "four")
val emptySet = mutableSetOf<String>()
A Map is an associative dictionary containing Pairs of keys and values.
funmain() {
// immutable reference, immutable map
val imap = mapOf(Pair(1, "a"), Pair(2, "b"), Pair(3, "c"))
println(imap)
// {1=a, 2=b, 3=c}
// immutable reference, mutable map (so contents can change)
val mmap = mutableMapOf(5 to "d", 6 to "e")
mmap.put(7,"f")
println(mmap)
// {5=d, 6=e, 7=f}
// lookup a value
println(mmap.get(5))
// d
// iterate over key and value
for ((k, v) in imap) {
print("$k=$v ")
}
// 1=a 2=b 3=c
// alternate syntax
imap.forEach { k, v -> print("$k=$v ") }
// 1=a 2=b 3=c
// `it` represents an implicit iterator
imap.forEach {
print("${it.key}=${it.value} ")
}
// 1=a 2=b 3=c
}
Array
Arrays are indexed, fixed-sized collection of objects and primitives. We prefer other collections, but these are offered for legacy and compatibility with Java.
funmain() {
// Create using the `arrayOf()` library function
arrayOf(1, 2, 3)
// Create using the Array class constructor
// Array<String> ["0", "1", "4", "9", "16"]
val asc = Array(5) {
i -> (i*i).toString()
}
asc.forEach { println(it) }
}
You can access array elements through using the [] operators, or the get() and set() methods.
A tuple is a data structure representing a sequence of n elements. ↩︎
Classes & Objects
Introduction
Object-oriented programming is a refinement of the structured programming model, that was discovered in 1966 by Ole-Johan Dahl and Kristen Nygaard.
It is characterized by the use of classes as a template for common behaviour (methods) and data (state) required to model that behaviour.
Abstraction: Model entities in the system to match real world objects. Each object exposes a stable high-level interface, that can be used to access its data and behaviours. “Changing the implementation should not break anything”.
Encapsulation: Keep state and implementation private.
Inheritance: Specialization of classes. Create a specialized (child) class by deriving from another (parent) class, and reusing the parent’s fields and methods.
Polymorphism: Base and derived classes share an interface, but have specialized implementations.
Object-oriented programming has benefits over iterative programming:
It supports abstraction, and allows us to express computation in a way that models our problem domain (e.g. customer classes, file classes).
It handles complex state more effectively, by delegating to classes.
It’s also a method of organizing your code. This is critical as programs grow.
There is some suggestion that it makes code reuse easier (debatable).
It became the dominant programming model in the 80s, and our most used languages are OO languages. e.g. C++, Java, Swift, Kotlin.
Classes
Kotlin is a class-based object-oriented language, with some advanced features that it shares with some other recent languages. The class keyword is used to define a Class. You create an instance of the class using the classname (no new keyword required!)
// define class
classPerson// create two instances and assign to p, q
// note that we have an implicit no-arg constructor
val p = Person()
val q = Person()
Classes include properties (values) and methods.
Properties
A property is a variable that is declared in a class, but outside of methods or functions. They are analogous to class members, or fields in other languages.
classPerson() {
var firstName = "Vanilla"var lastName = "Ice"}
funmain() {
val p = Person()
// we can access properties directly
// this calls an implicit get() method; default returns the value
println("${p.firstName}${p.lastName}${p.lastName} Baby")
}
Properties all have implicit backing fields that store their data. We can override the get() and set methods to determine how our properties interact with the backing fields.
For example, for a City class, we can decide that we want the city name always reported in uppercase, and we want the population always stored as thousands.
// the backing field is just referred to as `field`
// in the set() method, we use `value` as the argument
classCity() {
var name = ""get() = field.uppercase()
set(value) {
field = value }
var population = 0set(value) {
field = value/1_000
}
}
funmain() {
// create our city, using properties to access values
val city = City()
city.name = "Halifax" city.population = 431_000
println("${city.name} has a population of ${city.population} thousand people")
}
Behind-the-scenes, Kotlin is actually creating getter and setter methods, using the convention of getField and setField . In other words, you always have corresponding methods that are created for you. If you directly access the field name, these methods area actually getting called in the background.
Venkat Subramaniam has an excellent example of this [Subramaniam 2019]. Write the class Car in a separate file named Car.kt:
classCar(val yearOfMake: Int, var color: String)
Then compile the code and take a look at the bytecode using the javap tool, by running these commands:
$ kotlinc-jvm Car.kt
$ javap -p Car.class
This will display the bytecode generated by the Kotlin Compiler for the Car class:
publicfinalclassCar {
privatefinal int yearOfMake;
private java.lang.String color;
publicfinal int getYearOfMake();
publicfinal java.lang.String getColor();
publicfinal void setColor(java.lang.String);
public Car(int, java.lang.String);
}
That concise single line of Kotlin code for the Car class resulted in the creation of two fields—the backing fields for properties, a constructor, two getters, and a setter.
Constructors
Like other OO languages, Kotlin supports explicit constructors that are called when objects are created.
Primary Constructors
A primary constructor is the main constructor that your class will support (representing how you want it to be instantiated most of the time). You define it by expanding the class definition:
// class definition includes the primary constructor
classPersonconstructor() { }
// we can collapse this to define an explicit no-arg constructor
classPerson() {}
In the example above, the primary constructor is called when this class is instantiated.
Optionally, you can include parameters in the primary constructor, and use these to initialize parameters in the constructor body.
// constructor with arguments
// this uses the parameters to initialize properties (i.e. variables)
classPerson (first:String, last:String) {
val firstName = first.take(1).uppercase() + first.drop(1).lowercase()
val lastName = last.take(1).uppercase() + last.drop(1).lowercase()
// adding a statement like this will prevent the code from compiling
// println("${firstname} ${lastname}") // will not compile
}
funmain() {
// this does not work! we do not have a no-arg constructor
// val person = Person() // error since no matching constructor
// this works and demonstrates the properties
val person = Person("JEFF", "AVERY")
println("${person.firstName}${person.lastName}") // Jeff Avery
}
Constructors are designed to be minimal:
Parameters can only be used to initialize properties. They go out of scope immediately after the constructor executes.
You cannot invoke any other code in your constructor (there are other ways to handle that, which we will discuss below).
Secondary Constructors
What if you need more than a single constructor?
You can define secondary constructors in your class. Secondary constructors must delegate to the primary constructor. Let’s rewrite this class to have a primary no-arg constructor, and a second constructor with parameters.
// primary constructor
classPerson() {
// initialize properties
var firstName = "PAULA"var lastName = "ABDUL"// secondary constructor
// delegates to the no-arg constructor, which will be executed first
constructor(first: String, last: String) : this() {
// assign to the properties defined in the primary constructor
firstName = first.take(1).uppercase() + first.drop(1).lowercase()
lastName = last.take(1).uppercase() + last.drop(1).lowercase()
}
}
funmain() {
val person1 = Person() // primary constructor using default property values
println("${person1.firstName}${person1.lastName}")
val person2 = Person("JEFF", "AVERY") // secondary constructor
println("${person2.firstName}${person2.lastName}")
}
Init Blocks
How do we execute code in the constructor? We often want to do more than just initialize properties.
Kotlin has a special method called init() that is used to manage initialization code. You can have one or more of these init blocks in your code, which will be called in order after the primary constructor (they’re actually considered part of the primary constructor). The order of initialization is (1) primary constructor, (2) init blocks in listed order, and then finally (3) secondary constructor.
classInitOrderDemo(name: String) {
val first = "$name".also(::println)
init {
println("First init: ${first.length}")
}
val second = "$name".also(::println)
init {
println("Second init: ${second.length}")
}
}
funmain() {
InitOrderDemo("Jeff")
}
Info
Why does Kotlin split the constructor up like this? It’s a way to enforce that initialization MUST happen first, which results in cleaner, and safer code.
Class Methods
Similarly to other programming languages, functions defined inside of a class are called methods.
classPerson(var firstName: String, var lastName: String) {
fungreet() {
println("Hello! My name is $firstName")
}
}
funmain() {
val person = Person ("Jeff", "Avery")
println("${person.firstName}${person.lastName}")
}
Operator Overloading
Kotlin allows you to provide custom implementations for the predefined set of operators. These operators have predefined symbolic representation (like + or *) and predefined precedence if you combine them.
Basically, you use the operator keyword to define a function, and provide a member function or an extension function with a specific name for the corresponding type. This type becomes the left-hand side type for binary operations and the argument type for the unary ones.
Here’s an example that extends a class named ClassName by overloading the + operator.
We can override any operators by using the keyword that corresponds the symbol we want to override.
Note that this is the reference object on which we are calling the appropriate method. Parameters are available as usual.
Description
Expression
Translated to
Unary prefix
+a
a.unaryPlus()
-a
a.unaryMinus()
!a
a.not()
Increments, decrements
a++
a.inc()
a–
a.dec()
Arithmetic
a+b
a.plus(b)
a-b
a.minus(b)
a*b
a.times(b)
a/b
a.div(b)
a%b
a.rem(b)
a..b
a.rangeTo(b)
In
a in b
b.contains(a)
Augmented assignment
a+=b
a.plusAssign(b)
a-=b
a.minusAssign(b)
a*=b
a.timesAssign(b)
a/=b
a.divAssign(b)
a%b
a.remAssign(b)
Equality
a==b
a?.equals(b) ?: (b === null)
a!=b
!(a?.equals(b) ?: (b === null))
Comparison
a>b
a.compareTo(b) > 0
a<b
a.compareTo(b) < 0
a>=b
a.compareTo(b) >= 0
a<=b
a.compareTo(b) <= 0
Infix Functions
Functions marked with the infix keyword can also be called using the infix notation (omitting the dot and the parentheses for the call). Infix functions must meet the following requirements:
For example, we can add a “shift left” function to the built-in Int class:
infixfunInt.shl(x: Int): Int {
return (this shl x)
}
funmain() {
// calling the function using the infix notation
// shl 1 multiples an int by 2
println(212 shl 1)
// is the same as
println(212.shl(1))
}
Extension Functions
Kotlin supports extension functions: the ability to add functions to existing classes, even when you don’t have access to the original class’s source code, or cannot modify the class for some reason. This is also a great alternative to inheritance when you cannot extend a class.
For an simple example, imagine that you want to determine if an integer is even. The “traditional” way to handle this is to write a function:
In Kotlin, the Int class already has a lot of built-in functionality. It would be a lot more consistent to add this as an extension function to that class.
funInt.isEven() = this % 2==0funmain() {
println(4.isEven())
println(5.isEven())
}
You can use extensions with your own types and also types you do not control, like List, String, and other types from the Kotlin standard library.
Extension functions are defined in the same way as other functions, with one major difference: When you specify an extension function, you also specify the type the extension adds functionality to, known as the receiver type. In our earlier example, Int.isEven(), we need to include the class that the function extends, or Int.
Note that in the extension body, this refers to the instance of the type (or the receiver for this method invocation).
funString.addEnthusiasm(enthusiasmLevel: Int = 1) = this + "!".repeat(enthusiasmLevel)
funmain() {
val s1 = "I'm so excited"val s2 = s1.addEnthusiasm(5)
println(s2)
}
Defining an extension on a superclass
Extensions do not rely on inheritance, but they can be combined with inheritance to expand their scope. If you extend a superclass, all of it’s subclasses will inherit the extension method that you defined.
Define an extension on the Any class called print. Because it is defined on Any, it will be directly callable on all types.
// Any is the top-level class from which all classes derive i.e. the ultimate superclass.
A data class is a special type of class, which primarily exists to hold data, and doesn’t have custom methods. Classes like this are more common than you expect – we often create trivial classes to just hold data, and Kotlin makes it very easy.
Why would you use a data class over a regular class? It generates a lot of useful methods for you:
hashCode()
equals() // compares fields
toString()
copy() // using fields
destructuring
Here’s an example of how useful this can be:
dataclassPerson(val name: String, var age: Int)
funmain() {
val mike = Person("Mike", 23)
// toString() displays all properties
println(mike.toString())
// structural equality (==) compares properties
println(mike == Person("Mike", 23)) // True
println(mike == Person("Mike", 21)) // False
// referential equality (===) compares object references
println(mike === Person("Mike", 23)) // True
// hashCode based on primary constructor properties
println(mike.hashCode() == Person("Mike", 23).hashCode()) // True
println(mike.hashCode() == Person("Mike", 21).hashCode()) // False
// destructuring based on properties
val(name, age) = mike
println("$name$age") // Mike 23
// copy that returns a copy of the object
// with concrete properties changed
val jake = mike.copy(name = "Jake") // copy
}
Enum Classes
Enums in Kotlin are classes, so enum classes support type safety.
We can use them in expected ways. Enum num constants are separated with commas. We can also do interesting things with our enums, like use them in when clauses (Example from [Sommerhoff 2020]).
Each enum constant is an object, and can be instantiated.
enumclassDirection(val degrees: Double) {
NORTH(0.0), SOUTH(180.0), WEST(270.0), EAST(90.0)
}
funmain() {
val direction = Direction.EAST
print(direction.degrees)
}
Class Hierarchies
All classes in Kotlin have a common superclass Any, that is the default superclass for a class with no supertypes declared:
classExample// Implicitly inherits from Any
Any has three methods: equals(), hashCode() and toString(). Thus, they are defined for all Kotlin classes.
Inheritance
To derive a class from a supertype, we use the colon : operator. We also need to delegate to the base class constructor using ().
By default, classes and methods are closed to inheritance. If you want to extend a class or method, you need to explicitly mark it as open for inheritance.
classBaseclassDerived : Base() // error!
openclassBaseclassDerived : Base() // ok
openclassPerson(val name: String) {
openfunhello() = "Hello, I am $name" }
classPolishPerson(name: String) : Person(name) {
overridefunhello() = "Dzien dobry, jestem $name" }
funmain() {
val p1 = Person("Jerry")
val p2 = PolishPerson("Beth")
println(p1.hello())
println(p2.hello())
}
If the derived class has a primary constructor, the base class can (and must) be initialized, using the parameters of the primary constructor. If the derived class has no primary constructor, then each secondary constructor has to initialize the base type using the super keyword, or to delegate to another constructor which does that.
Classes can be declared abstract, which means that they cannot be instantiated, only used as a supertype. The abstract class can contain a mix of implemented methods (which will be inherited by subclasses) and abstract methods, which do not have an implementation.
// useful way to represent a 2D point
dataclassPoint(val x:Int, val y:Int)
abstractclassShape() {
// we can have a single representation of position
var x = 0var y = 0funposition(): Point {
return Point(x, y)
}
// subtypes will have their own calculations for area
abstractfunarea():Int
}
classRectangle (var width: Int, var height: Int): Shape() {
constructor(x: Int, y: Int, width: Int, height: Int): this(width, height) {
this.x = x
this.y = y
}
// must be overridden since our base is abstract
overridefunarea():Int {
return width * height
}
}
funmain() {
// this won't compile, since Shape() is abstract
// val shape = Shape()
// this of course is fine
val rect = Rectangle(10, 20, 50, 10)
println("Rectangle at (${rect.position().x},${rect.position().y}) with area ${rect.area()}")
// => Rectangle at (10,20) with area 500
}
Interfaces
Interfaces in Kotlin are similar to abstract classes, in that they can contain a mix of abstract and implemented methods. What makes them different from abstract classes is that they cannot store state. They can have properties but these need to be abstract or to provide accessor implementations.
Visibility Modifiers
Classes, objects, interfaces, constructors, functions, properties and their setters can have visibility modifiers. Getters always have the same visibility as the property. Kotlin defaults to public access if not visibility modifier is provided.
The possible visibility modifiers are:
public: visible to any other code.
private: visible inside this class only (including all its members).
protected: visible to any derived class, otherwise private.
internal: visible within the same module, where a module is a set of Kotlin files compiled together.
Destructuring
Sometimes it is convenient to destructure an object into a number of variables. This syntax is called a destructuring declaration. A destructuring declaration creates multiple variables at once. In the example below, you declared two new variables: name and age, and can use them independently:
dataclassPerson(val name: String, val age: Int)
funmain() {
val p = Person("Janine", 38)
val(name, age) = p // destructuring
println(name)
println(age)
}
A destructuring declaration is compiled down to the following code:
val name = person.component1()
val age = person.component2()
component1(), component() are aliases to the named properties in this class, in the order they were declared (and, of course, there can be component3() and component4() and so on). You would never normally refer to them using these aliases.
Here’s an example from the Kotlin documentation on how to use this to return multiple values from a function:
// data class with properties `result` and `status`
dataclassResult(val result: Int, val status: Status)
funfunction(...): Result {
// computations
return Result(result, status)
}
// Destructure into result and status
val(result, status) = function(...)
// We can also choose to not assign fields
// e.g. we could just return `result` and discard `status`
val(result, _) = function(...)
Companion Objects
OO languages typically have some idea of static members: methods that are associated with a class instead of an instance of a class. Static methods can be useful when attempting to implement the singleton pattern, for instance.
Kotlin doesn’t support static members directly. To get something comparable in Kotlin, you need to declare a companion object as an inner class of an existing class. Any methods that are created as part of the companion object are considered to be static methods in the enclosing class.
classHouse(val numberOfRooms: Int, val price: Double) {
companionobject {
val HOUSES_FOR_SALE = 10fungetNormalHouse() = House(6, 599_000.00)
fungetLuxuryHouse() = House(42, 7_000_000.00)
}
}
funmain() {
val normalHouse = House.getNormalHouse() // works
println(normalHouse.price)
println(House.HOUSES_FOR_SALE)
}
We can also use object types to implement singletons. All we need to do is use the object keyword.
classCountry(val name:String) {
var area = 0.0 }
// there can be only one
objectCountryFactory {
funcreateCountry() = Country("Canada")
}
funmain() {
val obj = CountryFactory.createCountry()
println(obj.name)
}
Packages & Modules
Java and Kotlin have two different levels of abstraction when it comes to grouping code: packages and modules.
Packages
Packages are meant to be a collection of related classes. e.g. graphics classes. Packages are primarily a mechanism for managing dependencies between parts of an application, and encourages clear separation of concerns. They are conceptually similar to namepsaces in C++.
Use the package declaration to the top of a source file to assign a file to a namespace. Classes or modules in the same package have full visibility to each other.
Info
Best practice is to use a reverse DNS name for a package name. e.g. com.sun.graphics if you developed the Graphics library at Sun Microsystems. Package names are always lowercase, dot-separateds with no underscores. If you want to use multiple-words, consider using camel case.
For example, in the file below, contents are contained in the ca.uwaterloo.cs346 package. The full name of the class could be qualified as ca.uwaterloo.cs346.ErrorMessage . If you were referring to it from a different package, you would need to use this fully qualified name.
To use a class in a different namespace, we need to import the related class by using the import keyword. This applies to any class that we wish to use. In the example below, we import our ErrorMessage class into a different namespace so that we can instantiate and use it.
import ca.uwaterloo.cs346.ErrorMessage
classLogger {
val error = ErrorMessage()
error.printMessage()
}
Modules
Modules serve a different purpose than packages: they are intended to expose permissions for external dependencies. Using modules, you can create higher-level constructs (modules) and have higher-level permissions that describe how that module can be reused. Modules are intended to support a more rigorous separation of concerns that you can obtain with packages.
You can think of each project as being represented by a single module, and each module can contain one or more packages.
Let’s use our starting Gradle directory as the starting point. We’ll expand the source code subdirectory to include a package (net.codebot) containing a single class (Main.kt).
To create a module for this project, we need to add a file named module-info.java in the src/main subdirectory. This will describe a module that contains this project, and also describes what classes will be exported and available to other projects.
Info
Modules are particularly important when working with multi-project builds, since the module-info.java describes what classes are available to other projects.
The file opens with the name of the module. Module names have the same naming restrictions as package names, and the convention is to use the same name for the module and package that it contains.
exports lists the packages that should be available to other modules that wish to import this module.
requires lists other modules on which this module depends.
Info
You’ll notice that the module-info.java file is located under the java/ folder, even though this is a Kotlin project! This is due to the relative newness of the Java module system. You can get around this by adding the following lines to your build.gradle file:
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
Functional Kotlin
Introduction
Functional programming is a programming style1 where programs are constructed by compositing functions together. Functional programming treats functions as first-class citizens: they can be assigned to a variable, passed as parameters, or returned from a function.
Functional programming also specifically avoids mutation: functions transform inputs to produce outputs, with no internal state. Functional programming can be described as declarative (describe what you want) instead of imperative (describe how to accomplish a result).
Functional programming constrains assignment, and therefore constrains side-effects (Martin 2003).
Kotlin is considered a hybrid language: it provides mechanisms for you to write in a functional style, but it also doesn’t prevent you from doing non-functional things. As a developer, it’s up to you to determine the most appropriate approach to a given problem.
Here are some common properties that we talk about when referring to “functional programming”:
First-class functions means that functions are treated as first-class citizens. We can pass them as to another function as a parameter, return functions from other functions, and even assignment functions to variables. This allows us to treat functions much as we would treat any other variable.
Pure functions are functions that have no side effects. More formally, the return values of a pure function are identical for identical arguments (i.e. they don’t depend on any external state). Also, by having no side effects, they do not cause any changes to the system, outside of their return value. Functional programming attempts to reduce program state, unlike other programming paradigms (imperative or object-oriented which are based on careful control of program state).
Immutable data means that we do not modify data in-place. We prefer immutable data that cannot be accidentally changed, especially as a side-effect of a function. Instead, if we need to mutate data, we pass it to a function that will return a new data structure containing the modified data, leaving the original data intact. This avoids unintended state changes.
Lazy evaluation is the notion that we only evaluate as expression when we need to operate on it (and we only evaluate what we need to evaluate at the moment!) This allows us to express and manipulate some expressions that would be extremely difficult to actually represent in other paradigms.
Info
In the next sections, we’ll focus on Kotlin support for higher-order functions. Avoiding mutation and side effects is partly a stylistic choice - you don’t require very many language features to program this way, but Kotlin encourages non-mutable data with the val keyword.
Function Types
Functions in Kotlin are “first-class citizens” of the language. This means that we can define functions, assign them to variables, pass functions as arguments to other functions, or return functions! Functions are types in Kotlin, and we can use them anywhere we would expect to use a regular type.
Bert’s Barber shop is creating a program to calculate the cost of a haircut, and they end up with 2 almost-identical functions.
funmain() {
val taxMultiplier = 1.10funcalculateTotalWithFiveDollarDiscount(initialPrice: Double): Double {
val priceAfterDiscount = initialPrice - 5.0val total = priceAfterDiscount * taxMultiplier
return total
}
funcalculateTotalWithTenPercentDiscount(initialPrice: Double): Double {
val priceAfterDiscount = initialPrice * 0.9val total = priceAfterDiscount * taxMultiplier
return total
}
}
These functions are identical except for the line that calculates priceAfterDiscount. If we could somehow pass in that line of code as an argument, then we could replace both with a single function that looks like this, where applyDiscount() represents the code that we would dynamically replace:
// applyDiscount = initialPrice * 0.9, or
// applyDiscount = initialPrice - 5.0
funcalculateTotal(initialPrice: Double, applyDiscount: ???): Double {
val priceAfterDiscount = applyDiscount(initialPrice)
val total = priceAfterDiscount * taxMultiplier
return total
}
This is a perfect scenario for passing in a function!
In this example, applyDiscount is now a reference to the discountFiveDollars function (note the :: notation when we have a function on the RHS of an assignment). We can even call it.
val discountedPrice = applyDiscount(20.0) // Result is 15.0
So what is the type of our function? The type of a function is the function signature, but with a different syntax that you might be accustomed to seeing.
// this is the original function signature
fundiscountFiveDollars(price: Double): Double = price - 5.0val applyDiscount = ::discountFiveDollars
// applyDiscount accepts a Double as an argument and returns a Double
// we use this format when specifying the type
val applyDiscount: (Double) -> Double
For functions with multiple parameters, separate them with a comma.
We can use this notation when explicitly specifying type.
fundiscountFiveDollars(price: Double): Double = price - 5.0// specifying type is not necessary since type inference works too
// we'll just do it here to demonstrate how it would appear
val applyDiscount : (Double) -> Double = ::discountFiveDollars
Pass a function to a function
We can use this information to modify the earlier example, and have Bert’s calculation function passed into the second function.
fundiscountFiveDollars(price: Double): Double = price - 5.0fundiscountTenPercent(price: Double): Double = price * 0.9funnoDiscount(price: Double): Double = price
funcalculateTotal(initialPrice: Double, applyDiscount: (Double) -> Double): Double {
val priceAfterDiscount = applyDiscount(initialPrice)
val total = priceAfterDiscount * taxMultiplier
return total
}
val withFiveDollarsOff = calculateTotal(20.0, ::discountFiveDollars) // $16.35
val withTenPercentOff = calculateTotal(20.0, ::discountTenPercent) // $19.62
val fullPrice = calculateTotal(20.0, ::noDiscount) // $21.80
Returning Functions from Functions
Instead of typing in the name of the function each time he calls calculateTotal(), Bert would like to just enter the coupon code from the bottom of the coupon that he receives from the customer. To do this, he just needs a function that accepts the coupon code and returns the right discount function.
I’ve taken liberties with Dave Leed’s example, but my notes can’t do it justice. I’d highly recommend a read through his site - he’s building an outstanding Kotlin book chapter-by-chapter with cartoons and illustrations.
Introduction to Lambdas
We can use this same notation to express the idea of a function literal, or a function as a value.
The code on the RHS of this expression is a function literal, which captures the body of this function. We also call this a lambda. A lambda is just a function, but written in this form:
the function is enclosed in curly braces { }
the parameters are listed, followed by an arror
the body comes after the arrow
What makes a lambda different from a traditional function is that it doesn’t have a name. In the expression above, we assigned the lambda to a variable, which we could them use to reference it, but the function itself isn’t named.
Note that due to type inference, we could rewrite this example without the type specified on the LHS. This is the same thing!
In cases where there’s only a single parameter for a lambda, you can omit the parameter name and the arrow. When you do this, Kotlin will automatically make the name of the parameter it.
val applyDiscount: (Double) -> Double = { it - 5.0 }
Lambdas and Higher-Order Functions
Passing Lambdas as Arguments
Higher-order functions have a function as an input or output. We can rewrite our earlier earlier example to use lambdas instead of function references:
// fun discountFiveDollars(price: Double): Double = price - 5.0
// fun discountTenPercent(price: Double): Double = price * 0.9
// fun noDiscount(price: Double): Double = price
funcalculateTotal(initialPrice: Double, applyDiscount: (Double) -> Double): Double {
val priceAfterDiscount = applyDiscount(initialPrice)
val total = priceAfterDiscount * taxMultiplier
return total
}
val withFiveDollarsOff = calculateTotal(20.0, { price - 5.0 }) // $16.35
val withTenPercentOff = calculateTotal(20.0, { price * 0.9 }) // $19.62
val fullPrice = calculateTotal(20.0, { price }) // $21.80
In cases where function’s last parameter is a function type, you can move the lambda argument outside of the parentheses to the right, like this:
The Kotlin standard library contains several functions whose sole purpose is to execute a block of code on an object. When you call such a function on an object with a lambda expression, it forms a temporary scope, and applies the lambda to that object.
There are five of these scope functions: let, run, with, apply, and also, and each of them has a slightly different purpose.
Here’s an example where we do not use one of these scope functions. There is a great deal of repetition, since we need a temporary variable, and then have to act on that object.
val alice = Person("Alice", 20, "Amsterdam")
println(alice)
alice.moveTo("London")
alice.incrementAge()
println(alice)
With a scope function, we can refer to the object without using a name. This is greatly simplified!
The scope functions have subtle differences in how they work, summarized from the Kotlin Standard Library documentation. Inside the lambda of a scope function, the context object is available by a short reference instead of its actual name. Each scope function uses one of two ways to access the context object: as a lambda receiver (this) or as a lambda argument (it).
The context object is available as an argument (it). The return value is the lambda result.
let can be used to invoke one or more functions on results of call chains. For example, the following code prints the results of two operations on a collection:
val numbers = mutableListOf("one", "two", "three", "four", "five")
val resultList = numbers.map { it.length }.filter { it > 3 }
println(resultList)
With let, you can rewrite it:
val numbers = mutableListOf("one", "two", "three", "four", "five")
numbers.map { it.length }.filter { it > 3 }.let {
println(it)
// and more function calls if needed
}
A non-extension function: the context object is passed as an argument, but inside the lambda, it’s available as a receiver (this). The return value is the lambda result.
We recommend with for calling functions on the context object without providing the lambda result. In the code, with can be read as “with this object, do the following.”
val numbers = mutableListOf("one", "two", "three")
with(numbers) {
println("'with' is called with argument $this")
println("It contains $size elements")
}
The context object is available as a receiver (this). The return value is the lambda result.
run does the same as with but invokes as let - as an extension function of the context object.
run is useful when your lambda contains both the object initialization and the computation of the return value.
val service = MultiportService("https://example.kotlinlang.org", 80)
val result = service.run {
port = 8080 query(prepareRequest() + " to port $port")
}
// the same code written with let() function:
val letResult = service.let {
it.port = 8080it.query(it.prepareRequest() + " to port ${it.port}")
}
The context object is available as a receiver (this). The return value is the object itself.
Use apply for code blocks that don’t return a value and mainly operate on the members of the receiver object. The common case for apply is the object configuration. Such calls can be read as “apply the following assignments to the object.”
val adam = Person("Adam").apply {
age = 32 city = "London"}
println(adam)
Having the receiver as the return value, you can easily include apply into call chains for more complex processing.
The context object is available as an argument (it). The return value is the object itself.
also is good for performing some actions that take the context object as an argument. Use also for actions that need a reference to the object rather than its properties and functions, or when you don’t want to shadow the this reference from an outer scope.
When you see also in the code, you can read it as “and also do the following with the object.”
val numbers = mutableListOf("one", "two", "three")
numbers
.also { println("The list elements before adding new one: $it") }
.add("four")
Collection Functions
Collection classes (e.g. List, Set, Map, Array) have built-in functions for working with the data that they contain. These functions frequently accept other functions as parameters.
filter produces a new list of those elements that return true from a predicate function.
val list = (1..100).toList()
val filtered = list.filter { it % 5==0 }
// 5 10 15 20 ... 100
val below50 = filtered.filter { itin0..49 }
// [5, 10, 15, 20]
map produces a new list that is the results of applying a function to every element that it contains.
val list = (1..100).toList()
val doubled = list.map { it * 2 }
// 2 4 6 8 ... 200
reduce accumulates value starting with the first element and applying an operation to each element from left to right.
We also have helper functions to extract specific elements from a list.
take returns a collection containing just the first n elements. drop returns a new collection with the first n elements removed.
val list = (1..50)
val first10 = list.take(10)
// 1 2 3 ... 10
val last40 = list.drop(10)
// 11 12 13 ... 50
first and last return those respective elements. slice allows us to extract a range of elements into a new collection.
val list = (1..50)
val even = list.filter { it % 2==0 }
// 2 4 6 8 10 ... 50
even.first() // 2
even.last() // 50
even.slice(1..3) // 4 6 8
Lazy Sequences
Lazy evaluation allows us to generate expressions representing large or infinite lists, and work on them without actually evaluating every element . For example, we can generate an infinite sequence and then extract the first n elements that we need.
// generate an infinite list of integers
// starting at zero, step 10
val list = generateSequence(0) { it + 10}
// 0 10 20 30 40 50 60 70 80 ...
val results = list.take(5).toList()
// 0 10 20 30 40
take from this list before attempting to do anything with it. It’s infinite so it’s possible to hang your system if you’re not careful2.
val list = generateSequence(0) { it + 10}
val results = list.drop(5).toList() // length is infinite - 5 ?!?
Chaining operations
Since our higher-order functions typically return a list, we can chain operations together, so the return value of one function is a list, which is acted on by the next function in the chain. For example, we can map and filter a collection without needing to store the intermediate collection.
val list = (1..999999).toList()
val results = list
.map { it * 2 }
.take(10)
// [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
The operations are performed in top-down order: map, then take. In this case, it means that we’re mapping the entire list and then discarding most of the resulting list with the take operation. This is really inefficient: filter your list first!
// better implementation
val veryLongList = listOf(0..9999999L).toList()
val results = veryLongList
.take(50)
.map { it * 2 }
// [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
Recursion
As a hybrid language, Kotlin supports a number of paradigms. Recursion is less likely than other languages, given that we have loops and other mechanisms to handle iteration.
However, the compiler certainly supports recursion, and can even optimize for tail recursion. To qualify, a function needs to:
be structured so that the last statement is a call to the function, with state being passed in the function call.
Other popular styles being imperative and object-oriented programming. ↩︎
Yes I tried this! The result was a runtime memory error: “Exception in thread “main” java.lang.OutOfMemoryError: Java heap space”. ↩︎
Idiomatic Kotlin
Urs Peters: Idiomatic Kotlin
This section summarizes a talk by Urs Peters, presented on Kotlin Dev Day 2022. No claim of authorship is intended – it’s a very interesting talk and worth watching in it’s entirety!
https://www.youtube.com/watch?v=zYH6zTtl-nc
Why Idiomatic Kotlin?
It’s possible to use Kotlin as a “better Java”, but you would be missing out on some of the features that make Kotlin unique and interesting.
Principles
1. Favor immutability over mutability.
Kotlin favors immutability by providing various immutable constucts and defaults.
dataclassProgrammer(val name: String,
val languages: List<String>)
funknown(language:String) = languages.contains(language)
val urs = Programmer("Urs", listOf("Kotlin", "Scale", "Java"))
val joe = urs.copy(name = "Joe")
What is so good about immutability?
Immutability: exactly one state that will never change.
Mutable: an infinite amount of potential states.
Criteria
Immutable
Mutable
Reasoning
Simple: one state only
Hard: many possible states
Safety
Safer: state remains the same and valid
Unsafe: accidental errors due to state changes
Testability
No side effects which makes tests deterministic
Side effects: can lead to unexpected failures
Thread-safety
Inherently thread-safe
Manual synchronization required
How to leverage it?
prefer vals over vars
prefer read-only collections (listOf instead of mutableListOf)
use immutable value objects instead of mutable ones (e.g. data classes over classes)
Local mutability that does not leak outside is ok (e.g. a var within a function is ok if nothing external to the function relies on it).
2. Use Nullability
Think twice before using !!
val uri = URI("...")
val res = loadResource(uri)
val lines = res!!read() // bad!
val lines = res?.read() ?:throw IAE("$uri invalid") // more reasonable
Stick to nullable types only
public Optional<Goody> findGoodyForAmount(amount:Double)
val goody = findGoodyForAmount(100)
if(goody.isPresent()) goody.get() ... else... // bad
val goody = findGoodyForAmount(100).orElse(null)
if(goody !=null) goody ... else... // good uses null consistently
Use nullability where applicable but don’t overuse it.
dataclassOrder(
val id: Int? = null,
val items: List<LineItem>? = null,
val state: OrderState? = null,
val goody: Goody? = null) // too much!
dataclassOrder(
val id: Int? = null,
val items: List<LineItem> = emptyList()),
val state: OrderState = UNPAID,
val goody: Goody? = null) // some types made more sense as not-null values
Avoid using nullable types in Collections
val items: List<LineItem?> = emptyList()
val items: List<LineItem>? = null,
val items: Lilst<LineItem?>? = null// all terribad
val items: List<LineItem> = emptyList() // that's what this is for
Use lateinit var for late initialization rather than nullability
Use immutable data classes for value classes, config classes etc.
classPerson(val name: String, val age: Int)
val p1 = Person("Joe", 42)
val p2 = Person("Joe", 42)
p1 == p2 // false
dataclassPerson(val name: String, val age: Int)
val p1 = Person("Joe", 42)
val p2 = Person("Joe", 42)
p1 == p2 // true
Use normal classes instead of data classes for services etc.
classPersonService(val dao: PersonDao) {
funcreate(p: Person) {
if (op.age >= MAX_AGE)
LOG.warn("$p${bornInYear(p.age)} too old")
dao.save(p)
}
companionobject {
val LOG = LogFactory.getLogger()
val MAX_AGE = 120funbornInYear(age: Int) = ...
}
}
Use value classes for domain specific types instead of common types.
valueclassEmail(val value: String)
valueclassPassword(val value: String)
funlogin(email: Email, pwd: Password) // no performance impact! type erased in bytecode
Seal classes for exhaustive branch checks
// problematic
dataclassSquare(val length: Double)
dataclassCircle(val radius: Double)
when (shape) {
is Circle ->"..."is Rectangle ->"..."else->throw IAE("unknown shape $shape") // annoying
}
// fixed
sealedinterfaceShape// prevents additions
dataclassSquare(val length: Double)
dataclassCircle(val radius: Double)
when (shape) {
is Circle ->"..."is Rectangle ->"..."}
4. Use Available Extensions
// bad
val fis = FileInputStream("path")
val text = try {
val sb = StringBuilder()
var line: String?
while(fis.readLine().apply {line = this} !=null) {
sb.append(line).append(System.lineSeparator())
}
sb.toString()
} finally {
try { fis.close() } catch (ex:Throwable) { }
}
// good, via extension functions
val text = FileInputStream("path").use { it.reader().readText() }
Extend third party classes
// bad
funtoDateString(dr: LocalDateTime) = dt.format(DateTimeFormatter.ofPattern("yyyy-MM-dd"))
// good, works with code completion!
funLocalDateTime.toDateString() = this.format(DateTimeFormatter.ofPattern("yyyy-MM-dd"))
5. Use Control-Flow Appropriately
Use if/else for single branch conditions rather than when
// too verbose
val reduction = when {
customer.isVip() ->0.05else->0.0}
// better
val reduction = if (customer.isVip()) 0.05else0.0
This is better because it results in more concise, deterministic, more easily testable and clearly scoped code that is easy to reason about compared to the imperative style.
if/else is an expression returning a result.
// imperative style
var result: String
if(number % 2==0)
result = "EVEN"else result = "ODD"// expression style, better
val result = if(number % 2==0) "EVEN"else"ODD"
when is an expression too, returning a result.
// imperative style
var hi: String
when(lang) {
"NL"-> hi = "Goede dag""FR"-> hi = "Bonjour"else-> hi = "Good day"}
// expression style, better
val hi = when(lang) {
"NL"->"Goede dag""FR"->"Bonjour"else->"Good day"}
try/catch also.
// imperative style
var text: String
try {
text = File("path").readText()
} catch (ex: Exception) {
text = ""}
// expression style, better
val text = try {
File("path").readText()
} catch (ex: IOException) {
""}
Most functional collections return a result, so the return keyword is rarely needed!
Program on a higher abstraction level with (chained) higher-order functions from the collection.
// bad!
val kids = mutableSetOf<Person>()
for(person in persons) {
if(person.age < 18) kids.add(person)
}
names.sorted()
// better!
val kids: mutableSetOf<Person> = persons.filter{ it.age < 18}
You are actually programming at a higher-level of abstraction, since you’re manipulating the collection directly instead of considering each of its elements. e.g. it’s obvious in the second example that we’re filtering, instead of needing to read the implementation to figure it out.
For readability, write multiple chained functions from top-down instead of left-right.
// bad!
val names = mutableSetOf<String>()
for(person in persons) {
if(person.age < 18)
names.add(person.name)
}
names.sorted()
// better!
val names = persons.filter{ it.age < 18}
.map{ it.name }
.sorted()
Use intermediate variables when chaining more than ~3-5 operators.
// old way
funclient(): RestClient {
val client = RestClient()
client.username = "xyz" client.secret = "secret" client.url = "https://..../employees"return client
}
// better way
funclient() = RestClient().apply {
username = "xyz" secret = "secret" url = "https://..../employee"}
Use let/run to manipulate the context object and return a different type.
// old way
val file = File("/path")
file.setReadOnly(true)
val created = file.createNewFile()
// new way
val created = File("/path").run {
setReadOnly(true)
createNewFile() // last expression so result from this function is returned
}
Use also to execute a side-effect.
// old way
if(amount <=0) {
val msg = "Payment amount is < 0"LOGGER.warn(msg)
throw IAE(msg)
} else...
// new way
require(amount > 0) {
"Payment amount is < 0".also(LOGGER::warn)
}
9. Embrace Coroutines
// No coroutines
// Using Mono from Sprint React and many combinators (flatMap)
// Standard constructs like if/else cannot be used
// Business intent cannot be derived from this code
@PutMapping("/users")
@ResponseBodyfunupsertUser(@RequestBody user: User): Mono<User> =
userByEmail(user.email)
.switchIfEmpty{
verifyEmail(user.email).flatMap{ valid ->if(valid) Mono.just(user)
elseMono.error(ResponseStatusException(BAD_REQUEST, "Bad Email"))
}
}.flatMap{ toUpsert -> save(toUpsert) }
// Coroutines clean this up
// Common language constructs can be used
// Reads like synchronous code
@PutMapping("/users")
@ResponseBodyfunupsertUser(@RequestBody user: User): User =
userByEmail(user.email).awaitSingle() ?:if (verifyEmail(user.email)) user elsethrow ResponseStatusException(BAD_REQUEST, "Bad Email")).let{
toUpsert -> save(toUpsert)}
Project Loom will (eventually) result in support for coroutines running on the JVM. This will greatly simplify running coroutines.
Chapter 6
Building Software
Software development topics related to building software. Why we build software, and desireable features of our software applications. Special-topics.
Subsections of Building Software
Introduction
What is application software?
Software is the set of programs, concepts, tools, and methods used to produce a running system on computing devices 1
We often think of software as a program that we execute; you’ve probably already written a large number of small programs. Every computer system consists of hundreds or thousands of very specialized, small-to-medium sized programs working in harmony to deliver a functioning system. No single program exists in isolation, and any programs that we write and execute depend on a number of other programs for their creation and operation. When we talk about “software”, we’re talking about a very broad category that includes:
System software: programs written to provide services to other software. This includes operating systems, drivers, compilers and similar systems.
Application software: programs written primarily for users, to help them solve a problem or perform a task, typically by manipulating and processing information. This includes text editors, messaging clients, productivity software, video games, graphing tools, computer-aided design and so on. This can span desktop, mobile, or even web-based applications.
In this course, we’re going to focus on application software, or software designed to solve tasks for people. This includes work and productivity software, games or other types of useful programs that people use everyday. e.g. software to manipulate numbers (Excel), compose documents (Word), write code (Vim) or entertain ourselves (League of Legends).
Although we will be writing self-contained software, we’ll also talk about full stack software.
Full-stack is an overloaded term in software development. Historically, it referred to the idea that an application often has a “front-end” (user interface) and a “back-end” (server, or process performing the actual computation). A full-stack developer in that context is someone who is capable of working on both “sides” of this boundary: the graphical front-end, and the back-end that does the “heavy lifting” of the application.
In modern-usage, we similarly use this term to refer to the ability to write code at different levels of the architectural stack. e.g. web-developers might use a mix of HTML/CSS/JS on the client-side and a more robust language like Java for a remote service. Modern applications often consist of multiple front-ends servicing different platforms e.g. desktop and mobile, and back-end services that provide data and other capabilities to these application layers.
Full-stack application development refers to designing and building applications that offer client-side functionality, but that can also leverage back-end services as required.
We’ll build full-stack applications in this course. In particular, we’ll build standalone applications to start, and then extend their functionality to leverage a remote service.
The role of software development?
“Software development refers to a set of computer science activities dedicated to the process of creating, designing, deploying and supporting software."
– IBM Research
Software development then is the complete process of taking a software system from inception, through design, implementation and ultimately delivery to a customer. Software development consists of many interrelated activities across different discipline.
This is a course on software development in this broad-sense: going from initial requirements and working through the process to deliver a final product that someone can use and enjoy. Although we will focus most of our attention on the design + implementation of software, we want to consider how to address these other concerns as well.
Software is only useful if it solves a problem for someone. We cannot develop software in a vacuum.
We’ll consider a more formal process to developer software in the next chapter.
Software is multidisciplinary
Software development can be seen as the convergence of math and computer science, engineering, arts and creative design. It’s inheritently multidisciplinary. It’s a unique discipline that can be creative and expressive, but still require formal mathematical rigor.
One central theme to this course is that there can be many different ways of determining what to build, or how to design something, or what the code should look like, based on your perspective.
Imagine that you’re part of a team building a product. What you consider important will be determined by your role, and your imemdiate concerns.
A designer might be concerned with the usability of a system for its users, and care about the user experience, aesthetics, branding and so on.
A programmer, trained in Computer Science might be concerned about functionality, scalability (designing a solution that can grow to handle more data), performance, readability of the code and so on.
A manager or project manager might be concerned with delivering a product on-time, and not going over budget.
A software engineer might be concerned with the quality of the product that’s being delivered, and the process that we use to ensure this.
These are all valid perspectives! Part of the challenge for you is knowing how to address all of these needs.
Info
“How to program correctly” is a contentious topic. There is rarely/never a definitive practice that everyone agrees with and uses consistently. Part of the reason for this is that we are often working in new domains, and solving new or unique problems. Part of the reason is also probably because there are many, many different perspectives that emphasize the unique needs of the writer. How do you deal this? Look beyond the surface and don’t be afraid to use what works for you in your particular situation.
Why is software important?
The growth of software systems has easily been the most pervasive technological advancement of the past 50 years. Software underpins most industries: banking, finance, transportation, manufacturing, research, education, retail. It is practically impossible to live and work in modern society without interacting with software systems on a daily basis2.
Keeping pace with the speed of technological advancements is challenging. The reality of software today3:
Software has become embedded in virtually every aspect of our lives. The number of people who have an interest in the features and functions provided by a specific application has grown dramatically. Although software was once the domain of experts, this is no longer the case, as software is routinely used by consumers. We should strive to build software to meet the needs of all people.
The information technology requirements demanded by individuals, businesses, and governments grow increasingly complex with each passing year. Large teams of people now create computer programs. Sophisticated software that was once implemented in a predictable, self-contained computing environment is now embedded inside everything from consumer electronics to medical devices to autonomous vehicles. Software is complex and pervasive. Design is a critical activity to ensure that we address requirements and build the “right thing”.
Individuals, businesses, and governments increasingly rely on software for strategic and tactical decision making as well as day-to-day operations and control. If the software fails, people and major enterprises can experience anything from minor inconvenience to catastrophic consequences. Software should be high-quality, safe and reliable.
As the perceived value of a specific application grows, the likelihood is that its user base and longevity will also grow. As its user base and time in use increase, demands for adaptation and enhancement will also grow. Software should be adaptable and scalable.
I’m writing this on a notebook computer, while wearing a smartwatch, with a smartphone in my pocket, a tablet on the table beside me and my speaker streaming music over the Internet. The amount of computing power at our disposal is astonishing. Also, I have too many gadgets. ↩︎
A console application (aka “command-line application”) is an application that is intended to run from within a shell, such as bash, zsh, or PowerShell, and uses text and block characters for input and output. This style of application dominated through the early days of computer science, until the 1980s when graphical user interfaces became much more common.
Console applications use text for both input and output: this can be as simple as plan text displayed in a window (e.g. bash), to systems that use text-based graphics and color to enhance their usability (e.g. Midnight Commander). This application style was really driven by the technical constraints of the time. Software was written for text-based terminals, often with very limited resources, and working over slow network connections. Text was faster to process, transmit and display than sophisticated graphics.
Some console applications remain popular, often due to powerful capabilities or features that are difficult to replicate in a graphical environment. Vim, for instance, is a text editor that originated as a console application, and is still used by many developers. Despite various attempts to build a “graphical vim”, the console version is often seen as more desireable due to the benefits of console applications i.e. faster, reduced latency, small memory footprint and ability to easily leverage other command-line tools.
Console applications often favor ‘batch-style processing‘, where you provide arguments to the program, and it executes a single task before exiting. This is very much due to the Unix Philosophy from the 70s, where this interaction style dominated:
Make each program do one thing well. To do a new job, build fresh rather than complicate old programs by adding new “features”.
Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.
Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.
Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them."
– Bell Systems Technical Journal (1978)
Although we tend to run graphical operating systems with graphical applications, console applications are still very common. e.g. Windows, macOS, Linux all ship with consoles and powerful tools that are common used, at ledast by a subset of users. e.g. ls, curl, wget, emacs, vim, git and so on.
For expert users in particular, this style has some advantages. Console applications:
can easily be scripted or automated to run without user intervention.
can redirect input and output using standard IO streams, to allow interaction with other console applications that support this standard.
tend to be small and performant, due to their relatively low-overhead (i.e. no graphics, sound, other application overhead).
The disadvantage is the steep learning curve, and lack of feature discoverability.
Design
Architecture
Command-line applications tend towards performing a single action and then exiting. This lends itself to a pipeline architecture, where processing steps are performed on intput, in order, to produce an output. This is also called a pipes and filters design pattern.
Examples of how this pattern might be applied:
A file rename utility that changes the date on one or more files that are supplied as arguments. The program would (1) process and validate the command-line arguments, (2) apply the rule to produce a set of target output files (likely original:revise filename pairs) and then (3) perform these operations. Any interruption or error would cause the rename to abort.
An image processing utility that attempts to filter a photo. Filters are applied in sequence on the input image producing a final output image.
Of course, not every command-line program works like this; there are some that are interactive, where the user provides successive input while the program is running. A program like this could use a mode traditional event-driven architecture:
Examples of this might include a text editor like Vi. It can certainly process command-line arguments, but it primarily operates in an interactive mode, where it waits for, and acts upon, user input (where keystrokes are represented as events).
Lifecycle
Command-line applications expect a single-entry point: a method with this familiar signature:
funmain(args:Array<String>) {
// code here
System.exit(0)
}
When your program is launched it executes this method. When it reaches the end of the method, the program stops executing. It’s considered “good form” to call System.exit(0) where the 0 is an error code returned to the OS. In this case, 0 means no errors i.e. a normal execution.
Input should be handled through either (a) arguments supplied on the command-line, or (b) values supplied through stdin. Batch-style applications should be able to run to completion without prompting the user for more input.
Output should be directed to stdout. You’re typically limited to textual output, or limited graphics (typically through the use of Unicode characters).
Errors should typically be directed to stderr.
As suggested above, we want to use a standard calling convention. Typically, command-line applications should use this format, or something similar:
$ program_name -option=value parameter
program_name is the name of your program. It should be meaningful, and reflect what your program does.
options represent a value that tells your program how to operate on the data. Typically options are prefixed with a dash (”-”) to distinguish them from parameters. If an option also requires a value (e.g. ”-bg=red”) then separate the option and value with a colon (”:”) or an equals sign (”=”).
parameter represents input, or data that your program would act upon (e.g. the name of a file containing data). If multiple parameters are required, separate them with whitespace (e.g. ”program_name parameter1 parameter2”).
The order of options and parameters should not matter.
Running a program with insufficient arguments should display information on how to successfully execute it.
$ rename
Usage: rename [source][dest]
Features
Command-line applications should have the following standard features:
Use IO Streams: your application should handle all input through stdin, channel output to stdout and errors to stderr. This ensures that it will work as-expected with other command-line programs.
Support conventions: The “target” (e.g. filename on which to operate) is usually provided as the primary argument. To disambiguate other input, it is normal to provide additional information using dashes using 1. For example, it is standard to use --help to display brief help that demonstrates how to use your application.
Provide user feedback for errors: your program should never print out “successful” messages. Save user feedback for errors. Error messages should be clear and help the user figure out what went wrong, or what they need to do to fix the error.
Typical command-line interaction is shown below:
% exa --help
Usage:
exa [options][files...]META OPTIONS
-?, --help show list of command-line options
-v, --version show version of exa
DISPLAY OPTIONS
-1, --oneline display one entry per line
-l, --long display extended file metadata as a table
-G, --grid display entries as a grid (default) -x, --across sort the grid across, rather than downwards
-R, --recurse recurse into directories
-T, --tree recurse into directories as a tree
-F, --classify display type indicator by file names
...
% exa -T
.
├── 01.syllabus.md
├── 02.introduction.md
├── 03.software_process.md
├── 04.sdlc.md
├── 05.architecture.md
├── 06.development.md
├── 07.testing.md
├── 08.kotlin_primer.md
├── 09.building_desktop.md
├── 10.building_mobile.md
├── 11.building_libraries.md
├── 12.building_services.md
├── 13.multiplatform.md
├── 99.unused.md
├── assets
│ ├── 2_tier_architecture.png
│ ├── 3_tier_architecture.png
│ ├── abstract_factory.png
│ ├── activity_lifecycle.png
...
Processing arguments
The main() method can optionally accept an array of Strings containing the command-line arguments for your program. To process them, you can simply iterate over the array.
This is effectively the same approach that you would take in C/C++.
fun main(args: Array<String>){ // args.size returns number of items
for(s in args){ println("Argument: $s")}}
Info
This is a great place to use the command design pattern to abstract commands. See the public repo for samples.
Reading/writing text
The Kotlin Standard Library (“kotlin-stdlib“) includes the standard IO functions for interacting with the console.
readLine() reads a single value from “stdin“
println() directs output to “stdout“
code/ucase.kts
// read single value from stdin
val str:String ?= readLine()
if (str !=null) {
println(str.toUpperCase())
}
It also includes basic methods for reading from existing files.
import java.io.*
var filename = "transactions.txt"// read up to 2GB as a string (incl. CRLF)
val contents = File(filename).readText(Charsets.UTF_8)
println(contents)
2020-06-06T14:35:44, 1001, 78.22, CDN
2020-06-06T14:38:18, 1002, 12.10, CDN
2020-06-06T14:42:51, 1003, 44.50, CDN
Example: rename utility
Let’s combine these ideas into a larger example.
Requirements: Write an interactive application that renames one or more files using options that the user provides. Options should support the following operations: add a prefix, add a suffix, or capitalize it.
We need to write a script that does the following:
Extracts options and target filenames from the arguments.
Checks that we have (a) valid arguments and (b) enough arguments to execute program properly (i.e. at least one filename and one rename option).
For each file in the list, use the options to determine the new filename, and then rename the file.
Usage: rename.kts [option list] [filename]
For this example, we need to manipulate a file on the local file system. The Kotlin standard library offers a File class that supports this. (e.g. changing permissions, renaming the file).
Construct a ‘File‘ object by providing a filename, and it returns a reference to that file on disk.
funmain(args: Array<String>) {
val files = getFiles(args)
val options = getOptions(args)
// check minimum required arguments
if (files.size ==0|| options.size ==0) {
println("Usage: [options] [file]")
} else {
applyOptions(files, options)
}
fungetOptions(args:Array<String>): HashMap<String, String> {
var options = HashMap<String, String>()
for (arg in args) {
if (arg.contains(":")) {
val(key, value) = arg.split(":")
options.put(key, value)
}
}
return options
}
fungetFiles(args:Array<String>) : List<String> {
var files:MutableList<String> = mutableListOf()
for (arg in args) {
if (!arg.contains(":")) {
files.add(arg)
}
}
return files
}
funapplyOptions(files:List<String>,options:HashMap<String, String>) {
for (filein files) {
var rFile = file// capitalize before adding prefix or suffix
if (options.contains("prefix")) rFile = options.get("prefix") + rFile
if (options.contains("suffix")) rFile = rFile + options.get("suffix")
File(file).renameTo(rFile)
println(file + " renamed to " + rFile)
}
}
Reading/writing binary data
These examples have all talked about reading/writing text data. What if I want to process binary data? Many binary data formats (e.g. JPEG) are defined by a standard, and will have library support for reading and writing them directly.
Kotlin also includes object-streams that support reading and writing binary data, including entire objects. You can, for instance, save an object and it’s state (serializing it) and then later load and restore it into memory (deserializing it).
classEmp(var name: String, var id:Int) : Serializable {}
var file = FileOutputStream("datafile")
var stream = ObjectOutputStream(file)
var ann = Emp(1001, "Anne Hathaway", "New York")
stream.writeObject(ann)
Cursor positioning
ANSI escape codes are specific codes that can be “printed” by the shell but will be interpreted as commands rather than output. Note that these are not standard across consoles, so you are encouraged to test with a specific console that you wish to support.
Packaging just refers to putting together your software components in a way that you can distribute to your end-user. As we’ve seen, this could be physical (a retail box including a DVD), or a digital image (ISO image, package file, installer executable). It’s the “bundling” part of building software.
Packaging is also one of those tasks that we tend to hand wave: just compile, link and hand-out the executable right? Unfortunately it’s not that simple. Apart from the executable, your application may also include images, sound clips and other resources, plus external libraries that you don’t necessarily want to statically link into your executable. You need to distribute all of these files, and ensure that they are installed in the correct location, registered with the OS and made usable by your application. You also need to perform steps to ensure that your application respects conventions of the particular environment: installing under Program Files in Windows, putting libraries under Windows\\System32, putting an icon on the desktop and so on.
The way we address the compexities of packaging is to break down the process into multiple steps, which are handled by different tools:
Step
Explanation
Step 1: Compiling classes
Use the kotlinc compiler to create classes from our source code.
Step 2: Creating archives
Use the jar command to create jar files of classes and related libraries.
Step 3: Creating scripts
Optionally, create scripts to allow a user to execute directly from the JAR files.
In the next sections, we’ll demonstrate using each of these tools.
1. Compiling classes
Let’s use the HelloWorld sample to demonstrate different methods for preparing, and then packaging our application.
To compile from the command-line, we can use the Kotlin compiler, kotlinc. By default, it takes Kotlin source files (.kt) and compiles them into corresponding class files (.class) that can be executed on the JVM.
We could also just do this with IntelliJ IDEA or Android Studio, where Gradle - build - build will generate class files. As we will see at the end of this section, we can often just generate the final installer in a single step without doing each step manually.
2. Creating archives
Class files by themselves are difficult to distribute. The Java platform includes the jar utility, used to create a single archive file from your classes.
A JAR file is a standard mechanism in the Java ecosystem for distributing applications. It’s effectively just a compressed file (just like a ZIP file) which has a specific structure and contents. Most distribution mechanisms expect us to create a JAR file first.
This example compiles Hello.kt, and packages the output in a Hello.jar file.
$ kotlinc Hello.kt -include-runtime -d Hello.jar
$ ls
Hello.jar Hello.kt
The -d option tells the compiler to package all of the required classes into our jar file. The -include-runtime flag tells it to also include the Kotlin runtime classes. These classes are needed for all Kotlin applications, and they’re small, so you should always include them in your distribution (if you fail to include them, you app won’t run unless your user has Kotlin installed).
HelloKt.class – a class wrapper generated by the compiler
META-INF/MANIFEST.MF – a file containing metadata.
kotlin/ – Kotlin runtime classes not included in the JDK.
The MANIFEST.MF file is autogenerated by the compiler, and included in the JAR file. It tells the runtime which main method to execute. e.g. HelloKt.main().
In IntelliJIDEA, Gradle - build - jar will create a jar file. As we will see at the end of this section, we can often just generate the final installer in a single step without doing each step manually.
3. Creating scripts
We can distribute JAR files like this to our users, but they’re awkward: users would need to have the Java JDK installed, and typejava -jar filename.jar to actually run our programs. For some scenarios this might be acceptable (e.g. launching a service from an init script), but most of the time we want a more user-friendly installation mechanism.
The simplest thing we can do it create a script to launch our application from a JAR file, with the same effect as executing from the command-line:
For simple applications, especially ones that we use ourselves, this may be sufficient. It has the downside of requiring the user to have the Java JDK installed on their system3. This particular script is also not very robust: it doesn’t set runtime parameters, and doesn’t handle more than a single JAR file.
For a more robust script, we can let Gradle generate one for us. In IntelliJIDEA, Gradle - distribution - distZip will create a zip file that includes a custom runtime script. Unzip it, and it will contain a lib directory with all of the JAR files, and a bin directory with a script that will run the application from the lib directory. Copy both to a new location and you can execute the app from there.
Here’s an example of the Archey project, with a distZip file that was generated by Gradle.
The bin/app script is quite robust, and will handle many different types of system configurations. It will also handle setting the CLASSPATH for all of the libraries that are included in the lib directory!
Here’s a portion of the app script. You don’t want to have to write this by-hand!
# OS specific support (must be 'true' or 'false').cygwin=false
msys=false
darwin=false
nonstop=false
case"$( uname )" in #( CYGWIN* ) cygwin=true ;; #( Darwin* ) darwin=true ;; #( MSYS* | MINGW* ) msys=true ;; #( NONSTOP* ) nonstop=true ;;
esacCLASSPATH=$APP_HOME/lib/app.jar:$APP_HOME/lib/kotlin-stdlib-jdk8-1.5.31.jar:$APP_HOME/lib/guava-30.1.1-jre.jar:$APP_HOME/lib/kotlin-stdlib-jdk7-1.5.31.jar:$APP_HOME/lib/kotlin-stdlib-1.5.31.jar:$APP_HOME/lib/kotlin-stdlib-common-1.5.31.jar:$APP_HOME/lib/failureaccess-1.0.1.jar:$APP_HOME/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:$APP_HOME/lib/jsr305-3.0.2.jar:$APP_HOME/lib/checker-qual-3.8.0.jar:$APP_HOME/lib/error_prone_annotations-2.5.1.jar:$APP_HOME/lib/j2objc-annotations-1.3.jar:$APP_HOME/lib/annotations-13.0.jar
# Determine the Java command to use to start the JVM.if[ -n "$JAVA_HOME"] ; thenif[ -x "$JAVA_HOME/jre/sh/java"] ; then# IBM's JDK on AIX uses strange locations for the executables JAVACMD=$JAVA_HOME/jre/sh/java
else JAVACMD=$JAVA_HOME/bin/java
fiif[ ! -x "$JAVACMD"] ; then die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
The script works well. Here’s archey being executed from the script:
$ build/distributions/app/bin/app
#### User: jaffe### Home: /Users/jaffe####### ####### Name: Bishop.local###################### OS: Mac OS X##################### Version: 11.3#################### CPU: x86_64#################### Cores: 10##################### Free Memory: 514 MB###################### Total Memory: 520 MB#################### Disk Size: 582 GB################ Free Space: 582 GB#### ##### IP Address: 127.0.0.1
Toolkits
In some ways, a terminal-based program is really just an extension of a line printer. It’s designed to output to stdout one line at a time, and doesn’t handle editing in-place very well (which would be required if you wanted to create a terminal-based UI, like Midnight Commander).
There are specific libraries that have been developed to provide more sophisticated capabilities. Here’s a couple of tile-based toolkits that allow you to build up a full UI in the console:
Mosaic is a Kotlin toolkit for building tiled applications e.g. Rogue-like games.
Thse toolkits really serve a rare circumstance, where you want graphical capabilities but you’re running in a non-graphical environment. Typically if you’re running a modern OS, you have more sophisticated graphical capabilities available.
Here’s a console toolkit that provides further capabilities.
There is precedent for either single or double-dashes, but be consistent in your program. ↩︎
In the same way that the Java compiler compiles Java code into IR code that will run on the JVM, the Kotlin compiler also compiles Kotlin code into compatible IR code. The java command will execute any IR code, regardless of which programming language and compiler was used to produce it. ↩︎
When Java was first introduced, the assumption was that every platform would have the JVM pre-installed, and we could just use that. Unfortunately, we often needed a specific version of the JVM, so companies ended up distributing and installing their own JVM alongside their applications. Not a great situation. ↩ ↩︎
Desktop Applications
Overview
Graphical applications arose in the early 80s as we moved from text-based terminals to more technically capable systems. This was part of the Personal Computer (PC) movement of that time, which aimed to put a “computer in every home” Graphical User Interfaces (GUIs) were seen as more “user-friendly” and considered an important factor in the adoption of these systems. Introduced in 1984, the Apple Macintosh introduced the first successful commercial graphical operating system; other vendors (e.g. Microsoft, Commodore, Sun) quickly followed suit. The conventions that were adopted on the Mac became standard on other platforms as well e.g. Windows.
Desktop applications refers to graphical applications designed for a notebook or desktop computer, typically running Windows, macOS or Linux. Users interact with these applications using a mouse and keyboard, although other devices may also be supported (e.g. camera, trackpad).
Although desktop computers can run console applications in a shell, for this discussion, we’re focusing on graphical applications.
Features
Graphical desktop application should have the following features:
Multiple application windows should be supported. Most applications will often present their interface within a single, interactive window, but it can sometimes be useful to have multiple simultaneous windows controlled by a single application1.
Support for full-screen or windowed interaction: although graphical applications tend to run windowed, they should normally be usable full-screen as well. The window contents should scale or reposition themselves as the window size changes.
Window decorations: Each window should have a titlebar, minimize/maximize/restore buttons (that work as expected).
Windows may or may not be resizable: if they are resizable, the contents should scale or adjust their layout based on window size (for this reason, it may make sense to either contrain window dimensions when resizing, or make some windows fixed size). Convention allows the main window to be resized, and option dialogs (or similar non-essential windows) to be fixed-size.
Interactive graphical elements: window contents could be any combination of graphics, animations, multimedia, or text that is desired for the target application. These contents should be dynamic (i.e. have the ability to change in response to system state) and should support a range of interactions - clicking, double-clicking, dragging - provided by both mouse and keyboard.
Standard menubars: every application should have the following menus (with shortcuts). Although some applications choose to eliminate menus (or replace with other controls), most of the time you should include them. Exact contents may vary, but users expect at-least this functionality:
File: New, Open, Close, Print, Quit.
Edit: Cut, Copy, Paste.
Window: Minimize, Maximize.
Help: About.
Keyboard shortcuts: you should strive to have keyboard shortcuts for common functionality. All standard shortcuts should be supported2. e.g.
Ctrl-N for File-New, Ctrl-O for File-Open, Ctrl-Q for Quit.
Ctrl-X for Cut, Ctrl-C for Copy, Ctrl-V for Paste.
F1 for Help.
Benefits
There are obvious benefits to a graphical display being able to display rich colours, graphics and multimedia. However, this application style also encourages encourages a certain style of interaction, where users point-and-click to elements on-screen to interact with them.
There are numerous benefits to this style of user interface:
The interface provides affordances: visual suggestions on how you might interact with the system. This can include hints like tooltips, or a graphical design that makes use of controlsobvious ((e.g. handles to show where to “grab” a window corner).
Systems provide continuous feedback to users. This includes obvious feedback (e.g. dialog boxes, status lines) and more subtle, continuous feedback (e.g. widgets animating when pressed).
Interfaces are explorable: users can use menus, and cues to discover new features.
Low cost of errors: undo-redo, and the ability to rollback to previous state makes exploration low-risk.
These environments encouraged developers to use consistent widgets and interactive elements. Standardization led to a common look-and-feel, and placement of common controls - which made software easier to learn, especially for novices. Many of the standard features that we take for granted are a direct result of this design standardization in the 80s. [ed. notice that Windows, macOS, Linux all share a very common interaction paradigm, and a common look-and-feel! You can move between operating systems and be quite comfortable because of this.]
Functionality
These are complex requirements, that outside of the scope of a programming language (in-part because they’re going to be intrinsically tied to the underlying operating system, so they’re difficult to standardize).
A widget or GUI toolkit is a UI framework which provides this functionality. This includes support for:
Creating and managing application windows, with standard window functionality e.g. overlapping windows, depth, min/max buttons, resizing.
Reusable components called widgets that can be combined in a window to build typical applications. e.g. buttons, lists, toolbars, images, text views.
Dynamic layout that adapts the interface to change in window size or dimensions.
Support for an event-driven architecture3 i.e. suport for standard and custom events. Includes event generation and propogation.
Implementation details will vary based on the toolkit that you’re using. We’ll discuss requirements first, and then in the next section we’ll provide implementation details for some common widgets toolkits.
Window Management
In the context of a applications, a window is simply a region of the screen that “belongs” to a specific application. Typically one application has one main window, an optionally other windows that may also be displayed. These are overlayed on a “desktop”, which is really just the screen background.
To manage many different windows, across many different applications, a part of the operating system called a windowing system is responsible for creating, destroying and managing running windows. The windowing system provides an API to applications to support for all window-related functionality, including:
provide an mechanism for applications to create, or destroy their own windows
handle window movement automatically and invisibly to the application (i.e. when you drag a window, the windowing system moves it).
handles overlapping windows across applicaitons (e.g. so that your application window can be brought to the ““front” and overlap another application’s window).
A windowing system or windowing technology is typically included as part of the operating system, though it’s possible in some systems to replace windowing systems (e.g. Linux).
Coordinate systems
A computer screen uses a Cartesean coordinate system to track window position. By convention, the top-left is the origin, with x increasing as you move right, and y increasing as you move down the screen. The bottom-right corner of the screen is maximum x and y, which equals the resolution of the screen.
Note that its possible for screen contents to move moved out-of-bounds and made inaccessible. We typically don’t want to do this.
In the example below, you can see that this is a 1600x1200 resolution screen4, with the four corner positions marked. It contains a single 400x400 window, positioned at (500, 475) using these screen, or global coordinates.
Given that the windowing system manages movement and positioning of windows on-screen, an application window doesn’t actually know where it’s located on-screen! The application that “owns” the window above doesn’t have access to it’s global coordinates. It does however, have access to it’s own internal, or local coordinates. For example, our window might contain other objects, and the application would know about their placement. In this local coordinate system, we use the top of the window as the origin, with the bottom-right coordinate being the (width, height) of the window. Objects within the window are referenced relative to the window’s origin.
Window creation
Typically, the toolkit will provide a mechanism to create a top-level application window, typically as a top-level class that can instantated. That class will have properties to control its behaviour (some of which is used by the Windowing system to setup the window correctly).
As application developers, we do not need to do anything to support window movement, since it’s provided by the windowing system. Any non-fullscreen windows created by a toolkit are automatically moveable.
Widgets and Layout
We’re going to refer to graphical on-screen elements as widgets. Most toolkits support a large number of similar widgets. The diagram below shows one desktop toolkit with drop-down lists, radio buttons, lists and so on. All of these elements are considered widgets.
Typically, using widgets us as simple as instantiating them, adding them to the window, and setting up a mechanism to detect when users interact with them so that appropriate actions can be taken.
Scene graph
It’s standard practice in graphical applications to represent the interface as a scene graph. This is a mechanism for modeling a graphical application as a tree of nodes (widgets), where each node has exactly one parent. Effects applied to the parent are applied to all of its children.
Toolkits support scene graphs directly. There is typically a distinction made between Container widgets and Node widgets. Containers are widgets that are meant to hold other widgets e.g. menus which hold menu_items, toolbars which can hold toolbar_buttons and so on. Nodes are widgets that are interactive and don’t have any children.
Building a UI involves explicitly setting up the scene graph, by instantiating nodes, and adding them to containers to build a scene graph. (For this reason, containers will always have a list of children, and a mechanism for adding and removing children from their list).
Layout
Layout is the mechanism by which nodes in the scene graph are positioned on the screen, and managed if the window is resized.
Fixed layout is a mechanism to place widgets in a static layout where the window will remain the same size. This is done by setting properties of the widgets to designate each one’s position, and ensuring that the containers do not attempt any dynamic layout.
Relative layout delegates responsibility for widget position and size to their parents (i.e. containers). Typically this means setting up the scene graph in such a way that the appropriate container is used based on how it adjusts position. Typical containers include a vertical-box that aligns it’s children in a vertical line, or a grid that places children in a grid with fixed rows and columns.
Design
Events
Applications often handle multiple types of processing: asynchronous, such as when a user types a few keystrokes, or synchronous, such as when we want a computation to run non-stop to completion.
User interfaces are designed around the idea of using events or messages as a mechanism for components to indicate state changes to other interested entities. This works well, due to the asynchronous nature of user-driven interaction, where there can be relatively long delays between inputs (i.e. humans type slowly compared to the rate at which a computer can process the keystrokes).
This type of system, designed around the production, transmission and consumption of events between loosely-coupled components, is called an Event-Driven Architecture. It’s the foundation to most user-interface centric applications (desktop, mobile), which common use messages to signal a user’s interaction with a viewable component in the interface.
What’s an event? An event is any significant occurrence or change in state for system hardware or software.
The source of an event can be from internal or external inputs. Events can generate from a user, like a mouse click or keystroke, an external source, such as a sensor output, or come from the system, like loading a program.
How does event-driven architecture work? Event-driven architecture is made up of event producers and event consumers. An event producer detects or senses the conditions that indicate thaat something has happened, and creates an event.
The event is transmitted from the event producer to the event consumers through event channels, where an event processing platform processes the event asynchronously. Event consumers need to be informed when an event has occurred, and can choose to act on it.
Events be generated from user actions, like a mouse click or keystroke, an external source, such as a sensor output, or come from the system, like loading a program.
An event driven system typically runs an event loop, that keeps waiting for these events. The process is illustrated in the diagram below:
An EventEmitter generates an event.
The event is placed in an event queue.
An event loop peels off events from the queue and dispatches them to event handlers (functions which have been registered to receive this specific type of event).
The event handlers receive and process the event.
To handle event driven architectures, we often subdivide application responsibility into separate components.
MVC divides any application into three distinct parts:
Model: the core component of the application that handles state (“business logic layer”).
View: a representation of the application state, often as a user-interface (“presentation layer”)
Controller: a component that accepts input, interprets user actions and converts to commands for the model or view.
Similar to the observer pattern, the views monitor the system state, represented by the model. When the state changes, the views are notified and they update their data to reflect these changes. Notifications frequently happen through events generated by, and managed by, the toolkit that you’re using.
Often this is realized as separate classes for each of these components, with an additional main class to bind everything together.
// main class
classMain {
val model = Model()
val controller = Controller(model)
val view = View(controller, model)
model.addView(model)
}
We use an interface to represent the views, which provides the flexibility to allow many different types of output for the program. Any class can be a view as long as it supports the appropriate method to allow notifications from the model.
interfaceIView {
funupdate()
}
classView(val controller: Controller, val model: Model): IView {
overridefunupdate() {
// fetch data from model
}
}
The model maintains a list of all views, and notifies them with state changes (indicating that they may wish to refresh their data, or respond to the state change in some way).
The controller just passes input from the user to the model.
classController(val model: Model) {
funhandle(event: Event) {
// pass event data to model
}
}
One issue with this version of MVC is that the controller often serves little purpose, except to pass along events that are captured by the View (the View contains the user-interface and widgets, and generates events as the user interacts with it).
MVC remains common for simple applications, but tends to be implemented as just a model and one or more views, with the controller code included in the view itself.
Model-View-Presenter
Model-View-Presenter (MVP) keeps the key concept in MVC - separating the business logic from the presentation - and introduces an intermediate Presenter which handles converting the model’s data into a useful format for the views. This is typically done explicitly by the Presenter class. MVP arose from Taligent in the 1990s, but was popularized by Martin Fowler around 2006.
There have been multiple variants of MVP. We’ll focus on MVVM, probably the most popular variant.
MVVM
Model-View-ViewModel was invented by Ken Cooper and Ted Peters to simplify event-driven programming of user interfaces in C#/.NET. It’s similar to MVP, but includes the notion of binding variables to widgets within the framework, so that changes in widget state are are automatically propogated from the view to other components.
MVVM includes the following components:
Model: as MVC, the core component that handles state. It can also map to a data access layer or database directly.
View: a representation of the application state, presented to the user.
ViewModel: a model that specifically interprets the underlying Model state for the particular view to which it is associated. Typically we rely on binding to map variables in the ViewModel directly to widgets in the View, so that updating one directly updated the other.
MVVM is much more common in modern languages and toolkits and has the advantage of replacing all “mapping” code with direct binding of variables and widgets by the toolkit. This greatly simplifies interface development.
Toolkits: JavaFX
We’re using Kotlin with the Java Virtual Machine (JVM) ecosystem, so we’ll discuss some toolkits that are available in that ecosystem.
Java launched in 1996, with AWT as its first GUI framework. AWT is a heavyweight toolkit that provided a thin abstraction layer over the system-specific widgets provided by OS vendors i.e. it provided wrappers for UI components that were built into the OS. However, this tight integration to the OS meant that AWT behaved very differently across different operating systems, which ran counter to Sun’s original goals of having a single cohesive toolkit that ran equally well on all platforms.
Swing was originally part of the Java Foundation Classes, and replaced the AWT in 1997. Unlike AWT, Swing is a lightweight toolkit: Swing components draw themselves using the Java2D Graphics Library, which makes Swing applications consistent across platforms. This also means that Swing can support a broader range of components, including some that aren’t directly supported by the OS. In other words, lightweight toolkits provide some tangible benefits:
The largest collection of widgets, not limited to just the subset that can be assumed to be present on each OS.
Consistency in how widgets behave, since they are designed as a set.
An OS independent look-and-feel.
JavaFX was originally designed by Sun Microsystems in 2008 as a replacement for the Java AWT and Swing toolkits, and was designed to compete directly with Adobe Flash/Flex and similar web toolkits. In 2010, Oracle released JavaFX into the community as part of the OpenJDK initiative. The open source version of JavaFX is currently maintained by Gluon and the community.
JavaFX is an imperative toolkit, where the programmer describes the layout and how it should be managed in code (and XML). [This contrasts with a declarative toolkit like Jetpack Compose, where the programmer describes a layout and the system reflects state in that layout].
JavaFX is a lightweight toolkit that runs well on Windows, Mac, Linux. It provides a native look-and-feel on each platform, and even supports hardware acceleration! It’s not included with the JRE, but because it’s open source, we can distrbute the libraries with our applications.
Setup
Although JavaFX can be installed from the main JavaFX site, the recommended way to bundle these libraries into your application is to add it to your Gradle configuration file. Gradle will then download and install JavaFX as-needed.
In your project’s build.gradle file, make the following changes to include the javafxplugin and related settings in the javafx block.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
application
kotlin("jvm") version "1.6.20" id("org.openjfx.javafxplugin") version "0.0.13" id("org.beryx.jlink") version "2.25.0"}
group = "net.codebot"version = "1.0.0"val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
repositories {
mavenCentral()
}
dependencies {
testImplementation(kotlin("test"))
}
tasks.test {
useJUnitPlatform()
}
tasks.withType<KotlinCompile> {
kotlinOptions.jvmTarget = "1.8"}
application {
mainModule.set("calculator")
mainClass.set("calculator.Main")
}
javafx {
// version is determined by the plugin above
version = "18.0.2" modules = listOf("javafx.controls", "javafx.graphics")
}
// https://stackoverflow.com/questions/74453018/jlink-package-kotlin-in-both-merged-module-and-kotlin-stdlib
jlink {
forceMerge("kotlin")
}
In your Gradle menu in IntelliJ, press “Sync” to load the changes, and the JavaFX libraries should be loaded. If you expand the “External Libraries” in the Project view, you can see the JavaFX libraries have been installed:
Example: HelloFX
The following application shows how to create a simple window with some graphics. Athough longer than our console version of “Hello Kotlin”, it accomplishes quite a lot with minimal code. We’ll discuss this in further detail below.
classApp: Application() {
overridefunstart(stage:Stage?) {
val image = Image("java.png", 175.0, 175.0)
val imageView = ImageView(image)
val label = Label(
System.getProperty("java.vendor")
+ System.getProperty("java.version") + "\n" + System.getProperty("javafx.version"))
val box = VBox(imageView, label)
VBox.setMargin(label, Insets(10.0))
val scene = Scene(box, 175.0, 225.0)
stage.setResizable(false)
stage.setScene(scene)
stage.show()
}
}
This is actually pretty impressive when you realize that we have just created:
A resizable window with min/max/restore buttons
A titlebar and content centred in the window.
A UI that will inherit the appearance of any platform where it runs. Execute this on Windows, and the buttons will have a standard appearance and positioning for that platform!
Classes
In JavaFX, our highest level abstractions are the Application class, with one or more Stage classes representing the application windows, and one or more Scene classes to manage the contents of a window. Nodes represent the individual graphical elements.
As we saw in the previous chapter with JavaFX, it’s standard practice in 2D graphical applications to represent the interface as a scene graph of objects. In JavaFX, the Scene class maintains this scene graph, consisting of various nodes, for each Scene that we display. Note that it’s possible to have multiple windows, each with multiple scenes, each of which manages a different scene graph. (Multiple windows can be displayed at once, but only once scene graph can be displayed at a given time in a window, representing the current window contents).
Application
The Application class is top-level representation of the application. It serves as the application entry point, replacing the main() method. During launch, a JavaFX application will perform the followin steps:
Constructs an instance of the specified Application class
The start() method is abstract and MUST be overridden. The init() and stop() methods are optional, but MAY be overridden. It’s fairly normal to just override start() and ignore the others most of the time.
Stage
The Stage class is the top-level container or application window. You can have multiple stages, representing multiple windows.
javafx.stage.Window
javafx.stage.Stage
A Stage instance is automatically created by the runtime, and passed into the start() method.
Stage methods operate at the window level:
setMinWidth(), setMaxWidth()
setResizable()
setTitle()
setScene()
show()
Scene
The Scene is a container for the content in a scene-graph. Although you can create multiple scenes, only one can be attached to a window at a time, representing the “current” contents of that window.
javafx.scene.Scene
To construct a scene, and set it up:
Create a scene graph consisting of a container holding on or more nodes;
Add the root node of the scene graph to the scene;
Add the scene to a stage and make the stage visible.
Scene methods manipulate the scene graph, or attempt to set properties for the entire graph:
setRoot(Node)
setFill(Paint)
getX(), getY()
Node
Node is the base class for all elements of a scene graph. Types of nodes include:
meta objects like Camera and LightBase to offer fine control of the scene.
Nodes have common properties for position (x, y), width and height, background colour and so on. These can be set manually in code, or in the case of visual properties, associated with a CSS stylesheet.
Info
JavaFX is pretty comprehensive, but you might want to implement something that isn’t built into that toolkit e.g. date widgets.
Luckily, you can include projects that expand the standard widgets. These are intended to be imported and used alongside the standard JavaFX widgets.
ControlsFX expands to include checklists, breadcrumb bars and other unique widgets.
JFxtras includes a calendar widget, gauges and other useful widgets.
Layouts
Layout is how items are arranged on the screen. Layout classes are branch nodes that have built-in layout behaviour. Your choice of parent class to hold the nodes determines how its children will be laid out.
Layout Class
Behaviour
HBox
Layout children horizontally in-order
VBox
Layout children vertically in-order
FlowPane
Layout left-right, top-bottom in-order
BorderPane
Layout across sides, centre in-order
GridPane
2D grid, with cells the same size
Example: Java Version
Here’s the Java Version example from above, annotated. The sequence to setup a window is:
Define the nodes (lines 4-11)
Create a layout as the root of the scene graph (line 14), which will hold the nodes.
Add the root node to the scene (line 18)
Add the scene to the stage (line 19)
Show the stage (line 23)
classApp: Application() {
overridefunstart(stage:Stage?) {
// imageView is our first node
val image = Image("java.png", 175.0, 175.0)
val imageView = ImageView(image)
// label is our second node
val label = Label(
System.getProperty("java.vendor")
+ System.getProperty("java.version") + "\n" + System.getProperty("javafx.version"))
// box is our layout that will manage the position of our nodes
val box = VBox(imageView, label)
VBox.setMargin(label, Insets(10.0))
// create a scene from the layout class, and attach to the stage
val scene = Scene(box, 175.0, 225.0)
stage.setScene(scene)
// set window properties and show it
stage.setResizable(false)
stage.show()
}
}
Events
JavaFX expands on the Listener model that was introduce in Java Swing, and provides support for a wide varieties of events. The Event class is the base class for a JavaFX event. Common events include:
MouseEvent − This is an input event that occurs when a mouse is clicked. It includes actions like mouse clicked, mouse pressed, mouse released, mouse moved.
KeyEvent − This is an input event that indicates the key stroke occurred over a node. This event includes actions like key pressed, key released and key typed.
WindowEvent − This is an event related to window showing/hiding actions. It includes actions like window hiding, window shown.
Nodes have convenience methods for handling common event types. They include:
setOnMouseClicked()
setOnMousePressed()
setOnMouseReleased()
setOnMouseMoved()
setOnKeyPressed()
setOnKeyReleased()
Additionally, there is a generic “action” handler which responds to the standard interaction with a control e.g. pressing a button, or selecting a menu item.
For example, here’s a handler for a “save” button (from sample-code/desktop/contacts)
val save = Button("Save")
save.setOnAction { event -> model.add(Contact(name.text, phone.text, email.text))
model.save()
}
Packaging
Scripts are a simple way to get your application to launch, but they struggle when you have complex dependencies, or resources that need to be included (like you often will with a GUI application). If you are building a JavaFX or Compose desktop application, you should consider using jlink or jpackage to build an installer.
JLink will let you build a custom runtime that will handle the module dependencies for JavaFX. The simplest way to do this is to add the JLink plugin to your build.gradle file and let Gradle handle it.
plugins {
id 'org.beryx.jlink' version '2.25.0'}
You can also configure it in the build.gradle file as well. For a full set of options see the Badass-JLink plugin page.
jlink{
launcher {
name = "clock" }
imageZip.set(project.file("${project.buildDir}/image-zip/clock-image.zip"))
}
We can rebuild the clock sample using Gradle - build - jLink to produce a runtime script in build/image
Here’s the resulting directory structure. Notice that it includes a number of libraries that our application needs to run.
Running the top-level bin/clock_advanced image will execute our application.
$ ./clock_advanced
Creating installers
Finally, we can use jpackage to create native installers for a number of supported operating systems. JPackage is included as a console application in Java JDK 16 or higher, and will work with any JVM language (e.g. Java, Kotlin, Scala). The full guide is here.
An installer is an application that when executed, installs a different application for the user. We need installers because most applications consists of many different files: executables, libraries, resources (images, sound files), preference files and so on. These need to be installed in the correct location, and sometimes registered, to function correctly.
Tasks that the installer performs include:
Copying application files to the correct location.
Installing and registering system libraries.
Making changes to the system registry (or similar system databases).
Creating icons on the desktop, or applications folder.
Prompting the user if any of these tasks require elevated privileges.
Instead of running jpackage manually, we will install a plugin into IntelliJ and use that environment to generate our installers. We can do this by installing the Badass-JLink plugin page. To use the plugin, include the following in your gradle.build script:
If you install the plugin correctly, then you should see the jpackage command in Gradle - build - jpackage. Run this and it will create platform installers in the build/distribution directory.
This is a standard macOS installer. Drag the clock_advanced icon to the Applications folder. You can then run it from that folder.
Info
Installers are meant for graphical applications. If you are building a JavaFX or Compose desktop application, this is the right choice. If you’re building a console application, you probably want a script instead (see previous step) so that you can execute it from the console directly.
Photoshop, for instance, famously has multiple windows tiled over the desktop. It’s also a very, very complex program, so it needs to split up functionality like this. ↩︎
Though not strictly required, all modern toolkits are built around the idea of an event-driven architecture, where events or messages are used to communicate changes to system state, or to signal user intentions. ↩︎
Note that this is the resolution that the screen is set to display, which may be different from the resolution that the screen natively supports. For example, my monitor is meant to run at 2880x1440, but I can set it to any desired resolution up to those values. ↩︎
Web Services
Overview
Up to this point, we’ve assumed that we’re designing a standalone application i.e. for a single computer. Historically, many software applications were designed to be standalone, and much of the software that we use is still standalone.
However, it can be useful to sometimes split processing across multiple systems. Here’s a partial list of the reasons why you might want to do this:
Resource sharing. We often need to share resources across users. For example, storing our customer data in a shared databases that everyone in the company can access.
Reliability. We might want to increase the reliability of our software by redunant copies running on different systems. This allows for fault tolerance - failover in case the first system fails. This is common with important resources like a web server.
Performance. It can be more cost effective to have one highly capable machine running the bulk of our processing, while cheaper/smaller systems can be used to access that shared machine. It can also be cheaper to spread computation across multiple systems, where tasks can be run in parallel. Distributed architectured provide flexibility to align the processing capabilities with the task to be performed.
Scalability. Finally, if designed correctly, distributing our work across multiple systems can allow us to grow our system to meet high demand. Amazon for example, needs to ensure that their systems remain responsive, even in times of heavy load (e.g. holiday season).
Openness. There is more flexibility, since we can mix systems from different vendors.
Earlier, we discussed a distributed application as a set of components, spread across more than one machine, and communicating with one another over a network. Each component has some capabilities that it provides to the other components, and many of them coordinate work to accomplish a specific task.
We have already described a number of different distributed architectures. These enforce the idea that distributed systems can take many different forms, each with it’s own advantages and disadvantages. When we’re considering building a service, we’re really focusing on a particular kind of distributed system, where our application is leveraging remote resources.
Service Architectures
Client-Server
A client-server architecture is a model where one centralized server provides capabilities to multiple client machines.
Client − The process that issues a request to the second process i.e. the server. From the point of view of the user, this is the application that they interact with.
Server − The process that receives the request, carries it out, and sends a reply to the client.
In this architecture, the application is modelled as a set of services that are provided by servers and a set of clients that use these services. The servers need not know about clients, but the clients must know the identity of servers.
Advantages
Separation of responsibilities such as user interface presentation and business logic processing (client).
Reusability of server components and potential for concurrency (single server).
It also makes effective use of resources when a large number of clients are accessing a high-performance server.
Disadvantages
Limited server availability and reliability.
Limited testability and scalability.
Fat clients with presentation and business logic together.
Limited ability to scale the server (need more processing, “buy a bigger server”).
Multi-tier architecture
A multi-tier architecture (also known as 2-tier, 3-tier or layered) is an architecture that separates an application into separate tiers or areas of concerns1. Often the user interface, business logic and data layers end up split apart.
The most common form is this architecture is 3-tier architecture, where each tier is a separate module, deployed on a separate computer. Tiers typically communicate over a network connection.
The top tier is the Presentation layer, which handles the UI logic. This is typically hosted on a client machine (i.e. where the user accesses it).
The middle layer is the Application or Logic tier, which handles “business logic”; the rules and state management of the application. This can often include logic for coordinating requests from a client across multiple services as well.
The bottom layer is the Data tier, which handles access, storage and management of the underlying data. This is often a database, or a wrapper around a database (or some other form of storage).
This is an extremely common architecture for Enterprise applications. It also aligns with the way that websites are traditionally served (Presentation tier is the browser, Logic tier is the web server and underlying code, and the Data tier is a database).
In some cases, the Logic and Presentation tiers are combined, for a 2-tier architecture consisting of just Presentation and Data tiers.
Advantages
Enhances the reusability and scalability − as demands increase, extra servers can be added.
Provides maintainability and flexibility.
Disadvantages
Greater complexity, more difficult to deploy and test across tiers.
More emphasis on server reliability and availability.
Service Oriented Architecture
A service-oriented architecture (SOA) is an architectural style that supports service orientation. A service is a discrete unit of functionality that can be accessed remotely and acted upon and updated independently, such as retrieving a credit card statement online.
In other words, services exist independently of clients, and provide services to any client that requires it. Services are loosely coupled to one another, and should act independently.
Principles of SOA, governing how services should be designed:
Service contract: there should be an agreed upon interface for accessing a service.
Longevity: services should be designed to be long-lived (long-running).
Autonomy: services should work independently of one another.
Service composibility: services can be used to compose other services.
Stateless: services should not track state, but either return a resulting value or throw an exception if necessary.
Because they are independent entities, we need a supporting infrastructure around services, and applications that are designed to leverage that infrastructure. This includes a repository, where an application can search for services that can meet its needs at runtime.
Advantages
A client or any service can access other services regardless of their platform, technology, vendors, or language implementations.
Each service component is independent from other services due to the stateless service feature.
The implementation of a service will not affect the application of the service as long as the exposed interface is not changed.
Enhances the scalability and provides standard connection between systems.
Disadvantages
Even more complexity in setting up a system, since we’re now distributing across multiple tiers.
Registry and other supporting infrastructure can be complex to setup and maintain.
Difficulty debugging, profiling and so on.
Microservices
A microservices architecture arranges an application as a collection of loosely coupled services, using fine-grained services and a lightweight protocol. Some of the defining characteristics of microservices:
Services are organized around business capabilities i.e. they provide specialized, domain-specific services to applications (or other services).
Service are not tied to any one programming language, platform or set of technologies.
Services are small, decentralized, and independently deployable.
A microservice based architecture is really a subtype of SOA, which an emphasis on smaller, domain-specific components with very narrow functions.
Advantages
Easier to design, build and deploy small targeted services.
Redundancy - you can always “spin up” a replacement service if something fails.
Performance - you can always “scale out” by firing up redundant services to share the workload, as required.
Disadvantages
Extremely difficult to test and debug.
Practically requires supporting services, like a registry for processes to locate service endpoints.
Web Servers
As originally designed, a web server and web browser are a great example of a client-server architecture.
A web server is service running on a server, listening for requests at a particular port over a network, and serving web documents (HTML, JSON, XML, images). The payload that is delivered to a web browser is the content, which the browser interprets and displays. When the user interacts with a web page, the web browser reacts by making requests for additional information to the web server.
Over time, both browser and web server have become more sophisticated, allowing servers to host additional content, run additional programs as needed, and work as part of a larger ecosystem that can distribute client requests across other systems.
Web technologies are interesting to us because they can be the basis for a robust service request mechanism. We’ll explore that in this section.
HTTP Protocol
The Hypertext Transfer Protocol (HTTP) is an application layer protocol that supports serving documents, and processing links to related documents, from a remote service.
A web browser is a typical client, which the user is accessing. A web server would be a typical server.
The user requests content through the browser, which results in an HTTP request message being sent to the server.
The server, which provides resources such as HTML files and other content or performs other functions on behalf of the client, returns a response message to the client. The response contains completion status information about the request and may also contain requested content in its message body.
Request Methods
HTTP defines methods to indicate the desired action to be performed on the identified resource.
What this resource represents, whether pre-existing data or data that is generated dynamically, depends on the implementation of the server. Often, the resource corresponds to a file or the output of an executable residing on the server. Method names are case sensitive.
GET: The GET method requests that the target resource transfers a representation of its state. GET requests should only retrieve data and should have no other effect.
HEAD: The HEAD method requests that the target resource transfers a representation of its state, like for a GET request, but without the representation data enclosed in the response body. Uses include looking whether a page is available through the status code, and quickly finding out the size of a file (Content-Length).
POST: The POST method requests that the target resource processes the representation enclosed in the request according to the semantics of the target resource. For example, it is used for posting a message to an Internet forum, or completing an online shopping transaction.
PUT: The PUT method requests that the target resource creates or updates its state with the state defined by the representation enclosed in the request. A distinction to POST is that the client specifies the target location on the server.
DELETE: The DELETE method requests that the target resource deletes its state.
Spring Boot
Spring is a popular Java framework. It’s opinionated, in that it provides a strict framework for a web service. You are expected to add classes, and customize behaviour as needed, but you are restricted to the overarching structure that is provided. This is a reasonable tradeoff. In return for giving up some flexibility, you get Dependency Injection and other advanced features, which make testing (along other things) much easier to achieve.
This power comes at a price: complexity. Spring has a large number of configuration files that allow you to tweak and customize the framework, but the options can be overwhelming. To help developers, Pivotal created Spring Boot, which is a program that creates a starting configuration for you - even going so far as to include a web server and other required libraries to get you up-and-running quickly.
We’ll walk through setting up a simple Spring Boot application. The steps are typically:
Use Spring Boot to create a working project.
Write controller, model, and other required classes.
Write and run tests.
Setup
It is highly recommended that you use Spring Boot to create your starting project. You can run it one of two ways:
Visit start.spring.io and use the web form to set the parameters for your project. Generate a project and download the project files.
Use the Spring project wizard in IntelliJ. Set the parameters for your project (Kotlin, Gradle) and follow instructions to generate an IntelliJ IDEA project.
Regardless of which you choose, you will be asked for dependencies. You will probably want to include at least Spring Web and Spring Data JPA and possibly others:
Spring Web: this will embed a web server in your project so that you can easily test it.
Spring Data JPA: JPA stands for Java Persistance API. JPA is an object-persistence layer that allows you to map classes directly to database tables and avoid writing SQL for simple requests.
JDBC API: this will allow you to use JDBC to access databases - helpful if your service needs to persist data.
H2 Database: an embedded database for testing - you can also swap out for a different database if desired.
Your starting project should look something like this:
Notice that the class is annotated with @SpringBootApplication. This tells the framework to treat this as the top-level application class. Spring uses extensive annotations like this to flag methods and properties for the framework.
The main() method calls the framework’s runApplication method to launch.
Your Spring project comes with an embedded web server, which runs on port 8080. You can test this by running the web server (click on the Play button beside the Main method), and then open the URL in a web browser: http://localhost:8080
Info
The Spring Web starter code always defaults to https://localhost:8080 to serve your web service. Use this URL for testing.
Unfortunately, this won’t return anything useful yet. Although the web service is running, we need to write code to handle the requests! That’s the job of the controller.
We’ll add a controller class to save messages (just a couple of strings), or retrieve a list of messages that were previously posted.
Writing a Controller Class
The controller class is responsible for handling requests from a client process. Our web service uses HTTP, so requests will be the request methods that we discussed earlier: GET to retrieve a list of messages, and POST to store a new message.
Here’s an example of a controller class, configured to handle Post and Get requests.
@RestController flags this class as our main controller class, which will be responsible for handling the HTTP requests. Within this class, we need to write code to handle the endpoints and requests for our application. The main endpoint will be /messages, since we set that mapping on the controller directly.
@GetMapping and @PostMapping indicate the methods that will handle GET and POST requests respectively. Our methods work with MessageService and Message classes, which we will need to define.
This means that our endpoint will be https://localhost:8080/messages (the default address and port for Spring, plus the endpoint that we defined). Our controller will handle GET and POST requests to that endpoint.
Info
The MessageResource() class declaration is an example of dependency injection: instead of instantiating the MessageService inside of the class, we pass it in as a parameter. MessageService is flagged as a class that the framework can manage directly.
Dependency injection makes testing earlier, since you don’t have unmanaged objects being allocated inside of a class. In this case, we can mock the MessageService during testing to isolate our MessageResource tests.
We can identify the following annotations:
Annotation
Use
@RestController
Indicates a controller class that should process requests
@GetMapping
A function that will be called when a GET request is received.
@PostMapping
A function that will be called when a POST request is received.
Using this code:
A client sending a GET request will be returned a list of all messages in JSON format.
A client sending a POST request, with well-formed data, will create a new message.
We could add in other mappings (e.g. PUT, DELETE) if required. All of these mappings would be handled in the Controller class.
We can finish our first pass at this service by adding a Message class and a MessageService class to store the data from our requests. The full service is listed below (also in the public repo: /service/spring-server-basic)
So far, we’re receiving and storing JSON objects as Messages. However, we’re only saving them in a list, which will be lost when we halt our service. What if we want to persist the data into a database? How do we convert our Message objects to a format that we can write out?
Spring Data JPA is a library that focuses on using JPA to store data in a relataional database. It greatly simplifies setting up a repository: we just setup the interface that we want, and it automatically creates the implementation code for us!
To start, add the JAP dependencies to your build.gradle file. Make sure to add these to the existing sections - don’t delete anything! Once added, click on the Sync icon
plugins { kotlin("plugin.jpa") version "1.6.10"}dependencies { implementation("org.springframework.boot:spring-boot-starter-data-jpa")}
Next we’ll add a repository interface that tells the service what operations we want to perform against our Message class. The modified code looks like this:
By inheriting from CrudRepository, the MessageRepositorygains methods for save, findByIndex and other common operations.
The findMessages method is undefined, so we use some annotations to map it to a query against a specific database table.
@Query indicates a database query that should be run, and the results mapped to the output of this function.
@Table describes a data class that corresponds to an underlying database table. This table will be created for us, using the parameters from the Message class as column names.
By default, our Spring framework projects support H2, and in-memory database. The database file is stored in ./data in our project folder. You can browse it in IntelliJ IDEA and see the underlying table that is created.
We can do a lot with just POST and GET operations, since we also have the ability to pass parameters in our requests. If your server is running in IntelliJ IDEA, you can create requests for testing directly in the IDE:
Click the drop-down menu beside the GET or POST mapping in your Controller code. You should see an option to
This should bring up a script where you can enter requests that you wish to run. Enter as many as you wish: to run them, click on the Run arrow beside the request. Pay careful attention to the format, and notice the content type is included in the request:
You can run simple tests directly from a browser, but IntelliJ IDEA for testing provides additional support like code completion and syntax highlighting for structuring your requests. This doesn’t replace proper automated tests, but it’s certainly helpful when configuring and debugging your services.
Making Client Requests
The big question, of course, is how do we make requests to this service from a client? How do we actually use it?
Kotlin (and Java) includes libraries that allow you to structure and execute requests from within your application.
This example opens a connection, and prints the results of a simple GET request:
URL("https://google.com").readText()
To use our service, we may want to set a few more parameters on our request. The HttpRequest class uses a builder to let us supply as many optional parameters as we need when building the request.
Here’s a method that fetches data from our server example above:
funget(): String {
val client = HttpClient.newBuilder().build()
val request = HttpRequest.newBuilder()