Subsections of Getting Started
Setup
We will be building Kotlin applications and services, and using the Java JDK as our deployment target. This means that you need to install the Java JDK and the Kotlin compiler on your development machine. It’s also highly recommended that you install the Intelli IDEA IDE for working with Kotlin, as it offers advanced language support and integrated well with our other tools and libraries.
Check the Course-Project/Technologies page for the recommended version numbers. Also, make sure that you and your team all install the same distribution and versions of these tools!
The following represents the minimum toolchain for the course.
Git
We use Git for version control, so you will need Git installed to perform operations on your source code.
Git is pre-installed on macOS and Linux; Windows users can install it from https://git-scm.org. Once it’s installed, you may need to update your path to include the Git installation directory. You can check your installation using the git version command.
❯ git version
git version 2.37.1 (Apple Git-137.1)
Java JDK
- Download and install the JDK from Azul or OpenJDK (make sure to match your system architecture).
- Add JAVA_HOME to your system’s environment variables, pointing to this installation.
- Update your path to include the directory containing the Java executables.
For example, I am using JDK 18 (at the time I’m writing this). I have the following lines in my .zshrc:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-18.jdk/Contents/Home
export PATH=$PATH:$JAVA_HOME/bin
You can check your installation using the java version command. Make sure the version matches what you expected to see.
$ java -version
openjdk version "18.0.2.1" 2022-08-18
OpenJDK Runtime Environment Zulu18.32+13-CA (build 18.0.2.1+1)
OpenJDK 64-Bit Server VM Zulu18.32+13-CA (build 18.0.2.1+1, mixed mode, sharing)
IntelliJ IDEA
IntelliJ IDEA is our recommended development environment. You can install it from https://www.jetbrains.com/idea/download/.
There is an Open Source Community version which will work for this course. There is also an Ultimate license, which includes better support for databases, web services and other frameworks that we’ll be using. This is normally a paid upgrade, but as a student you can get a free license to most of thier products, including this version of IntelliJ IDEA.
You can check the installed version by opening IntelliJ IDEA and looking at the IntelliJ IDEA - About dialog.
Kotlin
We will need a Kotlin compiler, and the Kotlin standard libraries. IntelliJ IDEA includes a Kotlin plugin, so if you have installed IntelliJ IDEA and you are working from the IDE, then you do not need to install Kotlin.
However, if you wish to compile from the command-line, or use a different editor, then you will need to install Kotlin manually. It can be installed from https://www.kotlinlang.org or from most package managers (e.g. brew install kotlin if you are a Mac user with Homebrew installed).
If you install the command-line version, you can check your installation using the kotlin -version command.
❯ kotlinc -version
info: kotlinc-jvm 1.8.20 (JRE 18.0.2.1+1)
Info
There are other libraries that are suggested on the Course-Project/Technologies page e.g. Ktor, Exposed, JUnit. You do NOT need to manually install any of these! Early in the course we will discuss Gradle, our build system, which has the ability to import these libraries into our project automatically. It’s much easier than trying to manually manage all of the dependencies!
As long as you have installed Git, Java/JVM, Kotlin, and have an editor/IDE setup, you can start working on your project.
Using Git
A Version Control Systems (VCS) is a software system designed to track changes to source code. It is meant to provide a canonical version of your project’s code and and other assets, and ensure that only desireable (and tested) changes are pushed into production. Common VCS systems include Mercurial (hg), Subversion (SVN), Perforce, and Microsoft Team Foundation Server. We’ll be using Git, a very popular VCS, in this course.
All VCS’s, including Git, let you take a snapshot of your source code at any given point in time. This example shows a project that starts with a single index.html file, adds about.html at a later time, and then finally makes some edits. The VCS tracks these changes, and provides functionality that we’ll discuss below.
https://www.git-tower.com/learn/git/ebook/en/desktop-gui/basics/what-is-version-control
Why use Version Control?
A VCS provides some major benefits:
- History: a VCS provides a long-term history of every file. This includes tracking when files were added, or deleted, and every change that you’ve made. Changes are grouped together, so you can look at (for instance) the set changes that introduced a feature.
- Versions: the ability to version your code, and compare different versions. Did you break something? You can always unwind back to the “last good” change that was saved, or ever compare your current code with the previously working version to identify an issue.
- Collaboration: a VCS provides the necessary capabilities for multiple people to work on the same code simultaneously, while keeping their changes isolated. You can create branches where your changes are separate from other ongoing changes, and the VCS can help you merge changes together once they’re tested.
Installing Git
Git binaries can be installed from the Git home page or through a package manager (e.g. Homebrew on Mac). Although there are graphical clients that you can install, Git is primarily a command-line tool. Commands are of the form: git <command>.
You’ll also want to make sure that the git executable (git or git.exe) is in your path.
Concepts
Version control is modeled around the concept of a changeset: a grouping of files that together represent a change to the system (e.g. a feature that you’ve implemented may impact multple source files). A VCS is designed to track changes to sets of files.
Git is designed around these core concepts:
- Repository: The location of the canonical version of your source code.
- Working Directory: A copy of your repository, where you will make your changes before saving them in the repository.
- Staging Area: A logical collection of changes from the working directory that you want to collect and work on together (e.g. it might be a feature that resulted in changes to multiple files that you want to save as a single change).
A repository can be local or remote:
- A local repository is where you might store projects that you don’t need to share with anyone else (e.g. these notes are in a local git repository on my computer).
- A remote repository is setup on a central server, where multiple users can access it (e.g. GitLab, GitHub effectively do this, by offering free hosting for remote repositories).
Git works by operating on a set of files (aka changeset): we git add files in the working directory to add them to the change set; we git commit to save the changeset to the local repository. We use git push and git pull to keep the local and remote repositories synchronized.
https://support.nesi.org.nz/hc/en-gb/articles/360001508515-Git-Reference-Sheet
Local Workflow
To create a local repository that will not need to be shared:
- Create a repository. Create a directory, and then use the
git init command to initialize it. This will create a hidden .git directory (where Git stores information about the repository).
$ mkdir repo
$ cd repo
$ git init
Initialized empty Git repository in ./repo/.git/
$ ls -a
. .. .git
- Make any changes that you want to your repository. You can add or remove files, or make change to existing files.
$ vim file1.txt
ls -a
. .. .git file1.txt
- Stage the changes that you want to keep. Use the
git add command to indicate which files or changes you wish to keep. This adds them to the “staging area”. git status will show you what changes you have pending.
$ git add file1.txt
$ git status
On branch master
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: file1.txt
- Commit your staging area.
git commit assigns a version number to these changes, and stores them in your local repository as a single changeset. The -m argument lets you specify a commit message. If you don’t provide one here, your editor will open so that you can type in a commit message. Commit messages are mandatory, and should describe the purpose of this change.
$ git commit -m "Added a new file"Remote Workflow
Remote Workflow
A remote workflow is almost the same, except that you start by making a local copy of a repository from a remote system.
- Clone a remote repository. This creates a new local repository which is a copy of a remote repository. It also establishes a link between them so that you can manually push new changes to the remote repo, or pull new changes that someone else has placed there.
# create a copy of the CS 346 public repository
$ git clone https://git.uwaterloo.ca/j2avery/cs346.git ./cs346
Making changes and saving/committing them is the same as the local workflow (above).
- Push to a remote repository to save any local changes to the remote system.
- Pull from remote repository to get a copy of any changes that someone else may have saved remotely since you last checked.
- Status will show you the status of your repository; log will show you a history of changes.
# status when local and remote repositories are in sync
$ git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
# condensed history of a sample repository
$ git log --oneline
b750c10 (HEAD -> master, origin/master, origin/HEAD) Update readme.md
fcc065c Deleted unused jar file
d12a838 Added readme
5106558 Added gitignore
Branching
The biggest challenge when working with multiple people on the same code is that you all may want to make changes to the code at the same time. Git is designed to simplify this process.
Git uses branches to isolate changes from one another. You think of your source code as a tree, with one main trunk. By default, everyone in git is working from the “trunk”, typically named master or main (you can see this when we used git status above).
https://www.nobledesktop.com/learn/git/git-branches
A branch is a fork in the tree, where we “split off” work and diverge from one of the commits (typically we split from a point where everything is working as expected)! Once we have our feature implemented and tested, we can merge our changes back into the master branch.
Notice that there is nothing preventing multiple users from doing this. Because we only merge changes back into master when they’re tested, the trunk should be relatively stable code.
We have a lot of branching commands:
$ git status // see the current branch
On branch master
$ git branch test // create a branch named test
Created branch test
$ git checkout test // switch to it
Switched to a new branch 'test'
$ git checkout master //switch back to master
Switched to branch 'master'
$ git branch -d test // delete branch
Deleted branch test (was 09e1947).
When you branch, you inherit changes from your starting branch. Any change that you make on that branch are isolated until you choose to merge them.
A typical workflow for adding a feature would be:
- Create a feature branch for that feature.
- Make changed on your branch only. Test everything.
- Code review it with the team.
- Switch back to
master and git merge from your feature branch to the master branch. If there are no conflicts with other change on the master branch, your changes will be automatically merged by git. If your changed conflict (e.g multiple people changed the same file and are trying to merge all changed) then git may ask you to manually merge them.
$ git checkout -b test // create branch
Switched to a new branch 'test'
$ vim file1.md // make some changes
$ git add file1.md
$ git commit -m "Committing changed to file1.md"
$ git checkout master // switch to master
$ git merge test // merge changes from test
Updating 09e1947..ebb5838
Fast-forward
file1.md | 136 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 118 insertions(+), 18 deletions(-)
$ git branch -d test // remove branch (optional)
Deleted branch test (was ebb5838).
Merging Code
This is a trivial case, demonstrating a merge that happens very soon after the branch was created. However, it’s more likely that branches will be created and worked on for a long period of time before you merge back to main.
When you merge, Git examines your copy of each file, and attempts to apply any other changes that may have been committed to main since you created the branch. (If there’s multiple people working on the project, it’s not unusual for multiple changes to be made to the same file). In many cases, as long as there are no conflicts, Git will merge the changes together. However, if Git is unable to do so (e.g. you and a colleague both changed the same file and your changes overlap), then you will be prompted to manually merge the changes together.
When this happens, Git will apply both changes to the source file, and add inline comments. You have to manually fix the file, and then commit the change before attempting to merge again.
Pull Requests (PRs)
One way to avoid merge issues is to review changes before they are merged into main (this also lets you review the code, manually run tests etc). The standard mechanism for this is a Pull Request (PR). A PR is simply a request to another developer (possibly the person responsible for maintaining the main branch) to git pull your feature branch and review it before merging.
We will not force PRs in this course, but you might find them useful within your team.
GitLab also calls these Merge Requests.
Best Practices
These are suggestions for working with Git effectively.
- Work iteratively. Learn to solve a problem in small steps: define the interface, write tests against that interface, and get the smallest functionality tested and working.
- Commit often! Once you have something work (even partly working) commit it! This gives you the freedom to experiment and always revert back to a known-good version.
- Branch as needed. Think of a branch as an efficient way to go down an alternate path with your code. Need to make a major change and not sure how it will work out? Branch and work on it without impacting your main branch.
- Store your projects in private, online repositories. Keep them private so that you don’t share them unless it’s appropriate. Being online provides a remote backup and makes it easy to add someone to your project later.
https://xkcd.com/1597
IntelliJ IDEA
An Integrated Development Environment (IDE) is custom software designed to support all aspects of software development. It typically includes a code editor, debugging, testing and profiling tools, and anything else that might be needed to support your workflow.
While you certainly can use command-line tools to build applications, it’s strongly encouraged to use an IDE which provides a large number of advanced features (e.g. debugging, profiling support, auto-completion in code, integration with other systems and so on).
In this course, we’re going to use IntelliJ IDEA, an IDE produces by JetBrains (the company that invented Kotlin), which provides all development functionality, and integrates with all of our tools.
Anything we can do to reduce the friction of common software activities is worth pursuing. This is why we like Integrated Development Environment (IDE)s like IntelliJ – they supports a range of common activities:
-
Producing new code: tools for navigating existing classes, maybe filling in code snippets for us.
-
Reading existing code: browsing classes, maybe diagramming the system to allow us to build a mental model of an existing system.
-
Refactoring: the ability to produce a series of possibly sweeping changes without breaking existing code. e.g. renaming a method, and everywhere it’s invoked; extracing an interface from an existing set of classes.
-
Debugging: visualizations and other tools designed to help diagnose.
-
Profiling: tools to help us understand performance and runtime behaviour.
The following section outlines some features that you should investigate. In the examples below, you can generally copy-paste the code into a main() method in IntelliJ and execute it.
Installation
IntelliJ IDEA can be downloaded from https://www.jetbrains.com/intellij
There are two versions of IntelliJ IDEA: the Community Edition, which is open source, and the Ultimate Edition which is more capable. JetBrains offers free educational licenses to students, which includes a license for IntelliJ IDEA Ultimate if you wish. Either one will work for this course.
Make sure to install the version appropriate for your system architecture (e.g. x86 for an Intel processor, or ARM for Apple M1/M2 processors).
IntelliJ will attempt to use the Java JDK that you installed previously. If intelliJ is unable to locate a suitable JDK, it may complain that it cannot locate a Project JDK (i.e. Java JDK). To fix this, click on Setup SDK and enter details for the JDK that you installed (i.e. where it is installed).
Creating a Project
IntelliJ fully supports Gradle, and we can create a new project directly in the IDE.
From the Splash screen, select Create New Project.
You will need to supply project parameters.
-
Kotlin as your programming language.,
-
Gradle for your build system.
-
Gradle can use either Groovy or Kotlin as a DSL.
-
Either is fine, though course examples are mostly in Groovy (they were created before Kotlin was widely supported).
-
JDK should point to your installation.
If successful, IntelliJ IDEA will open into the main window to an empty project.
After the project has loaded, use View -> Tools -> Gradle to show the Gradle tasks window, and use build -> build and application -> run to Build and Run respectively.
Navigating Projects
IntelliJ has a number of windows that it will display by default. You can use Cmd-Hotkey (or Ctrl-Hotkey) to navigate these windows[^3].
- Project: a list of all files (CMD-1).
- Structure: methods and properties of the current open class/source file (CMD-7).
- Source: the current source files (no hotkey).
- Git: Git status and log (CMD-9) - not shown.
- Gradle: tasks that are available to run (no hotkey) - not shown.
Producing New Code
Running Sample Code
We maintain a public Git repository of the source code shown in lectures. To get a copy, git clone the repository URL. This command, for instance, would create a working copy of the sample code in a directory named cs346.
$ git clone https://git.uwaterloo.ca/j2avery/cs346-public cs346
Each subfolder contains a project built using Gradle and IntelliJ, which should be runnable either from the command-line or from the IDE using Gradle.
You can build and execute these projects directly in IntelliJ:
- File -> Open and navigate to the top-level directory containing the
build.gradle file. Do NOT open a specific file, just the directory. Click Ok.
- After the project has loaded, use View -> Tools -> Gradle to show the Gradle tasks window, and use build -> build and application -> run to Build and Run respectively.
Reading and Understanding Code
IntelliJ can generate UML diagrams from existing code. These diagrams will reflect the structure of actual classes and methods in your application.
The documentation contains a section on source code navigation that is worth reading carefully.
- To navigate backwards, press ⌘ [. To navigate forward, press ⌘ ].
- To navigate to the last edited location, press ⇧ ⌘ ⌫.
- To move caret between matching code block braces, press ⌃ M.
- To move the caret to the next word or the previous word, press ⌥ → or ⌥ ←.
There are also built-in dialogs that help you navigate through existing code.
| Feature |
What it does |
Hotkey |
| Show recent locations |
Show the files and sections that have been viewed recently in a dialog, where you can quickly move between them. |
⇧ ⌘ E |
| Show type hierarchy |
Type hierarchies show parent and child classes of a class. |
⌃ H |
| Show method hierarchy |
Method hierarchies show subclasses where the method overrides the selected one as well as superclasses or interfaces where the selected method gets overridden. |
⇧ ⌘ H |
Gradle Builds
Introduction
When writing complex applications, there is potentially a large list of steps that need to be completed before we can deploy our software. We might need to:
- Download and import new versions of libraries that we’re using.
- Run a code analysis tool against your source code to check for suspicious code, formatting etc.
- Run a documentation tool to generate revised documentation.
- Build a directory structure containing images, fonts and other resources for our executable to use.
- Compile the code and run automated tests to ensure that its working correctly.
- Create an installer that you can use to deploy everything.
Performing these steps manually is error prone, and very time-consuming. Instead of doing this by-hand, we tend to rely on build systems: software that is used to build other software. Build systems provide consistency in how software is built, and let you automate much of the process. They addresses issues like:
- How do I make sure that all of my steps (above) are being handled properly?
- How do I ensure that everyone is building software the same way i.e. compiling with the same options?
- How do I know that I have the correct library versions?
- How do I ensure that tests are being run before changes are committed?
There are a number of build systems on the market that attempt to address these problems. They are often programming-language or toolchain dependent.
- C++: CMake, Scons, Premake
- Java: Ant, Maven, Gradle
Info
Make is one of the most widely used build systems, which allows you to script your builds (by creating a makefile to describe how to build your project). Using make, you can ensure that the same steps are taken every time your software is built.
For small or relatively simple projects, make is a perfectly reasonable choice. It’s easy to setup, and is pre-installed on many systems. However, make has limitations and may not be the best choice for large or more complex projects.
-
Build dependencies must be explicitly defined. Libraries must be present on the build machine, manually maintained, and explicitly defined in your makefile.
-
Make is fragile and tied to the underlying environment of the build machine.
-
Performance is poor. Make doesn’t scale well to large projects.
-
Its language isn’t very expressive, and has a number of inconsistencies.
-
It’s very difficult to fully automate and integrate with other systems.
We’re going to use Gradle in this course:
- It handles all of our requirements (which is frankly, pretty impressive).
- It’s the official build tool for Android builds, so you will need it for Android applications.
- It fits nicely into the Kotlin and JVM ecosystem.
- It’s cross-platform and language agnostic.
You write Gradle build scripts in a DSL (Groovy or Kotlin). You describe tasks, and Gradle figures out how to perform them. Gradle handles dependency management and manages complex dependencies automatically!
Gradle Tasks
Gradle works by running tasks - some are built-in, and you can define your own. Gradle tasks can be executed from the command-line. e.g.
- gradle help: shows available commands
- gradle init: create a new project and dir structure.
- gradle tasks: shows available tasks from build.gradle.
- gradle build: build project into build/
- gradle run: run from build/
$ gradle help
> Task :help
Welcome to Gradle 6.4.1.
To run a build, run gradle
$ gradle build
Starting a Gradle Daemon ...
BUILD PASSED in 2s
Creating a Project
A Gradle project is simply a set of source files, resources and configuration files structured so that Gradle can build it.
Gradle projects require a very specific directory structure. A typical Gradle project directory looks like this:
We could create this by hand, but for now let’s use Gradle to create a starting directory structure and build configuration file that we can modify.
gradle init will run the project wizard to create a new project in the current directory. Select application , Kotlin for a language, and one application project for this sample.
Info
Gradle supports using either Groovy or Kotlin as a DSL. We’ll use Kotlin DSL in all Gradle build files.
$ gradle init
Select type of project to generate:
1: basic
2: application
3: library
4: Gradle plugin
Enter selection (default: basic) [1..4] 2
Select implementation language:
1: C++
2: Groovy
3: Java
4: Kotlin
5: Scala
6: Swift
Enter selection (default: Java) [1..6] 4
Split functionality across multiple subprojects?:
1: no - only one application project
2: yes - application and library projects
Enter selection (default: no - only one application project) [1..2] 1
Select build script DSL:
1: Groovy
2: Kotlin
Enter selection (default: Kotlin) [1..2] 2
Generate build using new APIs and behavior (some features may change in the next minor release)? (default: no) [yes, no]
Project name (default: single-project):
Source package (default: single.project):
> Task :init
Get more help with your project: https://docs.gradle.org/7.6/samples/sample_building_kotlin_applications.html
BUILD SUCCESSFUL in 16s
2 actionable tasks: 2 executed
The directory structure will resemble this:
$ tree -L 4
.
├── app
│ ├── build.gradle
│ └── src
│ ├── main
│ │ ├── kotlin
│ │ └── resources
│ └── test
│ ├── kotlin
│ └── resources
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
└── settings.gradle
app is the application source code folder for our project. app/src is source code, and app/test is for unit tests.gradle is the gradle wrapper files, which allows gradle to bootstrap itself if required. gradlew, gradlew.bat are Gradle scripts that you should use to run commands.settings.gradle and build.gradle are configuration files.
You can use gradle tasks to see all supported actions. The available tasks will vary based on the type of project you create.
$ gradle tasks
> Task :tasks
------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------
Application tasks
-----------------
run - Runs this project as a JVM application
Build tasks
-----------
assemble - Assembles the outputs of this project.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
classes - Assembles main classes.
clean - Deletes the build directory.
jar - Assembles a jar archive containing the main classes.
A “standard” Gradle project has about 30 tasks. Many of them are called infrequently, or called by other tasks (e.g. build calling buildNeeded). The most commonly used commands are build, run and clean.
$ gradle build
BUILD SUCCESSFUL in 8s
8 actionable tasks: 8 executed
$ gradlew run
> Task :run
Hello world.
BUILD SUCCESSFUL in 1s
2 actionable tasks: 1 executed, 1 up-to-date
Single Project Setup
The settings.gradle file contains basic project settings. It specifies the project name, and the directory containing our project source code.
rootProject.name = 'single-project'
include('app')
The build.gradle file contains our project configuration.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
kotlin("jvm") version "1.8.20"
application
}
group = "net.codebot"
version = "1.0.0"
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
repositories {
mavenCentral()
}
dependencies {
testImplementation(kotlin("test"))
}
tasks.test {
useJUnitPlatform()
}
tasks.withType<KotlinCompile> {
kotlinOptions.jvmTarget = "1.8"
}
application {
mainClass.set("single.project.AppKt")
}
The build.gradle file contains information about your project, including the versions of all external libraries that you require. In this project file, you define how your project should be built:
- You define the versions of each tool that Gradle will use e.g. compiler version. This ensures that your toolchain is consistent.
- You define versions of each dependency e.g. library that your build requires. During the build, Gradle downloads and caches those libraries. This ensures that your dependencies remain consistent.
Gradle has a wrapper around itself: gradlew and gradlew.bat You define the version of the build tools that you want to use, and when you run Gradle commands using the wrapper script, it will download and use the correct version of Gradle. This ensures that your build tools are consistent.
Here’s how we run using the wrapper.
$ ./gradlew run
Downloading https://services.gradle.org/distributions/gradle-7.6-bin.zip
...........10%............20%...........30%............40%............50%...........60%............70%............80%...........90%............100%
> Task :app:run
Hello World!
BUILD SUCCESSFUL in 14s
2 actionable tasks: 2 executed
Example: Console
Let’s setup the build for a calculator application.
package calc
fun main(args: Array<String>) {
try {
println(Calc().calculate(args))
} catch (e: Exception ) {
print("Usage: number [+|-|*|/] number")
}
}
class Calc() {
fun calculate(args:Array<String>):Any {
if (args.size != 3) throw Exception("Invalid number of arguments")
val op1:String = args.get(0)
val operation:String = args.get(1)
val op2:String = args.get(2)
return(
when(operation) {
"+" -> op1.toInt() + op2.toInt()
"-" -> op1.toInt() - op2.toInt()
"*" -> op1.toInt() * op2.toInt()
"/" -> op1.toInt() / op2.toInt()
else -> "Unknown operator"
}
)
}
}
Let’s migrate this code into a Gradle project.
- Use Gradle to create the directory structure. Select “application” as the project type, and “Kotlin” as the language.
$ gradle init
Select type of project to generate:
1: basic
2: application
- Copy the
calc.kt file into src/main, and modify the build.gradle file to point to that source file.
application {
// Main class for the application.
// Kotlin generates a wrapper class for our main method
mainClassName = 'calc.CalcKt'
}
- Use gradle to make sure that it builds.
$ gradle build
BUILD SUCCESSFUL in 975ms
- If you use gradle run, you will see some unhelpful output:
$ gradle run
> Task :run
Usage: number [+|-|*|/] number
We need to pass arguments to the executable, which we can do with –args.
$ gradle run --args="2 + 3"
> Task :run
5
Multi-Project Setup
This configuration works well with a single program, but often you want to built related projects together.
e.g.
- console
- graphical client
- shared components
- service
Gradle supports multi-project configurations, so that you can track and manage sub-projects together.
You can add an extra project to the single-project above by adding a second project directory, and then modifying the settings.gradle to include the new project.
For example, here we have added a server project directory and then added it to the settings.gradle file.
This gives us the ability to build both client AND server from the same project.
.
├── app
│ ├── build.gradle
│ └── src
│ ├── main
│ │ ├── kotlin
│ │ └── resources
│ └── test
│ ├── kotlin
│ └── resources
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── server
│ ├── build.gradle
│ └── src
│ ├── main
│ │ ├── kotlin
│ │ └── resources
│ └── test
│ ├── kotlin
│ └── resources
└── settings.gradle
settings.gradle
rootProject.name = 'single-project'
include('app', 'server')
If you’re creating a new project, you can instead choose to run gradle init and select multiple-projects from the wizard. This will generate a multi-project setup with a client, server and shared libraries.
Split functionality across multiple subprojects?:
1: no - only one application project
2: yes - application and library projects
Enter selection (default: no - only one application project) [1..2] 2
Managing Dependencies
This works well if your projects are completely independent, but often you will have shared code that you want to share between projects. We call this relationship a project dependency. In this case, it’s an internal dependency, since we’re responsible for producing all relevant classes ourselves (within our organization).
Project dependencies
To add a shared library that can be used by both our client and server projects, you need to:
- Create a new shared project.
- Add it to the top-level settings.gradle file.
- Add this shared project to the build.gradle.kts file for any project that requires it.
If we modify our project from above, we now have app/, server/ and shared/ projects:
tree -L 4
.
├── app
│ ├── build.gradle
│ └── src
│ ├── main
│ │ ├── kotlin
│ │ └── resources
│ └── test
│ ├── kotlin
│ └── resources
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── server
│ ├── build.gradle
│ └── src
│ ├── main
│ │ ├── kotlin
│ │ └── resources
│ └── test
│ ├── kotlin
│ └── resources
├── shared
│ ├── build.gradle
│ └── src
│ ├── main
│ │ ├── kotlin
│ │ └── resources
│ └── test
│ ├── kotlin
│ └── resources
└── settings.gradle
To include the shared project in the client and server projects, we modify the app/build.gradle.kts and server/build.gradle.kts to include this dependency:
dependencies {
implementation(project(":shared"))
}
External Dependencies
It would be unusual for us to write all of the code in our application. Typically, you’re leveraging software that was written by someone else e.g. OpenGL for graphics, Kotlin standard library for everything in this course. We refer to this relationship between our source code and these external libraries as an external dependency.
The challenge with using libraries like this is ensuring that you are building, testing and deploying against a consistent version of the same libraries.
Traditionally, software was distributed in release archives, or tarballs, by the people that maintained these libraries. Their users would then download this code and compile it into tools or add it as a library to their own applications. This is extremely error prone (“what version did I test with again?”). The modern way to manage dependencies is using a package manager: a system used to manage dependencies and required libraries e.g. npm,pip, go mod, maven, gradle,apt-get,brew, etc.
All package managers work roughly the same at a high level:
- A user asks to install a package or set of packages (with specific versions of each one)
- The package manager performs some basic dependency resolution
- The package manager calculates the full set of transitive dependencies, including version conflict resolution
- The package manager installs them, often from a remote repository.
In this course, we’ll use Gradle for both building software and managing dependencies. Gradle can download specific versions of libraries for us, from an online *repository: a location where libraries are stored and made available. Typically a repository will offer a large collection of libraries, and include many years of releases, so that a package manager is able to request through some public interface, a specific version of a library and all of its dependencies.
Repositories can be local (e.g. a large company can maintain its own repository of internal or external libraries), or external (e.g. a collection of public libraries). The most popular Java/Kotlin repository is mavenCentral, and we’ll use it with Gradle to import any external dependencies that we might require.
You can control the repository that Gradle uses by specifying its location in the build.gradle file.
repositories {
jcenter()
mavenCentral()
}
You add a specific library or dependency by adding it into the dependencies section of the build.gradle file.
dependencies {
implementation("org.jfxtras:jfxtras-controls:17-r1")
}
To locate available packages, use an online package directory. e.g. https://package-search.jetbrains.com
The details include how to import it into your project.
Managing Dependencies
One challenge with setting up dependencies in multi-project builds is that you will have multiple build.gradle.kts files, each with their own list of dependencies and versions.
It’s important to keep your versions consistent across projects. How do we do this?
Version Catalogs
A Version Catalog is a list of versions, plugins and dependencies that we are using in our application. We can extract them from our build.gradle.kts files, and store them in a single location to avoid duplication.
In our settings.gradle.kts, add a Version Catalog like this:
dependencyResolutionManagement {
versionCatalogs {
create("libs") {
// constants
version("jdk", "17")
version("javafx", "18.0.2")
// https://plugins.gradle.org/
plugin("kotlin-lang", "org.jetbrains.kotlin.jvm").version("1.8.10")
plugin("jlink", "org.beryx.jlink").version("2.26.0")
plugin("javafx", "org.openjfx.javafxplugin").version("0.0.13")
plugin("javamodularity", "org.javamodularity.moduleplugin").version("1.8.12")
// https://mvnrepository.com/
library("kotlin-coroutines", "org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4")
library("junit-jupiter", "org.junit.jupiter:junit-jupiter:5.9.2")
}
}
}
version specifies a version string that we can use in our build scripts.plugin specifies a plugin and version number.library is an external dependency, along with group:artifact and version details.
To use these, just replace the hard-coded version numbers, plugins and dependencies in your build.gradle.kts file.
plugins {
application
alias(libs.plugins.kotlin.lang)
alias(libs.plugins.javamodularity)
}
dependencies {
implementation(project(":shared"))
testImplementation(libs.junit.jupiter)
}
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(libs.versions.jdk.get()))
}
}
The multi-project sample in the public-repository is an example that does this across all of the subprojects in the build.
Adding a Custom Task
You can add tasks by defining them in your build.gradle file.
tasks.register ('helloWorld') {
doLast {
println("Hello World")
}
}
$ ./gradlew helloWorld
> Task :app:helloWorld
Hello World
It’s staggering how much software is available through package repositories….
Gradle Template
Here are sample configuration files for a multi-project Gradle build. It has the following structure:
project-template/
├── application/
├── build
├── console/
├── gradle/
├── gradle.properties
├── gradlew
├── gradlew.bat
├── readme.md
├── settings.gradle.kts
└── shared/
application, console and shared all represent projects, with their own configuration files.
Tip
You can find the source code for this project in the GitLab Public Repository for this course, under sample-code/project-template.
Top-Level (Root)
The project root contains the settings.gradle.kts file, which defines the overall project structure.
// the name of the project
rootProject.name = "multi-project"
// which projects to include
include("application", "console", "shared")
// a way of tracking version numbers globally
// this ensures that we use the same version of libraries in each project
// not every project will use all of these plugins or libraries
// see individual project configuration files for the actual usage
dependencyResolutionManagement {
versionCatalogs {
create("libs") {
// constants
version("jdk", "17")
version("javafx", "18.0.2")
// https://plugins.gradle.org/
plugin("kotlin-lang", "org.jetbrains.kotlin.jvm").version("1.8.10")
plugin("jlink", "org.beryx.jlink").version("2.26.0")
plugin("javafx", "org.openjfx.javafxplugin").version("0.0.13")
plugin("javamodularity", "org.javamodularity.moduleplugin").version("1.8.12")
// https://mvnrepository.com/
library("kotlin-coroutines", "org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4")
library("sqlite", "org.xerial:sqlite-jdbc:3.40.1.0")
library("exposed-core", "org.jetbrains.exposed:exposed-core:0.40.1")
library("exposed-dao", "org.jetbrains.exposed:exposed-dao:0.40.1")
library("exposed-jdbc", "org.jetbrains.exposed:exposed-jdbc:0.40.1")
library("junit-jupiter", "org.junit.jupiter:junit-jupiter:5.9.2")
library("sl4j-api", "org.slf4j:slf4j-api:2.0.6")
library("sl4j-simple", "org.slf4j:slf4j-simple:2.0.6")
}
}
}
It also contains gradle.properties, which includes project definitions.
kotlin.code.style=official
Gradle wrapper settings
There is a top-level gradle directory, containing the Gradle bootstrap files (the means by which Gradle downloads and installs itself when you run gradlew). These files are auto-generated by Gradle when you setup the project.
You might want to update the gradle-wrapper.properties to point to a recent version fo Gradle by changing the distributionURL line. In this example, we’re specifying Gradle 8.0.2.
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.0.2-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
Application project
The Application project contains a single src folder, containing the directory tree, and a single build.gradle.kts file which includes the configuration for this specific project. This is a JavaFX application, so we should expect to see application-style plugins and dependencies.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
// notice the syntax for the plugin section uses alias
// this is how we pull in the plugins from the Version Catalog (above)
// e.g. libs.plugins.kotlin.lang inserts the kotlin-lang plugin details.
@Suppress("DSL_SCOPE_VIOLATION")
plugins {
application
alias(libs.plugins.kotlin.lang)
alias(libs.plugins.javamodularity)
alias(libs.plugins.javafx)
alias(libs.plugins.jlink)
}
// used for packaging only
group = "net.codebot"
version = "1.0-SNAPSHOT"
// telling Gradle to put Java and Kotlin output in the same build structure
// required since we have Java files (module-info.java) and Kotlin source
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
// pull all dependencies from here
repositories {
mavenCentral()
}
// libraries that we need, using versions from Version Catalog
// we also want to use the shared/ library, so we need to include it here
dependencies {
implementation(project(":shared"))
implementation(libs.kotlin.coroutines)
testImplementation(libs.junit.jupiter)
}
// fancy way of saying "use JUnit 5"
tasks.test {
useJUnitPlatform()
}
// tell Gradle to use a specific version of the JDK when compiling
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(libs.versions.jdk.get()))
}
}
// application plugin settings
// module name that we've set in the module-info.java
// fully-qualified classname to excecute when you use "gradle run"
application {
mainModule.set("net.codebot.application")
mainClass.set("net.codebot.application.Main")
}
// JavaFX plugin settings
// tell the JavaFX plugin which modules to include
// you may need to add others e.g. javafx.web
javafx {
version = libs.versions.javafx.get()
modules = listOf("javafx.controls", "javafx.graphics")
}
// get around small output bug
// https://stackoverflow.com/questions/74453018/jlink-package-kotlin-in-both-merged-module-and-kotlin-stdlib
jlink {
forceMerge("kotlin")
}
Console Application
The Console application has a similar structure to the Application project, with a single src folder and a build.gradle.kts file for this project. This is a console application so it doesn’t need JavaFX support.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
// notice the syntax for the plugin section uses alias
// this is how we pull in the plugins from the Version Catalog (above)
// e.g. libs.plugins.kotlin.lang inserts the kotlin-lang plugin details.
@Suppress("DSL_SCOPE_VIOLATION")
plugins {
application
alias(libs.plugins.kotlin.lang)
alias(libs.plugins.javamodularity)
}
// used for packaging only
group = "net.codebot"
version = "1.0.0"
// telling Gradle to put Java and Kotlin output in the same build structure
// required since we have Java files (module-info.java) and Kotlin source
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
// pull all dependencies from here
repositories {
mavenCentral()
}
// libraries that we need, using versions from Version Catalog
// we also want to use the shared/ library, so we need to include it here
dependencies {
implementation(project(":shared"))
testImplementation(libs.junit.jupiter)
}
// fancy way of saying "use JUnit 5"
tasks.test {
useJUnitPlatform()
}
// tell Gradle to use a specific version of the JDK when compiling
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(libs.versions.jdk.get()))
}
}
// application plugin settings
// module name that we've set in the module-info.java
// fully-qualified classname to excecute when you use "gradle run"
application {
mainModule.set("net.codebot.console")
mainClass.set("net.codebot.console.MainKt")
}
Shared Project
Finally, the Shared project provides services to both the Application and Console projects. It’s basically a library - not something that we execute directly, but code that we need to pull into the other projects. We keep it in a shared project to avoid code duplication.
Here’s the relevant build.gradle.kts.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
// notice the syntax for the plugin section uses alias
// this is how we pull in the plugins from the Version Catalog (above)
// e.g. libs.plugins.kotlin.lang inserts the kotlin-lang plugin details.
@Suppress("DSL_SCOPE_VIOLATION")
plugins {
`java-library`
alias(libs.plugins.kotlin.lang)
alias(libs.plugins.javamodularity)
}
// used for packaging only
group = "net.codebot"
version = "1.0.0"
// telling Gradle to put Java and Kotlin output in the same build structure
// required since we have Java files (module-info.java) and Kotlin source
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
// pull all dependencies from here
repositories {
mavenCentral()
}
// libraries that we need, using versions from Version Catalog
// notice that the shared project needs sqlite and exposed, since
// it manages a database that the other projects use (indirectly).
dependencies {
implementation(libs.sqlite)
implementation(libs.exposed.core)
implementation(libs.exposed.jdbc)
implementation(libs.sl4j.api)
implementation(libs.sl4j.simple)
testImplementation(libs.junit.jupiter)
}
// fancy way of saying "use JUnit 5"
tasks.test {
useJUnitPlatform()
}
// tell Gradle to use a specific version of the JDK when compiling
java {
toolchain {
languageVersion.set(JavaLanguageVersion.of(libs.versions.jdk.get()))
}
}
For details on module-setup for each of these projects, refer to packages and modules
GitLab for Projects
GitLab is a source code hosting platform similar to GitHub or BitBucket. The University of Waterloo maintains a hosted instance of GitLab, and you can login at https://git.uwaterloo.ca using your UW credentials .
We need some way to track project information, in a way that is transparent and makes information available to everyone on the team.
There are many ways of accomplishing this. Some organizations track project artifacts on paper, with written SRS documents, requirements and estimates on spreadsheets, written test plans and so on. However, this is very inefficient, requiring changes to be carefully coordinated across different documents and tracking systems. More recently, it has become common to use online project tracking. We’ll use GitLab to track all of our project artifacts.
Capabilities
GitLab offers the following functionality high-level functionality:
| Feature |
Description |
| Repository |
Version control for source code, or other files. |
| Issues |
Mechanism to track project tasks or issues. They can capture details, be assigned a due date, passed between people and have an associated status that changes as the project progresses (e.g. Open, Closed). |
| Wiki |
Create online documents using Markdown. This is very useful for longer-documents (e.g. design documents). |
| CI/CD |
Continuous Integration and Deployment. We can setup rules that will automatically test or deploy our source code when it’s committed (or other conditions are met). This includes powerful analytics and reporting capabilities (that are beyond what we will cover in this course). |
| Snippets |
Share snippets of source code outside of the project. |
Project Tracking
Here’s some suggestions on how we can use GitLab at each phase of the SDLC:
| Phase |
SDLC Activities |
GitLab Feature |
| Planning |
We need to document project goals and objectives; log and assign requirements (which we can track as issues). |
Wiki, Issues |
| Requirements |
Definition and elaboration of requirements that are identified during the Requirements stage. Update issues as required. |
Wiki, Issues |
| Analysis & Design |
Track design decisions that are made and documented during the Analysis and Design phase. Document these decisions and make information available. |
Wiki |
| Development |
Log and document issues that are discovered during the Development phase. Manage feature branches and store source code. |
Issues |
| Testing |
Log and document issues that are discovered during the Testing phase. Manage unit tests. Handle continuous integration. |
Issues |
| Evaluation |
Log issues that are found. Log change requests (CR), based on customer feedback. |
Issues |
| * |
Manage source code, tests and all related materials. Ensure that we are using consistent versions of everything. |
Repository |
Project Structure
Setup
To start using GitLab for this course, you should do the following:
- Create a new blank project in GitLab with a meaningful name and description.
- When it opens, select
Members from the left-hand menu. Invite each member of your team with the appropriate role: typically, Developer or Maintainer for full access.
- Check to ensure that each member can login on their own machine using their own credentials.
- Optional. Under
Settings - General, add a project description. Create and upload an avatar!
Organization
Most project information can be tracked in the Issues menu (left-hand side when the project is open).
Each major deliverable should have a milestone attached to it. For this project, this means that each sprint should be a separate milestone.
- Under Issues-Milestones, create suitable milestones (aka deadlines). Make sure to assign dates to these!
- Under Issues-Labels, create keywords that will help you organize your issues.
- Under Issues-List, create all of the issues that you wish to work towards. At the beginning of your project, none of your issues should be assigned to a person; they should be listed as “No milestone”, since you haven’t scheduled them yet.
Common Tasks
Writing Documents
GitLab projects have a Wiki attached to them, which you can use to create hyperlinked documents, with formatting, images and so on. It’s an ideal place to store any documentation that you might create, from your Project Plan, to an SRS, to Architecture digrams. A Wiki also has the advantage of staying up to date with your other project details. e.g. you can create a Project Plan that links to Issues in your GitLab project.
GitLab uses Markdown as its native format (specifically GitHub-Flavored Markdown), a common human-readable data format.
Tracking Issues
GitLab has the ability to attach and track issues to a project, as shown below.
An issue is meant to represent some unit of work that can be appled to the project. Historically, they often referred to “bugs” or software defects. However, for planning purposes, there is little difference between a feature, a change request and a bug - they all represent work that needs to prioritized, assigned to someone and scheduled.
As suggested above, your defaults for a new issue should be Assignee: Unassigned, Milestone: No milestone. When you schedule it into a Sprint, then you change the Milestone to reflect that sprint and the assignee to the team member responsible for it.
Updating Issues
Issues should be considered living documents, that reflect the current state of your work.
- Issues should be assigned as part of the sprint kickoff
- when you do something significant, you should add a comment to the issue! this helps you recall, and helps your team mates if they need to help out.
- when you complete it, mark it completed.
Subsections of Software Process
Motivation
Definitions
Software is the set of programs, concepts, tools, and methods used to produce a running system on computing devices – John Tukey, 1958.
A program is a set of instructions that inform a computer how to perform a very specific task. Every computer system consists of thousands of specialized, small-to-medium sized programs working in harmony to deliver a functioning system. No single program exists in isolation, and any programs that we write and execute depend on a number of other programs for their creation and operation.
When we talk about programs, we’re talking about a very broad category that includes:
- System software: programs written to provide services to other software. This includes operating systems, drivers, compilers and similar systems.
- Application software: programs written primarily for users, to help them solve a problem or perform a task, typically by manipulating and processing information. Applications are common on mobile or desktop devices, and on the web.
In this course, we’re going to focus on application software, or software designed to solve tasks for people. This includes work and productivity software, games or other types of useful programs that people use everyday. e.g. software to manipulate numbers (Excel), compose documents (Word), write code (Vim) or entertain ourselves (League of Legends).
In particular, we’re doing to talk about full-stack application software.
Full-stack is an overloaded term in software development. Historically, it referred to the idea that an application often has a “front-end” (user interface) and a “back-end” (server or process performing the actual computation). A full-stack developer in that context is someone who is capable of working on both “sides” of this boundary: the front-end that the user sees, and the back-end that does the “heavy lifting”.
In modern usage, we tend to think of full-stack in terms of a local “front-end” application, and one or more remote “back-end” services. e.g. a Twitter client fetches data from a remote system, and displays them on your local device. The application that you are running is responsible for communicating with, and fetching data from this remote system, as well as any local operations.
We focus on full-stack development because most software has some remote functionality that it leverages. In today’s world of “big data”, it’s common to write applications that rely on shared, remote data. Even so-called “standalone” applications will typically rely on a remote system to check that the user is licensed, or to check for application updates on execution.
Full-stack application development then refers to designing and building applications that offer client-side functionality, but that can also leverage back-end services when required.
“Software development refers to a set of computer science activities dedicated to the process of creating, designing, deploying and supporting software." – IBM Research, 2022.
Software development is the complete process of taking a software system through requirements, design, implementation and eventual delivery to a customer. Software development isn’t just programming, but consists of many interrelated activities across different disciplines. It’s important to consider the broader context so that we can be confident that
- we have a target user, and a problem that they need solved.
- we have determined the characteristics or requirements of a solution.
- we have delivered something that will address their problems in an efficient and elegant fashion.
This is a course on software development in this broader-sense: going from initial requirements and working through the process to deliver a final product that someone can use and enjoy. However, for the sake of expediency (and the need to fit everything in a single term!) we will focus most of our attention on the design + implementation of software applications.
Software is only useful if it solves a problem for someone. We cannot develop software in a vacuum.
The importance of quality
The growth of software systems has easily been the most pervasive technological advancement of the past 50 years. Software underpins most industries: banking, finance, transportation, manufacturing, research, education, retail. It is practically impossible to live and work in modern society without interacting with software on a daily basis.
Due to this, the software that we design and write can have significant real-world impact :
- Software has become embedded in virtually every aspect of our lives. Although software was once the domain of experts, this is no longer the case, and software is routinely used by consumers. We should strive to build software to meet the needs of all people.
- The information technology requirements demanded by individuals, businesses, and governments grow increasingly complex with each passing year. Design is a critical activity to ensure that we address requirements and build the “right thing”.
- If the software fails, people and major enterprises can experience anything from minor inconvenience to catastrophic consequences. Software should be high-quality, safe and reliable.
- As the perceived value of a specific application grows, the likelihood is that its user base and longevity will also grow, and demands for adaptation and enhancement will also grow. Software should be adaptable and scalable.
For these reasons, software design is mandatory to ensure that we build high-quality, stable, scalable systems. This will be a major topic as we progress through the course.
Development Lifecycle
Our ultimate goals as software developers is to build effective, high-quality software systems. This requires discipline:
Software Engineering: The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software.
A software process is the collection of activities, actions, and tasks that are performed when to create software. An activity is used to achieve a broad objective (e.g., communication with stakeholders) and ios applied during the process. Actions (e.g., architectural design) encompasses a specific set of tasks that may be used for a particular activity (e.g., an architectural model).
Process Activities
A process framework establishes the foundation for a complete software engineering process by identifying a small number of framework activities that are applicable to all software projects, regardless of their size or complexity. In addition, the process framework encompasses a set of umbrella activities that are applicable across the entire software process. A generic process framework for software engineering encompasses five activities:
-
Communication: We need to define the problem that we’re trying to solve, and discuss the goals and requirements with the customer, potential users, and other stakeholders. This is critical to ensure that we’re building the “right” product or solving the main problem that needs to be addressed.
-
Planning: This involves defining the tasks that need to be completed, and high level milestones that you need to achieve. It also includes identify people and resources, potential risks to the project, and developing a plan to mitigate these risks. These are what we would typically call “project management” activities.
-
Modeling: It’s always cheaper to think ahead prior to actually building something. This step includes design efforts to create abstract models or representations of the product that you need to build. By iterating on a design, you refine your understanding of the problem, and move towards a more-correct solution. This is crucial before you actually start building it, both to improve the accuracy and usefulness of what you build, but to minimize costs.
-
Construction: The process of building your product i.e. realizing your design. This may require successive iterations to complete all of the required features, and should also include some level of testing and validation.
-
Deployment: Tasks required to produce a working product, and deliver it to the customer. This includes collecting their ongoing feedback and making revisions as needed.
Let’s review these in more detail.
Communication
Products, in general, are designed and built to solve particular problems for users. The first, and most important step in any project is ensuring that you understand both the users, and the problem that you are attempting to address. Once that’s done, you need to ensure that there is agreement by everyone on both the problem and proposed solution.
Users
The people impacted by your product are called stakeholders. This includes the people who will use your solution, but can also include anyone else that is affected by it. For example, in a software project, stakeholders can include:
- users, the people who will directly utilize your software;
- the Information Technology (IT) department who will have to install and maintain your software;
- people who do not directly use your product, but who may need to provide input to your system, or work with it’s output;
- your company, and you personally since you presumably need to maintain the software over time.
As software designer and developers, we tend to focus on the actual users of our software and services, but we need to consider the needs of all of these stakeholders.
Requirements
Requirements analysis is the set of activities designed to identify problems in sufficient detail to determine a solution. Requirement specification is the identification and documentation of the capabilities or requirements that are needed to address a particular problem for stakeholders. Types of requirements include:
- Architectural requirements: Related to the system architecture of a system. e.g. how it will integrate with an existing system, or how it must be deployed to be successful.
- Business requirements: High-level organizational goals that your product or solution will help address, directly or indirectly.
- User requirements: Statements of the needs of a particular set of stakeholders (namely, those that will use your product or software).
- Implementation requirements: Changes that are required to facilitate adoption of your solution. This can include education and training, data migration, and any other work that is triggered by the adoption of your system.
- Quality of service requirements: Detailed statements the system’s qualities. Examples include: reliability, testability, maintainability, availability.
- Regulatory requirements: Laws or policies, imposed by a third-party. e.g. privacy rules related to health data that your product might collect and use.
In software development, we are mostly focused on the design and implementation of a system that meets user requirements. In other words, we focus on the end-users of our product and attempt to design a product that is useful and usable to meet their particular needs.
We can also think about these requirements as being about system capabilities versus system qualities:
-
The capabilities of a system refers to the functionality that we will design and implement. Capabilities are also known as functional requirements, and include user requirements from the list above, plus any other requirements that directly result in a product feature. These are the requirements that we will focus on in this phase.
-
The qualities that the solution should have, constraints under which it might operate, are called non-functional requirements. These include quality of service from the list above. For software, this includes capabilities like accuracy, speed, quality of service. We will focus on non-fnctional requirements in the Analysis & Design phase.
Our goal as designers is to design and build a system that meets both capabilities and qualities.
Planning
Planning activities determine what actions are required to meet requirement and project goals, and establish a plan to deliver the project on-time, on-budget, with the requirements met.
Project planning has to consider:
- Project goals: There may be additional goals outside of the product itself. e.g. Integrate a remote testing team into our development process; deliver X in revenue during delivery and so on.
- Resources: Who and what we have available to dedicate to the project. Typically this means allocating staff, budget, necessary resources. If you and the customer are both committing resources, this is the time to identify what those might be.
- Constraints: Other factors that we need to consider e.g. we need a demo for a tradeshow in Spring 2022.
The triple constraint model (“project management triangle”) is a model of the constraints of project management. While its origins are unclear, it has been used since at least the 1950s. It contends that: The quality of work is constrained by the project’s budget, deadlines and scope. In other words, quality depends on the relationship between project costs, what is being done (scope) and time required. The relationship between these constraints, and what tradeoffs work is rarely very straightforward, but the constraints themselves are real. Projects do have time and budget constraints that need to be respected, and we need reasonable confidence that we can deliver our scope within those constraints.
This model is pervasive, but also has it’s detractors who point out that there are many times when this model does not work. e.g. late projects are often also over-budget.
Modeling
Before actually constructing anything, we want to ensure that we have met both functional requirements (gathered from users), and non-functional requirements (qualities that we want our system to have). It’s useful to think of non-functional requirements as the constraints or conditions on how our solution works:
- Technical constraints: requirements made for technical reasons (e.g. must be implemented in C++ for compatibility with our existing libraries).
- Business constraints: requirements made for business reasons (e.g. must run on Windows 11 because that’s what we have deployed at customer sites, or must use Java because that’s where we have expertise as a development team).
- Quality attributes: scalability, security, performance, maintainability, evolvability, reliability, deployability (e.g. must complete a core task in less than 5 seconds; must demonstrate 99.999% uptime; must support 1000 concurrent users).
Ideally, before we actually attempt to build something, we want to think through and plan the work, confirm our designs and assumptions, and fine-tune our understanding of the problem. The exact nature of modeling will vary based on the type of project, but can include building prototypes (to show a customer and confirm our understanding), or mockups of screens (to verify that the interface is clear to users).
Typically modeling is done in iterations, where we design something, use that to confirm our understanding with users, make corrections and modifications to our design, and continue iterating until we feel that we can proceed with construction. Taking time to design first saves considerable time and cost in the construction phase.
Construction
This is the step of actually manufacturing a product, or a software system, based on your earlier designs. This is typically the most expensive and time-consuming step of the project, and consumes most of our time and resources. Although we prefer to have a near-perfect design by the time we arrive at this stage, it’s common to have to iterate on design elements prior to realizing the completed product. Details of this step are highly dependent on the type of product that you’re building; we won’t focus too much on it at this time.
Deployment
Similarly, deployment may include packaging and distribution of a physical product. In some cases it many even include installation and validation on behalf of a customer, or on a customer site. We’ll discuss this later in the context of software delivery. For now, just appreciate that this can be a very costly step; correcting mistakes at this point is nearly impossible without going through the entire process again.
Umbrella Activities
Software engineering process framework activities are complemented by a number of umbrella activities. In general, umbrella activities are applied throughout a software project and help a software team manage and control progress, quality, change, and risk. Typical umbrella activities include:
- Software project tracking and control. Allows the software team to assess progress against the project plan and take any necessary action to maintain the schedule.
- Risk management. Assesses risks that may affect the outcome of the project or the quality of the product.
- Software quality assurance. Defines and conducts the activities required to ensure software quality.
- Technical reviews. Assess software engineering work products in an effort to uncover and remove errors before they are propagated to the next activity.
- Measurement. Defines and collects process, project, and product measures that assist the team in delivering software that meets stakeholders’ needs; can be used in conjunction with all other framework and umbrella activities.
- Software configuration management. Manages the effects of change throughout the software process.
- Reusability management. Defines criteria for work product reuse (including software components) and establishes mechanisms to achieve reusable components.
- Work product preparation and production. Encompasses the activities required to create work products such as models, documents, logs[…]
Software engineering process is not a rigid prescription that must be followed dogmatically by a software team. Rather, it should be agile and adaptable (to the problem, to the project, to the team, and to the organizational culture). Therefore, a process adopted for one project might be significantly different than a process adopted for another project.
Process Models
We use the term process model to describe the structure that is given to these activities. That is, it defines the complete set of activities that are required to specify, design, develop, test and deploy a system, and describes how they fit together. A software process model is a type of process model adapted to describe for software systems.
Software activities tend to be named slightly differently than the generic activity names that we’ve been using:
Compared to the standard model, we’ve split Planning into separate Planning and Requirements definition. These activities are often performed by different departments or individuals, so traditionally they’re split apart. Analysis & Design corresponds to Modeling, and Implementation and Testing together correspond to Construction.
This is a simplified conceptual understanding of the different “pieces” of a software development project. People generally accept that we have planning, formal requirements, modeling and design, implementation and testing - but there’s lots of disagreement on how these pieces “fit together”. Let’s continue talking about different forms of process models that we could use:
Waterfall Model
In the 1970s, there was a concerted effort to formalize ‘known-good’ methods of project management. Software projects were seen as expensive and time-consuming, and there was considerable pressure to improve how they were managed. In a 1970 paper, Winston Royce laid out a mechnism for formalizing the large-scale management of software projects [Royce 1970], dubbed the Waterfall model. This envisions software production as a series of steps, each cascading into the next one, much like a waterfall. In this model, requirements are defined first, a design is created and then implemented, then tested and so on. Software development is treated as a set of linear steps that are followed strictly in-order.
The Waterfall Model, as understood and practiced for a long time, most closely resembles a linear project model, and is similar to how other construction or manufacturing projects are organized.
The Waterfall model the following characteristics:
- A project starts at the top and advances through stages. Each stage must be completed before the next stage begins.
- The stages are modeled after organizational units that are responsible for that particular stage (e.g. Product Management owns Requirements, Architects own Analysis & Design, QA owns Testing and so on).
- There are criteria that need to be met before the project can exit one stage and enter the subsequent stage. This can be informal (e.g. an email letting everyone know that the design is “finished”), to a more formal handoff that includes artifacts (e.g. Product Requirements documents, Design documents, Test Plans and so on).
This linear approach strongly suggests that you can and should define a project up-front (i.e. determine cost, time and so on). This can be a very appealing proposition to risk-adverse businesses, but as we’ll see, this may not be realistic: requirements change, often as the project is underway, which makes this style of project structure difficult.
V-Model
The V-Model is an alternative that attempts to line-up the testing phase with the area of responsibilityt that is being tested. It’s a useful conceptual model, but it’s unclear how this is supposed to address the criticisms of a straight linear model.
Spiral Model
A spiral model acknowledges that iteration is useful, and suggests iterating from a high-level of abstraction through to lower-levels of detail. The product manager in this case is supposed to define the levels of detail (soooo close, but still problematic).
The Agility Movement
The late 90s were a particularly active period in terms of advancing software process. There was a widespread recognition that old, manufacturing-based ways of building software just didn’t work - either for developers or for customers.
There are a large number of software process models that were developed at this time, including Extreme Programming (XP) [Beck 1999] , Scrum [Schwaber & Sutherland 1995], Lean [ Poppendieck & Poppendieck 2003]. Collectively, these are called “Agile Processes”.
This culminated in In 2001, when a group of software developers, writers, and consultants signed and published the Manifesto for Agile Software Development.
“Agile Software Development” isn’t a single process, but rather an approach to software development that encompasses this philosophy. It encourages team structures and attitudes that make communication easier (among team members, business people, and between software engineers and their managers). It emphasizes rapid delivery of operational software, but also recognizes that planning has its limits and that a project plan must be flexible .
What does this mean?
- Individuals and interactions (over process and tools): Emphasis on communication with the user and other stakeholders.
- Working software (over comprehensive documentation): Deliver small working iterations of functionality, get feedback and revise based on feedback. You will NOT get it right the first time.
- Customer collaboration (over contract negotiation): Software is a collaboration between you and your stakeholders. Plan on meeting and reviewing progress frequently. This allows you to be responsive and correct your course early.
- Responding to change (over following a plan): Software systems live past the point where you think you’re finished. Customer requirements will change as the business changes.
Info
Agile is also implicitly about shifting power and decision making from product managers and other business leaders to the development team, the ones actually building software. At least part of the failure of previous models is the failure to understand that development is not usually predictable. We’re often building something for the first time, or solving a unique problem, so it’s extremely difficult to predict the outcome far in advance.
What is the benefit?
Agility means recognizing that requirements and plans will change over time.
- Software is too complex to design and build all at once. It’s more manageable to add features and test incrementally.
- Software is in a constant state of change, and requirements will change during the development cycle.
The conventional wisdom in software development is that the cost of change increases nonlinearly as a project progresses. Agility is often characterized as “embracing change” since it expects project and requirements changes, and is constantly reassessing the state of the project. The benefit of Agile is that it reduces (and tries to eliminate) breaking late-project changes.
Agile assumptions
Any agile software process is characterized in a manner that addresses a number of key assumptions about the majority of software projects:
- It is difficult to predict in advance which software requirements will persist and which will change. It is equally difficult to predict how customer priorities will change as the project proceeds.
- For many types of software, design and construction are interleaved. That is, both activities should be performed in tandem so that design models are proven as they are created. It is difficult to predict how much design is necessary before construction is used to prove the design.
- Analysis, design, construction, and testing are not as predictable (from a planning point of view) as we might like.
Given these three assumptions, how do we create a process that can manage unpredictability?
Central to Agile processes is that any propcess must be adapatable to rapidly changing project and technical conditions. It must also be incremental and incorporate customer feedback so that the appropriate adaptations can be made.
This idea of an iterative, evolutionary development model remains central to all Agile processes (although they may present it differently). Instead of building a “complete” system and then asking for feedback, we instead attempt to deliver features in small increments, ina. way that we can solicit feedback continuously though the process. Over time, we will add more features, until the we reach a point where we have delivered sufficient functionality and value for the customer.
Note that in this process model, we still do some initial project planning and requirements definition, but the majority of our time is spent iterating over features. Every time we implement and validate some new functionality, we have the opportunity to deploy it (either for further customer testing, or as a release).
Let’s take some time and talk about the two most influential process models: Scrum and Extreme Programming (XP).
Scrum & Sprints
If you adopt only one agile practice, let it be retrospectives. Everything else will follow.
– Woody Zuill
The important thing is not your process. The important thing is your process for improving your process.
– Henrik Kniberg.
Scrum is the defacto process model for managing scope during a project iterations i.e. it’s focused on the overall project structure. Scrum breaks down a project into fixed-length iterations called sprints (typically 2-4 weeks in length for each sprint). Sprints are defined so that you iterate on prioritized features in that time, and produce a fully-tested and shippable product at the end of each sprint.
Typically a project will consist of many sprints, and you will iterate until you and the customer decide that you’re done (i.e. the only remaining requirements are deemed low enough priority that you decide to defer them or not complete them). Practically, having a buildable and potentially “shippable” product at the end of each cycle is incredibly valuable for testing, demonstrating functionality to customers, and it provides flexbibility in how you deply.
In Scrum, everything is structured around sprints:
Key Concepts
- Product Owner: the person responsible for gathering requirements and making them available in the product backlog. They are not considered part of the project team, but represent both the external business and customer. At the start of each sprint, they work with the team to prioritize features and decide what will be assigned to a sprint.
- Product Backlog: a list of all possible features and changes that the Product Owner thinks should be considered. There is no guarantee that these will all be developed! The team must agree to bring features forward into a sprint before they are developed.
- Sprint Backlog is the set of features that are assigned to a specific sprint. This is the “scope” for that sprint.
- The Scrum Master is the person that helps facilitate work during the sprint. They are not in charge (!) but track progress and help identify blocking issues that might prevent the team from meeting their deliverables.
- The Daily Scrum is a standup meeting where you discuss (a) what you’ve done since the last meeting, (b) what you intend to do today, and (c) any obstacles that might prevent you from accomplishing b. The Scrum Master runs this meeting, and the entire team attends.
Sprint Breakdown
The following steps are followed in each sprint:
- The project team and Product Owner collectively decide what requirements to address, and they are moved from the Product Backlog to the Sprint backlog. Once features have been decided, you do not allow any further scope changes (i.e. you cannot add anything to the sprint once its started). Work is actually assigned to team members (and you collctively agree that you believe it can be completed in the sprint).
- During the sprint, you iterate on the features in the Sprint Backlog. This includes design, development, testing etc. until you complete the feature or the sprint is finished. The Scrum Master facilitates aily meetings to make sure that nobody is “stuck” on their feature.
- At the end of the sprint, have a review with the team to see what was accomplished. Demo for the Product Owner (and sometimes the actual customer). Reflect on your progress, and be critical of how you might improve process the next sprint. (e.g. could we have communicated better? should we have done more testing during the sprint? did we take on too many features?)
Extreme Programming (XP)
The most important thing to know about Agile methods or processes is that there is no such thing. There are only Agile teams. The processes we describe as Agile are environments for a team to learn how to be Agile.
– Don Wells
Extreme Programming (XP) is an Agile methodology focused on best-practices for programmers. It was based on a large-scale project that Kent Beck managed at Chrysler in the late 90s, and attempted to capture what was working for them at that time. It aims to produce higher-quality software and a higher quality-of-life for the development team.
The five core values of XP are communication, simplicity, feedback, courage, and respect.
- Communication: The key to a successful project. It includes both communication within the team, and with the customer. XP empasizes face to face discussion with a white board (figurtively).
- Simplicity. Build the “simplest thing that will work”. Follow YAGNI (You Ain’t Gonna Need It) and DRY (Don’t Repeat Yourself ).
- Feedback. Team members solicit and react to feedback right away to improve their practices and their product.
- Courage: The courage to insist on doing the “right thing”. The course to be honest with yourselves if something isn’t working, and fix it.
- Respect: Respect your team members, development is a collaborative exercise.
XP launched with 12 best practices of software development [Beck 2004]. Some of these (e.g. The Planning Game, 40-Hour Week, Coding Standard) have fallen out of disuse. Others have been added or changed over time, so it is difficult to find a “definitive” list of commonly used XP practices .
Although some XP practices never really worked very well, many have been adopted as “best practices”. We’ll revisit these in the next secion.
Info
XP is rarely used as-is. It’s common for development teams to adopt one or more of these ideas based on what suits them, and their environment. For example, daily standups are very common, but very few places will implement pair programming.
Being Agile
We have generations of people claiming to have the “correct” solution, or the best process model. The reality is, every process model is a reflection of the organization that developed it. There are many process and all are equally valid in their domain.
In this section, we identify commonly used Agile principles and practices that we will use in this course. Although there will always be some contention on what constitutes “best practices”, this is a very common subset of approaches that in practice will work very well.
Info
Teams SHOULD be adopting their own best practices and process according to their specific domain. It’s reasonable to assume that healthcare, game development, telecom, compiler development all have unique work environments and constraints, that make it reasonable to customize how they work for that environment.
Principles
From these different Agile models, we can extract a set of useful guiding principles [Pressman 2018]. This is what we aspire to do with our practices.
- Principle 1. Be agile. The basic tenets of agile development are to be flexible and adaptable in your approach, so that you can adjust if needed between iterations. Keep your technical approach as simple as possible, keep the work products you produce as concise as possible, and make decisions locally whenever possible.
- Principle 2. Focus on quality at every step. The focus of every process activity and action should be the quality of the work produced.
- Principle 3. Be ready to adapt. When necessary, adapt your approach to constraints imposed by the problem, the people, and the project itself.
- Principle 4. Manage change. The approach may be either formal or informal, but mechanisms must be established to manage the way changes are requested, assessed, approved, and implemented.
- Principle 5. Build an effective team. Software engineering process and practice are important, but the bottom line is people. Build a self-organizing team that has mutual trust and respect.
- Principle 6. Establish mechanisms for communication and coordination. Projects fail because important information falls into the cracks and/or stakeholders fail to coordinate their efforts to create a successful end product. Keep lines of communication open. When in doubt, ask questions!
Process
Using the SDLC
We can describe our process as the Software Development Lifecycle (SDLC). This illustrates the process that we will follow for this project. Each block in the diagram represents a stage, containing related activities, that are performed in order.
To complete a project, you start with Planning activities, and move through Requirements, Analysis & Design and so on. The project is complete when you finish Evaluation and the team collectively decides that they are “done” (or you run our of time/resources!).
Note that the preliminary activities (Planning, Requirements, Analysis & Design) are only performed once.
Implementation and related activities are grouped together, since they are performed in-order, but we perform multiple passses over all of them. This iteration is called a sprint (taken from Scrum). In a typical development project, sprints should be relatively short, from two to four weeks in length, and the team typically works through multiple sprints until the project is completed.
Info
In our course, Sprints are two-weeks long, and we will have four sprints in total (i.e. 4 x 2-week sprints).
Each sprint includes the following activities:
- Feature Selection: On the first day of the Sprint, the team meets and decides what features to add (and what bugs to fix) during that iteration.
- Implementation/Testing. During most of the sprint, the team iterates on their features. As each feature is completed, it is tested.
- Evaluation. At the end of the Sprint, the team meets with the Product Owner to demo what they have completed, and get feedback. The team also has a Retrospective, where they reflect on how to improve.
The cycle repeats for however many Sprints the team has available (or until they decide they are “done”). The product should be usable and potentially shippable to a customer at the end of each Sprint (though obviously features may not be complete, but the ones that exist should be bug-free).
Best Practices
Along with the SDLC, we also have a set of best practices that we will use. The SDLC provides the overall organization (what we should do), and these practices provide more strict guidelines (how we should do it).
These will appear in later chapters related to specific activities.
-
User stories: describe features in a way that makes sense to customers.
-
Pair programming: critical code is written by two people working as a team; one codes while the other one watches, plans and makes suggestions. This results in demonstrably better code and is much more productive than working alone [Böckeler & Siessegger 2020] .
-
Test-driven development: tests are written before the code. This helps to enforce contracts/interfaces as a primary focus of your design. We will discuss this in the Implementation section.
-
Code reviews: before code changes are committed to the repository, they need to be reviewed by one or more other developers on the team, ideally more senior members. The claim is that (a) the developer gets feedback to help identify bugs, and improve their design, and (b) participating in code reviews helps spread knowledge about that code around the team. Research suggests that code reviews most often result in design recommendations, and aren’t particularly effective at finding bugs [Czerwonka et al. 2015].
https://xkcd.com/844/
Best Practices
Software process is just speculation unless we are actually able to apply these ideas to practice. This section discusses which practices we will adopt from the previous chapter, and how we will integrate them into our SDLC. In industry, these could be considered “best practices” - that is, something that is commonly held to be beneficial to a team, and almost universally adopted (in one form or another).
Tip
It’s tricky to position these as “best practices”, since it implies that (a) every team should or will do them, and (b) they are somehow the best of what we have available. Neither of these is strictly true. You’ll find that teams will gravitate towards the practices that benefit their particular situation and context. e.g. a team of experienced developers may prefer pair programming over code reviews. The term itself, “best practices” is just the common way that we refer to processes that teams generally agree are helpful.
As a reminder, our lifecycle diagram from the previous chapter suggests an iterative development sequence that we will follow when designing and building our project:
We’re more concerned about development practices, so we’ll focus on the interative part of that model. Here are the practices that we’ll discuss in this course. They will be introduced slowly, but by the end of the term, you should be doing all of them routinely.
Implementation
Testing
Deployment
Subsections of Best Practices
Pair Programming
Pair programming means that two people design and implement code together, on a single machine. This is a very collaborative way of working that involves a lot of communication and collaboration between them. While a pair of developers work on a task together, they do not only write code, they also plan and discuss their work. They clarify ideas on the way, discuss approaches and come to better solutions.
Originally an Extreme Programming practice, it is considered a best-practice today, used successfully on many programming teams.
Benefits
There are tangible benefits to pair programming - both to the organization and project team, and to the quality of the work produced.
For the organization:
- remove knowledge silos to increase team resiliency
- build collective code ownership
- help with team building
- accelerate on-boarding
- serve as a short feedback loop, similar to a code review happening live
For the team, and the developers:
- improve learning
- increase efficiency
- improve software design
- improve software quality
- reduce the incidence of bugs
- increase satisfaction
- increase safety and trust of the developers pairing
- increase developer confidence
Research supports the idea that a pair of programmers working together will produce higher quality software, in the same or less time as the developers working independently (Williams et al. 2000).
Surprising, research also suggests that there is no loss in productivity when pair programming. In other words, they produce at least the same amount of code as if they were working independntly, but it tends to be higher quality than if they worked alone.
Styles of Pairing
Styles adapted from https://martinfowler.com/articles/on-pair-programming.html#HowToPair
Driver and Navigator
This is the classic form of pair programming.
The Driver is the person at the wheel, i.e. the keyboard. She is focussed on completing the tiny goal at hand, ignoring larger issues for the moment. A driver should always talk through what she is doing while doing it.
The Navigator is in the observer position, while the driver is typing. She reviews the code on-the-go, gives directions and shares thoughts. The navigator also has an eye on the larger issues, bugs, and makes notes of potential next steps or obstacles.
Image from https://unruly.co/.
Remember that pair programming is a collaboration. A common flow goes like this:
- Start with a reasonably well-defined task. Pick a requirement or user story for example.
- Agree on one tiny goal at a time. This can be defined by a unit test, or by a commit message, or some goal that you’ve written down.
- Switch keyboard and roles regularly. Alternating roles keeps things exciting and fresh.
- As navigator, leave the details of the coding to the driver - your job is to take a step back and complement your pair’s more tactical mode with medium-term thinking. Park next steps, potential obstacles and ideas on sticky notes and discuss them after the tiny goal is done, so as not to interrupt the driver’s flow.
Ping-Pong
This technique is ideal when you have a clearly defined task that can be implemented in a test-driven way.
- “Ping”: Developer A writes a failing test
- “Pong”: Developer B writes the implementation to make it pass.
- Developer B then starts the next “Ping”, i.e. the next failing test.
- Each “Pong” can also be followed by refactoring the code together, before you move on to the next failing test.
Strong-Style Pairing
This is a technique particularly useful for knowledge transfer. In this style, the navigator is usually the person much more experienced with the setup or task at hand, while the driver is a novice (with the language, the tool, the codebase, …). The experienced person mostly stays in the navigator role and guides the novice.
An important aspect of this is the idea that the driver totally trusts the navigator and should be “comfortable with incomplete understanding”. Questions of “why”, and challenges to the solution should be discussed after the implementation session.
Switching Roles
Here’s some suggestions on role-switching from Shopify Engineering:
Switching roles while pairing is essential to the process—it’s also one of the trickiest things to do correctly. The navigator and driver have very different frames of reference.
The Wrong Way
Pairing is about working together. Anything that impedes one of the pairers from contributing or breaks their flow is bad. Two of the more obvious wrong ways are to “grab the keyboard” or “push the keyboard”.
Grabbing the keyboard: Sometimes when working as the navigator it’s tempting to take the keyboard control away to quickly do something. This puts the current driver in a bad position. Not only are they now not contributing, but such a forceful role change is likely to lead to conflict.
Pushing the keyboard: Other times, the driver feels a strong need to direct the strategy. It’s very tempting to just “push” the keyboard to the navigator, forcing them to take the driver’s seat, and start telling them what to do. This sudden context switch can be jarring and confusing to the unsuspecting navigator. It can lead to resentment and conflict as the navigator feels invalidated or ignored.
Finally, even a consensual role switch can be jarring and confusing if done too quickly and without structure.
The Right Way
The first step to switching roles is always to ask. The navigator needs to ask if they can grab the keyboard before doing so. The driver needs to ask if the navigator is willing to drive before starting to direct them. Sometimes, switching without asking works out but these situations are the exception.
It’s important to take some time when switching as well. Both pairers need to time to acclimatizing to their new roles. This time can be reduced somewhat by having a structure around switching (e.g. Ping-pong pairing) which allows the pairers to be mentally prepared for the switch to happen.
Pair Rotation
If a feature will take a long time, you might consider rotating people into and out of the pair over time (e.g. one person swaps out and a new person comes). You don’t want to do this frequently, no more than once per day over a full working day. It’s helpful to prevent the duo from become stale or frustrated with one another, and may help with knowledge transfer.
In a small team, with very short cycles, this may not be practical or necessary.
Setup for Pairing
Physical Setup
Ideally, you would work together in the same space. It is worth spending some time figuring out a comfortable setup for both of you.
- Make sure both of you have enough space, and that both of you can sit facing the computer.
- Agree on the computer setup (key bindings, IDE etc). Check if your partner has any particular preferences or needs (e.g. larger font size, higher contrast, …)
- It’s common to have a single keyboard, mouse for the driver, but some pairs setup with two keyboard - that’s your choice.
Remote Setup
If you’re working remotely, you may not be able to physically sit together. Luckily, there are practical solutions to pairing remotely. For remote pairing, you need a screen-sharing solution that allows you to not only see, but also control the other person’s machine, so that you are able to switch the keyboard.
There are also development tools that are designed to specifically address this:
Challenges
The biggest challenges with pair programming are not code related, but issues with communication and understanding one another. Don’t be afraid to switch roles, or take breaks when required, and be patient with one another!
Post-Mortem
If possible, spend some time reflecting on the pairing session when its over. Ask yourself what went well, and what you can improve. This can be help clear the air of any tension or issues that came up during the session, and help you improve as a team.
Code Reviews
What is the value of a code review?
When to perform them?
What feedback is useful?
What to do with the feedback you’re given?
TDD & Unit Testing
Test-Driven Development (TDD) is a strategy introduced by Kent Beck, which suggests writing tests first, before you start coding. You write tests against expected behaviour, and then write code which works without breaking the tests. TDD suggests this process:
- Think about the function (or class) that you need to create.
- Write tests that describe the behaviour of that function or class. As above, start with valid and invalid input.
- Your test will fail (since the implementation code doesn’t exist yet). Write just enough code to make the tests pass.
- Repeat until you can’t think of any more tests to write.
Running out of tests means that there is no more behaviour to implement… and you’re done, with the benefit of fully testable code.
Courtesy of https://medium.com/@tunkhine126/red-green-refactor-42b5b643b506
Why TDD?
There are some clear benefits:
- Early bug detection. You are building up a set of tests that can verify that your code works as expected.
- Better designs. Making your code testable often means improving your interfaces, having clean separation of concerns, and cohesive classes. Testable code is by necessity better code.
- Confidence to refactor. You can make changes to your code and be confident that the tests will tell you if you have made a mistake.
- Simplicity. Code that is built up over time this way tends to be simpler to maintain and modify.
What are unit tests?
Let’s define it more formally. A unit test is a test that meets the following three requirements [Khorikov 2020]:
- Verifies a single unit of behavior,
- Does it quickly, and
- Does it in isolation from other tests.
Unit tests should target classes or components in your program. i.e. they should exercise how a particular class works. They should be small, very focused, and quick to execute and return results.
Testing in isolation means removing the effects of any dependencies on other code, libraries or systems. This means that when you test the behaviour of your class, you are assured that nothing else is contributing to the results that you are observing. We’ll discuss strategies to accomplish this in the [Dependencies] section.
Info
This is also why designing cohesive, loosely coupled classes is critical: it makes testing them so much easier if they work independently!
Unit tests are just Kotlin functions that execute and check the results returned from other functions. Ideally, you would produce one unit or more unit tests for each function. You would then have a set of unit tests to check all of the methods in a class, and multiple sets of units tests to cover all of your implementation classes.
Your goal should be to have unit tests for every critical class and function that you produce.
Installing JUnit
We’re going to use JUnit, a popular testing framework to create and execute our unit tests. It can be installed in number of ways: directly from the JUnit home page, or one of the many package managers for your platform.
We will rely on Gradle to install it for us as project dependency. If you look at the section of a build.gradle file below, you can see that JUnit is included, which means that Gradle will download, install and run it as required. IntelliJ projects will typically include Gradle by default.
dependencies {
// Use the Kotlin test library.
testImplementation org.jetbrains.kotlin:kotlin-test'
// Use the Kotlin JUnit integration.
testImplementation'org.jetbrains.kotlin:kotlin-test-junit'
}
Structuring a test
A typical unit test uses a very particular pattern, known as the arrange, act, assert pattern. This suggests that each unit test should consist of these three parts:
- Arrange: bring the system under test (SUT) to the starting state.
- Act: call the method or methods that you want to test.
- Assert: verify the outcome of the above action. This can be based on return values, or some other conditions that you can check.
Note that your Act section should have a single action, reflecting that it’s testing a single behaviour. If you have multiple actions taking place, that’s a sign that this is probably an integration test (see below). As much as possible, you want to ensure that you are writing minimal-scope unit tests.
Info
Another anti-pattern is an if statement in a test. If you are branching, it means that you are testing multi-ple things, and you should really consider breaking that one test into multiple tests instead.
How to write tests
A unit test is just a Kotlin class, with annotated methods that tell the compiler to treat the code as a test. It’s best practice to have one test class for each implementation class that you want to test. e.g. class Main has a test class MainTest. This test class would contain multiple methods, each representing a single unit test.
Tests should be placed in a special test folder in the Gradle directory structure. When building, Gradle will automatically execute any tests that are placed in this directory structure.
Your unit tests should be structured to create instances of the classes that they want to test, and check the results of each function to confirm that they meet expected behaviour.
For example, here’s a unit test that checks that the Model class can add an Observer properly. The addObserver() method is the test (you can tell because it’s annotated with @Test). The @Before method runs before the test, to setup the environment. We could also have an @After method if needed to tear down any test structures.
class ObserverTests() {
lateinit var model: Model
lateinit var view: IView
class MockView() : IView {
override fun update() {
}
}
@Before
fun setup() {
model = Model()
view = MockView()
}
@Test
fun addObserver() {
val old = model.observers.count()
model.addObserver(view)
assertEquals(old+1, model.observers.count())
}
Gradle will automatically execute and display the test results when you build your project.
Let’s walkthrough creating a test.
To do this in IntelliJ, select a class in the editor, press Alt-Enter, and select “Create Test” from the popup menu. This will create a unit test using JUnit, the default test framwork. There is a detailed walkthrough on the IntelliJ support site.
Info
You can also just create the test files by-hand. Just make sure to save them in the correct location!
The convention is to name unit tests after their class they’re testing, with “Test” added as a suffix. In this example, we’re creating a test for a Model class so the test is automatically named ModelTest. This is just a convention - you can name it anything that you want.
We’ve manually added the addition() function. We can add as many functions as we want within this class. By convention, they should do something useful with that particular class.
Below, class ModelTest serves as container for our test functions. In it, we have two unit tests that will be automatically executed when we built. NOTE that you would need more than these two tests to adequately test this class - this is just an example.
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Before
import org.junit.Test
class ModelTests {
lateinit var model: Model
@Before
fun setup() {
model = Model()
model.counter = 10
}
@Test
fun checkAddition() {
val original = model.counter
model.counter++
assertEquals(original+1, model.counter)
}
@Test
fun checkSubtraction() {
val original = model.counter
model.counter--
assertEquals(original-1, model.counter)
}
@After
fun teardown() {
}
}
The kotlin.test package provides annotations to mark test functions, and denote how they are managed:
| Annotation |
Purpose |
| @Test |
Marks a function as a test to be executed |
| @BeforeTest |
Marks a function to be invoked before each test |
| @AfterTest |
Marks a function to be invoked after each test |
| @Ignore |
Mark a function to be ignored |
| @Test |
Marks a function as a test |
In our test, we call utility functions to perform assertions of how the function should successfully perform.
| Function |
Purpose |
| assertEquals |
Provided value matches the actual value |
| assertNotEquals |
The provided and actual values do not match |
| assertFalse |
The given block returns false |
| assertTrue |
The given block returns true |
How to run tests
Tests will be run automatically with gradle build or we can execute gradle test to just execute the tests.
$ gradle test
BUILD SUCCESSFUL in 760ms
3 actionable tasks: 3 up-to-date
Gradle will report the test results, including which tests - if any - have failed.
You can also click on the arrow beside the test class or name in the editor. For example, clicking on the arrow in the gutter would run this addObserver() test.
Code coverage
Tests shouldn’t verify units of code. Rather, they should verify units of behaviour: something that is meaningful for the problem domain and, ideally, something that a business person can recognize as useful. The number of classes it takes to implement such a unit of behaviour is irrelevant. — Khorikov (2020)
Code coverage is a metric comparing the number of lines of code with the number of lines of code that have unit tests covering their execution. In other words, what “percentage” of your code is tested?
This is a misleading statistic at the best of times (we can easily contrive cases where code will never be executed or tested).
TDD would suggest that you should have 100% unit test coverage but this is impractical and not that valuable. You should focus instead on covering key functionality. e.g. domain objects, critical paths of your source code.
One recommendation is to look at the coverage tools in IntelliJ, which will tell you how your code is being executed, as well as what parts of your code are covered by unit tests. Use this to determine which parts of the code should be tested more completely.
https://www.jetbrains.com/help/idea/code-coverage.html
https://www.jetbrains.com/help/idea/running-test-with-coverage.html
https://xkcd.com/2200/
Integration Testing
Unit tests are great at verifying business logic, but it’s not enough to check that logic in a vacuum. You have to validate how different parts of it integrate with each other and external systems: the database, the message bus, and so on." – Khorikov, 2020.
A unit test is a test that meets these criteria (from the previous chapter):
- Verifies a single unit of behaviour (typically a class),
- Does it quickly, and
- Does this in isolation from other dependencies and other tests.
An integration test is a test that fails to meet one or more of these criteria. In other words, if you determine that you need to test something outside of the scope of a unit test, it’s considered an integration test (typically because it’s integrating behaviours from multiple components). Performance tests, system tests, etc. are all kinds of integration tests.
There are a lot of different nuanced tests that are used in computer science, but we’ll focus on building generic integration tests.
Typically an integration test is one where you leave in selected dependencies so that you can test the combination of classes together. Integration tests are also suitable in cases where it is difficult to completely remove a dependency. This can happen with some critical, external dependencies like an external database.
This diagram demonstrates how unit tests primarily test the domain model, or business logic classes. Integration tests focus on the point where these business logic classes interact with external systems or dependencies.
Note that in this diagram, we’re also identifying code that we shouldn’t bother testing. Trivial code is low complexity, and typically has no dependencies or external impact so it doesn’t require extensive testing. Overcomplicated code likely has so many dependencies that it’s nearly impossible to test - and it should likely be refactored into something similer and more manageable before you attempt to add tests to it.
How many tests?
When discussing unit tests, we suggested that you should focus on core classes and their behaviours. This is reasonable for unit tests.
We can expand this to suggest that you should “check as many of the business scenario’s edge cases as possible with unit tests; use integration tests to cover one happy path, as well as any edge cases that can’t be covered by unit tests.”
A “happy path” in testing is a successful execution of some functionality. In other words, once your unit tests are done, you should write an integration tests that exercises the functionality that a customer would likely exercise if they were using your software with common features and a common workflow. Focus on that first and only add more integration tests once you have the main execution path identified and tested.
The primary purpose of integration tests is to exercise dependencies, so that should be your main goal. Your main integration test (the “happy path test”) should exercise all external dependencies (libraries, database etc). If it cannot satisfy this requirement, add more integration tests to satisfy this constraint.
Guidelines
Here’s some guidelines for creating integration tests.
- Make domain model boundaries explicit.
Try to always have an explicit, well-known place for the domain model in your code base. The domain model is the collection of domain knowledge about the problem your project is meant to solve. This can be a separate class, or even a package reserved for the model.
- Reduce the number of layers of abstraction
“All problems in computer science can be solved by another layer of indirection, except for the problem of too many layers of indirection.” – David J. Wheeler
Try to have as few layers of indirection as possible. In most backend systems, you can get away with just three: the domain model, application services layer (controllers), and infrastructure layer. Having excessive layers of abstraction can lead to bloated and confusing code.
- Eliminate circular dependencies
Try to keep dependencies flowing in a single direction. i.e. don’t have class A call into class B, and class B call back into class A with results. Circular dependencies like this create a huge cognitive load for the person reading the code, and make testing much more difficult.
Refactoring
We all want to write perfect code, but given , we often end-up with less-than-perfect solutions.
-
Rushed features: Sometimes we don’t have enough time, so we cut corners.
-
Lack of tests: We might think that the code is ready, but we haven’t tested it adequately.
-
Lack of communication: Perhaps we misunderstood requirements, or how a feature would integrate into the larger product.
-
Poor design: Possibly our design is rigid and makes adding or modifying features difficult.
These are all actions that may cost us time later. We may need to stop and redesign a rushed feature, or we may need to fix bugs later.
We refer to this as technical debt — the deferred cost of doing something poorly.
What is refactoring?
Martin Fowler (2000) also introduced the notion of refactoring: systematically transforming your working code base into a cleaner, improved version of itself that is easier to read, maintain and extend.
Refactoring is a controlled technique for improving the design of an existing code base. Its essence is applying a series of small behavior-preserving transformations, each of which “too small to be worth doing”. However the cumulative effect of each of these transformations is quite significant.
– Martin Fowler, 2018.
Refactoring suggests that code must be continually improved as we work with it. We need to be in a process of perpetual, small improvements.
The goal of refactoring is to reduce technical debt by making small continual improvements to our code. It doesn’t reduce the likelihood of technical debt, but it amortizes that debt over many small improvements.
Refactoring your code means doing things like:
-
- Cleaning up class interfaces and relationships.
- Fixing issues with class cohesion.
- Reducing or removing unnecessary dependencies.
- Simplifying code to reduce unnecessary complexity.
- Making code more understandable and readable.
- Adding more exhaustive tests.
In other words, refactoring involves code improvement not related to adding functionality.
TDD and Refactoring work together. You continually refactor as you expand your code, and you rely on the tests to guarantee that you aren’t making any breaking changes to your code.
How to refactor code?
The Rule of Three
- When you’re doing something for the first time, just get it done.
- When you’re doing something similar for the second time, do the same thing again.
- When you’re doing something for the third time, start refactoring.
When adding a feature
- Refactor existing code before you add a new feature, since it’s much easier to make changes to clean code. Also, you will improve it not only for yourself but also for those who use it after you.
When fixing a bug
- If you find or suspect a bug, refactoring to simplify the existing code can often reveal logic errors.
During a code review
- The code review may be the last chance to tidy up the code before it becomes available to the public.
Refactorings
Martin Fowler. 2018. Refactoring: Improving the Design of Existing Code. 2nd Edition. Addison-Wesley. ISBN 978-0134757599.
The author has also made them available at: https://refactoring.com/catalog/
IntelliJ IDEA also makes this easy by providing automated ways of safely transforming your code. These refactorings often involve operations that would be tricky to do by-hand but easy for the tool do perform for you (e.g. renaming a method that is called from multiple locations).
To invoke refactorings, select an item in your source code (e.g. variable or function name) and press Ctrl-T to invoke the refactoring menu. You can also access from the application menu.
They have a complete list of these in the IntelliJ IDEA documentation. Refactorings include:
| Refactoring |
Purpose |
| Rename |
Change an identifier to something that is more meaningful or memorable. |
| Move |
Move classes or functions to different packages; move methods between classes. |
| Extract method |
Take a code fragment that can be grouped, move it into a separated method, and replace the old code with a call to the method |
| Extract field |
Extract an expression into a variable, and insert the expression dynamically. |
| Safe delete |
Check for usage of a symbol before you are allowed to delete it. |
| Change signature |
Change the method name, add, remove, reorder, and rename parameters and exceptions. |
| Type migration |
Change a member type (e.g. from integer to string), method return types, local variables, parameters etc. across the entire project. |
| Replace constructor with factory |
Modify class the become a singleton (returns a single instance). |
Code Smells
A “code smell” is a sign that a chunk of code is badly designed or implemented. It’s a great indication that you may need to refactor the code.
-
Adjectives used to describe code:
-
- “neat”, “clean”, “clear”, “beautiful”, “elegant” <— the reactions that we want
- “messy”, “disorganized”, “ugly”, “awkward” <— the reactions we want to avoid
-
A negative emotional reaction is a flag that your brain doesn’t like something about the organization of the code - even you can’t immediately identify what that is.
-
Conversely, a positive reaction indicates that your brain can easily perceive and following the underlying structure.
The following categories and examples are taken from refactoring.guru.
Bloaters
Bloaters are code, methods and classes that have increased to such gargantuan proportions that they are hard to work with. These smells accumulate over time as the program evolves (and especially when nobody makes an effort to eradicate them).
- Long method: A method contains too many lines of code. Generally, any method longer than ten lines should make you start asking questions.
- Large class: A class contains many fields/methods/lines of code. This suggests that it may be doing too much. Consider breaking out a new class, or interface.
- Primitive obsession: Use of related primitives instead of small objects for simple tasks (such as currency, ranges, special strings for phone numbers, etc.). Consider creating a data class, or small class instead.
- Long parameters list: More than three or four parameters for a method. Consider passing an object that owns all of these. If many of them are optional, consider a builder pattern instead.
Object-Oriented Abusers
All these smells are incomplete or incorrect application of object-oriented programming principles.
- Alternative Classes with Different Interfaces: Two classes perform identical functions but have different method names. Consolidate methods into a single class instead, with support for both interfaces.
- Refused bequest: If a subclass uses only some of the methods and properties inherited from its parents, the hierarchy is off-kilter. The unneeded methods may simply go unused or be redefined and give off exceptions. This violates the Liskov-substitution principle! Add missing behaviour, or replace inheritance with delegation.
- Switch Statement: You have a complex switch operator or sequence of if statements. This sometimes indicates that you are switching on type, something that should be handled by polymorphism instead. Consider whether a class structure and polymorphism makes more sense in this case.
- Temporary Field: Temporary fields get their values (and thus are needed by objects) only under certain circumstances. Outside of these circumstances, they’re empty. This may be a place to introduce nullable types, to make it very clear what is actually happening (vs. constantly checking fields for the presence of data).
Dispensibles
A dispensable is something pointless and unneeded whose absence would make the code cleaner, more efficient and easier to understand.
- Comments: A method is filled with explanatory comments. These are usually well-intentioned, but they’re not a substitute for well-structured code. Comments are a maintenance burden. Replace or remove excessive comments.
- Duplicate Code: Two code fragments look almost identical. Typically, done accidentally by different programmers. Extract the methods into a single common method that is used instead. Alternately, if the methods solve the same problem in different ways, pick and keep the most efficient algorithm.
- Dead Code: A variable, parameter, field, method or class is no longer used (usually because it’s obsolete). Delete unused code and unneeded files. You can always find it in Git history.
- Lazy Class: Understanding and maintaining classes always costs time and money. So if a class doesn’t do enough to earn your attention, it should be deleted. This is tricky: sometimes a small data class is clearer than using primitives (e.g. a Point class, vs using x and y stored as doubles).
Couplers
All the smells in this group contribute to excessive coupling between classes or show what happens if coupling is replaced by excessive delegation.
- Feature envy: A method accesses the data of another object more than its own data. This smell may occur after fields are moved to a data class. If this is the case, you may want to move the operations on data to this class as well.
- Inappropriate intimacy: One class uses the internal fields and methods of another class. Either move those fields and methods to the second class, or extract a separate class that can handle that functionality.
- Middle man: If a class performs only one action, delegating work to another class, why does it exist at all? It can be the result of the useful work of a class being gradually moved to other classes. The class remains as an empty shell that doesn’t do anything other than delegate. Remove it.
https://xkcd.com/1513
Release Process
What do we need to do for a software release?
Versioning
- You should version your software, so that every release has a release number and date associated with it.
- The standard convention is a triple, separated by decimals, of the format:
major.minor.build. For example, 1.2.3 would be major version 1, minor version 2, build 3.
-
- Major signifies a major product release. This is somewhat arbitrary, but typically is released infrequently and includes major features changes or additions. If you charge by release, you would typically charge for every new major version. You might release a new major version as frequently as once per year, or as infrequently as once very few years.
- Minor indicates a minor product release, typically a combination of new minor features, and bug or compatibility fixes. You might release a minor version a few times per year and users would not ordinarily expect to pay for these.
- Build number is internal build number within a minor release. This is intended to reflect bug fixes only; you typically iterate over builds internally and release the final successful version publically.
Release Chart
Copyright & Licensing
Copyright
Before distributing your software, in any form, it’s critical to establish ownership and rights pertaining to that software.
A copyright is a type of intellectual property that gives its owner the exclusive right to copy and distribute a creative work, usually for a limited time. Copyright is intended to protect the original expression of an idea in the form of a creative work, but not the idea itself. A copyright is subject to limitations based on public interest considerations, such as the fair use doctrine.
Software copyright is the application of copyright in law to machine-readable software. Under Canadian and US law, all software is copyright protected, in both source code and object code forms.
Practically, this means that different companies can independently produce software that solves the same problem, and there is no law preventing that from occurring, provided that they do not reuse actual source or object code from their competitor.
How do I assert copyright?
In Canada, software is protected under the Copyright Act of Canada. Copyright is acquired automatically when an original work is generated; the creator is not required to register or mark the work with the copyright symbol in order to be protected. The rights holder is granted: the exclusive right of reproduction, the right to rent the software, the right to restrain others from renting the software and the right to assign or license the copyright to others.
It’s common practice to assert your copyright claim in the header of your software source files. Although is not required to assert copyright, it’s a flag for potential violators, and it might make it easier to defend in court.
Copyright (c) 2022. Jeff Avery.
Licensing
As the rights holder, you can grant others rights with respect to your software. A software license is a legal instrument that grants the licensee (i.e. an end-user) permission to use the software in a manner dictated by the license. These rights could include (but aren’t limited to) the right to install and use it, the right to modify the source code, or the right to redistribute the software with or without changes.
Authors of copyrighted software can also choose to donate their software to the public domain, in which case it is also not covered by copyright and, as a result, cannot be licensed.
Standard Licenses
There are a number of standard software licenses that have been developed and are commonly used, particularly with resepect to Open Source software. The following table is extracted from the Wikipedia page for software licensing.
A permissive software license, sometimes also called BSD-style is a free-software license which instead of copyleft protections, carries only minimal restrictions on how the software can be used, modified, and redistributed, usually including a warranty disclaimer. Examples include the GNU All-permissive License, MIT License, BSD licenses, Apple Public Source License and Apache license. As of 2016, the most popular free-software license is the permissive MIT license.
Copyleft is the practice of granting the right to freely distribute and modify intellectual property with the requirement that the same rights be preserved in derivative works created from that property.
Copyleft software licenses are considered protective in contrast with permissive free software licenses, and require that information necessary for reproducing and modifying the work must be made available to recipients of the software program. This information is most commonly in the form of source code files, which usually contain a copy of the license terms and acknowledge the authors of the code. Notable copyleft licenses include the GNU General Public License (GPL), originally written by Richard Stallman, and the Creative Commons share-alike license.
Non-commercial licenses are intended to be used only be entities with no profit motive, including charities and public institutions. This is not common, and is both difficult to interpret and enforce.
A proprietary license refers to any other license, but usually one which grants few to no rights to the end-user (e.g. most commercial software licenses, which do not grant you any rights apart from the ability to install and use on a single machine).
It’s standard practice to include your license agreement with your product, typically as a license.txt file or something similar in your distribution.
How do I apply a license?
- Distribute the license with your program.
- Include a license.txt file in your distribution.
- If you provide source code to anyone, include a statement about how it is licensed in the header of each file. Check the license to see what is required.
- Include a licensing statement on your website.
- See terms of each license to check what’s suitable. e.g.
Unless explicitly stated otherwise all files in this repository are licensed under the Apache Software License 2.0 [insert boilerplate notice here]
Resources
Open Source Initiative. 2022. Licenses and Standards. https://opensource.org/licenses
CI/CD
Test-Driven Development addresses the issue of doing local, small-scope testing as part of implementation. However, it doesn’t address issues related to the system as a whole, or that might only occur when components are integrated.
Martin Folwer introduced the term continuous integration to describe a system where we also perform integration testing at least once pr day.
The fundamental benefit of continuous integration is that it removes sessions where people spend time hunting bugs where one person’s work has stepped on someone else’s work without either person realizing what happened. These bugs are hard to find because the problem isn’t in one person’s area, it is in the interaction between two pieces of work.
– Fowler, 2000.
A system that supports continuous integration needs, at a minimum, the following capabilities:
- It requires a revision control system, with a centralized main revision that can be used.
- The build process should be automated so that anyone can manually launch the process. [it should also support automated testing based on other events, like integrating a branch in the source tree].
- Tests should be automated so that they can be launched manually as well.
- The system should produce a final distribution.
CI Systems
Continuous Integration Systems are software systems that provide these capabilities. Early standalone systems include Jenkins (Open Source), and CircleCI. Many source control platforms also provide CI functionality, including Bitbucket, GitHub and GitLab.
For example, you can automate GitLab so that it will build and run your tests anytime a specific action is performed like committing to a branch, or merging a PR. This is managed through the CI/CD section of the project configuration.
The GitLab configuration and terminology is pretty standard:
- A pipeline represents the work or the job that will be done.
- A stage represents a set of jobs that need to be executed together.
- Jobs are executed by runners, which define and where a job will be executed.
These all represent actions that will be taken against your source code repository at specific times. The examples that they provide include:
- A
build stage, with a job called compile.
- A
test stage, with two jobs called test1 and test2.
- A
staging stage, with a job called deploy-to-stage.
- A
production stage, with a job called deploy-to-prod.
In this way, you can setup your source code repository to build, test, stage and deploy your software automatically one or more times per day, as a result of some key event, or when manually executed.
Info
Although we have a GitLab instance running, we do not have access to a cluster that can run jobs for us. In other words, we cannot do this in production using our current setup – at least not without gaining access to a Kubernetes cluster somewhere.
Subsections of Learning Kotlin
Why Kotlin?
Kotlin is a modern language designed by JetBrains. Originally designed as a drop-in replacement for Java, Kotlin has grown in popularity to become the default language for Android development, and a popular back-end language.
Programming language selection is difficult, since languages tend to focus on front-end or back-end. It’s common to mix different programming languages depending on how they’re being used. For instance, Kotlin, Java and Go are all used for back-end systems, but typically paired with a JS/HTML client on the front-end.
|
| 2022 Stack Overflow Developer Survey of “Most Loved Languages” |
The Kotlin Foundation, which manages the language, is invested in Kotlin as a modern cross-platform language. We’ll discuss Kotlin Multiplatform later in the course - the idea that we use a mix of Kotlin and native code to support code reuse across mobile and desktop platforms.
|
| Kotlin allows you to build share components for networking, data storage and models, while building native user-interfaces. This supports cross-platform development on mobile and desktop platforms. |
Finally, Kotlin also has a number of language features that make it an outstanding language for full-stack development:
- With some popular toolkits, it can be used to build compelling front-end client applications as well as back-end services.
- Critically, it supports compilation to a number of deployment targets: JVM for Windows/macOS/Linux on the desktop, Android native, or Web. With Compose Multiplatform, it can also be used to build iOS and Web apps.
- It’s a hybrid language: it can be used for declarative programming or class-based object-oriented programming. It also supports a number of functional features, especially with the use of collection classes.
- It has a very clean syntax, and supports quality-of-life features like default arguments, variable argument lists and rich collection types. It’s syntax closely resembles modern languages like Swift or Scala.
- Kotlin is statically compiled, so it catches many potential errors during compilation (not just at runtime).
- It has outstanding tools support with IntelliJ IDEA.
- It has massive community support (libraries, toolkits).
Sources: StackOverflow Developer Survey 2022 and JetBrains The State of Developer Ecosystem 2022.
Getting Started
Compiling Code
Kotlin is a unique language, in that we can target many diferent kinds of platforms. Typically, we think of languages as compiled or interpreted:
Compiled languages require an explicit step to compile code and generate native executables. This is done ahead of time, and the executables are distributed to users. e.g. C++
- The compilation cost is incurred before the user runs the program, so we get optimal startup performance.
- The target system architecture must be known ahead of time, since we’re distributing native binaries.
Interpreted languages allow developers to distribute the raw source code which can be interpreted when the user executes it.
- This requires some ‘runtime engine‘ that can convert source code to machine code on-the-fly. Results of this operation can often be cached so that the compilation cost is only incurred when it first executes.
Some languages can be compiled to a secondary format (IR, ”intermediate representation”) and then interpreted. Languages running on the Java Virtual Machine (JVM) are compiled ahead of time to IR, and then interpreted at runtime.
Kotlin can be compiled or interpreted!
Kotlin/JVM compiles Kotlin code to JVM bytecode, which is interpreted on a Java virtual machine.Kotlin/Android compiles Kotlin code to native Android binaries, which leverage native versions of the Java Library and Kotlin standard libraries.Kotlin/Native compiles Kotlin code to native binaries, which can run without a virtual machine. It is an LLVM based backend for the Kotlin compiler and native implementation of the Kotlin standard library.Kotlin/JS transpiles (converts) Kotlin to JavaScript. The current implementation targets ECMAScript 5.1 (with plans to eventually target ECMAScript 2015).
In this course, we’ll focus on using Kotlin/JVM to build desktop applications.
Code Execution
There are three primary ways of executing Kotlin code:
- Read-Evaluate-Print-Loop (REPL): Interact directly with the Kotlin runtime, one line at-a-time. In this environment, it acts like a dynamic language.
- KotlinScript: Use Kotlin as a scripting language, by placing our code in a script and executing directly from our shell. The code is compiled automatically when we execute it, which eliminates the need to compile ahead-of-time.
- Application: We can compile standalone applications, targetting native or JVM [ed. we will use JVM in this course].
REPL
REPL is a paradigm where you type and submit expressions to the compiler one line-at-a-time. It’s commonly used with dynamic languages for debugging, or checking short expressions. It’s not intended as a means of writing full applications!
> kotlin
Welcome to Kotlin version 1.6.10 (JRE 17.0.2+8-86)
Type :help for help, :quit for quit
>>> val message="Hello Kotlin!"
>>> println(message)
Hello Kotlin!
KotlinScript
KotlinScript is Kotlin code in a script file that we can execute from our shell. This makes Kotlin an interesting alternative to a language like Python for shell scripting.
> cat hello.kts
#!/usr/bin/env kotlin
val message="Hello Kotlin!"
println(message)
> ./hello.kts
Hello Kotlin!
Kotlin compiles scripts in the background before executing them, so there’s a delay before it executes [ed. I fully expect that later versions of Kotlin will allow caching the compilation results to speedup script execution time].
This is a great way to test functionality, but not a straight-up replacement for shell scripts, due to the runtime costs.
Applications
Kotlin applications are fully-functional, and can be compiled to native code, or to the JVM. Kotlin application code looks a little like C, or Java. Here’s the world’s simplest Kotlin program, consisting of a single main method.
fun main() {
val message="Hello Kotlin!"
println(message)
}
To compile from the command-line, we can use the Kotlin compiler, kotlinc. By default, it takes Kotlin source files (.kt) and compiles them into corresponding class files (.class) that can be executed on the JVM.
> kotlinc Hello.kt
> ls
Hello.kt HelloKt.class
> kotlin HelloKt
Hello Kotlin!
Notice that the compiled class is named slightly differently than the source file. If your code isn’t contained in a class, Kotlin wraps it in an artificial class so that the JVM (which requires a class) can load it properly. Later when we use classes, this won’t be necessary.
This example compiles Hello.kt into Hello.jar and then executes it:
> kotlinc Hello.kt -include-runtime -d Hello.jar
> ls
Hello.jar Hello.kt
> java -jar Hello.jar
Hello Kotlin!
Modularization
Java and Kotlin have two different levels of abstraction when it comes to grouping code: packages and modules.
Packages
Packages are meant to be a collection of related classes. e.g. graphics classes. Packages are primarily a mechanism for managing dependencies between parts of an application, and encourage clear separation of concerns. They are conceptually similar to namepsaces in C++.
Use the package declaration at the top of a source file to assign a file to a namespace. Classes or modules in the same package have full visibility to each other.
For example, in the file below, contents are contained in the ca.uwaterloo.cs346 package. The full name of the class includes the package and class name: ca.uwaterloo.cs346.ErrorMessage . If you were referring to it from a different package, you would need to use this fully qualified name.
package ca.uwaterloo.cs346
class ErrorMessage(val msg:String) {
fun print() {
println(msg)
}
}
fun main() {
val error = ErrorMessage("testing an error condition")
error.print()
}
Info
Best practice is to use a reverse DNS name for a package name. e.g. com.sun.graphics if you developed the Graphics library at Sun Microsystems. Package names are always lowercase, dot-separateds with no underscores. If you want to use multiple-words, consider using camel case.
To use a class in a different namespace, we need to import the related class by using the import keyword. In the example below, we import our ErrorMessage class into a different namespace so that we can instantiate and use it.
import ca.uwaterloo.cs346.ErrorMessage
class Logger {
val error = ErrorMessage()
error.printMessage()
}
Modules
Modules serve a different purpose than packages: they are intended to expose permissions for external dependencies. Using modules, you can create higher-level constructs (modules) and have higher-level permissions that describe how that module can be reused. Modules are intended to support a more rigorous separation of concerns that you can obtain with packages.
A simple application would be represented as a single module containing one or more packages, representing different parts of our application.
Let’s use our starting Gradle directory as the starting point. We’ll expand the source code subdirectory to include a package (net.codebot) containing a single class (Main.kt).
.
├── app
│ ├── build.gradle
│ └── src
│ └── main
│ └── java
│ └── module-info.java
│ └── kotlin
│ └── net
│ └── codebot
│ └── Main.kt
│ └── resources
│ └── test
│ ├── kotlin
│ └── resources
├── gradle
│ └── wrapper
│ ├── gradle-wrapper.jar
│ └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
└── settings.gradle
Our top-level project is in the app/ directory.
Our top-level package is net.codebot, and contains a class net.codebot.Main.
To create a module for this project, we need to add a file named module-info.java in the src/main subdirectory. This will describe a module that contains this project, and also describes what classes will be exported and available to other projects.
Modules are particularly important when working with multi-project builds, since the module-info.java describes what classes are available to other projects.
module-info.java
module net.codebot.Main {
requires javafx.graphics;
exports net.codebot.Main;
}
The file opens with the name of the module. Module names have the same naming restrictions as package names, and the convention is to use the same name for the module and package that it contains.
- requires lists other modules on which this module depends.
- exports lists the packages that should be available to other modules that wish to import this module.
Info
You’ll notice that the module-info.java file is located under the java/ folder, even though this is a Kotlin project! This is due to the relative newness of the Java module system. You can get around this by adding the following lines to your build.gradle file:
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
Using Libraries
Kotlin has full access to it’s own class libraries, plus any others that are imported and made available. Kotlin is 100% compatible with Java libraries, and makes extensive use of Java libraries when possible. For example, Kotlin collection classes actually use some of the underlying Java collection libraries!
In this section, we’ll discuss how to use existing libraries in your code. We need to talk about namespaces and qualifying classes before we can talk about libraries.
Kotlin Standard Library
The Kotlin Standard Library is included with the Kotlin language, and contained in the kotlin package. This is automatically imported and does not need to be specified in an import statement.
Some of the features that will be discussed below are actually part of the standard library (and not part of the core language). This includes essential classes, such as:
- Higher-order scope functions that implement idiomatic patterns (let, apply, use, etc).
- Extension functions for collections (eager) and sequences (lazy).
- Various utilities for working with strings and char sequences.
- Extensions for JDK classes making it convenient to work with files, IO, and threading.
Using Java Libraries
Kotlin is completely 100% interoperable with Java, so all of the classes available in Java/JVM can also be imported and used in Kotlin.
// import all classes in the java.io package
// this allows us to refer to any of those classes in the current namespace
import java.io.*
// we can also just import a single class
// this allows us to refer to just the ListView class in code
import javafx.scene.control.ListView
// Kotlin code calling Java IO libraries
import java.io.FileReader
import java.io.BufferedReader
import java.io.FileNotFoundException
import java.io.IOException
import java.io.FileWriter
import java.io.BufferedWriter
if (writer != null) {
writer.write(
row.toString() + delimiter +
s + row + delimiter +
pi + endl
)
Info
Importing a class requires your compiler to locate the file containing these classes! The Kotlin Standard Library can always be referenced by the compiler, and as long as you’re compiling to the JVM, the Java class libraries will also be made available. However, to use any other Java or Kotlin library, you will need to take additional steps. We’ll discuss this when we cover build systems and Gradle.
Types & Mutability
Programming languages can take different approaches to enforcing how types are managed.
- Strong typing: The language has strict typing rules, which typically enforced at compile-time. The exact type of a variable must be declared or fixed before the variable is used. This has the advantage of catching many types of errors at compile-time (e.g. type-mismatch).
- Weak typing: These languages have looser typing rules, and will often attempt to infer types based on runtime usage. This means that some categories of errors are only caught at runtime.
Kotlin is a strongly typed language, where variables need to be declared before they are used. Kotlin also supports type infererence. If a type isn’t provided, Kotlin will infer the type at compile time (similar to ‘auto‘ in C++). The compiler is strict about this: if the type cannot be inferred at compile-time, an error will be thrown.
Variables
Kotlin uses the var keyword to indicate a variable and Kotlin expects variables to be declared before use. Types are always placed to the right of the variable name. Types can be declared explicitly, but will be inferred if the type isn’t provided.
fun main() {
var a:Int = 10
var b:String = "Jeff"
var c:Boolean = false
var d = "abc" // inferred as a String
var e = 5 // inferred as Int
var f = 1.5 // inferred as Float
}
All standard data-types are supported, and unlike Java, all types are objects with properties and behaviours. This means that your variables are objects with methods! e.g. "10".toInt() does what you would expect.
Integer Types
| Type |
Size (bits) |
Min value |
Max value |
| Byte |
8 |
-128 |
127 |
| Short |
16 |
-32768 |
32767 |
| Int |
32 |
-2,147,483,648 (-2 31) |
2,147,483,647 (2 31- 1) |
| Long |
64 |
-9,223,372,036,854,775,808 (-2 63) |
9,223,372,036,854,775,807 (2 63- 1) |
Floating Point Types
| Type |
Size (bits) |
Significant bits |
Exponent bits |
Decimal digits |
| Float |
32 |
24 |
8 |
6-7 |
| Double |
64 |
53 |
11 |
15-16 |
Boolean
The type Boolean represents boolean objects that can have two values: true and false. Boolean has a nullable counterpart Boolean? that also has the null value.
Built-in operations on booleans include:
|| – disjunction (logical OR)&& – conjunction (logical AND)!- negation (logical NOT)
|| and && work lazily.
Strings
Strings are often a more complex data type to work with, and deserve a callout. In Kotlin, they are represented by the String type, and are immutable. Elements of a string are characters that can be accessed by the indexing operation: s[i], and you can iterate over a string with a for-loop:
fun main() {
val str = "Sam"
for (c in str) {
println(c)
}
}
You can concatenate strings using the + operator. This also works for concatenating strings with values of other types, as long as the first element in the expression is a string (in which case the other element will be case to a String automatically):
fun main() {
val s = "abc" + 1
println(s + "def")
}
Kotlin supports the use of string templates, so we can perform variable substitution directly in strings. It’s a minor but incredibly useful feature that replaces the need to concatenate and build up strings to display them.
fun main() {
println("> Kotlin ${KotlinVersion.CURRENT}")
val str = "abc"
println("$str.length is ${str.length}")
var n = 5
println("n is ${if(n > 0) "positive" else "negative"}")
}
is and !is operators
To perform a runtime check whether an object conforms to a given type, use the is operator or its negated form !is:
fun main() {
val obj = "abc"
if (obj is String) {
print("String of length ${obj.length}")
} else {
print("Not a String")
}
}
In most cases, you don’t need to use explicit cast operators in Kotlin because the compiler tracks the is -checks and explicit casts for immutable values and inserts (safe) casts automatically when needed:
fun main() {
val x = "abc"
if (x !is String) return
println("x=${x.length}") // x is automatically cast to String
val y = "defghi"
// y is automatically cast to string on the right-hand side of `||`
if (y !is String || y.length == 0) return
println("y=${y.length}") // y must be a string with length > 0
}
Immutability
Kotlin supports the use of immutable variables and data structures [mutable means that it can be changed; immutable structures cannot be changed after they are initialized]. This follows best-practices in other languages (e.g. use of ‘final‘ in Java, ‘const‘ in C++), where we use immutable structures to avoid accidental mutation.
var - this is a standard mutable variable that can be changed or reassigned.val - this is an immutable variable that cannot be changed once initialized.
var a = 0 // type inferred as Int
a = 5 // a is mutable, so reassignment is ok
val b = 1 // type inferred as Int as well
b = 2 // error because b is immutable
var c:Int = 10 // explicit type provided in this case
Operators
Kotlin supports a wide range of operators. The full set can be found on the Kotlin Language Guide.
+, -, *, /, %- mathematical operators= assignment operator&&, ||, !- logical ‘and’, ‘or’, ’not’ operators==, != - structural equality operators compares members of two objects for equality===, !==- referential equality operators are true when both sides point to the same object.[, ]- indexed access operator (translated to calls of get and set)
NULL Safety
NULL is a special value that indicates that there is no data present (often indicated by the null keyword in other languages). NULL values can be difficult to work with in other programming languages, because once you accept that a value can be NULL, you need to check all uses of that variable against the possibility of it being NULL.
NULL values are incredibly difficult to manage, because to address them properly means doing constant checks against NULL in return values, data and so on. They add inherent instability to any type system.
In Kotlin, every type is non-nullable by default. This means that if you attempt to assign a NULL to a normal data type, the compiler is able to check against this and report it as a compile-time error. If you need to work with NULL data, you can declare a nullable variable using the ? annotation [ed. a nullable version of a type is actually a completely different type]. Once you do this, you need to use specific ? methods. You may also need to take steps to handle NULL data when appropriate.
Conventions
- By default, a variable cannot be assigned a NULL value.
? suffix on the type indicates that it’s NULL-able.?. accesses properties/methods if the object is not NULL (“safe call operator”)?: elvis operator is a ternary operator for NULL data- !! override operator (calls a method without checking for NULL, bad idea)
fun main() {
// name is nullable
var name:String? = null
// only returns value if name is not null
var length = name?.length
println(length) // null
// elvis operator provides an `else` value
length = name?.length ?: 0
println(length) // 0
}
Generics
Generics are extensions to the type system that allows us to parameterize classes or functions across different types. Generics expand the reusability of your class definitions, because they allow your definitions to work with many types.
We’ve already seen generics when dealing with collections:
val list: List<Int> = listOf(5, 10, 15, 20)
In this example, <Int> is specifying the type that is being stored in the list. Kotlin infers types where it can, so we typically write this as:
val list = listOf(5, 10, 15, 20)
We can use a generic type parameter in the place of a specific type in many places, which allows us to write code towards a generic type instead of a specific type. This prevents us from writing methods that might only differ by parameter or return type.
A generic type is a class that accepts an input of any type in its constructor. For instance, we can create a Table class that can hold a differing values.
You define the class and make it generic by specifying a generic type to use in that class, written in angle brackets < >. The convention is to use T as a placeholder for the actual type that will be used.
class Table<T>(t: T) {
var value = t
}
val table1: Table<Int> = Table<Int>(5)
val table2 = Table<Float>(3.14)
A more complete example:
import java.util.*
class Timeline<T>() {
val events : MutableMap<Date, T> = mutableMapOf()
fun add(element: T) {
events.put(Date(), element)
}
fun getLast(): T {
return events.values.last()
}
}
fun main() {
val timeline = Timeline<Int>()
timeline.add(5)
timeline.add(10)
}
Control-Flow
Kotlin supports the style of control flow that you would expect in an imperative language, but it uses more modern forms of these constructs
if then else
if... then has both a statement form (no return value) and an expression form (return value).
fun main() {
val a=5
val b=7
// we don't return anything, so this is a statement
if (a > b) {
println("a is larger")
} else {
println("b is larger")
}
val number = 6
// the value from each branch is considered a return value
// this is an expression that returns a result
val result =
if (number > 0)
"$number is positive"
else if (number < 0)
"$number is negative"
else
"$number is zero"
println(result)
}
Info
This is why Kotlin doesn’t have a ternary operator: if used as an expression serves the same purpose.
for in
A for in loop steps through any collection that provides an iterator. This is equivalent to the for each loop in languages like C#.
fun main() {
val items = listOf("apple", "banana", "kiwifruit")
for (item in items) {
println(item)
}
for (index in items.indices) {
println("item $index is ${items[index]}")
}
for (c in "Kotlin") {
print("$c ")
}
}
Kotlin doesn’t support a C/Java style for loop. Instead we use a range collection .. that generates a sequence of values.
fun main() {
// invalid in Kotlin
// for (int i=0; i < 10; ++i)
// range provides the same funtionality
for (i in 1..3) {
print(i)
}
println() // space out our answers
// descending through a range, with an optional step
for (i in 6 downTo 0 step 2) {
print("$i ")
}
println()
// we can step through character ranges too
for (c in 'A'..'E') {
print("$c ")
}
println()
// Check if a number is within range:
val x = 10
val y = 9
if (x in 1..y+1) {
println("fits in range")
}
}
while
while and do... while exist and use familiar syntax.
fun main() {
var i = 1
while ( i <= 10) {
print("$i ")
i++
}
}
when
when replaces the switch operator of C-like languages:
fun main() {
val x = 2
when (x) {
1 -> print("x == 1")
2 -> print("x == 2")
else -> print("x is neither 1 nor 2")
}
}
fun main() {
val x = 13
val validNumbers = listOf(11,13,17,19)
when (x) {
0, 1 -> print("x == 0 or x == 1")
in 2..10 -> print("x is in the range")
in validNumbers -> print("x is valid")
!in 10..20 -> print("x is outside the range")
else -> print("none of the above")
}
}
We can also return a value from when. Here’s a modified version of this example:
fun main() {
val x = 13
val validNumbers = listOf(11,13,17,19)
val response = when (x) {
0, 1 -> "x == 0 or x == 1"
in 2..10 -> "x is in the range"
in validNumbers -> "x is valid"
!in 10..20 -> "x is outside the range"
else -> "none of the above"
}
println(response)
}
When is flexible. To evaluate any expression, you can move the comparison expressions into the when statement itself:
fun main() {
val x = 13
val response = when {
x < 0 -> "negative"
x >= 0 && x <= 9 -> "small"
x >=10 -> "large"
else -> "how do we get here?"
}
println(response)
}
return
Kotlin has three structural jump expressions:
return by default returns from the nearest enclosing function or anonymous functionbreak terminates the nearest enclosing loopcontinue proceeds to the next step of the nearest enclosing loop
Functions
Functions are preceded with the fun keyword. Function parameters require types, and are immutable. Return types should be supplied after the function name, but in some cases may also be inferred by the compiler.
Named Functions
Named function have a name assigned to them that can be used to invoke them directly (this is the expected form of a “function” in most cases, and the form that you’re probably expecting).
// no parameters required
fun main() {
println(sum1(1, 2))
println(sum1(3,4))
}
// parameters which require type annotations
fun sum1(a: Int, b: Int): Int {
return a + b
}
// return types can be inferred based on the value you return
// it's better form to explicitly include the return type in the signature
fun sum2(a: Int, b: Int) {
a + b // Kotlin knows that (Int + Int) -> Int
}
Single-Expression Functions
Simple functions in Kotlin can sometimes be reduced to a single line aka a single-expression function.
// previous example
fun sum(a: Int, b: Int) {
a + b // Kotlin knows that (Int + Int) -> Int
}
// this is equivilant
fun sum(a: Int, b: Int) = a + b
// this works since we evaluate a single expression
fun minOf(a: Int, b: Int) = if (a < b) a else b
Function Parameters
Default arguments
We can use default arguments for function parameters. When called, a parameter with a default value is optional; if the value is not provided by the caller, the default will be used.
// Second parameter has a default value, so it’s optional
fun mult(a:Int, b:Int = 1): Int {
return a * b
}
fun main() {
mult(1) // 1
mult(5,2) // 10
// mult() will throw an error, `a` must be provided
}
Named parameters
You can (optionally) provide the parameter names when you call a function. If you do this, you can even change the calling order!
fun repeat(s:String="*", n:Int=1):String {
return s.repeat(n)
}
fun main() {
println(repeat()) // *
println(repeat(s="#")) // *
println(repeat(n=3)) // ***
println(repeat(s="#", n=5)) // #####
println(repeat(n=5, s="#")) // #####
}
Variable-length arguments
Finally, we can have a variable length list of arguments:
// Variable number of arguments can be passed!
// Arguments in the list need to have the same type
fun sum(vararg numbers: Int): Int {
var sum: Int = 0
for(number in numbers) {
sum += number
}
return sum
}
fun main() {
sum(1) // 1
sum(1,2,3) // 6
sum(1,2,3,4,5,6,7,8,9,10) // 55
}
Collections
Overview
A collection is a finite group of some variable number of items (possibly zero) of the same type. Objects in a collection are called elements.
Collections in Kotlin are contained in the kotlin.collections package, which is part of the Kotlin Standard Library.
These collection classes exists as generic containers for a group of elements of the same type e.g. List would be an ordered list of integers. Collections have a finite size, and are eagerly evaluated.
Kotlin offers functional processing operations (e.g. filter, map and so on) on each of these collections.
fun main() {
val list = (1..10).toList() // generate list of 1..10
println( list.take(5).map{it * it} ) // square the first 5 elements
}
Under-the-hood, Kotlin uses Java collection classes, but provides mutable and immutable interfaces to these classes. Kotlin best-practice is to use immutable for read-only collections whenever possible (since mutating collections is often very costly in performance).
Collection Classes
| Collection Class |
Description |
| Pair |
A tuple of two values. |
| Triple |
A tuple of three values. |
| List |
An ordered collection of objects. |
| Set |
An unordered collection of objects. |
| Map |
An associative dictionary of keys and values. |
| Array |
An indexed, fixed-size collection of objects. |
Pair
A Pair is a tuple of two values. Use var or val to indicate mutability. Theto keyword can be used to indicate a Pair.
fun main() {
// mutable
var nova_scotia = "Halifax Airport" to "YHZ"
var newfoundland = Pair("Gander Airport", "YQX")
var ontario = Pair("Toronto Pearson", "YYZ")
ontario = Pair("Billy Bishop", "YTZ") // reassignment is ok
// immutable, mixed types
val canadian_exchange = Pair("CDN", 1.38)
// accessing elements
val characters = Pair("Tom", "Jerry")
println(characters.first)
println(characters.second)
// destructuring
val (first, second) = Pair("Calvin", "Hobbes") // split a Pair
println(first)
println(second)
}
Pairs are extremely useful when working with data that is logically grouped into tuples, but where you don’t need the overhead of a custom class. e.g. Pair for 2D points.
List
A List is an ordered collection of objects.
fun main() {
// define an immutable list
var fruits = listOf( "advocado", "banana")
println(fruits.get(0))
// advocado
// add elements
var mfruits = mutableListOf( "advocado", "banana")
mfruits.add("cantaloupe")
mfruits.forEach { println(it) }
// sorted/sortedBy returns ordered collection
val list = listOf(2,3,1,4).sorted() // [1, 2, 3, 4]
list.sortedBy { it % 2 } // [2, 4, 1, 3]
// groupBy groups elements on collection by key
list.groupBy { it % 2 } // Map: {1=[1, 3], 0=[2, 4]}
// distinct/distinctBy returns unique elements
listOf(1,1,2,2).distinct() // [1, 2]
}
Set
A Set is a generic unordered collection of unique elements (i.e. it does not support duplicates, unlike a List which does). Sets are commonly constructed with helper functions:
val numbersSet = setOf("one", "two", "three", "four")
val emptySet = mutableSetOf<String>()
A Map is an associative dictionary containing Pairs of keys and values.
fun main() {
// immutable reference, immutable map
val imap = mapOf(Pair(1, "a"), Pair(2, "b"), Pair(3, "c"))
println(imap)
// {1=a, 2=b, 3=c}
// immutable reference, mutable map (so contents can change)
val mmap = mutableMapOf(5 to "d", 6 to "e")
mmap.put(7,"f")
println(mmap)
// {5=d, 6=e, 7=f}
// lookup a value
println(mmap.get(5))
// d
// iterate over key and value
for ((k, v) in imap) {
print("$k=$v ")
}
// 1=a 2=b 3=c
// alternate syntax
imap.forEach { k, v -> print("$k=$v ") }
// 1=a 2=b 3=c
// `it` represents an implicit iterator
imap.forEach {
print("${it.key}=${it.value} ")
}
// 1=a 2=b 3=c
}
Array
Arrays are indexed, fixed-sized collection of objects and primitives. We prefer other collections, but these are offered for legacy and compatibility with Java.
fun main() {
// Create using the `arrayOf()` library function
arrayOf(1, 2, 3)
// Create using the Array class constructor
// Array<String> ["0", "1", "4", "9", "16"]
val asc = Array(5) {
i -> (i*i).toString()
}
asc.forEach { println(it) }
}
You can access array elements through using the [] operators, or the get() and set() methods.
Collection Functions
Collection classes (e.g. List, Set, Map, Array) have built-in functions for working with the data that they contain. These functions frequently accept other functions as parameters.
Filter
filter produces a new list of those elements that return true from a predicate function.
val list = (1..100).toList()
val filtered = list.filter { it % 5 == 0 }
// 5 10 15 20 ... 100
val below50 = filtered.filter { it in 0..49 }
// [5, 10, 15, 20]
Map
map produces a new list that is the results of applying a function to every element that it contains.
val list = (1..100).toList()
val doubled = list.map { it * 2 }
// 2 4 6 8 ... 200
Reduce
reduce accumulates values starting with the first element and applying an operation to each element from left to right.
val strings = listOf("a", "b", "c", "d")
println(strings.reduce { acc, string -> acc + string }) // abcd
Zip
zip combines two collections together, associating their respective pairwise elements.
val foods = listOf("apple", "kiwi", "broccoli", "carrots")
val fruit = listOf(true, true, false, false)
// List<Pair<String, Boolean>>
val results = foods.zip(fruit)
// [(apple, true), (kiwi, true), (broccoli, false), (carrots, false)]
A more realistic scenario might be where you want to generate a pair based on the results of the list elements:
val list = listOf("123", "", "456", "def")
val exists = list.zip(list.map { !it.isBlank() })
// [(123, true), (, false), (456, true), (def, true)]
val numeric = list.zip(list.map { !it.isEmpty() && it[0] in ('0'..'9') })
[(123, true), (, false), (456, true), (def, false)]
ForEach
forEach calls a function for every element in the collection.
val fruits = listOf("advocado", "banana", "cantaloupe" )
fruits.forEach { print("$it ") }
// advocado banana cantaloupe
We also have helper functions to extract specific elements from a list.
Take
take returns a collection containing just the first n elements. drop returns a new collection with the first n elements removed.
val list = (1..50)
val first10 = list.take(10)
// 1 2 3 ... 10
val last40 = list.drop(10)
// 11 12 13 ... 50
First, Last, Slice
first and last return those respective elements. slice allows us to extract a range of elements into a new collection.
val list = (1..50)
val even = list.filter { it % 2 == 0 }
// 2 4 6 8 10 ... 50
even.first() // 2
even.last() // 50
even.slice(1..3) // 4 6 8
OO Kotlin
Introduction
Object-oriented programming is a refinement of the structured programming model, that was discovered in 1966 by Ole-Johan Dahl and Kristen Nygaard.
It is characterized by the use of classes as a template for common behaviour (methods) and data (state) required to model that behaviour.
- Abstraction: Model entities in the system to match real world objects. Each object exposes a stable high-level interface, that can be used to access its data and behaviours. “Changing the implementation should not break anything”.
- Encapsulation: Keep state and implementation private.
- Inheritance: Specialization of classes. Create a specialized (child) class by deriving from another (parent) class, and reusing the parent’s fields and methods.
- Polymorphism: Base and derived classes share an interface, but have specialized implementations.
Object-oriented programming has benefits over iterative programming:
- It supports abstraction, and allows us to express computation in a way that models our problem domain (e.g. customer classes, file classes).
- It handles complex state more effectively, by delegating to classes.
- It’s also a method of organizing your code. This is critical as programs grow.
- There is some suggestion that it makes code reuse easier (debatable).
- It became the dominant programming model in the 80s, and our most used languages are OO languages. e.g. C++, Java, Swift, Kotlin.
Classes
Kotlin is a class-based object-oriented language, with some advanced features that it shares with some other recent languages. The class keyword is used to define a Class. You create an instance of the class using the classname (no new keyword required!)
// define class
class Person
// create two instances and assign to p, q
// note that we have an implicit no-arg constructor
val p = Person()
val q = Person()
Classes include properties (values) and methods.
Properties
A property is a variable that is declared in a class, but outside of methods or functions. They are analogous to class members, or fields in other languages.
class Person() {
var firstName = "Vanilla"
var lastName = "Ice"
}
fun main() {
val p = Person()
// we can access properties directly
// this calls an implicit get() method; default returns the value
println("${p.firstName} ${p.lastName} ${p.lastName} Baby")
}
Properties all have implicit backing fields that store their data. We can override the get() and set methods to determine how our properties interact with the backing fields.
For example, for a City class, we can decide that we want the city name always reported in uppercase, and we want the population always stored as thousands.
// the backing field is just referred to as `field`
// in the set() method, we use `value` as the argument
class City() {
var name = ""
get() = field.uppercase()
set(value) {
field = value
}
var population = 0
set(value) {
field = value/1_000
}
}
fun main() {
// create our city, using properties to access values
val city = City()
city.name = "Halifax"
city.population = 431_000
println("${city.name} has a population of ${city.population} thousand people")
}
Behind-the-scenes, Kotlin is actually creating getter and setter methods, using the convention of getField and setField . In other words, you always have corresponding methods that are created for you. If you directly access the field name, these methods area actually getting called in the background.
Venkat Subramaniam has an excellent example of this [Subramaniam 2019]. Write the class Car in a separate file named Car.kt:
class Car(val yearOfMake: Int, var color: String)
Then compile the code and take a look at the bytecode using the javap tool, by running these commands:
$ kotlinc-jvm Car.kt
$ javap -p Car.class
This will display the bytecode generated by the Kotlin Compiler for the Car class:
public final class Car {
private final int yearOfMake;
private java.lang.String color;
public final int getYearOfMake();
public final java.lang.String getColor();
public final void setColor(java.lang.String);
public Car(int, java.lang.String);
}
That concise single line of Kotlin code for the Car class resulted in the creation of two fields—the backing fields for properties, a constructor, two getters, and a setter.
Constructors
Like other OO languages, Kotlin supports explicit constructors that are called when objects are created.
Primary Constructors
A primary constructor is the main constructor that your class will support (representing how you want it to be instantiated most of the time). You define it by expanding the class definition:
// class definition includes the primary constructor
class Person constructor() { }
// we can collapse this to define an explicit no-arg constructor
class Person() {}
In the example above, the primary constructor is called when this class is instantiated.
Optionally, you can include parameters in the primary constructor, and use these to initialize parameters in the constructor body.
// constructor with arguments
// this uses the parameters to initialize properties (i.e. variables)
class Person (first:String, last:String) {
val firstName = first.take(1).uppercase() + first.drop(1).lowercase()
val lastName = last.take(1).uppercase() + last.drop(1).lowercase()
// adding a statement like this will prevent the code from compiling
// println("${firstname} ${lastname}") // will not compile
}
fun main() {
// this does not work! we do not have a no-arg constructor
// val person = Person() // error since no matching constructor
// this works and demonstrates the properties
val person = Person("JEFF", "AVERY")
println("${person.firstName} ${person.lastName}") // Jeff Avery
}
Constructors are designed to be minimal:
- Parameters can only be used to initialize properties. They go out of scope immediately after the constructor executes.
- You cannot invoke any other code in your constructor (there are other ways to handle that, which we will discuss below).
Secondary Constructors
What if you need more than a single constructor?
You can define secondary constructors in your class. Secondary constructors must delegate to the primary constructor. Let’s rewrite this class to have a primary no-arg constructor, and a second constructor with parameters.
// primary constructor
class Person() {
// initialize properties
var firstName = "PAULA"
var lastName = "ABDUL"
// secondary constructor
// delegates to the no-arg constructor, which will be executed first
constructor(first: String, last: String) : this() {
// assign to the properties defined in the primary constructor
firstName = first.take(1).uppercase() + first.drop(1).lowercase()
lastName = last.take(1).uppercase() + last.drop(1).lowercase()
}
}
fun main() {
val person1 = Person() // primary constructor using default property values
println("${person1.firstName} ${person1.lastName}")
val person2 = Person("JEFF", "AVERY") // secondary constructor
println("${person2.firstName} ${person2.lastName}")
}
Init Blocks
How do we execute code in the constructor? We often want to do more than just initialize properties.
Kotlin has a special method called init() that is used to manage initialization code. You can have one or more of these init blocks in your code, which will be called in order after the primary constructor (they’re actually considered part of the primary constructor). The order of initialization is (1) primary constructor, (2) init blocks in listed order, and then finally (3) secondary constructor.
class InitOrderDemo(name: String) {
val first = "$name".also(::println)
init {
println("First init: ${first.length}")
}
val second = "$name".also(::println)
init {
println("Second init: ${second.length}")
}
}
fun main() {
InitOrderDemo("Jeff")
}
Info
Why does Kotlin split the constructor up like this? It’s a way to enforce that initialization MUST happen first, which results in cleaner, and safer code.
Class Methods
Similarly to other programming languages, functions defined inside of a class are called methods.
class Person(var firstName: String, var lastName: String) {
fun greet() {
println("Hello! My name is $firstName")
}
}
fun main() {
val person = Person ("Jeff", "Avery")
println("${person.firstName} ${person.lastName}")
}
Operator Overloading
Kotlin allows you to provide custom implementations for the predefined set of operators. These operators have predefined symbolic representation (like + or *) and predefined precedence if you combine them.
Basically, you use the operator keyword to define a function, and provide a member function or an extension function with a specific name for the corresponding type. This type becomes the left-hand side type for binary operations and the argument type for the unary ones.
Here’s an example that extends a class named ClassName by overloading the + operator.
data class Point(val x: Double, val y: Double)
// -point
operator fun Point.unaryMinus() = Point(-x, -y)
// p1+p2
operator fun Point.plus(other: Point) = Point(this.x + other.x, this.y + other.y)
// p1*5
operator fun Point.times(scalar: Int) = Point(this.x * scalar, this.y * scalar)
operator fun Point.times(scalar: Double) = Point(this.x * scalar, this.y * scalar)
fun main() {
val p1 = Point(5.0, 10.0)
val p2 = Point(10.0, 12.0)
println("p1=${p1}")
println("p2=${p2}\n")
println("-p1=${-p1}")
println("p1+p2=${p1+p2}")
print("p2*5=${p2*5}")
}
We can override any operators by using the keyword that corresponds the symbol we want to override.
Note that this is the reference object on which we are calling the appropriate method. Parameters are available as usual.
| Description |
Expression |
Translated to |
| Unary prefix |
+a |
a.unaryPlus() |
|
-a |
a.unaryMinus() |
|
!a |
a.not() |
| Increments, decrements |
a++ |
a.inc() |
|
a– |
a.dec() |
| Arithmetic |
a+b |
a.plus(b) |
|
a-b |
a.minus(b) |
|
a*b |
a.times(b) |
|
a/b |
a.div(b) |
|
a%b |
a.rem(b) |
|
a..b |
a.rangeTo(b) |
| In |
a in b |
b.contains(a) |
| Augmented assignment |
a+=b |
a.plusAssign(b) |
|
a-=b |
a.minusAssign(b) |
|
a*=b |
a.timesAssign(b) |
|
a/=b |
a.divAssign(b) |
|
a%b |
a.remAssign(b) |
| Equality |
a==b |
a?.equals(b) ?: (b === null) |
|
a!=b |
!(a?.equals(b) ?: (b === null)) |
| Comparison |
a>b |
a.compareTo(b) > 0 |
|
a<b |
a.compareTo(b) < 0 |
|
a>=b |
a.compareTo(b) >= 0 |
|
a<=b |
a.compareTo(b) <= 0 |
Infix Functions
Functions marked with the infix keyword can also be called using the infix notation (omitting the dot and the parentheses for the call). Infix functions must meet the following requirements:
For example, we can add a “shift left” function to the built-in Int class:
infix fun Int.shl(x: Int): Int {
return (this shl x)
}
fun main() {
// calling the function using the infix notation
// shl 1 multiples an int by 2
println(212 shl 1)
// is the same as
println(212.shl(1))
}
Extension Functions
Kotlin supports extension functions: the ability to add functions to existing classes, even when you don’t have access to the original class’s source code, or cannot modify the class for some reason. This is also a great alternative to inheritance when you cannot extend a class.
For an simple example, imagine that you want to determine if an integer is even. The “traditional” way to handle this is to write a function:
fun isEven(n: Int): Boolean = n % 2 == 0
fun main() {
println(isEven(4))
println(isEven(5))
}
In Kotlin, the Int class already has a lot of built-in functionality. It would be a lot more consistent to add this as an extension function to that class.
fun Int.isEven() = this % 2 == 0
fun main() {
println(4.isEven())
println(5.isEven())
}
You can use extensions with your own types and also types you do not control, like List, String, and other types from the Kotlin standard library.
Extension functions are defined in the same way as other functions, with one major difference: When you specify an extension function, you also specify the type the extension adds functionality to, known as the receiver type. In our earlier example, Int.isEven(), we need to include the class that the function extends, or Int.
Note that in the extension body, this refers to the instance of the type (or the receiver for this method invocation).
fun String.addEnthusiasm(enthusiasmLevel: Int = 1) = this + "!".repeat(enthusiasmLevel)
fun main() {
val s1 = "I'm so excited"
val s2 = s1.addEnthusiasm(5)
println(s2)
}
Defining an extension on a superclass
Extensions do not rely on inheritance, but they can be combined with inheritance to expand their scope. If you extend a superclass, all of it’s subclasses will inherit the extension method that you defined.
Define an extension on the Any class called print. Because it is defined on Any, it will be directly callable on all types.
// Any is the top-level class from which all classes derive i.e. the ultimate superclass.
fun Any.print() {
println(this)
}
fun main() {
"string".print()
42.print()
}
Extension Properties
In addition to adding functionality to a type by specifying extension functions, you can also define extension properties.
For example, here is an extension property that counts a string’s vowels:
val String.numVowels
get() = count { it.lowercase() in "aeiou" }
fun main() {
println("abcd".numVowels)
}
Special Classes
Data Classes
A data class is a special type of class, which primarily exists to hold data, and doesn’t have custom methods. Classes like this are more common than you expect – we often create trivial classes to just hold data, and Kotlin makes it very easy.
Why would you use a data class over a regular class? It generates a lot of useful methods for you:
- hashCode()
- equals() // compares fields
- toString()
- copy() // using fields
- destructuring
Here’s an example of how useful this can be:
data class Person(val name: String, var age: Int)
fun main() {
val mike = Person("Mike", 23)
// toString() displays all properties
println(mike.toString())
// structural equality (==) compares properties
println(mike == Person("Mike", 23)) // True
println(mike == Person("Mike", 21)) // False
// referential equality (===) compares object references
println(mike === Person("Mike", 23)) // True
// hashCode based on primary constructor properties
println(mike.hashCode() == Person("Mike", 23).hashCode()) // True
println(mike.hashCode() == Person("Mike", 21).hashCode()) // False
// destructuring based on properties
val (name, age) = mike
println("$name $age") // Mike 23
// copy that returns a copy of the object
// with concrete properties changed
val jake = mike.copy(name = "Jake") // copy
}
Enum Classes
Enums in Kotlin are classes, so enum classes support type safety.
We can use them in expected ways. Enum num constants are separated with commas. We can also do interesting things with our enums, like use them in when clauses (Example from [Sommerhoff 2020]).
enum class Suits {
HEARTS, SPADES, DIAMONDS, CLUBS
}
fun main() {
val color = when(Suits.SPADES) {
Suits.HEARTS, Suits.DIAMONDS -> "red"
Suits.SPADES, Suits.CLUBS -> "black"
}
println(color)
}
Each enum constant is an object, and can be instantiated.
enum class Direction(val degrees: Double) {
NORTH(0.0), SOUTH(180.0), WEST(270.0), EAST(90.0)
}
fun main() {
val direction = Direction.EAST
print(direction.degrees)
}
Class Hierarchies
All classes in Kotlin have a common superclass Any, that is the default superclass for a class with no supertypes declared:
class Example // Implicitly inherits from Any
Any has three methods: equals(), hashCode() and toString(). Thus, they are defined for all Kotlin classes.
Inheritance
To derive a class from a supertype, we use the colon : operator. We also need to delegate to the base class constructor using ().
By default, classes and methods are closed to inheritance. If you want to extend a class or method, you need to explicitly mark it as open for inheritance.
class Base
class Derived : Base() // error!
open class Base
class Derived : Base() // ok
Kotlin supports single-inheritance.
open class Person(val name: String) {
open fun hello() = "Hello, I am $name"
}
class PolishPerson(name: String) : Person(name) {
override fun hello() = "Dzien dobry, jestem $name"
}
fun main() {
val p1 = Person("Jerry")
val p2 = PolishPerson("Beth")
println(p1.hello())
println(p2.hello())
}
If the derived class has a primary constructor, the base class can (and must) be initialized, using the parameters of the primary constructor. If the derived class has no primary constructor, then each secondary constructor has to initialize the base type using the super keyword, or to delegate to another constructor which does that.
class MyView : View {
constructor(ctx: Context) : super(ctx)
constructor(ctx: Context, attrs: AttributeSet) : super(ctx, attrs)
}
Abstract Classes
Classes can be declared abstract, which means that they cannot be instantiated, only used as a supertype. The abstract class can contain a mix of implemented methods (which will be inherited by subclasses) and abstract methods, which do not have an implementation.
// useful way to represent a 2D point
data class Point(val x:Int, val y:Int)
abstract class Shape() {
// we can have a single representation of position
var x = 0
var y = 0
fun position(): Point {
return Point(x, y)
}
// subtypes will have their own calculations for area
abstract fun area():Int
}
class Rectangle (var width: Int, var height: Int): Shape() {
constructor(x: Int, y: Int, width: Int, height: Int): this(width, height) {
this.x = x
this.y = y
}
// must be overridden since our base is abstract
override fun area():Int {
return width * height
}
}
fun main() {
// this won't compile, since Shape() is abstract
// val shape = Shape()
// this of course is fine
val rect = Rectangle(10, 20, 50, 10)
println("Rectangle at (${rect.position().x},${rect.position().y}) with area ${rect.area()}")
// => Rectangle at (10,20) with area 500
}
Interfaces
Interfaces in Kotlin are similar to abstract classes, in that they can contain a mix of abstract and implemented methods. What makes them different from abstract classes is that they cannot store state. They can have properties but these need to be abstract or to provide accessor implementations.
Visibility Modifiers
Classes, objects, interfaces, constructors, functions, properties and their setters can have visibility modifiers. Getters always have the same visibility as the property. Kotlin defaults to public access if not visibility modifier is provided.
The possible visibility modifiers are:
public: visible to any other code.private: visible inside this class only (including all its members).protected: visible to any derived class, otherwise private.internal: visible within the same module, where a module is a set of Kotlin files compiled together.
Destructuring
Sometimes it is convenient to destructure an object into a number of variables. This syntax is called a destructuring declaration. A destructuring declaration creates multiple variables at once. In the example below, you declared two new variables: name and age, and can use them independently:
data class Person(val name: String, val age: Int)
fun main() {
val p = Person("Janine", 38)
val (name, age) = p // destructuring
println(name)
println(age)
}
A destructuring declaration is compiled down to the following code:
val name = person.component1()
val age = person.component2()
component1(), component() are aliases to the named properties in this class, in the order they were declared (and, of course, there can be component3() and component4() and so on). You would never normally refer to them using these aliases.
Here’s an example from the Kotlin documentation on how to use this to return multiple values from a function:
// data class with properties `result` and `status`
data class Result(val result: Int, val status: Status)
fun function(...): Result {
// computations
return Result(result, status)
}
// Destructure into result and status
val (result, status) = function(...)
// We can also choose to not assign fields
// e.g. we could just return `result` and discard `status`
val (result, _) = function(...)
Companion Objects
OO languages typically have some idea of static members: methods that are associated with a class instead of an instance of a class. Static methods can be useful when attempting to implement the singleton pattern, for instance.
Kotlin doesn’t support static members directly. To get something comparable in Kotlin, you need to declare a companion object as an inner class of an existing class. Any methods that are created as part of the companion object are considered to be static methods in the enclosing class.
Examples below are taken from: https://livevideo.manning.com/module/59_5_16/kotlin-for-android-and-java-developers
class House(val numberOfRooms: Int, val price: Double) {
companion object {
val HOUSES_FOR_SALE = 10
fun getNormalHouse() = House(6, 599_000.00)
fun getLuxuryHouse() = House(42, 7_000_000.00)
}
}
fun main() {
val normalHouse = House.getNormalHouse() // works
println(normalHouse.price)
println(House.HOUSES_FOR_SALE)
}
We can also use object types to implement singletons. All we need to do is use the object keyword.
class Country(val name:String) {
var area = 0.0
}
// there can be only one
object CountryFactory {
fun createCountry() = Country("Canada")
}
fun main() {
val obj = CountryFactory.createCountry()
println(obj.name)
}
Functional Kotlin
Introduction
Functional programming is a programming style where programs are constructed by compositing functions together. Functional programming treats functions as first-class citizens: they can be assigned to a variable, passed as parameters, or returned from a function.
Functional programming also specifically avoids mutation: functions transform inputs to produce outputs, with no internal state. Functional programming can be described as declarative (describe what you want) instead of imperative (describe how to accomplish a result).
Functional programming constrains assignment, and therefore constrains side-effects (Martin 2003).
Kotlin is considered a hybrid language: it provides mechanisms for you to write in a functional style, but it also doesn’t prevent you from doing non-functional things. As a developer, it’s up to you to determine the most appropriate approach to a given problem.
Here are some common properties that we talk about when referring to “functional programming”:
First-class functions means that functions are treated as first-class citizens. We can pass them as to another function as a parameter, return functions from other functions, and even assignment functions to variables. This allows us to treat functions much as we would treat any other variable.
Pure functions are functions that have no side effects. More formally, the return values of a pure function are identical for identical arguments (i.e. they don’t depend on any external state). Also, by having no side effects, they do not cause any changes to the system, outside of their return value. Functional programming attempts to reduce program state, unlike other programming paradigms (imperative or object-oriented which are based on careful control of program state).
Immutable data means that we do not modify data in-place. We prefer immutable data that cannot be accidentally changed, especially as a side-effect of a function. Instead, if we need to mutate data, we pass it to a function that will return a new data structure containing the modified data, leaving the original data intact. This avoids unintended state changes.
Lazy evaluation is the notion that we only evaluate as expression when we need to operate on it (and we only evaluate what we need to evaluate at the moment!) This allows us to express and manipulate some expressions that would be extremely difficult to actually represent in other paradigms.
Info
In the next sections, we’ll focus on Kotlin support for higher-order functions. Avoiding mutation and side effects is partly a stylistic choice - you don’t require very many language features to program this way, but Kotlin encourages non-mutable data with the val keyword.
Function Types
Functions in Kotlin are “first-class citizens” of the language. This means that we can define functions, assign them to variables, pass functions as arguments to other functions, or return functions! Functions are types in Kotlin, and we can use them anywhere we would expect to use a regular type.
Dave Leeds on Kotlin presents the following excellent example:
Bert’s Barber shop is creating a program to calculate the cost of a haircut, and they end up with 2 almost-identical functions.
fun main() {
val taxMultiplier = 1.10
fun calculateTotalWithFiveDollarDiscount(initialPrice: Double): Double {
val priceAfterDiscount = initialPrice - 5.0
val total = priceAfterDiscount * taxMultiplier
return total
}
fun calculateTotalWithTenPercentDiscount(initialPrice: Double): Double {
val priceAfterDiscount = initialPrice * 0.9
val total = priceAfterDiscount * taxMultiplier
return total
}
}
These functions are identical except for the line that calculates priceAfterDiscount. If we could somehow pass in that line of code as an argument, then we could replace both with a single function that looks like this, where applyDiscount() represents the code that we would dynamically replace:
// applyDiscount = initialPrice * 0.9, or
// applyDiscount = initialPrice - 5.0
fun calculateTotal(initialPrice: Double, applyDiscount: ???): Double {
val priceAfterDiscount = applyDiscount(initialPrice)
val total = priceAfterDiscount * taxMultiplier
return total
}
This is a perfect scenario for passing in a function!
Assign a function to a variable.
fun discountFiveDollars(price: Double): Double = price - 5.0
val applyDiscount = ::discountFiveDollars
In this example, applyDiscount is now a reference to the discountFiveDollars function (note the :: notation when we have a function on the RHS of an assignment). We can even call it.
val discountedPrice = applyDiscount(20.0) // Result is 15.0
So what is the type of our function? The type of a function is the function signature, but with a different syntax that you might be accustomed to seeing.
// this is the original function signature
fun discountFiveDollars(price: Double): Double = price - 5.0
val applyDiscount = ::discountFiveDollars
// applyDiscount accepts a Double as an argument and returns a Double
// we use this format when specifying the type
val applyDiscount: (Double) -> Double
For functions with multiple parameters, separate them with a comma.
We can use this notation when explicitly specifying type.
fun discountFiveDollars(price: Double): Double = price - 5.0
// specifying type is not necessary since type inference works too
// we'll just do it here to demonstrate how it would appear
val applyDiscount : (Double) -> Double = ::discountFiveDollars
Pass a function to a function
We can use this information to modify the earlier example, and have Bert’s calculation function passed into the second function.
fun discountFiveDollars(price: Double): Double = price - 5.0
fun discountTenPercent(price: Double): Double = price * 0.9
fun noDiscount(price: Double): Double = price
fun calculateTotal(initialPrice: Double, applyDiscount: (Double) -> Double): Double {
val priceAfterDiscount = applyDiscount(initialPrice)
val total = priceAfterDiscount * taxMultiplier
return total
}
val withFiveDollarsOff = calculateTotal(20.0, ::discountFiveDollars) // $16.35
val withTenPercentOff = calculateTotal(20.0, ::discountTenPercent) // $19.62
val fullPrice = calculateTotal(20.0, ::noDiscount) // $21.80
Returning Functions from Functions
Instead of typing in the name of the function each time he calls calculateTotal(), Bert would like to just enter the coupon code from the bottom of the coupon that he receives from the customer. To do this, he just needs a function that accepts the coupon code and returns the right discount function.
fun discountForCouponCode(couponCode: String): (Double) -> Double = when (couponCode) {
"FIVE_BUCKS" -> ::discountFiveDollars
"TAKE_10" -> ::discountTenPercent
else -> ::noDiscount
}
Info
I’ve taken liberties with Dave Leed’s example, but my notes can’t do it justice. I’d highly recommend a read through his site - he’s building an outstanding Kotlin book chapter-by-chapter with cartoons and illustrations.
Introduction to Lambdas
We can use this same notation to express the idea of a function literal, or a function as a value.
val applyDiscount: (Double) -> Double = { price: Double -> price - 5.0 }
The code on the RHS of this expression is a function literal, which captures the body of this function. We also call this a lambda. A lambda is just a function, but written in this form:
- the function is enclosed in curly braces { }
- the parameters are listed, followed by an arror
- the body comes after the arrow
What makes a lambda different from a traditional function is that it doesn’t have a name. In the expression above, we assigned the lambda to a variable, which we could them use to reference it, but the function itself isn’t named.
Note that due to type inference, we could rewrite this example without the type specified on the LHS. This is the same thing!
val applyDiscount = { price: Double -> price - 5.0 }
The implicit ‘it’ parameter
In cases where there’s only a single parameter for a lambda, you can omit the parameter name and the arrow. When you do this, Kotlin will automatically make the name of the parameter it.
val applyDiscount: (Double) -> Double = { it - 5.0 }
Lambdas and Higher-Order Functions
Passing Lambdas as Arguments
Higher-order functions have a function as an input or output. We can rewrite our earlier earlier example to use lambdas instead of function references:
// fun discountFiveDollars(price: Double): Double = price - 5.0
// fun discountTenPercent(price: Double): Double = price * 0.9
// fun noDiscount(price: Double): Double = price
fun calculateTotal(initialPrice: Double, applyDiscount: (Double) -> Double): Double {
val priceAfterDiscount = applyDiscount(initialPrice)
val total = priceAfterDiscount * taxMultiplier
return total
}
val withFiveDollarsOff = calculateTotal(20.0, { price - 5.0 }) // $16.35
val withTenPercentOff = calculateTotal(20.0, { price * 0.9 }) // $19.62
val fullPrice = calculateTotal(20.0, { price }) // $21.80
In cases where function’s last parameter is a function type, you can move the lambda argument outside of the parentheses to the right, like this:
val withFiveDollarsOff = calculateTotal(20.0) { price -> price - 5.0 }
val withTenPercentOff = calculateTotal(20.0) { price -> price * 0.9 }
val fullPrice = calculateTotal(20.0) { price -> price }
This is meant to be read as two arguments: one inside the brackets, and the lambda as the second parameter.
Returning Lambdas as Function Results
We can easily modify our earlier function to return a lambda as well.
fun discountForCouponCode(couponCode: String): (Double) -> Double = when (couponCode) {
"FIVE_BUCKS" -> { price -> price - 5.0 }
"TAKE_10" -> { price -> price * 0.9 }
else -> { price -> price }
}
Scope Functions
The Kotlin standard library contains several functions whose sole purpose is to execute a block of code on an object. When you call such a function on an object with a lambda expression, it forms a temporary scope, and applies the lambda to that object.
There are five of these scope functions: let, run, with, apply, and also, and each of them has a slightly different purpose.
Here’s an example where we do not use one of these scope functions. There is a great deal of repetition, since we need a temporary variable, and then have to act on that object.
val alice = Person("Alice", 20, "Amsterdam")
println(alice)
alice.moveTo("London")
alice.incrementAge()
println(alice)
With a scope function, we can refer to the object without using a name. This is greatly simplified!
Person("Alice", 20, "Amsterdam").let {
println(it)
it.moveTo("London")
it.incrementAge()
println(it)
}
The scope functions have subtle differences in how they work, summarized from the Kotlin Standard Library documentation. Inside the lambda of a scope function, the context object is available by a short reference instead of its actual name. Each scope function uses one of two ways to access the context object: as a lambda receiver (this) or as a lambda argument (it).
| Function |
Object reference |
Return value |
Is extension function |
let |
it |
Lambda result |
Yes |
run |
this |
Lambda result |
Yes |
run |
- |
Lambda result |
No: called without the context object |
with |
this |
Lambda result |
No: takes the context object as an argument. |
apply |
this |
Context object |
Yes |
also |
it |
Context object |
Yes |
The context object is available as an argument (it). The return value is the lambda result.
let can be used to invoke one or more functions on results of call chains. For example, the following code prints the results of two operations on a collection:
val numbers = mutableListOf("one", "two", "three", "four", "five")
val resultList = numbers.map { it.length }.filter { it > 3 }
println(resultList)
With let, you can rewrite it:
val numbers = mutableListOf("one", "two", "three", "four", "five")
numbers.map { it.length }.filter { it > 3 }.let {
println(it)
// and more function calls if needed
}
A non-extension function: the context object is passed as an argument, but inside the lambda, it’s available as a receiver (this). The return value is the lambda result.
We recommend with for calling functions on the context object without providing the lambda result. In the code, with can be read as “with this object, do the following.”
val numbers = mutableListOf("one", "two", "three")
with(numbers) {
println("'with' is called with argument $this")
println("It contains $size elements")
}
The context object is available as a receiver (this). The return value is the lambda result.
run does the same as with but invokes as let - as an extension function of the context object.
run is useful when your lambda contains both the object initialization and the computation of the return value.
val service = MultiportService("https://example.kotlinlang.org", 80)
val result = service.run {
port = 8080
query(prepareRequest() + " to port $port")
}
// the same code written with let() function:
val letResult = service.let {
it.port = 8080
it.query(it.prepareRequest() + " to port ${it.port}")
}
The context object is available as a receiver (this). The return value is the object itself.
Use apply for code blocks that don’t return a value and mainly operate on the members of the receiver object. The common case for apply is the object configuration. Such calls can be read as “apply the following assignments to the object.”
val adam = Person("Adam").apply {
age = 32
city = "London"
}
println(adam)
Having the receiver as the return value, you can easily include apply into call chains for more complex processing.
The context object is available as an argument (it). The return value is the object itself.
also is good for performing some actions that take the context object as an argument. Use also for actions that need a reference to the object rather than its properties and functions, or when you don’t want to shadow the this reference from an outer scope.
When you see also in the code, you can read it as “and also do the following with the object.”
val numbers = mutableListOf("one", "two", "three")
numbers
.also { println("The list elements before adding new one: $it") }
.add("four")
Lazy Sequences
Lazy evaluation allows us to generate expressions representing large or infinite lists, and work on them without actually evaluating every element . For example, we can generate an infinite sequence and then extract the first n elements that we need.
// generate an infinite list of integers
// starting at zero, step 10
val list = generateSequence(0) { it + 10}
// 0 10 20 30 40 50 60 70 80 ...
val results = list.take(5).toList()
// 0 10 20 30 40
take from this list before attempting to do anything with it. It’s infinite so it’s possible to hang your system if you’re not careful.
val list = generateSequence(0) { it + 10}
val results = list.drop(5).toList() // length is infinite - 5 ?!?
Chaining operations
Since our higher-order functions typically return a list, we can chain operations together, so the return value of one function is a list, which is acted on by the next function in the chain. For example, we can map and filter a collection without needing to store the intermediate collection.
val list = (1..999999).toList()
val results = list
.map { it * 2 }
.take(10)
// [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
The operations are performed in top-down order: map, then take. In this case, it means that we’re mapping the entire list and then discarding most of the resulting list with the take operation. This is really inefficient: filter your list first!
// better implementation
val veryLongList = listOf(0..9999999L).toList()
val results = veryLongList
.take(50)
.map { it * 2 }
// [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]
Recursion
As a hybrid language, Kotlin supports a number of paradigms. Recursion is less likely than other languages, given that we have loops and other mechanisms to handle iteration.
However, the compiler certainly supports recursion, and can even optimize for tail recursion. To qualify, a function needs to:
- be structured so that the last statement is a call to the function, with state being passed in the function call.
- use the
tailrec keyword.
import java.math.BigInteger
tailrec fun fibonacci(n: Int, a: BigInteger, b: BigInteger): BigInteger {
return if (n == 0) a else fibonacci(n-1, b, a+b)
}
fun main(args: Array<String>) {
println(fibonacci(100, BigInteger("0"), BigInteger("1")))
}
// 354224848179261915075
https://www.programiz.com/kotlin-programming/recursion
https://xkcd.com/1270/
Idiomatic Kotlin
Urs Peters: Idiomatic Kotlin
This section summarizes a talk by Urs Peters, presented on Kotlin Dev Day 2022. No claim of authorship is intended – it’s a very interesting talk and worth watching in it’s entirety!
https://www.youtube.com/watch?v=zYH6zTtl-nc
Why Idiomatic Kotlin?
It’s possible to use Kotlin as a “better Java”, but you would be missing out on some of the features that make Kotlin unique and interesting.
Principles
1. Favor immutability over mutability.
Kotlin favors immutability by providing various immutable constucts and defaults.
data class Programmer(val name: String,
val languages: List<String>)
fun known(language:String) = languages.contains(language)
val urs = Programmer("Urs", listOf("Kotlin", "Scale", "Java"))
val joe = urs.copy(name = "Joe")
What is so good about immutability?
- Immutability: exactly one state that will never change.
- Mutable: an infinite amount of potential states.
| Criteria |
Immutable |
Mutable |
| Reasoning |
Simple: one state only |
Hard: many possible states |
| Safety |
Safer: state remains the same and valid |
Unsafe: accidental errors due to state changes |
| Testability |
No side effects which makes tests deterministic |
Side effects: can lead to unexpected failures |
| Thread-safety |
Inherently thread-safe |
Manual synchronization required |
How to leverage it?
- prefer
vals over vars
- prefer read-only collections (
listOf instead of mutableListOf)
- use immutable value objects instead of mutable ones (e.g. data classes over classes)
Local mutability that does not leak outside is ok (e.g. a var within a function is ok if nothing external to the function relies on it).
2. Use Nullability
Think twice before using !!
val uri = URI("...")
val res = loadResource(uri)
val lines = res!!read() // bad!
val lines = res?.read() ?: throw IAE("$uri invalid") // more reasonable
Stick to nullable types only
public Optional<Goody> findGoodyForAmount(amount:Double)
val goody = findGoodyForAmount(100)
if(goody.isPresent()) goody.get() ... else ... // bad
val goody = findGoodyForAmount(100).orElse(null)
if(goody != null) goody ... else ... // good uses null consistently
Use nullability where applicable but don’t overuse it.
data class Order(
val id: Int? = null,
val items: List<LineItem>? = null,
val state: OrderState? = null,
val goody: Goody? = null
) // too much!
data class Order(
val id: Int? = null,
val items: List<LineItem> = emptyList()),
val state: OrderState = UNPAID,
val goody: Goody? = null
) // some types made more sense as not-null values
Avoid using nullable types in Collections
val items: List<LineItem?> = emptyList()
val items: List<LineItem>? = null,
val items: Lilst<LineItem?>? = null // all terribad
val items: List<LineItem> = emptyList() // that's what this is for
Use lateinit var for late initialization rather than nullability
// bad
class CatalogService(): ResourceAware {
var catalog: Catalog? = null
override fun onCreate(resource: Bundle) {
this.catalog = Catalog(resource)
}
fun message(key: Int, lang: String) =
catalog?.productDescription(key, lang) ?: throw IllegalStateException("Impossible")
}
// good
class CatalogService(): ResourceAware {
lateinit var catalog: Catalog
override fun onCreate(resource: Bundle) {
this.catalog = Catalog(resource)
}
fun message(key: Int, lang: String) =
catalog.productDescription(key, lang)
3. Get The Most Out Of Classes and Objects
Use immutable data classes for value classes, config classes etc.
class Person(val name: String, val age: Int)
val p1 = Person("Joe", 42)
val p2 = Person("Joe", 42)
p1 == p2 // false
data class Person(val name: String, val age: Int)
val p1 = Person("Joe", 42)
val p2 = Person("Joe", 42)
p1 == p2 // true
Use normal classes instead of data classes for services etc.
class PersonService(val dao: PersonDao) {
fun create(p: Person) {
if (op.age >= MAX_AGE)
LOG.warn("$p ${bornInYear(p.age)} too old")
dao.save(p)
}
companion object {
val LOG = LogFactory.getLogger()
val MAX_AGE = 120
fun bornInYear(age: Int) = ...
}
}
Use value classes for domain specific types instead of common types.
value class Email(val value: String)
value class Password(val value: String)
fun login(email: Email, pwd: Password) // no performance impact! type erased in bytecode
Seal classes for exhaustive branch checks
// problematic
data class Square(val length: Double)
data class Circle(val radius: Double)
when (shape) {
is Circle -> "..."
is Rectangle -> "..."
else -> throw IAE("unknown shape $shape") // annoying
}
// fixed
sealed interface Shape // prevents additions
data class Square(val length: Double)
data class Circle(val radius: Double)
when (shape) {
is Circle -> "..."
is Rectangle -> "..."
}
4. Use Available Extensions
// bad
val fis = FileInputStream("path")
val text = try {
val sb = StringBuilder()
var line: String?
while(fis.readLine().apply {line = this} != null) {
sb.append(line).append(System.lineSeparator())
}
sb.toString()
} finally {
try { fis.close() } catch (ex:Throwable) { }
}
// good, via extension functions
val text = FileInputStream("path").use { it.reader().readText() }
Extend third party classes
// bad
fun toDateString(dr: LocalDateTime) = dt.format(DateTimeFormatter.ofPattern("yyyy-MM-dd"))
// good, works with code completion!
fun LocalDateTime.toDateString() = this.format(DateTimeFormatter.ofPattern("yyyy-MM-dd"))
5. Use Control-Flow Appropriately
Use if/else for single branch conditions rather than when
// too verbose
val reduction = when {
customer.isVip() -> 0.05
else -> 0.0
}
// better
val reduction = if (customer.isVip()) 0.05 else 0.0
Use when with multi-branch conditions.
fun reduction(customerType: CustomerTypeEnum) = when (customerType) {
REGULAR -> 0
GOLD -> 0.1
PLATINUM -> 0.3
}
6. Expression Oriented Programming
Imperative Programming
Imperative programming relies on declaring variables that are mutated along the way.
var kotlinDevs = mutableListOf<Person>()
for (person in persons) {
if (person.langs.contains("Kotlin"))
kotlinDevs.add(person)
}
kotlinDevs.sort()
Think of
-
var, lops, mutable collections
-
Mutating data, side effects
Expression Oriented Programming (on the way to Functional Programming!)
Expression oriented programming relies on thinking in functions where every input results in an output.
persons.filter { it.langs.contains("Kotlin") }.sorted()
Think of:
- Val, (higher-order) functions, functional + read-only collections
- Input/Output, transforming data
This is better because it results in more concise, deterministic, more easily testable and clearly scoped code that is easy to reason about compared to the imperative style.
if/else is an expression returning a result.
// imperative style
var result: String
if(number % 2 == 0)
result = "EVEN"
else
result = "ODD"
// expression style, better
val result = if(number % 2 == 0) "EVEN" else "ODD"
when is an expression too, returning a result.
// imperative style
var hi: String
when(lang) {
"NL" -> hi = "Goede dag"
"FR" -> hi = "Bonjour"
else -> hi = "Good day"
}
// expression style, better
val hi = when(lang) {
"NL" -> "Goede dag"
"FR" -> "Bonjour"
else -> "Good day"
}
try/catch also.
// imperative style
var text: String
try {
text = File("path").readText()
} catch (ex: Exception) {
text = ""
}
// expression style, better
val text = try {
File("path").readText()
} catch (ex: IOException) {
""
}
Most functional collections return a result, so the return keyword is rarely needed!
fun firstAdult(ps: List<Person>, age: Int) =
ps.firstOrNull{ it.age >= 18 }
7. Favor Functional Collections Over For-Loops
Program on a higher abstraction level with (chained) higher-order functions from the collection.
// bad!
val kids = mutableSetOf<Person>()
for(person in persons) {
if(person.age < 18) kids.add(person)
}
names.sorted()
// better!
val kids: mutableSetOf<Person> = persons.filter{ it.age < 18}
You are actually programming at a higher-level of abstraction, since you’re manipulating the collection directly instead of considering each of its elements. e.g. it’s obvious in the second example that we’re filtering, instead of needing to read the implementation to figure it out.
For readability, write multiple chained functions from top-down instead of left-right.
// bad!
val names = mutableSetOf<String>()
for(person in persons) {
if(person.age < 18)
names.add(person.name)
}
names.sorted()
// better!
val names = persons.filter{ it.age < 18}
.map{ it.name }
.sorted()
Use intermediate variables when chaining more than ~3-5 operators.
// bad!
val sortedAgeGroupNames = persons
.filter{ it.age >= 18 }
.groupBy{ it.age / 10 * 10 }
.mapValues{ it.value.map{ it.name }}
.toList()
.sortedBy{ it.first }
// better, more readable
val ageGroups = persons.filter{ it.age >= 18 }
.groupBy{ it.age / 10 * 10 }
val sortedNamesByAgeGroup = ageGroups
.mapValues{ (_, group) -> group.map(Person::name) }
.toList()
.sortedBy{ (ageGroup, _) -> ageGroup }
8. Scope Your Code
Use apply/with to configure a mutable object.
// old way
fun client(): RestClient {
val client = RestClient()
client.username = "xyz"
client.secret = "secret"
client.url = "https://..../employees"
return client
}
// better way
fun client() = RestClient().apply {
username = "xyz"
secret = "secret"
url = "https://..../employee"
}
Use let/run to manipulate the context object and return a different type.
// old way
val file = File("/path")
file.setReadOnly(true)
val created = file.createNewFile()
// new way
val created = File("/path").run {
setReadOnly(true)
createNewFile() // last expression so result from this function is returned
}
Use also to execute a side-effect.
// old way
if(amount <= 0) {
val msg = "Payment amount is < 0"
LOGGER.warn(msg)
throw IAE(msg)
} else ...
// new way
require(amount > 0) {
"Payment amount is < 0".also(LOGGER::warn)
}
9. Embrace Coroutines
// No coroutines
// Using Mono from Sprint React and many combinators (flatMap)
// Standard constructs like if/else cannot be used
// Business intent cannot be derived from this code
@PutMapping("/users")
@ResponseBody
fun upsertUser(@RequestBody user: User): Mono<User> =
userByEmail(user.email)
.switchIfEmpty{
verifyEmail(user.email).flatMap{ valid ->
if(valid) Mono.just(user)
else Mono.error(ResponseStatusException(BAD_REQUEST, "Bad Email"))
}
}.flatMap{ toUpsert -> save(toUpsert) }
// Coroutines clean this up
// Common language constructs can be used
// Reads like synchronous code
@PutMapping("/users")
@ResponseBody
fun upsertUser(@RequestBody user: User): User =
userByEmail(user.email).awaitSingle() ?:
if (verifyEmail(user.email)) user else
throw ResponseStatusException(BAD_REQUEST, "Bad Email")).let{
toUpsert -> save(toUpsert)}
Project Loom will (eventually) result in support for coroutines running on the JVM. This will greatly simplify running coroutines.
Subsections of Software Design
Architecture
Formally, Software architecture is the “fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution” [IEEE 1471-200].
Architecture can also be seen as a shared understanding how the system is structured. Martin Fowler (2003) attempts to pin down the term in a couple of different ways:
Definition 1: “Expert developers working on a project have a shared understanding of the system design. This shared understanding is called ‘architecture’ [and] includes how the system is divided into components and how the components interact through interfaces.”
Definition 2: “Architecture is the set of design decisions that must be made early in a project [and that we would like to get right]”.
Architecture is the also holistic analysis of a system, and how it’s parts relate to one another. Instead of examining requirements in isolation, we instead want to look at the consequences of the structure itself, including the qualities that emerge from this structure.
Architecture can be said to address the intersection of business goals, user goals and technical (system) qualities. The architect needs to determine how to deliver the functional requirements in a way that also addresses these qualities, and other potential business needs (e.g. cost). This may very well include making tradeoff decisions ahead of time. e.g. a user may want a system to return the results of a query in less than 5 seconds, but the cost of doing this might be prohibitively expensive!
The benefit to a careful architecture is that we have a more stable initial design that reflects our project concerns, while still allowing for adaptability, flexibility and other desireable qualities. We’ll discuss different qualities of a system below.
Info
Diagrams and portions of the following sections have been taken from: Mark Richards & Neal Ford. 2020. Fundamentals of Software Architecture: An Engineering Approach. O’Reilly. ISBN 978-1492043454.
Imposing Structure
Architects need to be concerned with both the logical structure of systems, and the physical realization of that structure.
Modularity (Logical)
Modularity refers to the logical grouping of source code into related groups. This can be realized as namespaces (C++), packages (Java or Kotlin). Modularity is important because it helps reinforce a separation of concerns, and also encourages reuse of source code through modules.
When discussing modularity, we can identify two related concepts: cohesion, coupling.
Cohesion is a measure of how related the parts of a module are to one another. A good indication that the classes or other components belong in the same module is that there are few, if any, calls to source code outside of the module (and removing anything from the module would necessitate calls to it outside of the module).
Coupling refers to the calls that are made between components; they are said to be tightly coupled based on their dependency on one another for their functionality. Loosely coupled means that there is little coupling, or it could be avoided in practice; tight coupling suggests that the components probably belong in the same module, or perhaps even as part of a larger component.
When designing modules, we want high cohesion (where components in a module belong together) and low coupling between modules (meaning fewer dependencies). This increases the flexibility of our software, and makes it easier to achieve desireable characteristics e.g. scalability.
Info
In Kotlin, modules can be created by assigning classes to the same package (using the package keyword at the top of a class). If you do this, you also need to place your files in a directory with the same name as the namespace e.g. classes in the package graphics would need to be located in a common directory named /graphics.
Components (Physical)
Modules are logical collections of related code. Components are the physical manifestation of a module. Components can represent a number of different abstractions, from a simple wrapper of related classes, to an entire layer of software that runs independently and communicates with external systems.
- Library. A simple wrapper is often called a library, which tends to run in the same memory address as the calling code and communicate via language function call mechanisms. Libraries are usually compile-time dependencies.
- Layers or subsystems. Groups of related code deployed together that may communicate with one another directly.
- Distributed service. A service tends to run in its own address space and communicates via low-level networking protocols like TCP/IP or higher-level formats like REST or message queues, forming stand-alone, deployable units. These are useful in in architectures like microservices.
Info
In Kotlin, a jar file is the component that we most often create to represent a module. Jar files are designed to be distributed much like a library in other languages.
Top-Level Partitioning
Partitioning is the decision on how we organize and group functionality (we use the term top-level partitioning, because this is the highest level of organization).
There are multiple approaches to how we group functionality.
- Technical partitioning: we group functionality according to technical capabilities. e.g. presentation or UI layer, business rules or domain layer and so on.
- Domain partitioning: we group functionality according to the domain area or area of interest. e.g. a payment processing module, a shopping cart module, a reporting module and so on.
So which is correct? Good question.
Technical partitioning tends to be used more often. If we’re concerned about reusability, it’s much easier to design a third party library that can be injected into an application if you provide technical capabilities, as compared to designing a domain-specific library.
For example, we have lots of UI frameworks that sit at the presentation layer, that can be used to build any sort of application regardless of domain. There are very few CatalogCheckout libraries, since any code produced to address that functionality is likely designed around the assumptions of that specific instance of that domain - and is unlikely to be reusable.
From here, developers subdivide components into classes, functions, or subcomponents. In general, class and function design is the shared responsibility of architects, tech leads, and developers.
Determining Components
So we know what components are, but how do we determine what components we need to create? Here’s a couple of common approaches, both of which assume that you’ve identified Use Cases (in your Requirements phase).
Actor/Actions: From your Use Cases, identify actors who perform activities, and the actions that they may perform. This is simply a technique for discovering the typical users of the system and what kinds of things they might do with the system. These actions represent activities that can be mapped directly to a corresponding software component.
Workflow: This approach looks at the key activities being performed, determines workflows and attempts to build components to address those specific activities.
Architectural Styles
An architectural style (or architectural pattern) is the overall structure that we create to represent our software. In describes how our components are organized and structured. Similar to design patterns, an architectural style is a general solution that has been found to work well at solving specific types of problems. The key to using these is to understand the problem well enough that you can determine if a pattern is applicable, and useful to your particular situation.
An architectural pattern describes both the topology (organization of components) and the associated architectural characteristics.
Fundamental Patterns
There are some fundamental patterns that have appeared through history.
Big Ball of Mud
Architects refer to the absence of any discernible architecture structure as a Big Ball of Mud.
A Big Ball of Mud is a haphazardly structured, sprawling, sloppy, duct-tape-and-baling-wire, spaghetti-code jungle. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. – Foote& Yoder 1997.
A Big Ball of Mud isn’t intentional - it’s what happens when you fail to consider architecture in a software project. Treat this as an anti-pattern.
Unitary (Monolithic)
A monolithic structure simply means an application that is designed to run on a single system, and not communicate with any other systems. Source code had very little structure, these systems generally worked in isolation, on data that was carefully fed to them.
However, one inviolatable rule is that systems increase in complexity and capabilities over time. As systems grow, software has to be more carefully structures and managed to continue to meet these requirements.
Client-Server
Client-server architectures were the first major break away from a monolithic architecture, and split processing into front-end and back-end pieces. This is also called a two-tier architecture. There are different ways to divide up the system into front-end and back-end. Examples include splitting between desktop application (front-end) and shared relational database (back-end), or web browser (front-end) and web server (back-end).
Three-tier architectures were also popular in the 1990s and 2000s, which would also include a middle business-logic tier:
In this particular example, the presentation tier handled the UI, the logic tier handled business logic or applicaiton logic, and the data tier managed persistance.
These tiers are commonly used in other architectures, and we’ll revisit them shortly.
Monolithic Architectures
Monolithic architectures consist of a single deployment unit. i.e. the application is self-contained and deployed to a single system. (It may still communicate with external entities, but these are separate systems).
Layered
A layered or n-tier architecture is a very common architectural style that organizes software into horizontal layers, where each layer represents some logical functionality.
There is some similarlty to client-server, though we don’t assume that these layers are split across physically distinct systems (which is why we describe them as logical layers and not physical tiers).
Standard layers in this style of architecture include:
-
Presentation: UI layer that the user interacts with.
-
Business Layer: the application logic, or “business rules”.
-
Persistence Layer: describes how to manage and save application data.
-
Database Layer: the underlying data store that actually stores the data.
The major characteristic of a layered architecture is that it enforces a clear separation of concerns between layers: the Presentation layer doesn’t know anything about the application state or logic, it just displays the information; the Business layer knows how to manipulate the data, but not how it is stored and so on. Each layer is considered to be closed to all of the other layers, and can only be communicated with through a specific interface.
The layered architecture makes an excellent starting point for many simple applications that have few external interactions. However, be careful to ensure that your layers are actually adding functionality to a request, otherwise they are just added overhead with no added value. Layered is well-suited for small simple applications, but may not scale well if you need to expand your application’s functionality across more than a single tier.
Pipeline
A pipeline (or pipes and filters) architecture is appropriate when we want to transform data in a sequential manner. It consists of pipes and filters, linked together in a specific fashion:
Pipes form the communication channel between filters. Each pipe is unidirectional, accepting input on one end, and producing output at the other end.
Filters are entities that perform operation on data that they are fed. Each filter performs a single operation, and they are stateless. There are different types of filters:
- Producer: The outbound starting point (also called a source).
- Transformer: Accepts input, optionally transforms it, and then forwards to a filter (this resembles a map operation).
- Tester: Accepts input, optionally transforms it based on the results of a test, and then forwards to a filter (this resembles a reduce operation).
- Consumer: The termination point, where the data can be saved, displayed etc.
These abstractions may appear familiar, as they are used in shell programming. It’s broadly applicable anytime you want to process data sequentially according to fixed rules.
Examples include: photo manipulation software, shells.
Microkernel
A microkernel architecture (also called plugin architecture) is a popular pattern that provides the ability to easily extend application logic to external, pluggable components. e.g. IntelliJ IDEA which uses application plugins to add functionality for new programming languages.
This architecture works by focusing the primary functionality into the core system, and providing extensibility through the plugin system. This allows the developer, for instance, to invoke functionality in a plugin when the plugin is present, using a defined interface that describes how to invoke it (without need to understand the underlying code).
An example would be a payment processing system, where the core system handles shopping and payment calculations, and behaviour specific to a payment vendor could be contained within a plugin (e.g. Visa plugin, AMEX plugin and so on).
One final note: interaction between other system components and plugins is done through the core system as a mediator. This reduces coupling of components and the plugins, and retains the flexibility of this architecture.
Examples of this architecture include web browsers (which support extensions), and IDEA (which support plugins for various programming languages).
Distributed Architectures
Distributed architectures assume multiple deployments across different systems. These deployments communicate over a network, or similar medium using a defined protocol.
This overhead leads to some unique challenges that are referred to collectively as the fallacies of distributed computing. This includes concerns with network reliability, latency, bandwith, security and so on - things that are non-issues with monolithic architectures.
Services-Based
A services-based architecture splits functionality into small “portions of an application” (also called domain services) that are independent and separately deployed. This is demonstrated below with a separately deployed user interface, a separately deployed series of coarse-grained services, and a monolithic database. Each service is a separate monolithic application that provides services to the application, and they share a single monolithic database.
Each service provides coarse-grained domain functionality (i.e. operating at a relatively high level of abstraction) and addresses a particular business-need. e.g. a service might handle a customer checkout request to process an order.
Working at a coarse-grained level of abstraction like this means that these types of services can rely on regular ACID (atomicity, consistency, isolation, durability) database transactions to ensure data integrity. In other words, since the service is handling the logic of the entire operation, it can consolidate all of the steps in a single database transaction. If there is a failure of any kind, it can report the status to the customer and rollback the transaction.
e.g. a customer purchasing an items from your online storefront: the same service can handle updating the order details, adjusting available inventory and processing the payment.
Microservices
A microservices architecture arranges an application as a collection of loosely coupled services, using a lightweight protocol.
Some of the defining characteristics of microservices:
- Services are usually processes that communicate over a network.
- Services are organized around business capabilities i.e. they provide specialized, domain-specific services to applications (or other services).
- Service are not tied to any one programming language, platform or set of technologies.
- Services are small, decentralized, and independently deployable.
Each microservice is expected to operate independently, and contain all of the logic that it requires to perform its specialized task. Microservices are distributed, so that each can be deployed to a separate system or tier.
The advantage of microservices over services is that we have prioritized decoupling of components and maximized cohesion - each microservice has a specific role and no dependencies. This makes extending and scaling out new microservices trivial. However, the cost of this is performance – communication over the network is relatively slow compared to inter-process communication on the same system.
The driving philosophy of microservices is the notion of bounded context: each service models a domain or workflow. Thus, each service includes everything necessary to operate within the application, including classes, other subcomponents, and database schemas. – Mark Richards
Although the services themselves are independent, they need to be able to call one another to fulfil business requirements. e.g. a customer attempting to checkout online may have thier order sent to a Shipping service to organize the details of the shipment, but then a request would need to be sent from the Shipping service to the Payment service to actually process the payment.
This suggests that communication between microservices is a key requirement. The architect utilizing this architecture would typically define a standard communication protocol e.g. message queues, or REST.
Coordinating a multi-step process like this involves either cooperation between services (as described above), or a third coordinating service.
Coordination in Microservices [Richards 2020].
Orchestration in Microservices [Richards 2020].
Which style to choose?
Design Principles
The term “software design” is heavily overloaded, with many different interpretations of what it entails.
- A UX designer will treat design as the process of working with users to identify requirements, and iterating on the interaction and experience design with them to fine tune how they want the experience to work.
- A software engineer will want to consider ways of designing modules and source code that emphasize desireable characteristics like scalability, reliability and performance.
- A software developer may want to consider readability of the code, and compatibility with existing code bases (among other things).
In this course, we’ll treat design as the complete set of low-level implementation decisions that are made prior to coding a system. We’ll discuss some different approaches to design that have been impactful and useful.
Features of Good Design
It doesn’t take a huge amount of knowledge and skill to get a program working. Kids in high school do it all the time… The code they produce may not be pretty; but it works. It works because getting something to work once just isn’t that hard.
Getting software right is hard. When software is done right, it requires a fraction of the human resources to create and maintain. Changes are simple and rapid. Defects are few and far between. Effort is minimized, and functionality and flexibility are maximized.
– Bob Martin, Clean Architecture (2016).
One recurring theme keep cropping up: the notion that software should be enduring. Software that you produce should be able to function for a long period of time, in a changing environment, where adjustments will need to be made over time; defects will be found and fixed; new features will be introduced and old features phased out.
As “Uncle Bob” points out, It’s relatively easy to get something to compile and work once, in a restricted environment; it’s much more difficult to build something that can be extended and modified over time. If you want software that can be useful for a long time, you need to design for that as well.
Of course, first and foremost, we want to design software that performs its intended function, but we also want robust software that is a joy to extend and maintain. Let’s talk about the characteristics of “good” software that support this approach.
Code Reuse
Software is expensive and time-consuming to produce, so anything that reduces cost or time is welcome. Reusability, or code reuse is often positioned as the easiest way to accomplish this. It also reduces defects, since you’re presumably reusing code that is tested and known-good.
“I see three levels of reuse.
At the lowest level, you reuse classes: class libraries, containers, maybe some class “teams” like container/iterator.
Frameworks are at the highest level. They really try to distill your design decisions. They identify the key abstractions for solving a problem, represent them by classes and define relationships between them. JUnit is a small framework, for example. It is the “Hello, world” of frameworks. It has Test, TestCase, TestSuite and relationships defined.
A framework is typically larger-grained than just a single class. Also, you hook into frameworks by subclassing somewhere. They use the so-called Hollywood principle of “don’t call us, we’ll call you.” The framework lets you define your custom behavior, and it will call you when it’s your turn to do something. Same with JUnit, right? It calls you when it wants to execute a test for you, but the rest happens in the framework.
There also is a middle level. This is where I see patterns. Design patterns are both smaller and more abstract than frameworks. They’re really a description about how a couple of classes can relate to and interact with ”
– Shvets citing Erich Gamma: https://refactoring.guru/gamma-interview.
One of the reasons that we like design patterns is that they’re a different type of reuse: instead of reusing the software directly, we’re reusing designs in a way that results in better code. We’ll discuss these in detail below.
Extensibility
Extensibility implies the ability to modify your code, to expand existing features or add new features. e.g. an image editor adding support for a new image type; a plain text editor adding support for code fences and syntax highlighting. Conditions will change over the lifetime of your software, and you need to design in a way that allows you to respond to changes.
In the sections below, we will discuss different approaches to handling these challenges.
Readability
We’re programmers. Programmers are, in their hearts, architects, and the first thing they want to do when they get to a site is to bulldoze the place flat and build something grand. We’re not excited by incremental renovation: tinkering, improving, planting flower beds.
There’s a subtle reason that programmers always want to throw away the code and start over. The reason is that they think the old code is a mess. And here is the interesting observation: they are probably wrong. The reason that they think the old code is a mess is because of a cardinal, fundamental law of programming:
It’s harder to read code than to write it. —Joel Spolsky, Things You Should Never Do, Part I (2000)
It’s very likely that the software that you write will need to be read by someone else: your teammates, the people that follow you on a project, maybe even hundreds or thousands of other developers if you relase it publically.
For that reason, it’s not enough to have code that works; it should work, and be clear and understandable to other people that will need to read it. Keep in mind that the “other people” may include future-you. Will your code still make sense if you have to come back to it a year from now? Five years from now? Code comments (that describe why you made your design decisions), and consistent code structure go a long way to making code readable.
Design Principles
What is good software design? How would you measure it? What practices would you need to follow to achieve it? How can you make your architecture flexible, stable and easy to understand?
These are the great questions; but, unfortunately, the answers are different depending on the type of application you’re building.
– Shvets, Dive Into Design Patterns (2020).
We do have some universal principles that we can apply to any situation.
Encapsulate What Varies
Info
Identify the aspects of your application that vary and separate them from what stays the same.
The main goal of this principle is to minimize the effect caused by changes.
You can do this by encapsulating classes, or functions. In both cases, your goal is separate and isolate the code that is likely to change from the rest of your code. This minimizes what you need to change over time.
The following example is taken from Shvets (and rewritten in non-idiomatic Kotlin).
fun getOrderTotal(order) {
total = 0
for (item in order.lineItems)
total += item.price * item.quantity
if (order.country == "US")
total += total * 0.07 // US sales tax
else if (order.country == "EU"):
total += total * 0.20 // European VAT
return total
}
Given that the tax rates will likely vary, we should isolate them into a separate function. This way, when the rates change, we have much less code to modify.
fun getOrderTotal(order) {
total = 0
foreach item in order.lineItems
total += item.price * item.quantity
total += total * getTaxRate(order.country)
return total
}
fun getTaxRate(country) {
return when (country) {
"US" -> 0.07 // US sales tax
"EU" -> 0.20 // European VAT
else -> 0
}
}
Similarly, we can split up classes into smaller independent units. Here’s a monolithic class that could be refactored:
Here are the restructured classes.
Program to an Interface, Not an Implementation
Info
Program to an interface, not an implementation. Depend on abstractions, not on concrete classes.
When classes rely on one another, you want to minimize the dependency - we say that you want loose coupling between the classes. This allows for maximum flexibility.
Do do this, you extract an abstract interface, and use that to describe the desired behaviour between the classes.
For example, in the diagram below, our cat on the left can eat sausage, but only sausage. The cat on the right can eat anything that provides nutrition, including sausage. The introduction of the food interface complicates the model, but provides much more flexibility to our classes.
Favor Composition over Inheritance
Inheritance is a useful tool for reusing code. In principle, it sounds great - derive from a base class, and you get all of it’s behaviour for free!
Unfortunately it’s rarely that simply. There are sometimes negative side effects of inheritance.
- A subclass cannot reduce the interface of the base class. You have to implement all abstract methods, even if you don’t need them.
- When overriding methods, you need to make sure that your new behaviour is compatible with the old behaviour. In other words, the derived class needs to act like the base class.
- Inheritance breaks encapsulation, because the details of the parent class are potentially exposed to the derived class.
- Subclasses are tightly coupled to superclasses. A change in the superclass can break subclasses.
- Reusing code through inheritance can lead to parallel inheritance hierarchies, and an explosion of classes. See below for an example.
A useful alternative to inheritance is composition.
Where inheritance represents an is-a relationship (a car is a vehicle), composition represents a has-a relationship (a car has an engine).
Imagine a catalog application for cars and trucks:
Here’s the same application using composition. We can create unique classes representing each vehicle which just implement the appropriate interfaces, without the intervening abstract classes.
Design Patterns
Overview
A design pattern is a generalizable solution to a common problem that we’re attempting to address. Design patterns in software development are one case of a formalized best practice, specifically around how to structure your code to address specific, recurring design problems in software development.
We include design patterns in a discussion of software development because this is where they tend to be applied: they’re more detailed that architecture styles, but more abstract than source code. Often you will find that when you are working through high-level design, and describing the problem to address, you will recognize a problem as similar to something else that you’ve encoutered. A design pattern is a way to formalize that idea of a common, reusable solution, and give you a standard terminology to use when discussing this design with your peers.
Patterns originated with Christopher Alexander, an architect, in 1977. Design patterns in software gained popularity with the book Design Patterns: Elements of Reusable Object-Oriented Software, published in 1994 [Gamma 1994]. There have been many books and articles published since then, and during the early 2000s there was a strong push to expand Design Patterns and promote their use.
Design patterns have seen mixed-success. Some criticisms levelled:
- They are not comprehensive, and do not reflect all styles of software or all problems encountered.
- They are old-fashioned and do not reflect current software practices.
- They add flexibility, at the cost of increased code complexity.
Broad criticisms are likely unfair. While it’s true that not all patterns are used, many of them are commonly used in professonal practice, and new patterns are being suggested. Design patterns certainly can add complexity to code, but they also encourage designs that help avoid subtle bugs later on.
In this section, we’ll outline the more common patterns, and indicate where they may be useful. The original set of patterns were subdivided based on the types of problems they addressed.
We’ll examine a number of patterns below. The original patterns and categories are taken from Eric Gamma et al. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Examples and some explanations are from Alexander Shvets. 2019. Dive Into Design Patterns.
Categories
Creational Patterns
Creational Patterns control the dynamic creation of objects.
| Pattern |
Description |
| Abstract Factory |
Provide an interface for creating families of related or dependent objects without specifying their concrete classes. |
| Builder |
Separate the construction of a complex object from its representation, allowing the same construction process to create various representations. |
| Factory Method |
Provide an interface for creating objects in a superclass, but allows subclasses to alter the type of objects that will be created. |
| Prototype |
Specify the kinds of objects to create using a prototypical instance, and create new objects from the ‘skeleton’ of an existing object, thus boosting performance and keeping memory footprints to a minimum. |
| Singleton |
Ensure a class has only one instance, and provide a global point of access to it. |
Example: Builder Pattern
Builder is a creational design pattern that lets you construct complex objects step by step. The pattern allows you to produce different types and representations of an object using the same construction code.
Imagine that you have a class with a large number of variables that need to be specified when it is created. e.g. a house class, where you might have 15-20 different parameters to take into account, like style, floors, rooms, and so on. How would you model this?
You could create a single class to do this, but you would then need a huge constructor to take into account all of the different parameters.
- You would then need to either provide a long parameter list, or call other methods to help set it up after it was instantiated (in which case you have construction code scattered around).
- You could create subclasses, but then you have a potentially huge number of subclasses, some of which you may not actually use.
The builder pattern suggests that you put the object construction code into separate objects called builders. The pattern organizes construction into a series of steps. After calling the constructor, you call methods to invoke the steps in the correct order (and the object prevents calls until it is constructed). You only call the steps that you require, which are relevant to what you are building.
Even if you never utilize the Builder pattern directly, it’s used in a lot of complex Kotlin and Android libraries. e.g. the Alert dialogs in Android.
val dialog = AlertDialog.Builder(this)
.setTitle("Title")
.setIcon(R.mipmap.ic_launcher)
.show()
Example: Singleton
Singleton is a creational design pattern that lets you ensure that a class has only one instance, while providing a global access point to this instance.
Why is this pattern useful?
- Ensure that a class has just a single instance. The most common reason for this is to control access to some shared resource—for example, a database or a file.
- Provide a global access point to that instance. Just like a global variable, the Singleton pattern lets you access some object from anywhere in the program. However, it also protects that instance from being overwritten by other code.
All implementations of the Singleton have these two steps in common:
-
Make the default constructor private, to prevent other objects from using the new operator with the Singleton class.
-
Create a static creation method that acts as a constructor.
In languages like Java, you would express the implementation in this way:
public class Singleton {
private static Singleton instance = null;
private Singleton() {
}
public static Singleton getInstance() {
if (instance == null) {
synchronized (Singleton.class) {
if (instance == null) {
instance = new Singleton();
}
}
}
return instance;
}
}
In Kotlin, it’s significantly easier.
object Singleton{
init {
println("Singleton class invoked.")
}
fun print(){
println("Print method called")
}
}
fun main(args: Array<String>) {
Singleton.print()
// echos "Print method called" to the screen
}
The object keyword in Kotlin creates an instance of a generic class. i.e. it’s instantiated automatically. Like any other class, you can add properties and methods if you wish.
Singletons are useful for times when you want a single, easily accessible instance of a class. e.g. Database object to access your database, Configuration object to store runtime parameters and so on. You should also consider it instead of extensively using global variables.
Structural Patterns
Structural Patterns are about organizing classes to form new structures.
| Pattern |
Description |
| Adapter, Wrapper |
Convert the interface of a class into another interface clients expect. An adapter lets classes work together that could not otherwise because of incompatible interfaces. |
| Bridge |
Decouple an abstraction from its implementation allowing the two to vary independently. |
| Composite |
Compose objects into tree structures to represent part-whole hierarchies. Composite lets clients treat individual objects and compositions of objects uniformly. |
| Decorator |
Attach additional responsibilities to an object dynamically keeping the same interface. Decorators provide a flexible alternative to subclassing for extending functionality. |
| Proxy |
Provide a surrogate or placeholder for another object to control access to it. |
Example: Adapter
Adapter is a structural design pattern that allows objects with incompatible interfaces to collaborate.
Imagine that you have a data source that is in XML, but you want to use a charting library that only consumes JSON data. You could try and extend one of those libraries to work with a different type of data, but that’s risky and may not even be possible if it’s a third-party library.
An adapter is an intermediate component that converts from one interface to another. In this case, it could handle the complexities of converting data between formats. Here’s a great example from Shvets (2019):
The simplest way to implement this is using object composition: the adapter is a class that exposes an interface to the main application (client). The client makes calls using that interface, and the adapter performs necessary actions through the service (which is often a library, or something whose interface you cannot control).
- The client is the class containing business logic (i.e. an application class that you control).
- The client interface describes the interface that you have designed for your application to communicate with that class.
- The service is some useful library or service (typically which is closed to you), which you want to leverage.
- The adapter is the class that you create to serve as an intermediary between these interfaces.
- The client application isn’t coupled to the adapter because it works through the client interface.
Behavioural Patterns
Behavioural Patterns are about identifying common communication patterns between objects.
| Pattern |
Description |
| Command |
Encapsulate a request as an object, thereby allowing for the parameterization of clients with different requests, and the queuing or logging of requests. It also allows for the support of undoable operations. |
| Iterator |
Provide a way to access the elements of an aggregate object sequentially without exposing its underlying representation. |
| Memento |
Without violating encapsulation, capture and externalize an object’s internal state allowing the object to be restored to this state later. |
| Observer |
Define a one-to-many dependency between objects where a state change in one object results in all its dependents being notified and updated automatically. |
| Strategy |
Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it. |
| Visitor |
Represent an operation to be performed on the elements of an object structure. Visitor lets a new operation be defined without changing the classes of the elements on which it operates. |
Example: Command
Command is a behavioural design pattern that turns a request into a stand-alone object that contains all information about the request (a command could also be thought of as an action to perform).
Imagine that you are writing a user interface, and you want to support a common action like Save. You might invoke Save from the menu, or a toolbar, or a button. Where do you put the code that actually handles saving the data?
If you attach it to the object that the user is interacting with, then you risk duplicating the code. e.g.
The Command pattern suggests that you encapsulate the details of the command that you want executed into a separate request, which is then sent to the business logic layer of the application to process.
The command class relationship to other classes:
Example: Observer (MVC)
Observer is a behavioral design pattern that lets you define a subscription mechanism to notify multiple objects about any events that happen to the object they’re observing. This is also called publish-subscribe.
The object that has some interesting state is often called subject, but since it’s also going to notify other objects about the changes to its state, we’ll call it publisher. All other objects that want to track changes to the publisher’s state are called subscribers, or observers of the state of the publisher.
Subscribers register their interest in the subject, who adds them to an internal subscriber list. When something interest happens, the publisher notifies the subscribers through a provided interface.
The subscribers can then react to the changes.
A modified version of Observer is the Model-View-Controller (MVC) pattern, which puts a third intermediate layer - the Controller - between the Publisher and Subscriber to handle user input.
For more details on MVC, see the Building Clients section.
UML
Architecture and design are all about making important, critical decisions early in the process. It’s extremely valuable to have a standard way of documenting systems, components, and interactions to aid in visualizing and communicating our designs.
The Unified Modelling Language (aka UML) is a modeling language consisting of an integrated set of diagrams, useful for designers and developers to specify, visualize, construct and communicate a design. UML is a notation that resulted from the unification of three competing modelling techniques:
- Object Modeling Technique OMT [James Rumbaugh 1991] - was best for analysis and data-intensive information systems.
- Booch [Grady Booch 1994] - was excellent for design and implementation. Grady Booch had worked extensively with the Ada language, and had been a major player in the development of Object Oriented techniques for the language. Although the Booch method was strong, the notation was less popular.
- OOSE (Object-Oriented Software Engineering [Ivar Jacobson 1992]) - featured a model known as Use Cases.
All three designers joined Rational Software in the mid 90s, with the goal of standardizing this new method. Along with many industry partners, they drafted an intial proposal which was submitted to the Object Management Group in 1997. This led to a series of UML standards driven through this standards body, with UML 2.5 being the current version.
The primary goals of UML include:
-
providing users with a common, expressive language that they can use to share models.
-
provide mechanism to extend the language if needed.
-
remain independent of any particular programming language or development process
-
support higher-level organizational concepts like frameworks, patterns.
UML contains a large number of diagrams, intended to address the needs to a wide range of stakeholders. e.g. analysis, desigers, coders, testers, customers.
UML contains both structure and behaviour diagrams.
Structure diagrams show the structure of the system and its parts at different level of abstraction, and shows how they are related to one another. Behaviour diagrams show the changes in the system over time.
These diagrams are intended to cover the full range of possible scenarios that we want to model. It’s common (and completely reasonable!) to only use the diagrams that you actually need. You will find, for instance, that component and class diagrams are commonly used when discussing component-level behaviour; package and deployment diagrams are used when determining how to install and execute your deliverable and so on.
Below we’ll highlight the most commonly used UML diagrams. For more comprehensive coverage, see Visual Paradigm or Martin Fowler’s UML Distilled [Fowler 2004].
Info
You should NOT create diagrams for every components, interaction or state in your system. That’s overkill for most projects. Instead, focus on building a high-level component diagram that shows the basic component interactions, which you can use to plan your system. Secondly, use diagrams if you have a particular component or sequence that is exceptionally complex, or important to get “right”.
Structure Diagrams
These document the static components in a system.
Class Diagram
The class diagram is a central modeling technique that runs through nearly all object-oriented methods. This diagram describes the types of objects in the system and various kinds of static relationships which exist between them.
There are three principal kinds of relationships which are important to model:
- Association - represent relationships between instances of types (a person works for a company, a company has a number of offices.
- Inheritance - the most obvious addition to ER diagrams for use in OO. It has an immediate correspondence to inheritance in OO design.
- Aggregation - Aggregation, a form of object composition in object-oriented design.
Component Diagram
A component diagram depicts how components are wired together to form larger components or software systems. It illustrates the architectures of the software components and the dependencies between them. Those software components including run-time components, executable components also the source code components.
Deployment Diagram
The Deployment Diagram helps to model the physical aspect of an Object-Oriented software system. It is a structure diagram which shows architecture of the system as deployment (distribution) of software artifacts to deployment targets. Artifacts represent concrete elements in the physical world that are the result of a development process. It models the run-time configuration in a static view and visualizes the distribution of artifacts in an application. In most cases, it involves modeling the hardware configurations together with the software components that lived on.
Behaviour Diagrams
These document behaviours of the system over time.
Use Case Diagram
A use-case model describes a system’s functional requirements in terms of use cases. It is a model of the system’s intended functionality (use cases) and its environment (actors). Use cases enable you to relate what you need from a system to how the system delivers on those needs. The use-case model is generally used in all phases of the development cycle by all team members and is extremely popular for that reason.
Info
Keep in mind that UML was created before UX. Use Cases are very much like User Stories, but with more detail. Think of Use Cases == User Stories, and Actors == Personas (except that they can also model non-human actors like other systems).
Activity Diagram
Activity diagrams are graphical representations of workflows of stepwise activities and actions with support for choice, iteration and concurrency. It describes the flow of control of the target system. Activity diagrams are intended to model both computational and organizational processes (i.e. workflows).
Info
I use these a lot when designing interactive systems, and modelling transitions between screens or areas of functionality.
Interaction Overview Diagram
The Interaction Overview Diagram focuses on the overview of the flow of control of the interactions. It is a variant of the Activity Diagram where the nodes are the interactions or interaction occurrences. The Interaction Overview Diagram describes the interactions where messages and lifelines are hidden. You can link up the “real” diagrams and achieve high degree navigability between diagrams inside the Interaction Overview Diagram.
Info
I don’t know that I’ve ever seen these used, but I’m specifically calling them out b/c you might be tempted to use them for interactive applications. I would probably choose a more straightforward activity diagram, and invest my remaining design time in prototyping and iterating with users.
State Machine Diagram
A state diagram is a type of diagram used in UML to describe the behavior of systems which is based on the concept of state diagrams by David Harel. State diagrams depict the permitted states and transitions as well as the events that effect these transitions. It helps to visualize the entire lifecycle of objects and thus help to provide a better understanding of state-based systems.
Sequence Diagram
The Sequence Diagram models the collaboration of objects based on a time sequence. It shows how the objects interact with others in a particular scenario of a use case.
Info
When do you use a Sequence Diagram over a more simple Activity Diagram? When you have a more complex interaction between components. I’ve used these to model client-server authentication for instance, or passing data between systems (where it might need to be encrypted/decrypted at each side).
Subsections of Building Clients
Console Applications
Overview
A console application (aka “command-line application”) is an application that is intended to run from within a shell, such as bash, zsh, or PowerShell, and uses text and block characters for input and output. This style of application dominated through the early days of computer science, until the 1980s when graphical user interfaces became much more common.
Console applications use text for both input and output: this can be as simple as plan text displayed in a window (e.g. bash), to systems that use text-based graphics and color to enhance their usability (e.g. Midnight Commander). This application style was really driven by the technical constraints of the time. Software was written for text-based terminals, often with very limited resources, and working over slow network connections. Text was faster to process, transmit and display than sophisticated graphics.
Some console applications remain popular, often due to powerful capabilities or features that are difficult to replicate in a graphical environment. Vim, for instance, is a text editor that originated as a console application, and is still used by many developers. Despite various attempts to build a “graphical vim”, the console version is often seen as more desireable due to the benefits of console applications i.e. faster, reduced latency, small memory footprint and ability to easily leverage other command-line tools.
Console applications often favor ‘batch-style processing‘, where you provide arguments to the program, and it executes a single task before exiting. This is very much due to the Unix Philosophy from the 70s, where this interaction style dominated:
-
Make each program do one thing well. To do a new job, build fresh rather than complicate old programs by adding new “features”.
-
Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.
-
Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.
-
Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them."
– Bell Systems Technical Journal (1978)
Although we tend to run graphical operating systems with graphical applications, console applications are still very common. e.g. Windows, macOS, Linux all ship with consoles and powerful tools that are common used, at ledast by a subset of users. e.g. ls, curl, wget, emacs, vim, git and so on.
For expert users in particular, this style has some advantages. Console applications:
- can easily be scripted or automated to run without user intervention.
- can redirect input and output using standard IO streams, to allow interaction with other console applications that support this standard.
- tend to be small and performant, due to their relatively low-overhead (i.e. no graphics, sound, other application overhead).
The disadvantage is the steep learning curve, and lack of feature discoverability.
Design
Architecture
Command-line applications tend towards performing a single action and then exiting. This lends itself to a pipeline architecture, where processing steps are performed on intput, in order, to produce an output. This is also called a pipes and filters design pattern.
Examples of how this pattern might be applied:
- A file rename utility that changes the date on one or more files that are supplied as arguments. The program would (1) process and validate the command-line arguments, (2) apply the rule to produce a set of target output files (likely original:revise filename pairs) and then (3) perform these operations. Any interruption or error would cause the rename to abort.
- An image processing utility that attempts to filter a photo. Filters are applied in sequence on the input image producing a final output image.
Of course, not every command-line program works like this; there are some that are interactive, where the user provides successive input while the program is running. A program like this could use a mode traditional event-driven architecture:
Examples of this might include a text editor like Vi. It can certainly process command-line arguments, but it primarily operates in an interactive mode, where it waits for, and acts upon, user input (where keystrokes are represented as events).
Lifecycle
Command-line applications expect a single-entry point: a method with this familiar signature:
fun main(args:Array<String>) {
// code here
System.exit(0)
}
When your program is launched it executes this method. When it reaches the end of the method, the program stops executing. It’s considered “good form” to call System.exit(0) where the 0 is an error code returned to the OS. In this case, 0 means no errors i.e. a normal execution.
Input should be handled through either (a) arguments supplied on the command-line, or (b) values supplied through stdin. Batch-style applications should be able to run to completion without prompting the user for more input.
Output should be directed to stdout. You’re typically limited to textual output, or limited graphics (typically through the use of Unicode characters).
Errors should typically be directed to stderr.
As suggested above, we want to use a standard calling convention. Typically, command-line applications should use this format, or something similar:
$ program_name -option=value parameter
- program_name is the name of your program. It should be meaningful, and reflect what your program does.
- options represent a value that tells your program how to operate on the data. Typically options are prefixed with a dash (”-”) to distinguish them from parameters. If an option also requires a value (e.g. ”-bg=red”) then separate the option and value with a colon (”:”) or an equals sign (”=”).
- parameter represents input, or data that your program would act upon (e.g. the name of a file containing data). If multiple parameters are required, separate them with whitespace (e.g. ”program_name parameter1 parameter2”).
The order of options and parameters should not matter.
Running a program with insufficient arguments should display information on how to successfully execute it.
$ rename
Usage: rename [source] [dest]
Features
Command-line applications should have the following standard features:
- Use IO Streams: your application should handle all input through
stdin, channel output to stdout and errors to stderr. This ensures that it will work as-expected with other command-line programs.
- Support conventions: The “target” (e.g. filename on which to operate) is usually provided as the primary argument. To disambiguate other input, it is normal to provide additional information using dashes using . For example, it is standard to use
--help to display brief help that demonstrates how to use your application.
- Provide user feedback for errors: your program should never print out “successful” messages. Save user feedback for errors. Error messages should be clear and help the user figure out what went wrong, or what they need to do to fix the error.
Typical command-line interaction is shown below:
% exa --help
Usage:
exa [options] [files...]
META OPTIONS
-?, --help show list of command-line options
-v, --version show version of exa
DISPLAY OPTIONS
-1, --oneline display one entry per line
-l, --long display extended file metadata as a table
-G, --grid display entries as a grid (default)
-x, --across sort the grid across, rather than downwards
-R, --recurse recurse into directories
-T, --tree recurse into directories as a tree
-F, --classify display type indicator by file names
...
% exa -T
.
├── 01.syllabus.md
├── 02.introduction.md
├── 03.software_process.md
├── 04.sdlc.md
├── 05.architecture.md
├── 06.development.md
├── 07.testing.md
├── 08.kotlin_primer.md
├── 09.building_desktop.md
├── 10.building_mobile.md
├── 11.building_libraries.md
├── 12.building_services.md
├── 13.multiplatform.md
├── 99.unused.md
├── assets
│ ├── 2_tier_architecture.png
│ ├── 3_tier_architecture.png
│ ├── abstract_factory.png
│ ├── activity_lifecycle.png
...
Processing arguments
The main() method can optionally accept an array of Strings containing the command-line arguments for your program. To process them, you can simply iterate over the array.
This is effectively the same approach that you would take in C/C++.
fun main(args: Array<String>) {
// args.size returns number of items
for (s in args) {
println("Argument: $s")
}
}
Info
This is a great place to use the command design pattern to abstract commands. See the public repo for samples.
Reading/writing text
The Kotlin Standard Library (“kotlin-stdlib“) includes the standard IO functions for interacting with the console.
- readLine() reads a single value from “stdin“
- println() directs output to “stdout“
code/ucase.kts
// read single value from stdin
val str:String ?= readLine()
if (str != null) {
println(str.toUpperCase())
}
It also includes basic methods for reading from existing files.
import java.io.*
var filename = "transactions.txt"
// read up to 2GB as a string (incl. CRLF)
val contents = File(filename).readText(Charsets.UTF_8)
println(contents)
2020-06-06T14:35:44, 1001, 78.22, CDN
2020-06-06T14:38:18, 1002, 12.10, CDN
2020-06-06T14:42:51, 1003, 44.50, CDN
Example: rename utility
Let’s combine these ideas into a larger example.
Requirements: Write an interactive application that renames one or more files using options that the user provides. Options should support the following operations: add a prefix, add a suffix, or capitalize it.
We need to write a script that does the following:
- Extracts options and target filenames from the arguments.
- Checks that we have (a) valid arguments and (b) enough arguments to execute program properly (i.e. at least one filename and one rename option).
- For each file in the list, use the options to determine the new filename, and then rename the file.
Usage: rename.kts [option list] [filename]
For this example, we need to manipulate a file on the local file system. The Kotlin standard library offers a File class that supports this. (e.g. changing permissions, renaming the file).
Construct a ‘File‘ object by providing a filename, and it returns a reference to that file on disk.
fun main(args: Array<String>) {
val files = getFiles(args)
val options = getOptions(args)
// check minimum required arguments
if (files.size == 0 || options.size == 0) {
println("Usage: [options] [file]")
} else {
applyOptions(files, options)
}
fun getOptions(args:Array<String>): HashMap<String, String> {
var options = HashMap<String, String>()
for (arg in args) {
if (arg.contains(":")) {
val (key, value) = arg.split(":")
options.put(key, value)
}
}
return options
}
fun getFiles(args:Array<String>) : List<String> {
var files:MutableList<String> = mutableListOf()
for (arg in args) {
if (!arg.contains(":")) {
files.add(arg)
}
}
return files
}
fun applyOptions(files:List<String>,options:HashMap<String, String>) {
for (file in files) {
var rFile = file
// capitalize before adding prefix or suffix
if (options.contains("prefix")) rFile = options.get("prefix") + rFile
if (options.contains("suffix")) rFile = rFile + options.get("suffix")
File(file).renameTo(rFile)
println(file + " renamed to " + rFile)
}
}
Reading/writing binary data
These examples have all talked about reading/writing text data. What if I want to process binary data? Many binary data formats (e.g. JPEG) are defined by a standard, and will have library support for reading and writing them directly.
Kotlin also includes object-streams that support reading and writing binary data, including entire objects. You can, for instance, save an object and it’s state (serializing it) and then later load and restore it into memory (deserializing it).
class Emp(var name: String, var id:Int) : Serializable {}
var file = FileOutputStream("datafile")
var stream = ObjectOutputStream(file)
var ann = Emp(1001, "Anne Hathaway", "New York")
stream.writeObject(ann)
Cursor positioning
ANSI escape codes are specific codes that can be “printed” by the shell but will be interpreted as commands rather than output. Note that these are not standard across consoles, so you are encouraged to test with a specific console that you wish to support.
| ^ |
C0 |
Abbr |
Name |
Effect |
| ^G |
7 |
BEL |
Bell |
Makes an audible noise. |
| ^H |
8 |
BS |
Backspace |
Moves the cursor left (but may “backwards wrap” if cursor is at start of line). |
| ^I |
9 |
HT |
Tab |
Moves the cursor right to next multiple of 8. |
| ^J |
0xA |
LF |
Line Feed |
Moves to next line, scrolls the display up if at bottom of the screen. Usually does not move horizontally, though programs should not rely on this. |
| ^L |
0xC |
FF |
Form Feed |
Move a printer to top of next page. Usually does not move horizontally, though programs should not rely on this. Effect on video terminals varies. |
| ^M |
0xD |
CR |
Carriage Return |
Moves the cursor to column zero. |
| ^[ |
0x1B |
ESC |
Escape |
Starts all the escape sequences |
These escape codes can be used to move the cursor around the console. All of these start with ESC plus a suffix.
| Cursor |
ESC |
Code |
Effect |
| Left |
\u001B |
[1D |
Move the cursor one position left |
| Right |
\u001B |
[1C |
Move the cursor one position right |
| Up |
\u001B |
[1A |
Move the cursor one row up |
| Down |
\u001B |
[1B |
Move the cursor one row down |
| Startline |
\u001B |
[250D |
Move the cursor to the start of the line |
| Home |
\u001B |
[H |
Move the cursor to the home position |
| Clear |
\u001B |
[2J |
Clear the screen |
| Reset |
\u001B |
|
Reset to default settings |
For instance, this progress bar example draws a line and updates the starting percentage in-place.
val ESC = "\u001B";
val STARTLINE = ESC + "[250D";
for (i in 1..100) {
print(STARTLINE);
print(i + "% " + BLOCK.repeat(i));
}
// output
100% ▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋▋
Printing in colour
We can also use escape sequences to change the color that is displayed.
| Cursor |
ESC |
Code |
Effect |
| Colour: Black |
\u001B |
[30m |
Set colour Black for anything printed after this |
| Colour: Yellow |
\u001B |
[33m |
Set colour Yellow for anything printed after this |
| Colour: White |
\u001B |
[37m |
Set colour White for anything printed after this |
Below, we use ANSI colour codes to display colored output:
val ANSI_BLACK = "\u001B[30m"
val ANSI_YELLOW = "\u001B[33m"
val ANSI_WHITE = "\u001B[37m"
println(ANSI_WHITE + " ;)(; ");
println(ANSI_YELLOW + " :----: " + ANSI_BLACK + " Vendor: " + System.getProperty("java.vendor"));
println(ANSI_YELLOW + "C|" + ANSI_WHITE + "====" + ANSI_YELLOW + "| " + ANSI_BLACK + " JDK Name: " + System.getProperty("java.vm.name"));
println(ANSI_YELLOW + " | | " + ANSI_BLACK + " Version: " + System.getProperty("java.version"));
println(ANSI_YELLOW + " `----' ");
println();
Applications
Packaging just refers to putting together your software components in a way that you can distribute to your end-user. As we’ve seen, this could be physical (a retail box including a DVD), or a digital image (ISO image, package file, installer executable). It’s the “bundling” part of building software.
Packaging is also one of those tasks that we tend to hand wave: just compile, link and hand-out the executable right? Unfortunately it’s not that simple. Apart from the executable, your application may also include images, sound clips and other resources, plus external libraries that you don’t necessarily want to statically link into your executable. You need to distribute all of these files, and ensure that they are installed in the correct location, registered with the OS and made usable by your application. You also need to perform steps to ensure that your application respects conventions of the particular environment: installing under Program Files in Windows, putting libraries under Windows\\System32, putting an icon on the desktop and so on.
The way we address the compexities of packaging is to break down the process into multiple steps, which are handled by different tools:
| Step |
Explanation |
| Step 1: Compiling classes |
Use the kotlinc compiler to create classes from our source code. |
| Step 2: Creating archives |
Use the jar command to create jar files of classes and related libraries. |
| Step 3: Creating scripts |
Optionally, create scripts to allow a user to execute directly from the JAR files. |
In the next sections, we’ll demonstrate using each of these tools.
1. Compiling classes
Let’s use the HelloWorld sample to demonstrate different methods for preparing, and then packaging our application.
fun main(args: Array<String>) {
println(“Hello Kotlin!”)
}
To compile from the command-line, we can use the Kotlin compiler, kotlinc. By default, it takes Kotlin source files (.kt) and compiles them into corresponding class files (.class) that can be executed on the JVM.
$ kotlinc Hello.kt
$ ls
Hello.kt HelloKt.class
$ kotlin HelloKt
Hello Kotlin!
We could also just do this with IntelliJ IDEA or Android Studio, where Gradle - build - build will generate class files. As we will see at the end of this section, we can often just generate the final installer in a single step without doing each step manually.
2. Creating archives
Class files by themselves are difficult to distribute. The Java platform includes the jar utility, used to create a single archive file from your classes.
A JAR file is a standard mechanism in the Java ecosystem for distributing applications. It’s effectively just a compressed file (just like a ZIP file) which has a specific structure and contents. Most distribution mechanisms expect us to create a JAR file first.
This example compiles Hello.kt, and packages the output in a Hello.jar file.
$ kotlinc Hello.kt -include-runtime -d Hello.jar
$ ls
Hello.jar Hello.kt
The -d option tells the compiler to package all of the required classes into our jar file. The -include-runtime flag tells it to also include the Kotlin runtime classes. These classes are needed for all Kotlin applications, and they’re small, so you should always include them in your distribution (if you fail to include them, you app won’t run unless your user has Kotlin installed).
To run from a jar file, use the java command.
$ java -jar Hello.jar
Hello Kotlin!
Our JAR file from above looks like this if you uncompress it:
$ unzip Hello.jar -d contents
Archive: Hello.jar
inflating: contents/META-INF/MANIFEST.MF
inflating: contents/HelloKt.class
inflating: contents/META-INF/main.kotlin_module
inflating: contents/kotlin/collections/ArraysUtilJVM.class
...
$ tree -L 2 contents/
.
├── META-INF
│ ├── MANIFEST.MF
│ ├── main.kotlin_module
│ └── versions
└── kotlin
├── ArrayIntrinsicsKt.class
├── BuilderInference.class
├── DeepRecursiveFunction.class
├── DeepRecursiveKt.class
├── DeepRecursiveScope.class
...
The JAR file contains these main features:
HelloKt.class – a class wrapper generated by the compilerMETA-INF/MANIFEST.MF – a file containing metadata.kotlin/ – Kotlin runtime classes not included in the JDK.
The MANIFEST.MF file is autogenerated by the compiler, and included in the JAR file. It tells the runtime which main method to execute. e.g. HelloKt.main().
$ cat contents/META-INF/MANIFEST.MF
Manifest-Version: 1.0
Created-By: JetBrains Kotlin
Main-Class: HelloKt
In IntelliJIDEA, Gradle - build - jar will create a jar file. As we will see at the end of this section, we can often just generate the final installer in a single step without doing each step manually.
3. Creating scripts
We can distribute JAR files like this to our users, but they’re awkward: users would need to have the Java JDK installed, and typejava -jar filename.jar to actually run our programs. For some scenarios this might be acceptable (e.g. launching a service from an init script), but most of the time we want a more user-friendly installation mechanism.
The simplest thing we can do it create a script to launch our application from a JAR file, with the same effect as executing from the command-line:
$ cat hello
#!/bin/bash
java -jar hello.jar
$ chmod +x hello
$ ./hello
Hello Kotlin!
For simple applications, especially ones that we use ourselves, this may be sufficient. It has the downside of requiring the user to have the Java JDK installed on their system. This particular script is also not very robust: it doesn’t set runtime parameters, and doesn’t handle more than a single JAR file.
For a more robust script, we can let Gradle generate one for us. In IntelliJIDEA, Gradle - distribution - distZip will create a zip file that includes a custom runtime script. Unzip it, and it will contain a lib directory with all of the JAR files, and a bin directory with a script that will run the application from the lib directory. Copy both to a new location and you can execute the app from there.
Here’s an example of the Archey project, with a distZip file that was generated by Gradle.
$ tree build/distributions -L 3
build/distributions
├── app
│ ├── bin
│ │ ├── app
│ │ └── app.bat
│ └── lib
│ ├── annotations-13.0.jar
│ ├── app.jar
│ ├── checker-qual-3.8.0.jar
│ ├── error_prone_annotations-2.5.1.jar
│ ├── failureaccess-1.0.1.jar
│ ├── guava-30.1.1-jre.jar
│ ├── j2objc-annotations-1.3.jar
│ ├── jsr305-3.0.2.jar
│ ├── kotlin-stdlib-1.5.31.jar
│ ├── kotlin-stdlib-common-1.5.31.jar
│ ├── kotlin-stdlib-jdk7-1.5.31.jar
│ ├── kotlin-stdlib-jdk8-1.5.31.jar
│ └── listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
└── app.zip
The bin/app script is quite robust, and will handle many different types of system configurations. It will also handle setting the CLASSPATH for all of the libraries that are included in the lib directory!
Here’s a portion of the app script. You don’t want to have to write this by-hand!
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "$( uname )" in #(
CYGWIN* ) cygwin=true ;; #(
Darwin* ) darwin=true ;; #(
MSYS* | MINGW* ) msys=true ;; #(
NONSTOP* ) nonstop=true ;;
esac
CLASSPATH=$APP_HOME/lib/app.jar:$APP_HOME/lib/kotlin-stdlib-jdk8-1.5.31.jar:$APP_HOME/lib/guava-30.1.1-jre.jar:$APP_HOME/lib/kotlin-stdlib-jdk7-1.5.31.jar:$APP_HOME/lib/kotlin-stdlib-1.5.31.jar:$APP_HOME/lib/kotlin-stdlib-common-1.5.31.jar:$APP_HOME/lib/failureaccess-1.0.1.jar:$APP_HOME/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:$APP_HOME/lib/jsr305-3.0.2.jar:$APP_HOME/lib/checker-qual-3.8.0.jar:$APP_HOME/lib/error_prone_annotations-2.5.1.jar:$APP_HOME/lib/j2objc-annotations-1.3.jar:$APP_HOME/lib/annotations-13.0.jar
# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD=$JAVA_HOME/jre/sh/java
else
JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
The script works well. Here’s archey being executed from the script:
$ build/distributions/app/bin/app
#### User: jaffe
### Home: /Users/jaffe
####### ####### Name: Bishop.local
###################### OS: Mac OS X
##################### Version: 11.3
#################### CPU: x86_64
#################### Cores: 10
##################### Free Memory: 514 MB
###################### Total Memory: 520 MB
#################### Disk Size: 582 GB
################ Free Space: 582 GB
#### ##### IP Address: 127.0.0.1
In some ways, a terminal-based program is really just an extension of a line printer. It’s designed to output to stdout one line at a time, and doesn’t handle editing in-place very well (which would be required if you wanted to create a terminal-based UI, like Midnight Commander).
There are specific libraries that have been developed to provide more sophisticated capabilities. Here’s a couple of tile-based toolkits that allow you to build up a full UI in the console:
Here’s an example of a GUI written with Zircon:
Thse toolkits really serve a rare circumstance, where you want graphical capabilities but you’re running in a non-graphical environment. Typically if you’re running a modern OS, you have more sophisticated graphical capabilities available.
Here’s a console toolkit that provides further capabilities.
Desktop Applications
Overview
Graphical applications arose in the early 80s as we moved from text-based terminals to more technically capable systems. This was part of the Personal Computer (PC) movement of that time, which aimed to put a “computer in every home” Graphical User Interfaces (GUIs) were seen as more “user-friendly” and considered an important factor in the adoption of these systems. Introduced in 1984, the Apple Macintosh introduced the first successful commercial graphical operating system; other vendors (e.g. Microsoft, Commodore, Sun) quickly followed suit. The conventions that were adopted on the Mac became standard on other platforms as well e.g. Windows.
Desktop applications refers to graphical applications designed for a notebook or desktop computer, typically running Windows, macOS or Linux. Users interact with these applications using a mouse and keyboard, although other devices may also be supported (e.g. camera, trackpad).
Although desktop computers can run console applications in a shell, for this discussion, we’re focusing on graphical applications.
Features
Graphical desktop application should have the following features:
- Multiple application windows should be supported. Most applications will often present their interface within a single, interactive window, but it can sometimes be useful to have multiple simultaneous windows controlled by a single application.
- Support for full-screen or windowed interaction: although graphical applications tend to run windowed, they should normally be usable full-screen as well. The window contents should scale or reposition themselves as the window size changes.
- Window decorations: Each window should have a titlebar, minimize/maximize/restore buttons (that work as expected).
- Windows may or may not be resizable: if they are resizable, the contents should scale or adjust their layout based on window size (for this reason, it may make sense to either contrain window dimensions when resizing, or make some windows fixed size). Convention allows the main window to be resized, and option dialogs (or similar non-essential windows) to be fixed-size.
- Interactive graphical elements: window contents could be any combination of graphics, animations, multimedia, or text that is desired for the target application. These contents should be dynamic (i.e. have the ability to change in response to system state) and should support a range of interactions - clicking, double-clicking, dragging - provided by both mouse and keyboard.
- Standard menubars: every application should have the following menus (with shortcuts). Although some applications choose to eliminate menus (or replace with other controls), most of the time you should include them. Exact contents may vary, but users expect at-least this functionality:
- File: New, Open, Close, Print, Quit.
- Edit: Cut, Copy, Paste.
- Window: Minimize, Maximize.
- Help: About.
- Keyboard shortcuts: you should strive to have keyboard shortcuts for common functionality. All standard shortcuts should be supported. e.g.
- Ctrl-N for File-New, Ctrl-O for File-Open, Ctrl-Q for Quit.
- Ctrl-X for Cut, Ctrl-C for Copy, Ctrl-V for Paste.
- F1 for Help.
Benefits
There are obvious benefits to a graphical display being able to display rich colours, graphics and multimedia. However, this application style also encourages encourages a certain style of interaction, where users point-and-click to elements on-screen to interact with them.
There are numerous benefits to this style of user interface:
- The interface provides affordances: visual suggestions on how you might interact with the system. This can include hints like tooltips, or a graphical design that makes use of controlsobvious ((e.g. handles to show where to “grab” a window corner).
- Systems provide continuous feedback to users. This includes obvious feedback (e.g. dialog boxes, status lines) and more subtle, continuous feedback (e.g. widgets animating when pressed).
- Interfaces are explorable: users can use menus, and cues to discover new features.
- Low cost of errors: undo-redo, and the ability to rollback to previous state makes exploration low-risk.
- These environments encouraged developers to use consistent widgets and interactive elements. Standardization led to a common look-and-feel, and placement of common controls - which made software easier to learn, especially for novices. Many of the standard features that we take for granted are a direct result of this design standardization in the 80s. [ed. notice that Windows, macOS, Linux all share a very common interaction paradigm, and a common look-and-feel! You can move between operating systems and be quite comfortable because of this.]
Functionality
These are complex requirements, that outside of the scope of a programming language (in-part because they’re going to be intrinsically tied to the underlying operating system, so they’re difficult to standardize).
A widget or GUI toolkit is a UI framework which provides this functionality. This includes support for:
- Creating and managing application windows, with standard window functionality e.g. overlapping windows, depth, min/max buttons, resizing.
- Reusable components called widgets that can be combined in a window to build typical applications. e.g. buttons, lists, toolbars, images, text views.
- Dynamic layout that adapts the interface to change in window size or dimensions.
- Support for an event-driven architecture i.e. suport for standard and custom events. Includes event generation and propogation.
Implementation details will vary based on the toolkit that you’re using. We’ll discuss requirements first, and then in the next section we’ll provide implementation details for some common widgets toolkits.
Window Management
In the context of a applications, a window is simply a region of the screen that “belongs” to a specific application. Typically one application has one main window, an optionally other windows that may also be displayed. These are overlayed on a “desktop”, which is really just the screen background.
To manage many different windows, across many different applications, a part of the operating system called a windowing system is responsible for creating, destroying and managing running windows. The windowing system provides an API to applications to support for all window-related functionality, including:
- provide an mechanism for applications to create, or destroy their own windows
- handle window movement automatically and invisibly to the application (i.e. when you drag a window, the windowing system moves it).
- handles overlapping windows across applicaitons (e.g. so that your application window can be brought to the ““front” and overlap another application’s window).
A windowing system or windowing technology is typically included as part of the operating system, though it’s possible in some systems to replace windowing systems (e.g. Linux).
Coordinate systems
A computer screen uses a Cartesean coordinate system to track window position. By convention, the top-left is the origin, with x increasing as you move right, and y increasing as you move down the screen. The bottom-right corner of the screen is maximum x and y, which equals the resolution of the screen.
Note that its possible for screen contents to move moved out-of-bounds and made inaccessible. We typically don’t want to do this.
In the example below, you can see that this is a 1600x1200 resolution screen, with the four corner positions marked. It contains a single 400x400 window, positioned at (500, 475) using these screen, or global coordinates.
Given that the windowing system manages movement and positioning of windows on-screen, an application window doesn’t actually know where it’s located on-screen! The application that “owns” the window above doesn’t have access to it’s global coordinates. It does however, have access to it’s own internal, or local coordinates. For example, our window might contain other objects, and the application would know about their placement. In this local coordinate system, we use the top of the window as the origin, with the bottom-right coordinate being the (width, height) of the window. Objects within the window are referenced relative to the window’s origin.
Window creation
Typically, the toolkit will provide a mechanism to create a top-level application window, typically as a top-level class that can instantated. That class will have properties to control its behaviour (some of which is used by the Windowing system to setup the window correctly).
- Sample properties: minWidth, width, maxWidth; minHeight, height, maxHeight; title; isFullScreen; isResizable
- Sample methods: close(), toFront(), toBack()
Window Movement
As application developers, we do not need to do anything to support window movement, since it’s provided by the windowing system. Any non-fullscreen windows created by a toolkit are automatically moveable.
We’re going to refer to graphical on-screen elements as widgets. Most toolkits support a large number of similar widgets. The diagram below shows one desktop toolkit with drop-down lists, radio buttons, lists and so on. All of these elements are considered widgets.
Typically, using widgets us as simple as instantiating them, adding them to the window, and setting up a mechanism to detect when users interact with them so that appropriate actions can be taken.
Scene graph
It’s standard practice in graphical applications to represent the interface as a scene graph. This is a mechanism for modeling a graphical application as a tree of nodes (widgets), where each node has exactly one parent. Effects applied to the parent are applied to all of its children.
Toolkits support scene graphs directly. There is typically a distinction made between Container widgets and Node widgets. Containers are widgets that are meant to hold other widgets e.g. menus which hold menu_items, toolbars which can hold toolbar_buttons and so on. Nodes are widgets that are interactive and don’t have any children.
Building a UI involves explicitly setting up the scene graph, by instantiating nodes, and adding them to containers to build a scene graph. (For this reason, containers will always have a list of children, and a mechanism for adding and removing children from their list).
Layout
Layout is the mechanism by which nodes in the scene graph are positioned on the screen, and managed if the window is resized.
- Fixed layout is a mechanism to place widgets in a static layout where the window will remain the same size. This is done by setting properties of the widgets to designate each one’s position, and ensuring that the containers do not attempt any dynamic layout.
- Relative layout delegates responsibility for widget position and size to their parents (i.e. containers). Typically this means setting up the scene graph in such a way that the appropriate container is used based on how it adjusts position. Typical containers include a vertical-box that aligns it’s children in a vertical line, or a grid that places children in a grid with fixed rows and columns.
Design
Events
Applications often handle multiple types of processing: asynchronous, such as when a user types a few keystrokes, or synchronous, such as when we want a computation to run non-stop to completion.
User interfaces are designed around the idea of using events or messages as a mechanism for components to indicate state changes to other interested entities. This works well, due to the asynchronous nature of user-driven interaction, where there can be relatively long delays between inputs (i.e. humans type slowly compared to the rate at which a computer can process the keystrokes).
This type of system, designed around the production, transmission and consumption of events between loosely-coupled components, is called an Event-Driven Architecture. It’s the foundation to most user-interface centric applications (desktop, mobile), which common use messages to signal a user’s interaction with a viewable component in the interface.
What’s an event? An event is any significant occurrence or change in state for system hardware or software.
The source of an event can be from internal or external inputs. Events can generate from a user, like a mouse click or keystroke, an external source, such as a sensor output, or come from the system, like loading a program.
How does event-driven architecture work? Event-driven architecture is made up of event producers and event consumers. An event producer detects or senses the conditions that indicate thaat something has happened, and creates an event.
The event is transmitted from the event producer to the event consumers through event channels, where an event processing platform processes the event asynchronously. Event consumers need to be informed when an event has occurred, and can choose to act on it.
Events be generated from user actions, like a mouse click or keystroke, an external source, such as a sensor output, or come from the system, like loading a program.
An event driven system typically runs an event loop, that keeps waiting for these events. The process is illustrated in the diagram below:
- An EventEmitter generates an event.
- The event is placed in an event queue.
- An event loop peels off events from the queue and dispatches them to event handlers (functions which have been registered to receive this specific type of event).
- The event handlers receive and process the event.
To handle event driven architectures, we often subdivide application responsibility into separate components.
MVC Patterns
Model-View-Controller
The most basic structure is Model-View-Controller (MVC), which leverages the Observer design pattern to separate business logic from the user interface.
MVC divides any application into three distinct parts:
- Model: the core component of the application that handles state (“business logic layer”).
- View: a representation of the application state, often as a user-interface (“presentation layer”)
- Controller: a component that accepts input, interprets user actions and converts to commands for the model or view.
Similar to the observer pattern, the views monitor the system state, represented by the model. When the state changes, the views are notified and they update their data to reflect these changes. Notifications frequently happen through events generated by, and managed by, the toolkit that you’re using.
Often this is realized as separate classes for each of these components, with an additional main class to bind everything together.
// main class
class Main {
val model = Model()
val controller = Controller(model)
val view = View(controller, model)
model.addView(model)
}
We use an interface to represent the views, which provides the flexibility to allow many different types of output for the program. Any class can be a view as long as it supports the appropriate method to allow notifications from the model.
interface IView {
fun update()
}
class View(val controller: Controller, val model: Model): IView {
override fun update() {
// fetch data from model
}
}
The model maintains a list of all views, and notifies them with state changes (indicating that they may wish to refresh their data, or respond to the state change in some way).
class Model {
val views = listOf()
fun addView(view: IView) {
views.add(view)
}
fun update() {
for (view : views) {
view.update()
}
}
}
The controller just passes input from the user to the model.
class Controller(val model: Model) {
fun handle(event: Event) {
// pass event data to model
}
}
One issue with this version of MVC is that the controller often serves little purpose, except to pass along events that are captured by the View (the View contains the user-interface and widgets, and generates events as the user interacts with it).
MVC remains common for simple applications, but tends to be implemented as just a model and one or more views, with the controller code included in the view itself.
Model-View-Presenter
Model-View-Presenter (MVP) keeps the key concept in MVC - separating the business logic from the presentation - and introduces an intermediate Presenter which handles converting the model’s data into a useful format for the views. This is typically done explicitly by the Presenter class. MVP arose from Taligent in the 1990s, but was popularized by Martin Fowler around 2006.
There have been multiple variants of MVP. We’ll focus on MVVM, probably the most popular variant.
MVVM
Model-View-ViewModel was invented by Ken Cooper and Ted Peters to simplify event-driven programming of user interfaces in C#/.NET. It’s similar to MVP, but includes the notion of binding variables to widgets within the framework, so that changes in widget state are are automatically propogated from the view to other components.
MVVM includes the following components:
- Model: as MVC, the core component that handles state. It can also map to a data access layer or database directly.
- View: a representation of the application state, presented to the user.
- ViewModel: a model that specifically interprets the underlying Model state for the particular view to which it is associated. Typically we rely on binding to map variables in the ViewModel directly to widgets in the View, so that updating one directly updated the other.
MVVM is much more common in modern languages and toolkits and has the advantage of replacing all “mapping” code with direct binding of variables and widgets by the toolkit. This greatly simplifies interface development.
We’re using Kotlin with the Java Virtual Machine (JVM) ecosystem, so we’ll discuss some toolkits that are available in that ecosystem.
Java launched in 1996, with AWT as its first GUI framework. AWT is a heavyweight toolkit that provided a thin abstraction layer over the system-specific widgets provided by OS vendors i.e. it provided wrappers for UI components that were built into the OS. However, this tight integration to the OS meant that AWT behaved very differently across different operating systems, which ran counter to Sun’s original goals of having a single cohesive toolkit that ran equally well on all platforms.
Swing was originally part of the Java Foundation Classes, and replaced the AWT in 1997. Unlike AWT, Swing is a lightweight toolkit: Swing components draw themselves using the Java2D Graphics Library, which makes Swing applications consistent across platforms. This also means that Swing can support a broader range of components, including some that aren’t directly supported by the OS. In other words, lightweight toolkits provide some tangible benefits:
- The largest collection of widgets, not limited to just the subset that can be assumed to be present on each OS.
- Consistency in how widgets behave, since they are designed as a set.
- An OS independent look-and-feel.
JavaFX was originally designed by Sun Microsystems in 2008 as a replacement for the Java AWT and Swing toolkits, and was designed to compete directly with Adobe Flash/Flex and similar web toolkits. In 2010, Oracle released JavaFX into the community as part of the OpenJDK initiative. The open source version of JavaFX is currently maintained by Gluon and the community.
JavaFX is an imperative toolkit, where the programmer describes the layout and how it should be managed in code (and XML). [This contrasts with a declarative toolkit like Jetpack Compose, where the programmer describes a layout and the system reflects state in that layout].
JavaFX is a lightweight toolkit that runs well on Windows, Mac, Linux. It provides a native look-and-feel on each platform, and even supports hardware acceleration! It’s not included with the JRE, but because it’s open source, we can distrbute the libraries with our applications.
Setup
Although JavaFX can be installed from the main JavaFX site, the recommended way to bundle these libraries into your application is to add it to your Gradle configuration file. Gradle will then download and install JavaFX as-needed.
In your project’s build.gradle file, make the following changes to include the javafxplugin and related settings in the javafx block.
import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
plugins {
application
kotlin("jvm") version "1.6.20"
id("org.openjfx.javafxplugin") version "0.0.13"
id("org.beryx.jlink") version "2.25.0"
}
group = "net.codebot"
version = "1.0.0"
val compileKotlin: KotlinCompile by tasks
val compileJava: JavaCompile by tasks
compileJava.destinationDirectory.set(compileKotlin.destinationDirectory)
repositories {
mavenCentral()
}
dependencies {
testImplementation(kotlin("test"))
}
tasks.test {
useJUnitPlatform()
}
tasks.withType<KotlinCompile> {
kotlinOptions.jvmTarget = "1.8"
}
application {
mainModule.set("calculator")
mainClass.set("calculator.Main")
}
javafx {
// version is determined by the plugin above
version = "18.0.2"
modules = listOf("javafx.controls", "javafx.graphics")
}
// https://stackoverflow.com/questions/74453018/jlink-package-kotlin-in-both-merged-module-and-kotlin-stdlib
jlink {
forceMerge("kotlin")
}
In your Gradle menu in IntelliJ, press “Sync” to load the changes, and the JavaFX libraries should be loaded. If you expand the “External Libraries” in the Project view, you can see the JavaFX libraries have been installed:
Example: HelloFX
The following application shows how to create a simple window with some graphics. Athough longer than our console version of “Hello Kotlin”, it accomplishes quite a lot with minimal code. We’ll discuss this in further detail below.
class App: Application() {
override fun start(stage:Stage?) {
val image = Image("java.png", 175.0, 175.0)
val imageView = ImageView(image)
val label = Label(
System.getProperty("java.vendor")
+ System.getProperty("java.version") + "\n"
+ System.getProperty("javafx.version"))
val box = VBox(imageView, label)
VBox.setMargin(label, Insets(10.0))
val scene = Scene(box, 175.0, 225.0)
stage.setResizable(false)
stage.setScene(scene)
stage.show()
}
}
This is actually pretty impressive when you realize that we have just created:
- A resizable window with min/max/restore buttons
- A titlebar and content centred in the window.
- A UI that will inherit the appearance of any platform where it runs. Execute this on Windows, and the buttons will have a standard appearance and positioning for that platform!
Classes
In JavaFX, our highest level abstractions are the Application class, with one or more Stage classes representing the application windows, and one or more Scene classes to manage the contents of a window. Nodes represent the individual graphical elements.
As we saw in the previous chapter with JavaFX, it’s standard practice in 2D graphical applications to represent the interface as a scene graph of objects. In JavaFX, the Scene class maintains this scene graph, consisting of various nodes, for each Scene that we display. Note that it’s possible to have multiple windows, each with multiple scenes, each of which manages a different scene graph. (Multiple windows can be displayed at once, but only once scene graph can be displayed at a given time in a window, representing the current window contents).
Application
The Application class is top-level representation of the application. It serves as the application entry point, replacing the main() method. During launch, a JavaFX application will perform the followin steps:
- Constructs an instance of the specified Application class
- Calls the
init() method
- Calls the
start(javafx.stage.Stage) method (passing in a default stage)
- Waits for the application to finish, which happens when either of the following occur:
- the application calls
Platform.exit()
- the last window has been closed and the
implicitExit attribute on Platform is true
- Calls the
stop() method
The start() method is abstract and MUST be overridden. The init() and stop() methods are optional, but MAY be overridden. It’s fairly normal to just override start() and ignore the others most of the time.
Stage
The Stage class is the top-level container or application window. You can have multiple stages, representing multiple windows.
javafx.stage.Window
javafx.stage.Stage
A Stage instance is automatically created by the runtime, and passed into the start() method.
Stage methods operate at the window level:
setMinWidth(), setMaxWidth()setResizable()setTitle()setScene()show()
Scene
The Scene is a container for the content in a scene-graph. Although you can create multiple scenes, only one can be attached to a window at a time, representing the “current” contents of that window.
To construct a scene, and set it up:
- Create a scene graph consisting of a container holding on or more nodes;
- Add the root node of the scene graph to the scene;
- Add the scene to a stage and make the stage visible.
Scene methods manipulate the scene graph, or attempt to set properties for the entire graph:
setRoot(Node)setFill(Paint)getX(), getY()
Node
Node is the base class for all elements of a scene graph. Types of nodes include:
Nodes have common properties for position (x, y), width and height, background colour and so on. These can be set manually in code, or in the case of visual properties, associated with a CSS stylesheet.
Info
JavaFX is pretty comprehensive, but you might want to implement something that isn’t built into that toolkit e.g. date widgets.
Luckily, you can include projects that expand the standard widgets. These are intended to be imported and used alongside the standard JavaFX widgets.
- ControlsFX expands to include checklists, breadcrumb bars and other unique widgets.
- JFxtras includes a calendar widget, gauges and other useful widgets.
Layouts
Layout is how items are arranged on the screen. Layout classes are branch nodes that have built-in layout behaviour. Your choice of parent class to hold the nodes determines how its children will be laid out.
| Layout Class |
Behaviour |
| HBox |
Layout children horizontally in-order |
| VBox |
Layout children vertically in-order |
| FlowPane |
Layout left-right, top-bottom in-order |
| BorderPane |
Layout across sides, centre in-order |
| GridPane |
2D grid, with cells the same size |
Example: Java Version
Here’s the Java Version example from above, annotated. The sequence to setup a window is:
- Define the nodes (lines 4-11)
- Create a layout as the root of the scene graph (line 14), which will hold the nodes.
- Add the root node to the scene (line 18)
- Add the scene to the stage (line 19)
- Show the stage (line 23)
class App: Application() {
override fun start(stage:Stage?) {
// imageView is our first node
val image = Image("java.png", 175.0, 175.0)
val imageView = ImageView(image)
// label is our second node
val label = Label(
System.getProperty("java.vendor")
+ System.getProperty("java.version") + "\n"
+ System.getProperty("javafx.version"))
// box is our layout that will manage the position of our nodes
val box = VBox(imageView, label)
VBox.setMargin(label, Insets(10.0))
// create a scene from the layout class, and attach to the stage
val scene = Scene(box, 175.0, 225.0)
stage.setScene(scene)
// set window properties and show it
stage.setResizable(false)
stage.show()
}
}
Events
JavaFX expands on the Listener model that was introduce in Java Swing, and provides support for a wide varieties of events. The Event class is the base class for a JavaFX event. Common events include:
- MouseEvent − This is an input event that occurs when a mouse is clicked. It includes actions like mouse clicked, mouse pressed, mouse released, mouse moved.
- KeyEvent − This is an input event that indicates the key stroke occurred over a node. This event includes actions like key pressed, key released and key typed.
- WindowEvent − This is an event related to window showing/hiding actions. It includes actions like window hiding, window shown.
Nodes have convenience methods for handling common event types. They include:
setOnMouseClicked()setOnMousePressed()setOnMouseReleased()setOnMouseMoved()setOnKeyPressed()setOnKeyReleased()
Additionally, there is a generic “action” handler which responds to the standard interaction with a control e.g. pressing a button, or selecting a menu item.
For example, here’s a handler for a “save” button (from sample-code/desktop/contacts)
val save = Button("Save")
save.setOnAction { event ->
model.add(Contact(name.text, phone.text, email.text))
model.save()
}
Packaging
Scripts are a simple way to get your application to launch, but they struggle when you have complex dependencies, or resources that need to be included (like you often will with a GUI application). If you are building a JavaFX or Compose desktop application, you should consider using jlink or jpackage to build an installer.
JLink will let you build a custom runtime that will handle the module dependencies for JavaFX. The simplest way to do this is to add the JLink plugin to your build.gradle file and let Gradle handle it.
plugins {
id 'org.beryx.jlink' version '2.25.0'
}
You can also configure it in the build.gradle file as well. For a full set of options see the Badass-JLink plugin page.
jlink{
launcher {
name = "clock"
}
imageZip.set(project.file("${project.buildDir}/image-zip/clock-image.zip"))
}
We can rebuild the clock sample using Gradle - build - jLink to produce a runtime script in build/image
Here’s the resulting directory structure. Notice that it includes a number of libraries that our application needs to run.
$ tree build/image -L 2
AnalogClock/build/image
├── bin
│ ├── clock_advanced
│ ├── clock_advanced.bat
│ ├── java
│ ├── jrunscript
│ └── keytool
├── conf
│ ├── net.properties
│ ├── security
│ └── sound.properties
├── include
│ ├── classfile_constants.h
│ ├── darwin
│ ├── jawt.h
│ ├── jni.h
│ ├── jvmti.h
│ └── jvmticmlr.h
├── legal
│ ├── java.base
│ ├── java.datatransfer
│ ├── java.desktop
│ ├── java.prefs
│ ├── java.scripting
│ ├── java.xml
│ └── jdk.unsupported
├── lib
│ ├── classlist
│ ├── fontconfig.bfc
│ ├── fontconfig.properties.src
│ ├── jrt-fs.jar
│ ├── jspawnhelper
│ ├── jvm.cfg
│ ├── libawt.dylib
..... (continues)
Running the top-level bin/clock_advanced image will execute our application.
Creating installers
Finally, we can use jpackage to create native installers for a number of supported operating systems. JPackage is included as a console application in Java JDK 16 or higher, and will work with any JVM language (e.g. Java, Kotlin, Scala). The full guide is here.
An installer is an application that when executed, installs a different application for the user. We need installers because most applications consists of many different files: executables, libraries, resources (images, sound files), preference files and so on. These need to be installed in the correct location, and sometimes registered, to function correctly.
Tasks that the installer performs include:
- Copying application files to the correct location.
- Installing and registering system libraries.
- Making changes to the system registry (or similar system databases).
- Creating icons on the desktop, or applications folder.
- Prompting the user if any of these tasks require elevated privileges.
Instead of running jpackage manually, we will install a plugin into IntelliJ and use that environment to generate our installers. We can do this by installing the Badass-JLink plugin page. To use the plugin, include the following in your gradle.build script:
plugins {
id 'org.beryx.jlink' version '2.25.0'
}
JPackage itself has a number of other options that you can specify in the build.gradle file. The full list of options is on the plugin website.
// build.gradle file options for jpackage
jlink {
options = ['--strip-debug', '--compress', '2', '--no-header-files', '--no-man-pages']
launcher{
name = 'hello'
jvmArgs = ['-Dlog4j.configurationFile=./log4j2.xml']
}
}
If you install the plugin correctly, then you should see the jpackage command in Gradle - build - jpackage. Run this and it will create platform installers in the build/distribution directory.
This is a standard macOS installer. Drag the clock_advanced icon to the Applications folder. You can then run it from that folder.
Info
Installers are meant for graphical applications. If you are building a JavaFX or Compose desktop application, this is the right choice. If you’re building a console application, you probably want a script instead (see previous step) so that you can execute it from the console directly.
Imagine that you want to build an application that runs on multiple platforms. e.e. Windows and macOS, or Android and iOS. How should you do it?
The first thing you would probably try is to just build the application in your favorite language, and then compile it wherever you want to run it. If it was a simple application, and you were using a standard language that worked identically on platforms, you might be able to do this. e.g. the C++ 14 console applications that you built in CS 246 could probably build and run anywhere! You’re using a well-defined language that is supported across multiple platforms so this probably works fine.
Unfortunately it’s not always that simple. Imagine that you want to add graphics: you would quickly realize that the fastest and most sophisticated graphics libraries are platform-dependent. You can do some amazing things with DirectX, but it only runs on Windows, and doesn’t help you build for macOS or Linux (or iOS or Android).
This is a common problem. A lot of the functionality that you will want to use in your application is tied to the underlying platform - including UI, graphics, sound, networking and so on. This means that any libraries or frameworks that you want to use are also tied that platform. Microsoft C++ developers might have a rich ecosystem of Windows-specific libraries to use, but they aren’t portable to macOS or Linux. SwiftUI is incredible for building macOS applications, but doesn’t work for Windows.
So how to you write sophisticated cross-platform applications?
Option 1. Develop separately for each platform.
One solution is to not bother chasing cross-platform code. Use the best tools and libraries, often the native ones that the vendor provides, and create entirely different applications for each platform. This has the advantage of producing the highest-quality software, but you cannot reuse your code so it’s often a very expensive approach (i.e. hire n teams, where n is the number of platforms).
e.g. You design a mobile app, and have separate projects for Swift/iOS and Kotlin/Android builds.
Option 2: Use technologies that exist on every platform.
Instead of building for each platform, you build for a runtime environment that exists on each of your platforms. This is one of the major justifications for companies targeting the Java JVM: “Write once, run anywhere”, at least for some types of software. This is also one of the main benefits of targeting the web: as long as a compliant browser exists, your webpage, or web-based application can run on every platform.
e.g. Gmail, Netflix and Facebook for web applications; Kotlin applications using cross-platform UI toolkits.
This is an extremely successful strategy for reaching a wide audience, but it faces two main challenges:
- You are restricted to the capabilities that that runtime platform offers you. Platform-specific features that require access to native libraries may not be available. This is the situation when writing JVM applications in Kotlin. The Java ecosystem contains a lot of intrinsic functionality, but you will be limited in your ability to access anything platform specific. e.g. this is one of the reason why Java is rarely used to develop commercial quality games: the JVM does not provide low-level access to advanced graphics capabilities of the platform.
- The runtime environment may not exist on your target platform. This is rarely a case when talking about web applications, since browsers exists on most platforms. However, the JVM isn’t always available. e.g. Apple doesn’t directly support a JVM and Just-In-Time (JIT) compilation on iOS: everything needs to be Ahead-of-Time (AoT) compiled, which prevents us from directly deploying Kotlin JVM apps on iOS.
Kotlin/JVM helps to address cross-platform compatibility, but it suffers from both of these restrictions.
Kotlin Multiplatform (KMP) offers a solution to this problem, by allowing us to produce native binaries for multiple platforms from the same code base. It helps us organize our code into reusable sections, while supporting the ability to interoperate with native code on each platform when required. This drastically reduces the effort required to write and maintain code across different platforms, and lets us bypass the restrictions of other solutions like the JVM.
Kotlin Multiplatform (KMP) is the Kotlin framework to support compilation on multiple platforms. As we’ve seen, Kotlin can already be used on any platform that has JVM support. KMP extends this to native builds on other platforms where a JVM is unavailable or undesireable for some reason. You use KMP to write native code using Kotlin for:
- Android
- iOS, WatchOS, TVOS
- macOS
- Windows
- Linux
- Web
Support for multiplatform programming is one of Kotlin’s key benefits. Kotlin provides common code that will run everywhere, plus native-platform capabilities.
- Common Kotlin includes the language, core libraries, and basic tools. Code written in common Kotlin works everywhere on all supported platforms, including JVM, Native (iOS, Android, Windows, Linux, macOS), Web (JS). Common multiplatform libraries cover everyday tasks such as HTTP, serialization, and managing coroutines. These libraries can be used on any platform.
- Kotlin also includes platform-specific versions of Kotlin libraries and tools (Kotlin/JVM, Kotlin/JS, Kotlin/Native). This includes native compilers for each of these platforms that produce a suitable target (e.g. bytecode for Kotlin/JVM, JS for Kotlin/JS). Through these platforms you can access the platform native code (JVM, JS, and Native) and leverage all native capabilities.
Info
“Platform-specific” functionality includes the user-interface. If you want a 100% “native look-and-feel” to your application, you would want to build the application using the native UI toolkit for that platform. e.g. Swift and SwiftUI for iOS. Cross-platform toolkits that we’ve used in this course, like JavaFX and Compose solve the problem of cross-platform compatibiltiy by providing a UI framework that runs everywhere, at the cost of being somewhat “non-native” feeling.
KMP allows you to build projects that use a combination of common and native libraries, and which can build to any one of the supported platforms - from the same codebase.
Kotlin multiplatform organizes the source code in hierarchies, with common-code at the base, and branches representing platform specific modules. All platform-specific source sets depend upon the common source set by default.
Common code can depend on many libraries that Kotlin provides for typical tasks like making HTTP calls, performing data serialization, and managing concurrency. Further, the platform-specific versions of Kotlin provide libraries we can use to can leverage the platform-specific capabilities of the target platforms.
For example, in the diagram above, commonMain code is available to all platforms (leaf nodes). desktopMain code is available to the desktop targets (linuxX64Main, mingwX64Main and macosX64Main) but not the other platforms like iosArm64Main.
In some cases, it may be desirable to define and access platform-specific APIs in common. This is particularly useful for areas where certain common and reusable tasks are specialized for leveraging platform-specific capabilities.
Kotlin multiplatform provides the mechanism of expected and actual declarations to achieve this objective. For instance, the common source set can declare a function as expected and the platform-specific source sets will be required to provide a corresponding function with the actual declaration:
Here, as we can see, we are using a function declared as expected in the common source set. The common code does not care how it’s implemented. So far, the targets provide platform-specific implementations of this function.
We can use these declarations for functions, classes, interfaces, enumerations, properties, and annotations.
Creating a KMP Project
In IntelliJ IDEA, select Kotlin - Multiplatform - Library.
This generates a project with the kotlin-multiplatform Gradle plugin. This plugin is added to our build.gradle file.
plugins {
kotlin("multiplatform") version "1.4.0"
}
This kotlin-multiplatform plugin configures the project for creating an application or library to work on multiple platforms.
Our project will contain shared, and native specific source folders.
It also generates source sets and Gradle build targets for each platform.
Writing Common Code
Let’s write a cross-platform version of the calculator application that we used at the very start of the course. We’ll define some common code and place it in the commonMain folder. This will be available to all of our platforms. Notice that this is basic Kotlin code, with no platform specific code included.
fun add(num1: Double, num2: Double): Double {
val sum = num1 + num2
writeLogMessage("The sum of $num1 & $num2 is $sum", LogLevel.DEBUG)
return sum
}
fun subtract(num1: Double, num2: Double): Double {
val diff = num1 - num2
writeLogMessage("The difference of $num1 & $num2 is $diff", LogLevel.DEBUG)
return diff
}
fun multiply(num1: Double, num2: Double): Double {
val product = num1 * num2
writeLogMessage("The product of $num1 & $num2 is $product", LogLevel.DEBUG)
return product
}
fun divide(num1: Double, num2: Double): Double {
val division = num1 / num2
writeLogMessage("The division of $num1 & $num2 is $division", LogLevel.DEBUG)
return division
}
The writeLogMessage() function should be platform specific, since each OS wil handle this differently. We will add a top-level declaration to our common code defining how that function should look:
enum class LogLevel {
DEBUG, WARN, ERROR
}
internal expect fun writeLogMessage(message: String, logLevel: LogLevel)
The expect keyword tells the compiler that the definition will be handled at the platform level, in another module. For example, we can flesh this out in the jvmMain module for Kotlin/JVM platform. The build for that platform will use the platform-specific version of this function.
internal actual fun writeLogMessage(message: String, logLevel: LogLevel) {
println("Running in JVM: [$logLevel]: $message")
}
Our goal is to define as much functionality as we can in the commonMain module, but recognize that we sometimes need to use platform-specific code for the results that we want to achieve.
Writing Common Unit Tests
Let’s write a few tests for our common calculator functions:
@Test
fun testAdd() {
assertEquals(4.0, add(2.0, 2.0))
}
@Test
fun testSubtract() {
assertEquals(0.0, subtract(2.0, 2.0))
}
@Test
fun testMultiply() {
assertEquals(4.0, multiply(2.0, 2.0))
}
@Test
fun testDivide() {
assertEquals(1.0, divide(2.0, 2.0))
}
There’s nothing unusual - we can easily write unit tests against common code. However, when we run them, we get a new window asking us to select a target. Select one or more targets for your tests.
Kotlin/Native
Kotlin/Native attempts to compile the Kotlin source directly to native binaries specific to the supported target platform. Kotlin/Native is primarily designed to allow compilation for platforms on which virtual machines are not desirable or possible, such as embedded devices or iOS.
Kotlin/Native provides an LLVM based backend for the Kotlin/Native compiler and native implementations of the Kotlin standard library. The Kotlin/Native compiler itself is known as Konan. LLVM is basically a compiler infrastructure that we can use to develop a front end for any programming language and a back end for any instruction set architecture.
It provides a portable, high-level assembly language optimized for various transformations that serve as a language-independent intermediate representation. Originally implemented for C and C++, today there are several languages with a compiler that supports LLVM, including Kotlin:
Kotlin/Native supports a number of platforms that we can conveniently select through the Gradle configuration:
- Linux (x86_64, arm32, arm64, MIPS, MIPS little-endian)
- Windows (mingw x86_64, x86)
- Android (arm32, arm64, x86, x86_64)
- iOS (arm32, arm64, simulator x86_64)
- macOS (x86_64)
- tvOS (arm64, x86_64)
- watchOS (arm32, arm64, x86)
- WebAssembly (wasm32)
Now, we should notice that in our Gradle configuration, there is a check for the host operating system. This is used to determine what native platform to target i.e. you need to be on macOS to build for that platform and so on.
kotlin {
val hostOs = System.getProperty("os.name")
val isMingwX64 = hostOs.startsWith("Windows")
val nativeTarget = when {
hostOs == "Mac OS X" -> macosX64("native")
hostOs == "Linux" -> linuxX64("native")
isMingwX64 -> mingwX64("native")
else -> throw GradleException("Host OS is not supported in Kotlin/Native.")
}
}
Interoperability
Kotlin/Native also supports two-way interoperability with native programming languages for different operating systems. The compiler creates:
- an executable for many platforms
- a static library or dynamic library with C headers for C/C++ projects
- an Apple framework for Swift and Objective-C projects
Kotlin/Native supports interoperability to use existing libraries directly from Kotlin/Native:
It is easy to include compiled Kotlin code in existing projects written in C, C++, Swift, Objective-C, and other languages. It is also easy to use existing native code, static or dynamic C libraries, Swift/Objective-C frameworks, graphical engines, and anything else directly from Kotlin/Native.
Finally, Multiplatform projects allow sharing common Kotlin code between multiple platforms, including Android, iOS, JVM, JavaScript, and native. Multiplatform libraries provide required APIs for common Kotlin code and help develop shared parts of a project in Kotlin in one place and share it with some or all target platforms.
Getting Started
https://kotlinlang.org/docs/native-get-started.html
Open the build.gradle.kts file, the build script that contains the project settings. To create Kotlin/Native applications, you need the Kotlin Multiplatform Gradle plugin installed. Ensure that you use the latest version of the plugin:
plugins {
kotlin("multiplatform") version "1.6.10"
}
Build your project. It will produce a native executable under
build/bin/native/debugExecutable/<your_app_name>.kexe
Example: Native Interop
This tutorial demonstrates how to use IntelliJ IDEA to create a command-line application. You’ll learn how to create a simple HTTP client that can run natively on specified platforms using Kotlin/Native and the libcurl library.
https://kotlinlang.org/docs/native-app-with-c-and-libcurl.html
The full code for this sample is here: https://github.com/Kotlin/kotlin-hands-on-intro-kotlin-native
- Create the project.
-
Update the build.gradle file.
kotlin {
def hostOs = System.getProperty("os.name")
def isMingwX64 = hostOs.startsWith("Windows")
def nativeTarget
if (hostOs == "Mac OS X") nativeTarget = macosX64('native')
else if (hostOs == "Linux") nativeTarget = linuxX64("native")
else if (isMingwX64) nativeTarget = mingwX64("native")
else throw new FileNotFoundException("Host OS is not supported in Kotlin/Native.")
nativeTarget.with {
binaries {
executable {
entryPoint = 'main'
}
}
}
}
-
Create a definition file.
When writing native applications, you often need access to certain functionalities that are not included in the Kotlin standard library, such as making HTTP requests, reading and writing from disk, and so on.
Kotlin/Native helps consume standard C libraries, opening up an entire ecosystem of functionality that exists for pretty much anything you may need. Kotlin/Native is already shipped with a set of prebuilt platform libraries, which provide some additional common functionality to the standard library. We’ll link in a standard C library.
Create a directory named src/nativeInterop/cinterop.
Create a file libcurl.def with the following contents.
headers = curl/curl.h
headerFilter = curl/*
compilerOpts.linux = -I/usr/include -I/usr/include/x86_64-linux-gnu
linkerOpts.osx = -L/opt/local/lib -L/usr/local/opt/curl/lib -lcurl
linkerOpts.linux = -L/usr/lib/x86_64-linux-gnu -lcurl
This defined kotlin header files to be created from the C headers on our system.
- Add interoperrability to your builds.
Add this to your build.gradle file.
nativeTarget.with {
compilations.main { // NL
cinterops { // NL
libcurl // NL
} // NL
} // NL
binaries {
executable {
entryPoint = 'main'
}
}
- Write the application code.
Update the source file Main.kt with the following source.
import kotlinx.cinterop.*
import libcurl.*
fun main(args: Array<String>) {
val curl = curl_easy_init()
if (curl != null) {
curl_easy_setopt(curl, CURLOPT_URL, "https://example.com")
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L)
val res = curl_easy_perform(curl)
if (res != CURLE_OK) {
println("curl_easy_perform() failed ${curl_easy_strerror(res)?.toKString()}")
}
curl_easy_cleanup(curl)
}
}
If you build it, you should get a native executable, linked to the curl libraries.
You should be able to run it to see output!
$ ./httpclient.kexe www.example.com
<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
....
The most common requirement for KMP is to support both Android and iOS targets. Kotlin Multiplatform Mobile (KMM) leverages KMP to simplify the development of cross-platform mobile applications. You can share common code between iOS and Android apps and write platform-specific code only where it’s necessary, typically to support a native UI or when working with platform-specific APIs.
A basic KMM project consists of three components:
- Shared module – a Kotlin module that contains common logic for both Android and iOS applications. Builds into an Android library and an iOS framework. Uses Gradle as a build system.
- Android application – a Kotlin module that builds into the Android application. Uses Gradle as a build system.
- iOS application – an Xcode project that builds into the iOS application.
Kotlin supports two-way interop with iOS: Kotlin can call into iOS libraries, and vice-versa using the Objective-C bindings. (Swift bindings are being developed). Android is a native target for Kotlin, and is much easier to support.
KMM is exciting because we can use Kotlin for both targets, and share probably 50-75% of the code between platform. The native code is just those modules that are very specific to each platform, typically the UI. A KMM application could potentially offer identical functionality on Android and iOS, while delivering a complately native UI experience with Jetpack Compose on Android, and SwiftUI on iOS.
For examples, see the list of KMM Samples.
Subsections of Building Services
Web Services
Overview
Up to this point, we’ve assumed that we’re designing a standalone application i.e. for a single computer. Historically, many software applications were designed to be standalone, and much of the software that we use is still standalone.
However, it can be useful to sometimes split processing across multiple systems. Here’s a partial list of the reasons why you might want to do this:
- Resource sharing. We often need to share resources across users. For example, storing our customer data in a shared databases that everyone in the company can access.
- Reliability. We might want to increase the reliability of our software by redunant copies running on different systems. This allows for fault tolerance - failover in case the first system fails. This is common with important resources like a web server.
- Performance. It can be more cost effective to have one highly capable machine running the bulk of our processing, while cheaper/smaller systems can be used to access that shared machine. It can also be cheaper to spread computation across multiple systems, where tasks can be run in parallel. Distributed architectured provide flexibility to align the processing capabilities with the task to be performed.
- Scalability. Finally, if designed correctly, distributing our work across multiple systems can allow us to grow our system to meet high demand. Amazon for example, needs to ensure that their systems remain responsive, even in times of heavy load (e.g. holiday season).
- Openness. There is more flexibility, since we can mix systems from different vendors.
Earlier, we discussed a distributed application as a set of components, spread across more than one machine, and communicating with one another over a network. Each component has some capabilities that it provides to the other components, and many of them coordinate work to accomplish a specific task.
We have already described a number of different distributed architectures. These enforce the idea that distributed systems can take many different forms, each with it’s own advantages and disadvantages. When we’re considering building a service, we’re really focusing on a particular kind of distributed system, where our application is leveraging remote resources.
Service Architectures
Client-Server
A client-server architecture is a model where one centralized server provides capabilities to multiple client machines.
- Client − The process that issues a request to the second process i.e. the server. From the point of view of the user, this is the application that they interact with.
- Server − The process that receives the request, carries it out, and sends a reply to the client.
In this architecture, the application is modelled as a set of services that are provided by servers and a set of clients that use these services. The servers need not know about clients, but the clients must know the identity of servers.
Advantages
- Separation of responsibilities such as user interface presentation and business logic processing (client).
- Reusability of server components and potential for concurrency (single server).
- It also makes effective use of resources when a large number of clients are accessing a high-performance server.
Disadvantages
- Limited server availability and reliability.
- Limited testability and scalability.
- Fat clients with presentation and business logic together.
- Limited ability to scale the server (need more processing, “buy a bigger server”).
Multi-tier architecture
A multi-tier architecture (also known as 2-tier, 3-tier or layered) is an architecture that separates an application into separate tiers or areas of concerns. Often the user interface, business logic and data layers end up split apart.
The most common form is this architecture is 3-tier architecture, where each tier is a separate module, deployed on a separate computer. Tiers typically communicate over a network connection.
- The top tier is the Presentation layer, which handles the UI logic. This is typically hosted on a client machine (i.e. where the user accesses it).
- The middle layer is the Application or Logic tier, which handles “business logic”; the rules and state management of the application. This can often include logic for coordinating requests from a client across multiple services as well.
- The bottom layer is the Data tier, which handles access, storage and management of the underlying data. This is often a database, or a wrapper around a database (or some other form of storage).
This is an extremely common architecture for Enterprise applications. It also aligns with the way that websites are traditionally served (Presentation tier is the browser, Logic tier is the web server and underlying code, and the Data tier is a database).
In some cases, the Logic and Presentation tiers are combined, for a 2-tier architecture consisting of just Presentation and Data tiers.
Advantages
- Enhances the reusability and scalability − as demands increase, extra servers can be added.
- Provides maintainability and flexibility.
Disadvantages
- Greater complexity, more difficult to deploy and test across tiers.
- More emphasis on server reliability and availability.
Service Oriented Architecture
A service-oriented architecture (SOA) is an architectural style that supports service orientation. A service is a discrete unit of functionality that can be accessed remotely and acted upon and updated independently, such as retrieving a credit card statement online.
In other words, services exist independently of clients, and provide services to any client that requires it. Services are loosely coupled to one another, and should act independently.
Principles of SOA, governing how services should be designed:
- Service contract: there should be an agreed upon interface for accessing a service.
- Longevity: services should be designed to be long-lived (long-running).
- Autonomy: services should work independently of one another.
- Service composibility: services can be used to compose other services.
- Stateless: services should not track state, but either return a resulting value or throw an exception if necessary.
Because they are independent entities, we need a supporting infrastructure around services, and applications that are designed to leverage that infrastructure. This includes a repository, where an application can search for services that can meet its needs at runtime.
Advantages
- A client or any service can access other services regardless of their platform, technology, vendors, or language implementations.
- Each service component is independent from other services due to the stateless service feature.
- The implementation of a service will not affect the application of the service as long as the exposed interface is not changed.
- Enhances the scalability and provides standard connection between systems.
Disadvantages
- Even more complexity in setting up a system, since we’re now distributing across multiple tiers.
- Registry and other supporting infrastructure can be complex to setup and maintain.
- Difficulty debugging, profiling and so on.
Microservices
A microservices architecture arranges an application as a collection of loosely coupled services, using fine-grained services and a lightweight protocol. Some of the defining characteristics of microservices:
- Services are organized around business capabilities i.e. they provide specialized, domain-specific services to applications (or other services).
- Service are not tied to any one programming language, platform or set of technologies.
- Services are small, decentralized, and independently deployable.
A microservice based architecture is really a subtype of SOA, which an emphasis on smaller, domain-specific components with very narrow functions.
Advantages
- Easier to design, build and deploy small targeted services.
- Redundancy - you can always “spin up” a replacement service if something fails.
- Performance - you can always “scale out” by firing up redundant services to share the workload, as required.
Disadvantages
- Extremely difficult to test and debug.
- Practically requires supporting services, like a registry for processes to locate service endpoints.
Web Servers
As originally designed, a web server and web browser are a great example of a client-server architecture.
A web server is service running on a server, listening for requests at a particular port over a network, and serving web documents (HTML, JSON, XML, images). The payload that is delivered to a web browser is the content, which the browser interprets and displays. When the user interacts with a web page, the web browser reacts by making requests for additional information to the web server.
Over time, both browser and web server have become more sophisticated, allowing servers to host additional content, run additional programs as needed, and work as part of a larger ecosystem that can distribute client requests across other systems.
Web technologies are interesting to us because they can be the basis for a robust service request mechanism. We’ll explore that in this section.
HTTP Protocol
The Hypertext Transfer Protocol (HTTP) is an application layer protocol that supports serving documents, and processing links to related documents, from a remote service.
HTTP functions as a request–response protocol:
- A web browser is a typical client, which the user is accessing. A web server would be a typical server.
- The user requests content through the browser, which results in an HTTP request message being sent to the server.
- The server, which provides resources such as HTML files and other content or performs other functions on behalf of the client, returns a response message to the client. The response contains completion status information about the request and may also contain requested content in its message body.
Request Methods
HTTP defines methods to indicate the desired action to be performed on the identified resource.
What this resource represents, whether pre-existing data or data that is generated dynamically, depends on the implementation of the server. Often, the resource corresponds to a file or the output of an executable residing on the server. Method names are case sensitive.
-
GET: The GET method requests that the target resource transfers a representation of its state. GET requests should only retrieve data and should have no other effect.
-
HEAD: The HEAD method requests that the target resource transfers a representation of its state, like for a GET request, but without the representation data enclosed in the response body. Uses include looking whether a page is available through the status code, and quickly finding out the size of a file (Content-Length).
-
POST: The POST method requests that the target resource processes the representation enclosed in the request according to the semantics of the target resource. For example, it is used for posting a message to an Internet forum, or completing an online shopping transaction.
-
PUT: The PUT method requests that the target resource creates or updates its state with the state defined by the representation enclosed in the request. A distinction to POST is that the client specifies the target location on the server.
-
DELETE: The DELETE method requests that the target resource deletes its state.
Spring Boot
Spring is a popular Java framework. It’s opinionated, in that it provides a strict framework for a web service. You are expected to add classes, and customize behaviour as needed, but you are restricted to the overarching structure that is provided. This is a reasonable tradeoff. In return for giving up some flexibility, you get Dependency Injection and other advanced features, which make testing (along other things) much easier to achieve.
This power comes at a price: complexity. Spring has a large number of configuration files that allow you to tweak and customize the framework, but the options can be overwhelming. To help developers, Pivotal created Spring Boot, which is a program that creates a starting configuration for you - even going so far as to include a web server and other required libraries to get you up-and-running quickly.
We’ll walk through setting up a simple Spring Boot application. The steps are typically:
- Use Spring Boot to create a working project.
- Write controller, model, and other required classes.
- Write and run tests.
Setup
It is highly recommended that you use Spring Boot to create your starting project. You can run it one of two ways:
- Visit start.spring.io and use the web form to set the parameters for your project. Generate a project and download the project files.
- Use the Spring project wizard in IntelliJ. Set the parameters for your project (Kotlin, Gradle) and follow instructions to generate an IntelliJ IDEA project.
Regardless of which you choose, you will be asked for dependencies. You will probably want to include at least Spring Web and Spring Data JPA and possibly others:
- Spring Web: this will embed a web server in your project so that you can easily test it.
- Spring Data JPA: JPA stands for Java Persistance API. JPA is an object-persistence layer that allows you to map classes directly to database tables and avoid writing SQL for simple requests.
- JDBC API: this will allow you to use JDBC to access databases - helpful if your service needs to persist data.
- H2 Database: an embedded database for testing - you can also swap out for a different database if desired.
Your starting project should look something like this:
Notice that the class is annotated with @SpringBootApplication. This tells the framework to treat this as the top-level application class. Spring uses extensive annotations like this to flag methods and properties for the framework.
The main() method calls the framework’s runApplication method to launch.
Your Spring project comes with an embedded web server, which runs on port 8080. You can test this by running the web server (click on the Play button beside the Main method), and then open the URL in a web browser: http://localhost:8080
Info
The Spring Web starter code always defaults to https://localhost:8080 to serve your web service. Use this URL for testing.
Unfortunately, this won’t return anything useful yet. Although the web service is running, we need to write code to handle the requests! That’s the job of the controller.
We’ll add a controller class to save messages (just a couple of strings), or retrieve a list of messages that were previously posted.
Writing a Controller Class
The controller class is responsible for handling requests from a client process. Our web service uses HTTP, so requests will be the request methods that we discussed earlier: GET to retrieve a list of messages, and POST to store a new message.
Here’s an example of a controller class, configured to handle Post and Get requests.
@RestController
@RequestMapping("/messages")
class MessageResource(val service: MessageService) {
@GetMapping
fun index(): List<Message> = service.findMessages()
@PostMapping
fun post(@RequestBody message: Message) {
service.post(message)
}
}
@RestController flags this class as our main controller class, which will be responsible for handling the HTTP requests. Within this class, we need to write code to handle the endpoints and requests for our application. The main endpoint will be /messages, since we set that mapping on the controller directly.
@GetMapping and @PostMapping indicate the methods that will handle GET and POST requests respectively. Our methods work with MessageService and Message classes, which we will need to define.
This means that our endpoint will be https://localhost:8080/messages (the default address and port for Spring, plus the endpoint that we defined). Our controller will handle GET and POST requests to that endpoint.
Info
The MessageResource() class declaration is an example of dependency injection: instead of instantiating the MessageService inside of the class, we pass it in as a parameter. MessageService is flagged as a class that the framework can manage directly.
Dependency injection makes testing earlier, since you don’t have unmanaged objects being allocated inside of a class. In this case, we can mock the MessageService during testing to isolate our MessageResource tests.
We can identify the following annotations:
| Annotation |
Use |
| @RestController |
Indicates a controller class that should process requests |
| @GetMapping |
A function that will be called when a GET request is received. |
| @PostMapping |
A function that will be called when a POST request is received. |
Using this code:
-
A client sending a GET request will be returned a list of all messages in JSON format.
-
A client sending a POST request, with well-formed data, will create a new message.
We could add in other mappings (e.g. PUT, DELETE) if required. All of these mappings would be handled in the Controller class.
We can finish our first pass at this service by adding a Message class and a MessageService class to store the data from our requests. The full service is listed below (also in the public repo: /service/spring-server-basic)
package demo
import org.springframework.boot.autoconfigure.SpringBootApplication
import org.springframework.boot.runApplication
import org.springframework.stereotype.Service
import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.PostMapping
import org.springframework.web.bind.annotation.RequestBody
import org.springframework.web.bind.annotation.RestController
@SpringBootApplication
class DemoApplication
fun main(args: Array<String>) {
runApplication<DemoApplication>(*args)
}
@RestController
class MessageResource(val service: MessageService) {
@GetMapping
fun index(): List<Message> = service.findMessages()
@PostMapping
fun post(@RequestBody message: Message) {
service.post(message)
}
}
data class Message(val id: String, val text: String)
@Service
class MessageService {
var messages: MutableList<Message> = mutableListOf()
fun findMessages() = messages
fun post(message: Message) {
messages.add(message)
}
}
Using JPA for Object Mapping
So far, we’re receiving and storing JSON objects as Messages. However, we’re only saving them in a list, which will be lost when we halt our service. What if we want to persist the data into a database? How do we convert our Message objects to a format that we can write out?
Spring Data JPA is a library that focuses on using JPA to store data in a relataional database. It greatly simplifies setting up a repository: we just setup the interface that we want, and it automatically creates the implementation code for us!
To start, add the JAP dependencies to your build.gradle file. Make sure to add these to the existing sections - don’t delete anything! Once added, click on the Sync icon
plugins {
kotlin("plugin.jpa") version "1.6.10"
}
dependencies {
implementation("org.springframework.boot:spring-boot-starter-data-jpa")
}
Next we’ll add a repository interface that tells the service what operations we want to perform against our Message class. The modified code looks like this:
@Service
class MessageService(val db: MessageRepository) {
fun findMessages(): List<Message> = db.findMessages()
fun post(message: Message){
db.save(message)
}
}
interface MessageRepository : CrudRepository<Message, String>{
@Query("select * from messages")
fun findMessages(): List<Message>
}
@Table("MESSAGES")
data class Message(@Id val id: String?, val text: String)
By inheriting from CrudRepository, the MessageRepositorygains methods for save, findByIndex and other common operations.
The findMessages method is undefined, so we use some annotations to map it to a query against a specific database table.
@Query indicates a database query that should be run, and the results mapped to the output of this function.@Table describes a data class that corresponds to an underlying database table. This table will be created for us, using the parameters from the Message class as column names.
By default, our Spring framework projects support H2, and in-memory database. The database file is stored in ./data in our project folder. You can browse it in IntelliJ IDEA and see the underlying table that is created.
Generating Requests
We can do a lot with just POST and GET operations, since we also have the ability to pass parameters in our requests. If your server is running in IntelliJ IDEA, you can create requests for testing directly in the IDE:
- Click the drop-down menu beside the GET or POST mapping in your Controller code. You should see an option to
- This should bring up a script where you can enter requests that you wish to run. Enter as many as you wish: to run them, click on the Run arrow beside the request. Pay careful attention to the format, and notice the content type is included in the request:
You can run simple tests directly from a browser, but IntelliJ IDEA for testing provides additional support like code completion and syntax highlighting for structuring your requests. This doesn’t replace proper automated tests, but it’s certainly helpful when configuring and debugging your services.
Making Client Requests
The big question, of course, is how do we make requests to this service from a client? How do we actually use it?
Kotlin (and Java) includes libraries that allow you to structure and execute requests from within your application.
This example opens a connection, and prints the results of a simple GET request:
URL("https://google.com").readText()
To use our service, we may want to set a few more parameters on our request. The HttpRequest class uses a builder to let us supply as many optional parameters as we need when building the request.
Here’s a method that fetches data from our server example above:
fun get(): String {
val client = HttpClient.newBuilder().build()
val request = HttpRequest.newBuilder()
.uri(URI.create("http://127.0.0.1:8080"))
.GET()
.build()
val response = client.send(request, HttpResponse.BodyHandlers.ofString())
return response.body()
}
We often need to package data into our requests. Here is an example of a POST request sending JSON data to our service, and returning the response:
fun post(message: Message): String {
val string = Json.encodeToString(message)
val client = HttpClient.newBuilder().build();
val request = HttpRequest.newBuilder()
.uri(URI.create("http://127.0.0.1:8080"))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(string))
.build()
val response = client.send(request, HttpResponse.BodyHandlers.ofString());
return response.body()
}
GET and POST represent the most common requests. You can similarly support other request types that have been discussed.
See the public repo for these samples, under /service/sprint-client and /service/spring-server.
Packaging
Unlike applications, which are hosted by users on their own systems, services are typically deployed on servers. These can be physical systems, VMs or containers running in the cloud, or any combination of these targets.
Web services, the type that we’ve been considering, need to be deployed to a web server. When we were building projects with Spring Boot, it quietly launched a web server in the background to support us testing our application. To deploy in a production environment though, we would need to install our application in an environment where a web server is already installed.
From Spring, we can produce one of two packages:
- WAR file: a Web Application Archive (WAR) file is a standard deployment package for services that run on a web server.
- JAR file: produce a standalone JAR file and deploy it (see instructions above).
We’ll focus on the most common situation: packaging for distribution to a standalone web server, or cloud platform. This isn’t something that we will be doing in this course, but it’s helpful to know that there is an easy road to deploy your application in a production environment.
JAR Archives
The sections below are taken in part from the official Spring deployment documentation.
Spring Boot’s executable jars are ready-made for most popular cloud PaaS (Platform-as-a-Service) providers. These providers tend to require that you “bring your own container”. They manage application processes (not Java applications specifically), so they need an intermediary layer that adapts your application to the cloud’s notion of a running process.
Amazon Web Services (AWS)
Amazon Web Services offers multiple ways to install Spring Boot-based applications, either as traditional web applications (war) or as executable jar files with an embedded web server. The options include:
- AWS Elastic Beanstalk
- AWS Code Deploy
- AWS OPS Works
- AWS Cloud Formation
- AWS Container Registry
As an example, AWS Elastic Beanstalk allows deploying a WAR file directly to a Tomcat Platform (a well-known web/application server), or the “Java SE Platform”. For web services, we would follow their documentation for configuring the cloud platform, and then upload the WAR file that we produce in IntelliJ.
Google Cloud
Google Cloud has several options that can be used to launch Spring Boot applications. The easiest to get started with is probably App Engine, but you could also find ways to run Spring Boot in a container with Container Engine or on a virtual machine with Compute Engine.
To run in App Engine, you can create a project in the UI first, which sets up a unique identifier for you and also sets up HTTP routes. Add a Java app to the project and leave it empty and then use the Google Cloud SDK to push your Spring Boot app into that slot from the command line or CI build.
App Engine Standard requires you to use WAR packaging.
Standalone Service
Spring also supports running your service as a standalone application. This is very similar to what we did above, when we generated scripts and a JAR file to launch our application. For details, refer to the Spring documentation.
Asynchronous Programming
Programs typically consists of functions, or subroutines, that we call in order to perform some operations. One underlying assumption is that subroutines will run to completion before returning control to the calling function. Subroutines don’t save state between executions, and can be called multiple times to produce a result. For example:
Because they run to completion, a subroutine will block your program from executing any further until the subroutine completes. This may not matter with very quick functions, but in some cases, this can cause your application to appear to “hang” or “lock up” waiting for a result.
This is not an unusual situation: applications often connect with outside resources to download data, query a database, or make a request to a web API, all of which take considerable time and can cause blocking behaviours. Delays like this are unacceptable.
The solution is to design software so that long-running tasks can be run in the background, or asynchronously. Kotlin has support for a number of mechanisms that support this. The most common approach involves manipulating threads.
Threads
A thread is the smallest unit of execution in an application. Typically an application will have a “main” thread, but we can also create threads to do other work that execute independently of this main thread. Every thread has its own instructions that it executes, and the processor is responsible for allocating time for each thread to run. .
Multi-threading is the idea of splitting up computation across threads – having multiple threads running in-parallel to complete work more quickly. This is also a potential solution to the blocking problem, since one thread can wait for the blocking operation to complete, while the other threads continue processing. Threads like this are also called background threads.
Info
Note from the diagram below that threads have a common heap, but each thread has its own registers and stack. Care must be taken to ensure that threads aren’t modifying the same memory at the same time! I’d recommend taking a course that covers concurrency in greater detail.
Concurrent vs. Parallel Execution
Concurrent Execution means that an application is making progress on more than one task at a time. This diagram illustrates how a single processor, handling one task at a time, can process them concurrently (i.e. with work being done in-order).
In this example, tasks 1 and 2 are being processed concurrently, and the single CPU is switching between them. Note that there is never a period of overlap where both are executing at the same time, but we have progress being made on both over the same time period.
Parallel Execution is where progress can be made on more than one task simultaneously. This is typically done by having multiple threads that can be addressed at the same time by the processor/cores involved..
In this example, both task 1 and task 2 can excecute through to completion without interfering with one another, because the system has multiple processors or cores to support parallel execution.
Finally, it is possible to have parallel and concurrent execution happening together. In this example, there are two processors/cores, each responsible for 2 tasks (i.e. CPU 1 handles task 1 and task 2, while CPU 2 handles task 3 and task 4). The CPU can slide up each task and alternate between them concurrently, while the other processor executes tasks in parallel.
So which do we prefer? Both have their uses, depending on the nature of the computation, the processor capabilities and other factors.
Parallel operations happen at the same time, and the operations logically do not interfere with one another. Parallel execution is typically managed by using threads to split computation into different units of execution that the processor can manage them independently. Modern hardware is capable of executing many threads simultaneously, although doing this is very resource intensive, and it can be difficult to manage threads correctly.
Concurrent tasks can be much easier to manage, and make sense for tasks where you need to make progress, but the tasks isn’t impeded by being interrupted occasionally.
Managing Threads
Kotlin has native support for creating and managing threads. This is done by
- Creating a user thread (distinct from the main thread where you application code typically runs).
- Defining some task for it to perform.
- Starting the thread.
- Cleaning up when it completes.
In this way, threads can be used to provide asynchronous execution, typically running in parallel with the main thread (i.e. running “in the background” of the program).
val t = object : Thread() {
override fun run() {
// define the task here
// this method will run to completion
}
}
t.start() // launch the thread, execution will continue here in the main thread
t.stop() // if we want to halt it from executing
Kotlin also provides a helper method that simplifies the syntax.
fun thread(
start: Boolean = true,
isDaemon: Boolean = false,
contextClassLoader: ClassLoader? = null,
name: String? = null,
priority: Int = -1,
block: () -> Unit
): Thread
// a thread can be instantiated quite simply.
thread(start = true) {
// the thread will end when this block completes
println("${Thread.currentThread()} has run.")
}
There are additional complexities of working with threads if they need to share mutable data: this a a significant problem. If you cannot avoid having threads share access to data, then read the Kotlin docs on concurrency first.
Threads are also very resource-intensive to create and manage. It’s temping to spin up multiple threads as we need them, and delete them when we’re done, but that’s just not practical most of the time.
Although threads are the underlying mechanism for many other solutions, they are too low-level to use directly much of the time. Instead we rely on other abstractions that leverage threads safely behind-the-scenes. We’ll present a few of these next.
Callbacks
Another solution is to use a callback function. Essentially, you provide the long-running function with a reference to a function and let it run on a thread in the background. When it completes, it calls the callback function with any results to process.
The code would look something like this:
fun postItem(item: Item) {
preparePostAsync { token ->
submitPostAsync(token, item) { post ->
processPost(post)
}
}
}
fun preparePostAsync(callback: (Token) -> Unit) {
// make request and return immediately
// arrange callback to be invoked later
}
This is still not an ideal solution.
- Difficulty of nested callbacks. Usually a function that is used as a callback, often ends up needing its own callback. This leads to a series of nested callbacks which lead to incomprehensible code.
- Error handling is complicated. The nesting model makes error handling and propagation of these more complicated.
Promises
Using a promise involves a long-running process, but instead of blocking it returns with a Promise - an object that we can reference immediately but which will be processed at a later time (conceptually like a data structure missing data).
The code would look something like this:
fun postItem(item: Item) {
preparePostAsync()
.thenCompose { token ->
submitPostAsync(token, item)
}
.thenAccept { post ->
processPost(post)
}
}
fun preparePostAsync(): Promise<Token> {
// makes request and returns a promise that is completed later
return promise
}
This approach requires a series of changes in how we program:
- Different programming model. Similar to callbacks, the programming model moves away from a top-down imperative approach to a compositional model with chained calls. Traditional program structures such as loops, exception handling, etc. usually are no longer valid in this model.
- Different APIs. Usually there’s a need to learn a completely new API such as
thenCompose or thenAccept, which can also vary across platforms.
- Specific return type. The return type moves away from the actual data that we need and instead returns a new type
Promise which has to be introspected.
- Error handling can be complicated. The propagation and chaining of errors aren’t always straightforward.
Coroutines
Kotlin’s approach to working with asynchronous code is to use coroutines, which are suspendable computations, i.e. the idea that a function can suspend its execution at some point and resume later on. By default, coroutines are designed to mimic sequentual behaviour and avoid by-default concurrency (i.e. it defaults to a simple case, and concurrency has to be explicitly declared).
Info
Many of the explanations and examples were taken directly from the Coroutines documentation. It’s worth looking at the original source for more details. The flight-data sample was taken from: Andrew Bailey, David Greenhalgh & Josh Skeen. 2021. Kotlin Programming: The Big Nerd Ranch Guide. 2nd Edition. Pearson. ISBN 978-0136891055.
Think of a coroutine as a light-weight thread. Like threads, coroutines can run in parallel, wait for each other and communicate. However, unlike threads, coroutines are not tied to one specific thread, and can be moved around as needed, making them very efficient. Also, compared to threads, coroutines are very cheap. We can easily create thousands of them with very little performance cost.
Coroutines are functions, but they behave differently than regular subroutines. Unlike subroutines which have a single point-of-entry, a coroutine may have multiple points-of-entry and may remember state between calls. This means that we can use coroutines to have cooperating functions, where control is passed back and forth between them. Coroutines can be suspended, or paused, while waiting on results, and the cooperating function can take over execution.
Here’s a quick-and-dirty example that spins up 1000 coroutines quite easily (don’t worry about the syntax yet).
import kotlinx.coroutines.*
fun main() = runBlocking {
repeat(1000) { // launch a lot of coroutines
launch {
delay(100L) // pause each coroutine for 100 ms
print(".") // print something to indicate that it's run
}
}
}
// ....................................................................................................
Kotlin provides the kotlinx.coroutines library with a number of high-level coroutine-enabled primitives. You will need to add the dependency to your build.gradle file, and then import the library.
// build.gradle
implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.0'
// code
import kotlinx.coroutines.*
A simple coroutine
Here’s a simple coroutine that demonstrates their use. This example is taken from the Kotlin Docs.
import kotlinx.coroutines.*
fun main() = runBlocking { // this: CoroutineScope
launch { // launch a new coroutine and continue
delay(1000L) // non-blocking delay for 1 second (default time unit is ms)
println("World!") // print after delay
}
println("Hello") // main coroutine continues while a previous one is delayed
}
//output
Hello
World
The section of code within the launch { } scope is delayed for 1 second. The program runs through the last line, prints “Hello” and then prints “World” after a delay.
- runBlocking is a coroutine builder that bridges the non-coroutine world of a regular
fun main() and the code with coroutines inside of the runBlocking { ... } curly braces. This is highlighted in an IDE by this: CoroutineScope hint right after the runBlocking opening curly brace.
- launch is also a coroutine builder. It launches a new coroutine concurrently with the rest of the code, which continues to work independently. That’s why
Hello has been printed first.
- delay is a special suspending function. It suspends the coroutine for a specific time. Suspending a coroutine does not block the underlying thread, but allows other coroutines to run and use the underlying thread for their code.
Info
If you remove or forget runBlocking in this code, you’ll get an error on the launch call, since launch is declared only in the CoroutineScope. Error: Unresolved reference: launch".
The name of runBlocking means that the thread that runs it (in this case — the main thread) gets blocked for the duration of the call, until all the coroutines inside runBlocking { ... } complete their execution. You will often see runBlocking used like that at the very top-level of the application and quite rarely inside the real code, as threads are expensive resources and blocking them is inefficient and is often not desired.
Suspending functions
We can extract the block of code inside launch { ... } into a separate function. When you perform “Extract function” refactoring on this code, you get a new function with the suspend modifier. This is your first suspending function. Suspending functions can be used inside coroutines just like regular functions, but their additional feature is that they can, in turn, use other suspending functions (like delay in this example) to suspend execution of a coroutine.
fun main() = runBlocking { // this: CoroutineScope
launch { doWorld() }
println("Hello")
}
// this is your first suspending function
suspend fun doWorld() {
delay(1000L)
println("World!")
}
// output
Hello
World!
In this case, the main() method runs doWorld() asynchronously in the background, then prints “Hello"before pausing and waiting for the runBlocking context to complete. This is the same behaviour that we had above, but the code is cleaner.
Structured concurrency
Coroutines follow a principle of structured concurrency which means that new coroutines can be only launched in a specific CoroutineScope which delimits the lifetime of the coroutine. The above example shows that runBlocking establishes the corresponding scope and that is why the previous example waits until everything completes before exiting the program.
In a real application, you will be launching a lot of coroutines. Structured concurrency ensures that they are not lost and do not leak. An outer scope cannot complete until all its children coroutines complete.
Coroutine Builders
A coroutine builder is a function that creates a new coroutine. Most coroutine builders also start the coroutine immediately after creating it. The most commonly used coroutine builder is launch, which takes a lambda argument, representing the function that will be executed.
Launch
launch will launch a new coroutine concurrently with the rest of the code, which continues to work independently.
Here’s an example that attempts to fetch data from a remote URL, which takes a few seconds to complete.
import kotlinx.coroutines.*
import java.net.URL
val ENDPOINT = "http://kotlin-book.bignerdranch.com/2e/flight"
fun fetchData(): String = URL(ENDPOINT).readText()
@OptIn(DelicateCoroutinesApi::class)
fun main() {
println("Started")
GlobalScope.launch {
val data = fetchData()
println(data)
}
println("Finished")
}
// output
Started
Finished
However, when we run this program, it completes immediately. This is because after the fetchData() function is called, the program continues executing and completes.
This is due to how the launch builder is designed to behave. Unfortunately running the entire program asynchronously isn’t really what we want. We actually want the fetchData() task to run to completion in the background, and the program halt and wait until that function is complete. To do this, we need a different builder that behaves differently.
runBlocking
runBlocking is also a coroutine builder. The name of runBlocking means that the thread that runs it (in this case — the main thread) gets blocked for the duration of the call, until all the coroutines inside runBlocking { ... } complete their execution.
The runBlocking function is a coroutine builder that blocks its thread until execution of its coroutine is complete. You can use runBlocking to launch coroutines that must all complete before execution continues. As you can see, it reaches the “Finished” statement, but pauses at the end of the scope until the fetchData() completes and returns data.
You can see the difference in behaviour here:
val ENDPOINT = "http://kotlin-book.bignerdranch.com/2e/flight"
fun fetchData(): String = URL(ENDPOINT).readText()
fun main() {
runBlocking {
println("Started")
launch {
val data = fetchData()
println(data)
}
println("Finished")
}
}
// output
Started
Finished
BI0262,ATQ,MXF,Delayed,115
In this case, the launch {} coroutine runs in the background fetching data, while the program continues running. After println("Finished") executes, and it’s at the end of the runBlocking scope, it halts and waits for the launch coroutine to complete before exiting the program.
coroutineScope
In addition to the coroutine scope provided by different builders, it is possible to declare your own scope using the coroutineScope builder. It creates a coroutine scope and does not complete until all launched children complete.
runBlocking and coroutineScope builders may look similar because they both wait for their body and all its children to complete. The main difference is that the runBlocking method blocks the current thread for waiting, while coroutineScope just suspends, releasing the underlying thread for other usages.
You can use coroutineScope from any suspending function. For example, you can move the concurrent printing of Hello and World into a suspend fun doWorld() function:
fun main() = runBlocking {
doWorld()
}
suspend fun doWorld() = coroutineScope { // this: CoroutineScope
launch {
delay(1000L)
println("World!")
}
println("Hello")
}
// output
Hello
World!
A coroutineScope builder can be used inside any suspending function to perform multiple concurrent operations. Let’s launch two concurrent coroutines inside a doWorld suspending function:
// Sequentially executes doWorld followed by "Done"
fun main() = runBlocking {
doWorld()
println("Done")
}
// Concurrently executes both sections
suspend fun doWorld() = coroutineScope { // this: CoroutineScope
launch {
delay(2000L)
println("World 2")
}
launch {
delay(1000L)
println("World 1")
}
println("Hello")
}
// output
Hello
World 1
World 2
Done
Managing a coroutine
A launch coroutine builder returns a Job object that is a handle to the launched coroutine and can be used to explicitly wait for its completion. For example, you can wait for completion of the child coroutine and then print “Done” string:
val job = launch { // launch a new coroutine and keep a reference to its Job
delay(1000L)
println("World!")
}
println("Hello")
job.join() // wait until child coroutine completes
println("Done")
In a long-running application you might need fine-grained control on your background coroutines. For example, a user might have closed the page that launched a coroutine and now its result is no longer needed and its operation can be cancelled. The launch function returns a Job that can be used to cancel the running coroutine:
val job = launch {
repeat(1000) { i ->
println("job: I'm sleeping $i ...")
delay(500L)
}
}
delay(1300L) // delay a bit
println("main: I'm tired of waiting!")
job.cancel() // cancels the job
job.join() // waits for job's completion
println("main: Now I can quit.")
// output
job: I'm sleeping 0 ...
job: I'm sleeping 1 ...
job: I'm sleeping 2 ...
main: I'm tired of waiting!
main: Now I can quit.
As soon as main invokes job.cancel, we don’t see any output from the other coroutine because it was cancelled.
Composing suspending functions
Sequential (default)
Assume that we have two suspending functions defined elsewhere that do something useful like some kind of remote service call or computation. What do we do if we need them to be invoked sequentially — first doSomethingUsefulOne and then doSomethingUsefulTwo, and compute the sum of their results? We use a normal sequential invocation, because the code in the coroutine, just like in the regular code, is sequential by default.
suspend fun doSomethingUsefulOne(): Int {
delay(1000L) // pretend we are doing something useful here
return 13
}
suspend fun doSomethingUsefulTwo(): Int {
delay(1000L) // pretend we are doing something useful here, too
return 29
}
val time = measureTimeMillis {
val one = doSomethingUsefulOne()
val two = doSomethingUsefulTwo()
println("The answer is ${one + two}")
}
println("Completed in $time ms")
// output
The answer is 42
Completed in 2017 ms
Concurrent (async)
What if there are no dependencies between invocations of doSomethingUsefulOne and doSomethingUsefulTwo and we want to get the answer faster, by doing both concurrently? Use async, another builder.
Conceptually, async is just like launch. It starts a separate coroutine which is a light-weight thread that works concurrently with all the other coroutines. The difference is that launch returns a Joband does not carry any resulting value, while async returns a Deferred — a light-weight non-blocking future that represents a promise to provide a result later. You can use .await() on a deferred value to get its eventual result, but Deferred is also a Job, so you can cancel it if needed.
val time = measureTimeMillis {
val one = async { doSomethingUsefulOne() }
val two = async { doSomethingUsefulTwo() }
println("The answer is ${one.await() + two.await()}")
}
println("Completed in $time ms")
// output
The answer is 42
Completed in 1017 ms
For more examples, the Kotlin Docs Coroutine Tutorial is highly recommended!
Data & Databases
Overview
Most software operates on data. This might be a primary function of your application (e.g. image editor) or a secondary function (e.g. user preferences). As a software developer, you need to give considerable thought to how you will handle user data.
Some examples of data that you might care about:
- The data that the user has captured and wishes to save e.g. an image that they have created in an image-editor, or the text from a note.
- The position and size of the application window, window size, or other settings that you might want to save and restore when the application is relaunched.
- User preferences that they might set in your application e.g. preferred font or font size.
- The public key to connect to a remote machine.
- The software license key for your application.
Operations you may need to do:
- Store data in a file so that you can load and restore it later.
- Transfer the data to another process that can use it, or to a completely different system e.g. over the internet.
- Filter it to a subset of your original data. Sort it. Modify it.
Let’s start by reviewing something we already know - data types.
Data Types
A type is a way of categorizing our data so that the computer knows how we intend to use it. There are many different kinds of types that you already know how to work with:
-
Primitive Types: these are intrinsically understood by a compiler, and will include boolean, and numeric data types. e.g. boolean, integer, float, double.
-
Strings: Any text representation which can include characters (char) or longer collections of characters (string). Strings are complicated to store and manipulate because there are a large number of potential values for each character, and the strings themselves are variable length e.g. they can span a few characters to many thousands of words in a single string!
-
Compound data types: A combination of primitive types. e.g. an Array of Integers.
-
Pointers: A data type whose value points to a location in memory. The underlying data may resemble a long Integer, but they are treated as a separate type to allow for specific behaviours to help protect our programs e.g. to prevent invalid memory access or invalid operations.
-
Abstract data type: We treat ADTs as different because they describe a structure, and do not actually hold any concrete data. We have a template for an ADT (e.g. class) and concrete realizations (e.g. objects). These can be singular, or stored as Compound types as well e.g. an Array of Objects.
You’re accustomed to working with most of these types – we declare variables in a programming language using the types for that language. For example, in Kotlin, we can assign the value 4 to different variables, each representing a different type. Note that we’ve had to make some adjustments to how we represent the value so that they match the expected type, and avoid compiler errors:
val a:Int = 4
val b:Double = 4.0
val c:String = "4"
By using different types, we’ve made it clear to the compiler that a and b do not represent the same value.
>>> a==b
error: operator '==' cannot be applied to 'Int' and 'Double'
Types have properties and behaviours that are specific to that type. For example, we can determine the length of c, a String, but not the length of a, an Int. Similarly, we can perform mathematical operations on some types but not others.
>>> c.length
res13: kotlin.Int = 1
>>> a.length
error: unresolved reference: length
a.length
^
>>> b/4
res15: kotlin.Double = 1.0
>>> c/4
error: unresolved reference. None of the following candidates is applicable because of receiver type mismatch:
public inline operator fun BigDecimal.div(other: BigDecimal): BigDecimal defined in kotlin
public inline operator fun BigInteger.div(other: BigInteger): BigInteger defined in kotlin
c/4
^
In order for our programs to work with data, it needs to be stored in one of these types: either a series of primitives or strings, or a collection of these types, or an instance of an ADT (i.e. an Object).
Info
Keep in mind that this is not the actual data representation, but the abstraction that we are using to describe our data. In memory, a string might be stored as a continuous number of bytes, or scattered through memory in chunks - for this discussion, that doesn’t matter. We’re relying on the programming language to hide away the underlying details.
Data can be simple, consisting of a single field (e.g. the user’s name), or complex, consisting of a number of related fields (e.g. a customer with a name, address, job title). Typically we group related data into classes, and class instances are used to represent specific instances of that class.
data class Customer(id:Int, name:String, city:String)
val new_client = Customer(1001, "Jane Bond", "Waterloo")
Data items can be singular (e.g. one particular customer), or part of a collection (e.g. all of my customers). A singular example would be a custom record.
val new_client1 = Customer(1001, "Jane Bond", "Waterloo")
val new_client2 = Customer(1002, "Bruce Willis", "Kitchener")
A bank might also keep track of transactions that a customer makes, where each transaction represents the deposit or withdrawal, the date when it occurred, the amount and so on. [ed. This date format is ISO 8601, a standard date/time representation.]
To represent this in memory, we might have a transaction data class, with individual transactions being stored in a collection.
data class Tx(id:Int, date:String, amount:Double, currency: String)
val transactions = mutableList()
transactions.add(Tx(1001, "2020-06-06T14:35:44", "78.22", "CDN"))
transactions.add(Tx(1001, "2020-06-06T14:38:18", "12.10", "USD"))
transactions.add(Tx(1002, "2020-06-06T14:42:51", "44.50", "CDN"))
These structures represent how we are storing this data in this particular case. Note that there may be multiple ways of representing the same data, we’ve just chosen one that makes sense to use.
Data Models
A data model is a abstraction of how our data is represented and how data elements relate to one another. We will often have a canonical data model, which we might then use as the basis for different representations of that data.
There are different forms of data models, including:
-
Database model: describes how to structure data in a database (flat, hierarchical, relational).
-
Data structure diagrams (DSD): describes data as entities (boxes) and their relationships (arrows connecting them).
-
Entity-Relationship Model: Similar to DSDs, but with some notational differences. Don’t scale out very well.
Data structure diagrams aren’t commonly used to diagram a complete system, but can be used to show how entities relate to one another. They can range in complexity from very large and formals diagrams, to quick illustrative sketches that just show the relationships between different entities.
Here’s a data structure diagram, showing different entities. These would likely be converted to multiple classes, each one responsible for their own data.
Info
We also use these terms when referring to complex data structures:
- field: a particular piece of data, corresponding to variables in a program.
- record: a collection of fields that together comprise a single instance of a class or object.
Data Representation
One of the challenges is determining how to store our data for different purposes.
Let’s consider our customer data. We can represent this abstractly as a Customer class. In code, we can have instances of Customers. How do we store that data, or transfer it to a different system?
The “easy” answer would be to share the objects directly, but that’s often not realistic. The target system would need to be able to work with that object format directly, and it’s likely not a very robust (or safe) way of transmitting your data.
Often we need to convert our data into different representations of the same data model.
- We need to ensure that we don’t lose any data
- We need to maintain the relationships between data elements.
- We need to be able to “convert back” as needed.
In the diagram below, you can see this in action. We might want to save our customer data in a database, or into a data file for backup (or transmission to a different system).
Part of the challenge in working with data is determining what you need to do with it, and what data representation will be appropriate for different situations. As we’ll see, it’s common to need to convert and manage data across these different representations.
Data Files
A file format is a standard way of encoding data in a file. There are a large number of predefined file formats that have been designed and maintained by different standards bodies. If you are working with data that matches a predefined format, then you should use the correct file format for that data! Examples of standard file formats include HTML, Scalable vector graphics (SVG), MPEG video files, JPEG images, PDF documents and so on.
If you are working with your own data (e.g. our Customer data), then you are free to define your own encoding.
A fundamental distinction is whether you want text or binary encoding:
- Text encoding is storing data as a stream of characters in a standard encoding scheme (e.g. UTF8). Text files have the advantage of being human-readable, and can be easy to process and debug. However, for large amounts of data, they can also be slow to process.
- Binary encoding allows you to store data in a completely open and arbitrary way. It won’t be human readable, but you can more easily store non-textual data in an efficient manner e.g. images from a drawing program.
Kotlin has libraries to support both. You can easily write a stream of characters to a text file, or push a stream of bytes to a binary file. We will explore how to do both in the next section.
A “rule of thumb” is that non-private data, especially if realtively small, should often be stored in a text file. The ability to read the file for debugging purposes is invaluable, as is the ability to edit the file in a standard text editor. This is why Preferences files, Log files and other similar data files are stored in human-readable formats.
Private data, or data that is difficult to process is probably better served in a binary format. It’s not human-readable, but that’s probably what you want if the data is sensitive. As we will discover, it can also be easier to process. We’ll discuss this below.
Character Encoding
Writing text into a file isn’t as simple as it sounds. Like other data, characters in memory are stored as binary i.e. numeric values in a range. To display them, we need some agreement on what number represents each character. This sound like a simple problem but it’s actually very tricky, due to the number of different characters that are in use around the world.
Our desire to be able to encode all characters in all language is balanced by our need to be efficient. i.e. we want to use the least number of bytes per character that we’re storing.
In the early days of computing, we used US-ASCII as a standard, which stored each character in 7 bits (range of 0-127). Although it was sufficient for “standard” English language typewriter symbols, this is rarely used anymore since it cannot handle languages other than English, nor can it handle very many extra symbols.
The Unicode standard is used for modern encoding. Every character known is represented in Unicode, and requires one or more bytes to store (i.e. it’s a multi-byte format). UTF-8 refers to unicode encoding for those characters which only require a single byte (the 8 refers to 8-bit encoding). UTF-16 and UTF-32 are used for characters that require 2 and 4 bytes respectively.
UTF-8 is considered standard encoding unless the data format required additional complexity.
Structuring Text Files
We know that we will use UTF-8, but that only describes how the characters will be stored. We also need to determine how to structure our data in a way that reflects our data model. We’ll talk about three different data structure formats for managing text data, all of which will work with UTF-8.
Comma-separated values (CSV)
The simplest way to store records might be to use a CSV (comma-separated values) file.
We use this structure:
- Each row corresponds to one record (i.e. one object)
- The values in the row are the fields separated by commas.
For example, our transaction data file stored in a comma-delimited file would look like this:
1001, 2020-06-06T14:35:44, 78.22, CDN
1001, 2020-06-06T14:38:18, 12.10, USD
1002, 2020-06-06T14:42:51, 44.50, CDN
CSV is literally the simplest possible thing that we can do, and sometimes it’s good enough.
It has some advantages:
- its extremely easy to work with, since you can write a class to read/write it in a few lines of code.
- it’s human readable which makes testing/debugging much easier.
- its fairly space efficient.
However, this comes with some pretty big disadvantages too:
- It doesn’t work very well if your data contains a delimiter (e.g. a comma).
- It assumes a fixed structure and doesn’t handle variable length records.
- It doesn’t work very well with complex or multi-dimensional data. e.g. a Customer class.
// how do you store this as XML? the list is variable length
data class Customer(id:Int, name:String, transactions:List<Transactions>)
Data streams are used to provide support for reading and writing “primitives“ from streams. They are very commonly used.
- File: FileInputStream, FileOutputStream
- Data: DataInputStream, DataOutputStream
var file = FileOutputStream("hello.txt")
var stream = DataOutputStream(file)
stream.writeInt(100)
stream.writeFloat(2.3f)
stream.writeChar('c')
stream.writeBoolean(true)
We can use streams to write our transaction data to a file quite easily.
val filename = "transactions.txt"
val delimiter = "," // comma-delimited values
// add a new record to the data file
fun append(txID:Int, amount:Float, curr:String = "CDN") {
val datetime = LocalDateTime.now()
File(filename).appendText("$txID $delimiter $datetime $delimiter $amount\n", Charsets.UTF_8)
}
Extensible Markup Language (XML)
XML is a human-readable markup language that designed for data storage and transmission.
Defined by the World Wide Web Consortium’s XML specification, it was the first major standard for markup languages. It’s structurally similar to HTML, with a focus on data transmission (vs. presentation).
- Structure consists of pairs of tags that enclose data elements:
<name>Jeff</name>
- Attributes can extend an element:
<img src="madonna.jpg"></img>
Example of a music collection structured in XML [ed. If you don’t know these albums, you should look them up. Steve Wonder is a musical genius. Dylan is, well, Dylan.]
An album is a record, and each album contains fields for title, artist etc.
<catalog>
<album>
<title>Empire Burlesque</title>
<artist>Bob Dylan</artist>
<country>USA</country>
<company>Columbia</company>
<price>10.90</price>
<year>1985</year>
</album>
<album>
<title>Innervisions</title>
<artist>Stevie Wonder</artist>
<country>US</country>
<company>The Record Plant</company>
<price>9.90</price>
<year>1973</year>
</album>
</catalog>
XML is useful, but doesn’t handle repeating structured particularly well. It’s also verbose when working with large amount of data. Finally, although it’s human-readable, it’s not particularly easy to read.
We’ll talk about processing XML in a moment. First, let’s talk about JSON.
JavaScript Object Notation (JSON)
JSON is an open standard file and data interchange format that’s commonly used on the web.
JSON consists of attribute:value pairs and array data types. It’s based on JavaScript object notation, but is language independent. It was standardized in 2013 as ECMA-404.
JSON has become extremely popular due to its simpler syntax compared to XML.
- Data elements consist of name/value pairs
- Fields are separated by commas
- Curly braces hold objects
- Square brackets hold arrays
Here’s the music collection in JSON.
{ "catalog": {
"albums": [
{
"title":"Empire Burlesque",
"artist":"Bob Dylan",
"country":"USA",
"company":"Columbia",
"price":"10.90",
"year":"1988"
},
{
"title":"Innervision",
"artist":"Stevie Wonder",
"country":"US",
"company":"The Record Plant",
"price":"9.90",
"year":"1973"
}
]
}}
Advantages of JSON:
- Simplifying closing tags makes JSON easier to read.
{ "employees":[
{ "first":"John", "last":"Zhang", "dept":"Sales"},
{ "first":"Anna", "last":"Smith", "dept":"Engineering"}
]}
Compare this to the corresponding XML:
<employees>
<employee><first>John</first> <last>Zhang</last> <dept>Sales</dept></employee>
<employee><first>Anna</first> <last>Smith</last> <dept>Engineering</dept></employee>
</employees>
- JSON also handles arrays better.
Array in XML:
<name>Celia</name>
<age>30</age>
<cars>
<model>Ford</model>
<model>BMW</model>
<model>Fiat</model>
</cars>
Array in JSON:
{
"name":"Celia",
"age":30,
"cars":[ "Ford", "BMW", "Fiat" ]
}
So, our application data resides in data structures, in memory. How do we make use of JSON or XML?
- To save: we convert objects into XML or JSON format, then save the raw XML or JSON in a data file.
- To restore: we load XML or JSON from our data files, and instantiate objects (records) based on the file contents.
Although you could write your own parser, there are a number of libraries out there than handle conversion to and from JSON quite easily.
XML parsing can be addressed by a number of parsers:
We’ll focus on using Kotlin’s serialization libraries to convert our objects to and from JSON directly!
Serialization
Info
Serialization is the process of converting your program to a binary stream, so that you can transmit it, or persist it to a file or database. Deserialization is the process of converting the stream back into an object.
This is really cool technology that we can use to save our objects directly without trying to save the indivual property values (like we would for a CSV file).
To include the serialization libraries in your project, add these dependencies to the build.gradle file.
plugins {
id 'org.jetbrains.kotlin.multiplatform' version '1.6.10'
id 'org.jetbrains.kotlin.plugin.serialization' version '1.6.10'
}
Also make sure to add the dependency:
dependencies {
implementation "org.jetbrains.kotlinx:kotlinx-serialization-json:1.3.2"
}
You can then save your objects directly as a stream which you can save to a file, or a database. You can later use this stream to recreate your objects. See https://github.com/Kotlin/kotlinx.serialization for details.
Example from: https://blog.jetbrains.com/kotlin/2020/10/kotlinx-serialization-1-0-released/
@Serializable
data class Project(
val name: String,
val owner: Account,
val group: String = "R&D"
)
@Serializable
data class Account(val userName: String)
val moonshot = Project("Moonshot", Account("Jane"))
val cleanup = Project("Cleanup", Account("Mike"), "Maintenance")
fun main() {
val string = Json.encodeToString(listOf(moonshot, cleanup))
println(string)
// [{"name":"Moonshot","owner":{"userName":"Jane"}},{"name":"Cleanup","owner":
// {"userName":"Mike"},"group":"Maintenance"}]
val projectCollection = Json.decodeFromString<List<Project>>(string)
println(projectCollection)
// [Project(name=Moonshot, owner=Account(userName=Jane), group=R&D),
// Project(name=Cleanup, owner=Account(userName=Mike), group=Maintenance)]
}
Note that to make this work, you have to annotate your class as @Serializable. It also needs to contain data that the compiler can convert to JSON.
Databases
A more scalable solution is to store data in a database: a formal system for organizing and managing data.
Databases range in size and complexity from simple file-based databases that run on your local computer, to large scalable databases that run on a server cluster. They are optimized for fetching text and numeric data, and performing set operations on this data.
They also have the advantage of scaling really well. They’re optimized not just for efficient storage and retrieval of large amounts of data, but also for concurrent access by hundreds or thousands of users simulaneously [ed. This is a huge topic and we’re not even scratching the surface. I’d strongly encourage you to take a more comprehensive database course e.g. CS 348.]
A relational database is a particular database design that structures data as tables:
- A table represents some logical entity e.g. Customer, Transactions. It consists of columns and rows i.e. like a grid.
- A column is a field or data element in that table. e.g. a “Customer“ table might have “name“, “city“, “birthdate“ fields.
- A row is a record, containing values for each field. e.g. ”Jeff Avery”, ”Waterloo”, ”June 23, 1985”. Each row in each table needs to be unique identified by a key. This can be a part of the data (e.g. timestamp) but is often a unique, generated numeric identifier (e.g. “customerid“, “transactionid“).
There are many other types of databases! We’ll use relational databases in this course b/c (a) they’re very common, and (b) they are suitable for any type of data/relations that you will encounter in this course.
Here’s our earlier example shown as a class, a set of records in a CSV file and as a set of relational database tables.
Why is this approach valuable? Relational databases can quickly oprerate on these tables, and support operations on sets of records. For example:
- Return a list of all purchases greater than $100.
- Return a list of customers from ”Paris”.
- Delete customers from ”Ottawa”.
Our example is pretty trivial, but imagine useful queries like:
- ”Find all transactions between 2:00 and 2:30”, or
- ”Find our which salesperson sold the greatest amount during last Saturday’s sale”.
What is SQL?
SQL (pronounced ”Ess-que-ell”) is a Domain-Specific Language (DSL) for describing your queries. Using SQL, you write statements describing the operation to perform and which tables to use, and the database performs the operations for you.
SQL is a standard, so SQL commands work across different databases [ed. SQL was adopted as a standard by ANSI in 1986 as SQL-86, and by ISO in 1987].
Using SQL, you can:
- C reate new records
- R etrieve sets of existing records
- U pdate the fields in one or more records
- D elete one or more records
SQL has a particular syntax for managing sets of records:
<operation> FROM [table] [WHERE [condition]]
operations: SELECT, UPDATE, INSERT, DELETE, ...
conditions: WHERE [col] <operator> <value>
For example:
/* SELECT returns data from a table, or a set of tables.
* an asterix (*) means "all"
*/
SELECT * FROM customers
SELECT * FROM Customers WHERE city = "Ottawa"
SELECT name FROM Customers WHERE custid = 1001
/* UPDATE modifies one or more column values based on some criteria.*/
UPDATE Customer SET city = "Kitchener" WHERE cust_id = 1001
UPDATE Customer SET city = "Kitchener" // uh oh, what is wrong with this?
/* INSERT adds new records to your database. */
INSERT INTO Customer(cust_id, name, city)
VALUES ("1005", "Sandra Avery", "Kitchener")
INSERT INTO Customer(cust_id, name, city)
VALUES ("1005", "Austin Avery", "Kitchener") // uh oh, what have I done?
It’s common to have a record spread across multiple tables. A join describes how to relate data across tables.
SELECT c.customer_id, c.first_name + “ “ + c.last_name, t.date, p.name, p.cost
FROM Customer c, Transactions t, Products p
WHERE c.customer_id = t.customer_id
AND t.product_id = p.product_id
Returns data that spans multiple tables:
$ 1001, Jeff Avery, 12-Aug-2020, T-shirt, 29.95
https://imgur.com/t/logic/v23nUwQ
SQLite
Relational databases are often large and complex systems. However, the relational approach can also be scaled-down.
SQLite (pronounced ESS-QUE-ELL-ITE) is a small-scale relational DBMS that supports SQL. It is small enough for local, standalone use.
SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine. SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers…"
https://www.sqlite.org/index.html
You can install the SQLite database u
