# User Interfaces

Graphical applications aren't restricted to desktop computing; graphical user interfaces exist on all modern computing devices, from smartwatches, to phones and tablets, to car entertainment systems.

Mobile and desktop toolkits solve similar problems for their respective platforms, while also addressing the unique challenges of each one (e.g. touch input is a major smartphone-specific feature, but almost irrelevant for desktop environments).

A widget or UI toolkit is a framework that provides support for building applications. Essentially, toolkits provide an abstraction of underlying operating system functionality, with a focus on application features. e.g. graphics, sound, reusable widgets and events.

Common features include:

• Creating and managing application windows, with standard window functionality e.g. overlapping windows, depth, min/max buttons, resizing. This is more important on desktop than mobile, although mobile toolkits are expanding to include mobile-specific windowing functionality.

• Graphical output, including 2D graphics, animations, sound and other methods of communicating information to the user.

• Providing reusable components called widgets that can be used to assemble a typical applications. e.g. buttons, lists, toolbars, images, text views. Promoting common components ensures that applications on that platform have common interaction mechanisms i.e. that they "look and feel similar", which is beneficial for users.
• Support for an event-driven architecture, where events (messages) can be published and circulated through your application. This is the primary mechanism that we use to intercept and handle user input, or other system messages e.g., indicating that your phone has changed orientation, or that a window has closed.

When building applications with toolkits and widgets, a developer needs to write code to control the appearance and position of these widgets, as well as code to handle user input. A lot of the complexity with this model is ensuring that application state is managed properly across UI components, business-objects, and models (e.g. if a user updates something on-screen, you need to make sure that the data is updated everywhere in your application, including possibly other windows that show that data).

This is how imperative toolkits like JavaFX and Qt work. The developer has to write the "glue code" that tells the system how to update the user interface in response to state changes, either from the user interacting with on-screen widgets, or from external events received by the application. This is the source of a lot of complexity in user interface development, and a frequent cause of errors.

By contrast, a declarative toolkit automatically manages how the UI reacts to state changes. The developer focuses on describing what state is required, and how state is used to initialize on-screen components, but doesn't need to write any glue-code. As state changes occur in your application, the UI is automatically changed to reflect that state. This technique works by conceptually regenerating the entire screen from scratch, and then applying any changes that are required to reflect state. The result is a simpler conceptual model for developers.

A window is simply a region of the screen that "belongs" to a specific application. Typically one application has one main window, but it may also own and control additional windows. These are overlayed on a "desktop", which is really just the screen background.

To manage different windows across many different applications, a part of the operating system called a windowing system is responsible for creating, destroying and managing running windows. The windowing system provides an API to applications to support for all window-related functionality, including:

• provide an mechanism for applications to create, or destroy their own windows
• handle window movement automatically and invisibly to the application (i.e. when you drag a window, the windowing system moves it).
• handles overlapping windows across applications (e.g. so that your application window can be brought to the ""front" and overlap another application's window).

Typically, the toolkit will provide an API that allows the developer to pass through requests to the windowing system to create and manage windows. Often, this includes window methods properties that can be manipulated to control window behaviour.

• Sample class: Stage, Window.
• Sample properties: minWidth, prefWidth, maxWidth; minHeight, prefHeight, maxHeight; title; isFullScreen; isResizable
• Sample methods: close(), toFront(), toBack()

Graphical output is a broad category that includes drawing and positioning elements on-screen. This can include adding arbitrary elements (e.g. circles, rectangles or other primitives), structured data (e.g. PNG or JPG images, MP4 video) or reusable widgets to the window.

We'll briefly cover standard concepts before discussing specifics of any of these.

A computer screen uses a Cartesean coordinate system to track window position. By convention, the top-left is the origin, with x increasing as you move right, and y increasing as you move down the screen. The bottom-right corner of the screen is maximum x and y, which equals the resolution of the screen¹.

Note that its possible for screen contents to move moved out-of-bounds and made inaccessible. We typically don't want to do this.

In the example below, you can see that this is a 1600x1200 resolution screen², with the four corner positions marked. It contains a single 400x400 window, positioned at (500, 475) using these global coordinates.

Given that the windowing system manages movement and positioning of windows on-screen, an application window doesn't actually know where it's located on-screen! The application that "owns" the window above doesn't have access to it's global coordinates (i.e. where it resides on the screen). It does however, have access to it's own internal, or local coordinates. For example, our window might contain other objects, and the application would know about their placement. In this local coordinate system, we use the top of the window as the origin, with the bottom-right coordinate being the (width, height) of the window. Objects within the window are referenced relative to the window's origin.

We're going to refer to graphical on-screen elements as widgets. Most toolkits support a large number of similar widgets. The diagram below shows one desktop toolkit with drop-down lists, radio buttons, lists and so on. All of these elements are considered widgets.

Typically, using widgets us as simple as instantiating them, adding them to the window, and setting up a mechanism to detect when users interact with them so that appropriate actions can be taken.

It's standard practice in graphical applications to represent the interface as a scene graph. This is a mechanism for modelling a graphical application as a tree of nodes (widgets), where each node has exactly one parent. Effects applied to the parent are applied to all of its children.

Toolkits support scene graphs directly. There is typically a distinction made between Container widgets and Node widgets. Containers are widgets that are meant to hold other widgets e.g. menus which hold menu_items, toolbars which can hold toolbar_buttons and so on. Nodes are widgets that are interactive and don't have any children.

Building a UI involves explicitly setting up the scene graph by instantiating nodes, and adding them to containers to build a scene graph. (For this reason, containers will always have a list of children, and a mechanism for adding and removing children from their list).

Layout is the mechanism by which nodes in the scene graph are positioned on the screen, and managed if the window is resized.

• Fixed layout is a mechanism to place widgets in a static layout where the window will remain the same size. This is done by setting properties of the widgets to designate each one's position, and ensuring that the containers do not attempt any dynamic layout.
• Relative layout delegates responsibility for widget position and size to their parents (i.e. containers). Typically this means setting up the scene graph in such a way that the appropriate container is used based on how it adjusts position. Typical containers include a vertical-box that aligns it's children in a vertical line, or a grid that places children in a grid with fixed rows and columns.

Applications often handle multiple types of processing: asynchronous, such as when a user types a few keystrokes, or synchronous, such as when we want a computation to run non-stop to completion.

User interfaces are designed around the idea of using events or messages as a mechanism for components to indicate state changes to other interested entities. This works well, due to the asynchronous nature of user-driven interaction, where there can be relatively long delays between inputs (i.e. humans type slowly compared to the rate at which a computer can process the keystrokes).

This type of system, designed around the production, transmission and consumption of events between loosely-coupled components, is called an Event-Driven Architecture. It's the foundation to most user-interface centric applications (desktop, mobile), which common use messages to signal a user's interaction with a viewable component in the interface.

What’s an event? An event is any significant occurrence or change in state for system hardware or software.

The source of an event can be from internal or external inputs. Events can generate from a user, like a mouse click or keystroke, an external source, such as a sensor output, or come from the system, like loading a program.

How does event-driven architecture work? Event-driven architecture is made up of event producers and event consumers. An event producer detects or senses the conditions that indicate that something has happened, and creates an event.

The event is transmitted from the event producer to the event consumers through event channels, where an event processing platform processes the event asynchronously. Event consumers need to be informed when an event has occurred, and can choose to act on it.

Events be generated from user actions, like a mouse click or keystroke, an external source, such as a sensor output, or come from the system, like loading a program.

An event driven system typically runs an event loop, that keeps waiting for these events. The process is illustrated in the diagram below:

1. An EventEmitter generates an event.
2. The event is placed in an event queue.
3. An event loop peels off events from the queue and dispatches them to event handlers (functions which have been registered to receive this specific type of event).
4. The event handlers receive and process the event.

To handle event driven architectures, we often subdivide application responsibility into separate components.

The most basic UI structure is Model-View-Controller (MVC), which leverages the Observer design pattern to separate business logic from the user interface.

MVC divides any application into three distinct parts:

• Model: the core component of the application that handles state ("business logic layer").
• View: a representation of the application state, often as a user-interface ("presentation layer")
• Controller: a component that accepts input, interprets user actions and converts to commands for the model or view.

Similar to the observer pattern, the views monitor the system state, represented by the model. When the state changes, the views are notified and they update their data to reflect these changes. Notifications frequently happen through events generated by, and managed by, the toolkit that you're using.

Often this is realized as separate classes for each of these components, with an additional main class to bind everything together.

// main class
class Main {
  val model = Model()
  val controller = Controller(model)
  val view = View(controller, model)
	model.addView(model) 
}

We use an interface to represent the views, which provides the flexibility to allow many different types of output for the program. Any class can be a view as long as it supports the appropriate method to allow notifications from the model.

There have been many variations on MVC. We'll present a couple of common ones below.

interface IView {
  fun update() 
}

class View(val controller: Controller, val model: Model): IView {
  override fun update() {
    // fetch data from model
  }
}

The model maintains a list of all views, and notifies them with state changes (indicating that they may wish to refresh their data, or respond to the state change in some way).

class Model {
  val views = listOf()
  fun addView(view: IView) {
    views.add(view)
  }
  fun update() {
    for (view : views) {
      view.update()
    }
  }
}

The controller just passes input from the user to the model.

class Controller(val model: Model) {
  fun handle(event: Event) {
    // pass event data to model
  }
}

One issue with this version of MVC is that the controller often serves little purpose, except to pass along events that are captured by the View (the View contains the user-interface and widgets, and generates events as the user interacts with it).

MVC remains common for simple applications, but tends to be implemented as just a model and one or more views, with the controller code included in the view itself.

Model-View-Presenter (MVP) keeps the key concept in MVC - separating the business logic from the presentation - and introduces an intermediate Presenter which handles converting the model's data into a useful format for the views. This is typically done explicitly by the Presenter class. This variation is also popular when you have multiple views that you wish to control centrally (e.g. subscreens where the Presenter is responsible for determining which screen is active).

MVP arose from Taligent in the 1990s, but was popularized by Martin Fowler around 2006.

Model-View-ViewModel was invented by Ken Cooper and Ted Peters to simplify event-driven programming of user interfaces in C#/.NET. It's similar to MVP, but includes the notion of binding variables to widgets within the framework, so that changes in widget state are are automatically propagated from the view to other components.

MVVM includes the following components:

• Model: as MVC, the core component that handles state. It can also map to a data access layer or database directly.
• View: a representation of the application state, presented to the user.
• ViewModel: a model that specifically interprets the underlying Model state for the particular view to which it is associated. Typically we rely on binding to map variables in the ViewModel directly to widgets in the View, so that updating one directly updates the other.

MVVM is common in modern languages and toolkits and has the advantage of replacing all "mapping" code with direct binding of variables and widgets by the toolkit. This greatly simplifies interface development.

There are many other variants (e.g. Model-View-ViewController) which deviate from "standard" MVC, usually in an attempt to solve a particular problem in a more elegant fashion. They all build on the same observer foundation.

MVC and it's variants are really meant to accomplish two things:

1. Enforce separation of concerns. "Business logic" or application data is stored in the model, and the view can make decisions on how to present that data, and how the user should interact with it.
2. Handle state management. It provides a strict division of state, so that the model is meant to contain the "source of truth", and it establishes rules on how state should be changed and propagated through your application.

However, these are difficult questions to address, and MVC designs that appear reasonable, will often feel inelegant as applications grow.

How do we reconcile the Clean Architecture model with MVC and it's variants? If you put them side-by-side, you can see that there are similarities in the way they are structured.

Both models are an attempt to force dependencies down towards the models or entities. In other words, the external layers This helps to ensure that the business logic is separated from the UI, and that the UI is as simple as possible. This is essential for control-flow, or flow of input through the application.

See the Clean UI sample posted alongside the course slides. This is a simple example of how you might structure a UI in a Clean Architecture style.

classDiagram
    View "1" ..> "1" UserController
    UserController "*" ..> "1" UserModel 
    
    ISubscriber "1" <|.. "1" UserViewModel
    IPublisher <|.. UserModel
    ISubscriber "*" <.. "*" IPublisher

    View "1" <-- "1" UserViewModel 
    UserViewModel "*" <-- "*" UserModel
    
    class View {
        -UserController controller    
        -UserViewModel viewModel
    }
    
    class ISubscriber {
        <<Interface>>
        +update()    
    }  
    
    class UserViewModel {
        -View view
        -UserModel model
        +update()
    }
    
    class UserController {
        +UserModel model
        +invoke(Event)
    }

    class IPublisher {
        <<Interface>>
        -List~Subscriber~ subscribers
        +notify()
    }
    
    class UserModel {
        -String firstname
        -String lastname
        +subscribe(ISubscriber)
        +unsubscribe(ISubscriber)
    }