#User Interfaces

Graphical applications aren't restricted to desktop computing; graphical user interfaces exist on all modern computing devices, from smartwatches, to phones and tablets, to car entertainment systems.

Mobile and desktop toolkits solve similar problems for their respective platforms, while also addressing the unique challenges of each one (e.g. touch input is a major smartphone-specific feature, but almost irrelevant for desktop environments).

A widget or UI toolkit is a framework that provides support for building applications. Essentially, toolkits provide an abstraction of underlying operating system functionality, with a focus on application features. e.g. graphics, sound, reusable widgets and events.

Common features include:

• Creating and managing application windows, with standard window functionality e.g. overlapping windows, depth, min/max buttons, resizing. This is more important on desktop than mobile, although mobile toolkits are expanding to include mobile-specific windowing functionality.

• Graphical output, including 2D graphics, animations, sound and other methods of communicating information to the user.

• Providing reusable components called widgets that can be used to assemble a typical applications. e.g. buttons, lists, toolbars, images, text views. Promoting common components ensures that applications on that platform have common interaction mechanisms i.e. that they "look and feel similar", which is beneficial for users.
• Support for an event-driven architecture, where events (messages) can be published and circulated through your application. This is the primary mechanism that we use to intercept and handle user input, or other system messages e.g., indicating that your phone has changed orientation, or that a window has closed.

When building applications with toolkits and widgets, a developer needs to write code to control the appearance and position of these widgets, as well as code to handle user input. A lot of the complexity with this model is ensuring that application state is managed properly across UI components, business-objects, and models (e.g. if a user updates something on-screen, you need to make sure that the data is updated everywhere in your application, including possibly other windows that show that data).

This is how imperative toolkits like JavaFX and Qt work. The developer has to write the "glue code" that tells the system how to update the user interface in response to state changes, either from the user interacting with on-screen widgets, or from external events received by the application. This is the source of a lot of complexity in user interface development, and a frequent cause of errors.

By contrast, a declarative toolkit automatically manages how the UI reacts to state changes. The developer focuses on describing what state is required, and how state is used to initialize on-screen components, but doesn't need to write any glue-code. As state changes occur in your application, the UI is automatically changed to reflect that state. This technique works by conceptually regenerating the entire screen from scratch, and then applying any changes that are required to reflect state. The result is a simpler conceptual model for developers.

A window is simply a region of the screen that "belongs" to a specific application. Typically one application has one main window, but it may also own and control additional windows. These are overlayed on a "desktop", which is really just the screen background.

To manage different windows across many different applications, a part of the operating system called a windowing system is responsible for creating, destroying and managing running windows. The windowing system provides an API to applications to support for all window-related functionality, including:

• provide an mechanism for applications to create, or destroy their own windows
• handle window movement automatically and invisibly to the application (i.e. when you drag a window, the windowing system moves it).
• handles overlapping windows across applications (e.g. so that your application window can be brought to the ""front" and overlap another application's window).

Typically, the toolkit will provide an API that allows the developer to pass through requests to the windowing system to create and manage windows. Often, this includes window methods properties that can be manipulated to control window behaviour.

• Sample class: Stage, Window.
• Sample properties: minWidth, prefWidth, maxWidth; minHeight, prefHeight, maxHeight; title; isFullScreen; isResizable
• Sample methods: close(), toFront(), toBack()

Graphical output is a broad category that includes drawing and positioning elements on-screen. This can include adding arbitrary elements (e.g. circles, rectangles or other primitives), structured data (e.g. PNG or JPG images, MP4 video) or reusable widgets to the window.

We'll briefly cover standard concepts before discussing specifics of any of these.

A computer screen uses a Cartesean coordinate system to track window position. By convention, the top-left is the origin, with x increasing as you move right, and y increasing as you move down the screen. The bottom-right corner of the screen is maximum x and y, which equals the resolution of the screen¹.

Note that its possible for screen contents to move moved out-of-bounds and made inaccessible. We typically don't want to do this.

In the example below, you can see that this is a 1600x1200 resolution screen², with the four corner positions marked. It contains a single 400x400 window, positioned at (500, 475) using these global coordinates.

Given that the windowing system manages movement and positioning of windows on-screen, an application window doesn't actually know where it's located on-screen! The application that "owns" the window above doesn't have access to it's global coordinates (i.e. where it resides on the screen). It does however, have access to it's own internal, or local coordinates. For example, our window might contain other objects, and the application would know about their placement. In this local coordinate system, we use the top of the window as the origin, with the bottom-right coordinate being the (width, height) of the window. Objects within the window are referenced relative to the window's origin.

We're going to refer to graphical on-screen elements as widgets. Most toolkits support a large number of similar widgets. The diagram below shows one desktop toolkit with drop-down lists, radio buttons, lists and so on. All of these elements are considered widgets.

Typically, using widgets us as simple as instantiating them, adding them to the window, and setting up a mechanism to detect when users interact with them so that appropriate actions can be taken.

It's standard practice in graphical applications to represent the interface as a scene graph. This is a mechanism for modelling a graphical application as a tree of nodes (widgets), where each node has exactly one parent. Effects applied to the parent are applied to all of its children.

Toolkits support scene graphs directly. There is typically a distinction made between Container widgets and Node widgets. Containers are widgets that are meant to hold other widgets e.g. menus which hold menu_items, toolbars which can hold toolbar_buttons and so on. Nodes are widgets that are interactive and don't have any children.

Building a UI involves explicitly setting up the scene graph by instantiating nodes, and adding them to containers to build a scene graph. (For this reason, containers will always have a list of children, and a mechanism for adding and removing children from their list).

Layout is the mechanism by which nodes in the scene graph are positioned on the screen, and managed if the window is resized.

• Fixed layout is a mechanism to place widgets in a static layout where the window will remain the same size. This is done by setting properties of the widgets to designate each one's position, and ensuring that the containers do not attempt any dynamic layout.
• Relative layout delegates responsibility for widget position and size to their parents (i.e. containers). Typically this means setting up the scene graph in such a way that the appropriate container is used based on how it adjusts position. Typical containers include a vertical-box that aligns it's children in a vertical line, or a grid that places children in a grid with fixed rows and columns.

Applications often handle multiple types of processing: asynchronous, such as when a user types a few keystrokes, or synchronous, such as when we want a computation to run non-stop to completion.

User interfaces are designed around the idea of using events or messages as a mechanism for components to indicate state changes to other interested entities. This works well, due to the asynchronous nature of user-driven interaction, where there can be relatively long delays between inputs (i.e. humans type slowly compared to the rate at which a computer can process the keystrokes).

This type of system, designed around the production, transmission and consumption of events between loosely-coupled components, is called an Event-Driven Architecture. It's the foundation to most user-interface centric applications (desktop, mobile), which common use messages to signal a user's interaction with a viewable component in the interface.

What’s an event? An event is any significant occurrence or change in state for system hardware or software.

The source of an event can be from internal or external inputs. Events can generate from a user, like a mouse click or keystroke, an external source, such as a sensor output, or come from the system, like loading a program.

How does event-driven architecture work? Event-driven architecture is made up of event producers and event consumers. An event producer detects or senses the conditions that indicate that something has happened, and creates an event.

The event is transmitted from the event producer to the event consumers through event channels, where an event processing platform processes the event asynchronously. Event consumers need to be informed when an event has occurred, and can choose to act on it.

Events be generated from user actions, like a mouse click or keystroke, an external source, such as a sensor output, or come from the system, like loading a program.

An event driven system typically runs an event loop, that keeps waiting for these events. The process is illustrated in the diagram below:

1. An EventEmitter generates an event.
2. The event is placed in an event queue.
3. An event loop peels off events from the queue and dispatches them to event handlers (functions which have been registered to receive this specific type of event).
4. The event handlers receive and process the event.

To handle event driven architectures, we often subdivide application responsibility into separate components.