CS 346 (W23)
Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage

Building Desktop

Graphical applications arose in the early 80s as we moved from text-based terminals to more technically capable systems. This was part of the Personal Computer (PC) movement of that time, which aimed to put a “computer in every home” Graphical User Interfaces (GUIs) were seen as more “user-friendly” and considered an important factor in the adoption of these systems. Introduced in 1984, the Apple Macintosh introduced the first successful commercial graphical operating system; other vendors (e.g. Microsoft, Commodore, Sun) quickly followed suit. The conventions that were adopted on the Mac became standard on other platforms as well e.g. Windows.

Early Mac desktop

Desktop applications refers to graphical applications designed for a notebook or desktop computer, typically running Windows, macOS or Linux. Users interact with these applications using a mouse and keyboard, although other devices may also be supported (e.g. camera, trackpad).

Although desktop computers can run console applications in a shell, for this discussion, we’re focusing on graphical applications.

Features

Graphical desktop application should have the following features:

  • Multiple application windows should be supported. Most applications will often present their interface within a single, interactive window, but it can sometimes be useful to have multiple simultaneous windows controlled by a single application1.
  • Support for full-screen or windowed interaction: although graphical applications tend to run windowed, they should normally be usable full-screen as well. The window contents should scale or reposition themselves as the window size changes.
  • Window decorations: Each window should have a titlebar, minimize/maximize/restore buttons (that work as expected).
  • Windows may or may not be resizable: if they are resizable, the contents should scale or adjust their layout based on window size (for this reason, it may make sense to either contrain window dimensions when resizing, or make some windows fixed size). Convention allows the main window to be resized, and option dialogs (or similar non-essential windows) to be fixed-size.
  • Interactive graphical elements: window contents could be any combination of graphics, animations, multimedia, or text that is desired for the target application. These contents should be dynamic (i.e. have the ability to change in response to system state) and should support a range of interactions - clicking, double-clicking, dragging - provided by both mouse and keyboard.
  • Standard menubars: every application should have the following menus (with shortcuts). Although some applications choose to eliminate menus (or replace with other controls), most of the time you should include them. Exact contents may vary, but users expect at-least this functionality:
    • File: New, Open, Close, Print, Quit.
    • Edit: Cut, Copy, Paste.
    • Window: Minimize, Maximize.
    • Help: About.
  • Keyboard shortcuts: you should strive to have keyboard shortcuts for common functionality. All standard shortcuts should be supported2. e.g.
    • Ctrl-N for File-New, Ctrl-O for File-Open, Ctrl-Q for Quit.
    • Ctrl-X for Cut, Ctrl-C for Copy, Ctrl-V for Paste.
    • F1 for Help.

Benefits

There are obvious benefits to a graphical display being able to display rich colours, graphics and multimedia. However, this application style also encourages encourages a certain style of interaction, where users point-and-click to elements on-screen to interact with them.

There are numerous benefits to this style of user interface:

  • The interface provides affordances: visual suggestions on how you might interact with the system. This can include hints like tooltips, or a graphical design that makes use of controlsobvious ((e.g. handles to show where to “grab” a window corner).
  • Systems provide continuous feedback to users. This includes obvious feedback (e.g. dialog boxes, status lines) and more subtle, continuous feedback (e.g. widgets animating when pressed).
  • Interfaces are explorable: users can use menus, and cues to discover new features.
  • Low cost of errors: undo-redo, and the ability to rollback to previous state makes exploration low-risk.
  • These environments encouraged developers to use consistent widgets and interactive elements. Standardization led to a common look-and-feel, and placement of common controls - which made software easier to learn, especially for novices. Many of the standard features that we take for granted are a direct result of this design standardization in the 80s. [ed. notice that Windows, macOS, Linux all share a very common interaction paradigm, and a common look-and-feel! You can move between operating systems and be quite comfortable because of this.]

Functionality

These are complex requirements, that outside of the scope of a programming language (in-part because they’re going to be intrinsically tied to the underlying operating system, so they’re difficult to standardize).

A widget or GUI toolkit is a UI framework which provides this functionality. This includes support for:

  • Creating and managing application windows, with standard window functionality e.g. overlapping windows, depth, min/max buttons, resizing.
  • Reusable components called widgets that can be combined in a window to build typical applications. e.g. buttons, lists, toolbars, images, text views.
  • Dynamic layout that adapts the interface to change in window size or dimensions.
  • Support for an event-driven architecture3 i.e. suport for standard and custom events. Includes event generation and propogation.

Implementation details will vary based on the toolkit that you’re using. We’ll discuss requirements first, and then in the next section we’ll provide implementation details for some common widgets toolkits.

Window Management

In the context of a applications, a window is simply a region of the screen that “belongs” to a specific application. Typically one application has one main window, an optionally other windows that may also be displayed. These are overlayed on a “desktop”, which is really just the screen background.

To manage many different windows, across many different applications, a part of the operating system called a windowing system is responsible for creating, destroying and managing running windows. The windowing system provides an API to applications to support for all window-related functionality, including:

  • provide an mechanism for applications to create, or destroy their own windows
  • handle window movement automatically and invisibly to the application (i.e. when you drag a window, the windowing system moves it).
  • handles overlapping windows across applicaitons (e.g. so that your application window can be brought to the ““front” and overlap another application’s window).

A windowing system or windowing technology is typically included as part of the operating system, though it’s possible in some systems to replace windowing systems (e.g. Linux).

Coordinate systems

A computer screen uses a Cartesean coordinate system to track window position. By convention, the top-left is the origin, with x increasing as you move right, and y increasing as you move down the screen. The bottom-right corner of the screen is maximum x and y, which equals the resolution of the screen.

Screen coordinates grown down and right from the origin (Dea et al. <u>JavaFX By Example</u>. 2017)

Note that its possible for screen contents to move moved out-of-bounds and made inaccessible. We typically don’t want to do this.

Moving items out of bounds is permitted (Dea et al. <u>JavaFX By Example</u>. 2017)

In the example below, you can see that this is a 1600x1200 resolution screen4, with the four corner positions marked. It contains a single 400x400 window, positioned at (500, 475) using these screen, or global coordinates.

Screen and window coordinates

Given that the windowing system manages movement and positioning of windows on-screen, an application window doesn’t actually know where it’s located on-screen! The application that “owns” the window above doesn’t have access to it’s global coordinates. It does however, have access to it’s own internal, or local coordinates. For example, our window might contain other objects, and the application would know about their placement. In this local coordinate system, we use the top of the window as the origin, with the bottom-right coordinate being the (width, height) of the window. Objects within the window are referenced relative to the window’s origin.

Local coordinate system

Window creation

Typically, the toolkit will provide a mechanism to create a top-level application window, typically as a top-level class that can instantated. That class will have properties to control its behaviour (some of which is used by the Windowing system to setup the window correctly).

  • Sample properties: minWidth, width, maxWidth; minHeight, height, maxHeight; title; isFullScreen; isResizable
  • Sample methods: close(), toFront(), toBack()

Window Movement

As application developers, we do not need to do anything to support window movement, since it’s provided by the windowing system. Any non-fullscreen windows created by a toolkit are automatically moveable.

Widgets and Layout

We’re going to refer to graphical on-screen elements as widgets. Most toolkits support a large number of similar widgets. The diagram below shows one desktop toolkit with drop-down lists, radio buttons, lists and so on. All of these elements are considered widgets.

The Zebra UI toolkit, showcasing a range of UI widgets

Typically, using widgets us as simple as instantiating them, adding them to the window, and setting up a mechanism to detect when users interact with them so that appropriate actions can be taken.

Scene graph

It’s standard practice in graphical applications to represent the interface as a scene graph. This is a mechanism for modeling a graphical application as a tree of nodes (widgets), where each node has exactly one parent. Effects applied to the parent are applied to all of its children.

Toolkits support scene graphs directly. There is typically a distinction made between Container widgets and Node widgets. Containers are widgets that are meant to hold other widgets e.g. menus which hold menu_items, toolbars which can hold toolbar_buttons and so on. Nodes are widgets that are interactive and don’t have any children.

This diagram shows a scene graph, containing a menubar container with file, edit and window menu item nodes, and a toolbar holding table, chart and text button nodes.

Building a UI involves explicitly setting up the scene graph, by instantiating nodes, and adding them to containers to build a scene graph. (For this reason, containers will always have a list of children, and a mechanism for adding and removing children from their list).

Layout

Layout is the mechanism by which nodes in the scene graph are positioned on the screen, and managed if the window is resized.

  • Fixed layout is a mechanism to place widgets in a static layout where the window will remain the same size. This is done by setting properties of the widgets to designate each one’s position, and ensuring that the containers do not attempt any dynamic layout.
  • Relative layout delegates responsibility for widget position and size to their parents (i.e. containers). Typically this means setting up the scene graph in such a way that the appropriate container is used based on how it adjusts position. Typical containers include a vertical-box that aligns it’s children in a vertical line, or a grid that places children in a grid with fixed rows and columns.

Graphics and Animation

It’s possible to also draw arbitrary graphics. This allows for more creative freedom than using just widgets. (Note that it’s entirely possible to do both - combine widgets and drawing in a single application).

  • Drawing
  • Graphics context
  • Transformations? (Lite)
  • Images
  • Animations

Current thinking: leave details on graphics to CS 349.


  1. Photoshop, for instance, famously has multiple windows tiled over the desktop. It’s also a very, very complex program, so it needs to split up functionality like this. ↩︎

  2. Ctrl on Windows and Linux, CMD on Mac. ↩︎

  3. Though not strictly required, all modern toolkits are built around the idea of an event-driven architecture, where events or messages are used to communicate changes to system state, or to signal user intentions. ↩︎

  4. Note that this is the resolution that the screen is set to display, which may be different from the resolution that the screen natively supports. For example, my monitor is meant to run at 2880x1440, but I can set it to any desired resolution up to those values. ↩︎