2. Adding Interactivity to the Renderer

A renderer is a collection of algorithms that take a Scene data structure as their input and produce a FrameBuffer data structure as their output. After the algorithms build the FrameBuffer object, we would like to see the image that it represents. With the previous renderer, client programs that used the renderer would store the FrameBuffer data in an image file. Then the client program's user could open the image file with a separate image viewing program. But, by the time the user opens the image file, the renderer's client program will have probably terminated. That means that the client program cannot respond to any kind of user feedback. The user cannot interact with the renderer while the renderer is running.

The previous renderer is an "offline" renderer. After the rendering pipeline fills a FrameBuffer with pixel data, the client program must save the FrameBuffer as an image file in the file system (this is the "offline" part; the pixel data needs to be stored outside of the client program). The client program can continue to modify the Scene data structure, re-renderer the modified scene into the FrameBuffer, and then save another image file. But the client program cannot update the Scene based on input from the program's user, who does not get to see the image files until later. An offline renderer is good for creating the frames of an animation since we do not expect to interact with an animation.

We want to create a system that allows user interaction with the renderer. We want a system that allows a client program to display a FrameBuffer on the computer's screen while the client program is running. This will allow the client program to interactively respond to user input. The client program can react to a user's mouse click or keyboard input by modifying the Scene data structure, re-rendering the Scene into the FrameBuffer, and then updating the screen display with the FrameBuffer's new contents.

We want to create an "online" (interactive) version of the renderer. There are two distinct parts to an online renderer. First, we need a way to transfer pixel data from a FrameBuffer object to a computer's display screen. Second, we need a way to communicate user inputs from the computer's devices (mouse, keyboard, etc.) to a client program.

With the first part in place, as soon as a client program has the rendering pipeline fill a FrameBuffer with the pixel data that represents a Scene, the client program can immediately have that FrameBuffer displayed on the computer's screen for the user to see. With the second part in place, if the user interacts with the program in any way, by moving the mouse, by clicking on the displayed image, by using the keyboard to send a command to the program, then the program can use that information to update the Scene, re-render it into the FrameBuffer, and initiate an update of the display screen.

Once we have those two pieces in place, by implementing a cycle of

  • 1) display the FrameBuffer to the user,
  • 2) use user input to update the Scene,
  • 3) re-render the Scene into the FrameBuffer,
  • 4) got to step 1,

a program that uses this renderer becomes interactive.

This document describes the two parts of this interactive system.

A new class in the renderer.framebuffer package, the FrameBufferPanel class, implements the first part.

We use the Java language's build in GUI framework to implement the second part. The Java language has an extensive, and sophisticated, library of GUI components and event handlers. We will briefly outline the basics of Java event-driven GUI programming.

2.1 Renderer source code

The Java source code for this renderer is publicly available as a zip file.

Here is a link to the renderer's source code.

Download and unzip the source code to any convenient location in your computer's file system. The renderer does not have any dependencies other than the Java 11 (or later) JDK. Once you have downloaded and unzipped the distribution, you are ready to compile the renderer and run the renderer's (interactive) example programs.

This renderer has one new file in the framebuffer package,

  • FrameBufferPanel.java.

All the other packages in the renderer are unchanged from the previous renderer. The client programs are modified to be interactive clients by using this new GUI compoent.

2.2. The FrameBufferPanel class

With an offline renderer client, after a FrameBuffer object is filled with pixel data the client program saves the FrameBuffer data to an image file in the file system and then a separate image viewer program is used to open the image file, retrieve the pixel data from the storage device, and copy the pixel data into a screen window owned by the image viewer program.

To create an online renderer client, we need a way to copy pixel data directly from a FrameBuffer object into a screen window that is owned by the client program.

This renderer adds a file, FrameBufferPanel.java, to the renderer.framebuffer package. A FrameBufferPanel object is the link between a FrameBuffer object and the Java GUI system. A FrameBufferPanel is a subclass of the JPanel class from Java's Swing GUI library. A JPanel is a GUI component that can act as a drawing surface. We can "draw" the pixel information contained in a FrameBuffer into the graphics context of a JPanel.

Since a JPanel is a GUI component, it can be added to any GUI container. We can use an instance of FrameBufferPanel as part of a Java GUI along side of any other Java GUI components, like buttons, sliders, checkboxes, and menus. We can also use event handlers to tie those GUI components to the 3D scene we are displaying in the FrameBufferPanel.

The main difficulty in designing the FrameBufferPanel class is that we want the transfer of pixel data from a FrameBuffer object to a JPanel object to be as fast as possible. To get the speed that we need we use Java's MemoryImageSource class. Also, we designed the pixel_buffer array in the FrameBuffer class to be in the exact same data format required by Java's MemoryImageSource. This way, the data in a FrameBuffer object can be transferred by a MemoryImageSource object directly to the JPanel object.

In a later section of this document we will explain a bit more about how the FrameBufferPanel class uses a JPanel to paint the pixels from a FrameBuffer on the computer screen.

The class FrameBufferPanel is to our renderer much as the GLUT library is to the OpenGL renderer. The OpenGL renderer only knows how to compute a framebuffer full of pixel data. OpenGL does not have the ability to display pixel data in a screen window. The GLUT library (or any one of many other similar libraries) takes the pixel data off of OpenGL's hands and displays the data in a graphics window. So GLUT acts as an interface between the OpenGL renderer and the operating system's GUI, which displays the pixel data and handles user events. (What is a bit weird about the OpenGL case is that OpenGL computes all of the pixel data in the graphics card, so all the pixel data is right where it needs to be, and it never leaves the graphics card, but the OpenGL library itself does not have a way to make that pixel data appear on the screen. That must be done by some other library.)

Here is another interesting analogy for our FrameBufferPanel class. Google's Chrome web browser is built on top of a renderer (but it is not a 3D-graphics renderer, it is an HTML renderer). This HTML renderer is called Blink. You could say that Chrome is to Blink much as our FrameBufferPanel class is to our 3D renderer (or as GLUT is to the OpenGL renderer). In each case, the renderer computes values for all the pixels within a framebuffer but the renderer does not by itself know how to display these pixels nor how to handle user events. So the renderer needs an interface between itself and the operating system's GUI. So FrameBufferPanel is an interface between our renderer and the Java GUI, Chrome is an interface between the Blink renderer and the operating system's GUI, and GLUT is an interface between the OpenGL renderer and the operating system's GUI.

2.3. GUI Programming

The FrameBufferPanel class lets us use our renderer with the Java GUI framework. The FrameBufferPanel class makes a FrameBuffer object look, to the Java GUI framework, like a JPanel component. In this section we will explain what we mean by a GUI component and how we allow a user to interact with the FrameBufferPanel.

GUI programming can be divided into two aspects, the appearance and the behavior of a GUI. The appearance of a GUI is a special case of 2D graphics programming. The behavior of a GUI is an example of event-driven programming.

Below we will first describe how to code the appearance of a GUI. Then we will describe how to code a GUI's behavior.

Here are a few introductory references to Java GUI programming. Most of these references are chapters from introductory Java textbooks, so they are basic references that assume you have never written event-driven GUI programs. Please use these references to supplement the discussion of Java GUI programming given below.

2.4. Java's GUI Framework

Java's GUI system is made up of a large number of classes, most of them in two packages (and their sub-packages). The two packages are java.awt and javax.swing. The java.awt package was written first. The javax.swing package came later. Some classes in javax.swing are meant to replace classes fromjava.awt. Most classes in javax.swing supplement the classes in java.awt.

A good way to understand the Java GUI framework is to divide up (most) of its classes into five categories.

Here is an outline of the five main parts of the Java GUI framework.

  I) Components
 II) Containers
III) Layout Managers
 IV) Events
  V) Event Listeners

I) Components                         II) Containers
    1) Label,     JLabel                   1) Window, JWindow
    2) Button,    JButton                  2) Frame,  JFrame
    3) Checkbox,  JCheckBox                3) Panel,  JPanel
    4) ComboBox,  JComboBox                4) Dialog, JDialog
    5) List,      JList                    5) Box
    6) TextField, JTextField
    7) TextArea,  JTextArea          III) Layout Managers
    8) ScrollBar, JScrollBar               1) BorderLayout
    9) JRadioButton                        2) FlowLayout
   10) JSlider                             3) GridLayout
   11) JScrollPane                         4) BoxLayout
   12) JToolBar                            5) CardLayout
   13) JTabbedPane                         6) GridBagLayout
   14) Choice                              7) BoxLayout
   15) Canvas
   16) menus

IV) Events                             V) Event Listener Interfaces
    1) ActionEvent                         1) ActionListener       (1 method)
        a) button                          2) ItemListener         (1 method)
        b) combo box                       3) AdjustmentListener   (1 method)
        c) list                            4) ChangeListener       (1 method)
        d) radio button                    5) TextListener         (1 method)
        e) text field                      6) FocusListener        (2 methods)
        f) menu item                       7) KeyListener          (3 methods)
        g) timer                           8) MouseListener        (5 methods)
    2) ItemEvent                              MouseMotionListener  (2 methods)
        a) check box                       9) MouseWheelListener   (1 method)
        b) combo box                      10) ComponentListener    (4 methods)
        c) list                           11) WindowListener       (7 methods)
        d) choices                            WindowFocusListener  (2 methods)
        e) check box menu item                WindowStateListener  (1 method)
        f) radio button menu item
    3) AdjustmentEvent
        a) scroll bar
    4) ChangeEvent
        a) slider
    5) TextEvent
        a) text field
        b) text area
        c) text pane
    6) FocusEvent
        a) all components
    7) KeyEvent
        a) all components
    8) MouseEvent
        a) all components
    9) MouseWheelEvent
        a) all components
   10) ComponentEvent
        a) all components
   11) WindowEvent
        a) windows (including frames and dialogs)

The components, containers, and layout managers determine the appearance of a GUI.

The events and event listeners determine the behavior of a GUI.

Components are the building blocks of a GUI's appearance. Things like buttons, check boxes, drop down lists, text boxes, menus, etc. These are the elements of a GUI that the user interacts with. Here is a visual summary of the most common Java GUI components.

The word "Component" is used by the Java GUI system but other GUI systems have other names for components. For example, in the Microsoft Windows GUI system components are called "Controls". In the Python GUI system components are called "Widgets". In the HTML GUI system they are called "Elements" (more specifically, "interactive elements").

A Container is used to group components together in a GUI. A typical GUI can usually be divided into logical parts. The GUI components within each part will be grouped into a container. Look at whatever GUI program you are using to read this document. Try to find the logical groups of components in that GUI. Can you see the probable outlines of their containers? (Look for toolbars, menu bars, side panels, button groups, etc.)

A Layout Manager is responsible for automatically positioning components within a container. In most GUI systems, the programmer is not responsible for the exact position of every component in a screen window. The programmer places components in a container and gives the container a layout manager. When the program runs, the layout manager assigns each component its exact location in the screen window. If the GUI's user resizes the screen window, then the layout manager does the work of figuring out a new location for each component. This scheme makes GUI programming easier for the programmer and the results more reliable for the user.

Here is a visual summary of the most common Java layout managers.

An Event is a Java object that represents a user's interaction with some component in the GUI. If we click on a button in the GUI, that button click is represented by an ActionEvent object. If we click on a check box in the GUI, that click is represented by an ItemEvent object. An event object stores information about the GUI event (which button (or check box) was clicked, when the click occurred, etc.). Every event object originates in the operating system, which delivers the object to the Java Virtual Machine (JVM), which then delivers the object to an event listener method in the Java GUI program.

Event Listeners are the building blocks of a GUI's behavior. An event listener is a method that we write as part of our GUI program. Our event listener method will be called by the JVM when the JVM determines that the method is responsible for handling a particular event object. Notice that while we write the event listener methods (they are part of our code), our code never calls the event listeners. They are only called by the JVM. This is sometimes referred to as "Inversion of Control" (IoC). We write the methods but we relinquish the calling of those methods to some other code, in this case the JVM.

2.5. Creating A GUI

First we will look at code that creates Java GUIs without any event handlers. We will emphasize the code that gets a GUI up on a screen and adds components to the GUI. In later sections we will see how to make those components do something.

For code examples of creating Java GUIs, download the following zip file.

An effective way to understand the Java code that builds a GUI is to execute the code using the Java Shell program. The Java Shell (jshell.exe) lets us run one line of Java code at a time. This gives us a way to experiment with individual lines of GUI code, to see the visual effect each line of code has on the GUI.

Open a command-prompt window and on the command-line type the following command.

   > jshell

You should see a response that looks something like this.

|  Welcome to JShell -- Version 11.0.7
|  For an introduction type: /help intro

jshell>

At the jshell prompt, type the following five lines of Java code.

    jshell> import java.awt.*
    jshell> import javax.swing.*
    jshell> var jf = new JFrame("Hello")
    jshell> jf.setSize(300, 300)
    jshell> jf.setVisible(true)

You should see a Java GUI window appear near the upper left-hand corner of your computer screen. Now type the following three additional lines of code, one line at a time. Notice the visual effect that each line has on the window.

    jshell> js.setTitle("Interesting!")
    jshell> jf.getContentPane().setBackground(Color.green)
    jshell> jf.setLocation(0, 400)

We can make the window disappear and then reappear.

    jshell> jf.setVisible(false)
    jshell> jf.setVisible(true)

Let us build up a simple GUI with a few components. The JFrame window has a BorderLayout object as its layout manager. But BorderLayout is not the easiest layout manager to use. So let us replace the default Borderlayout object with a FlowLayout object.

    jshell> jf.setLayout(new FlowLayout())

Now try the following lines of code, one line at a time. Be sure to notice the visual effect (or lack of an effect) of each line.

    jshell> var label = new JLabel("Notice Me")
    jshell> jf.add(label)
    jshell> jf.pack()
    jshell> var button = new JButton("Press Me")
    jshell> jf.add(button)
    jshell> jf.pack()

Notice that creating a GUI component object, like a JButton, does not make that component part of the GUI. When we use the new operator to create a component object, we are creating a Java object in the Java heap. That does not (yet) have any effect on the screen's GUI. Even when we add the component to the JFrame object, the component does not (yet) appear on the screen (though it is now part of the GUI). We need to tell the GUI to "rebuild" itself in order to see the new component as part of the visual GUI.

Now try the following three slightly more complex lines of code. You can copy-and-paste these lines into your jshell prompt, one line at a time.

    jshell> jf.add(new JLabel(UIManager.getIcon("FileView.directoryIcon")))
    jshell> jf.add(new JButton(UIManager.getIcon("FileView.fileIcon")))
    jshell> jf.pack()

At this point our GUI has four components in it. Try resizing the JFrame window. In particular, make the window less wide. Notice how the components rearrange themselves under the direction of the FlowLayout manager.

To remind ourselves what code we have typed into the jshell prompt, use the following JShell command.

    jshell> /list

Let us give the JFrame a different layout manager and see what difference it makes.

    jshell> jf.setLayout(new GridLayout(2, 2))
    jshell> jf.pack()

Be sure to resize the window and see how it now behaves.

Let's try one more layout manager.

    jshell> jf.getContentPane().setLayout(new BoxLayout(jf.getContentPane(), BoxLayout.Y_AXIS))
    jshell> jf.pack()

Resize the window. Try changing Y_AXIS to X_AXIS and re-executing those lines of code. How does this compare to the flow layout?

Let's return to the flow layout.

    jshell> jf.setLayout(new FlowLayout())
    jshell> jf.pack()

Let us now add a container to our GUI.

    jshell> var jp = new JPanel()
    jshell> jf.add(jp)
    jshell> jf.pack()

Let's put three buttons in the JPanel container.

    jshell> jp.add(new JButton("Button 1"))
    jshell> jp.add(new JButton("Button 2"))
    jshell> jf.pack()
    jshell> jp.add(new JButton("Button 3"))
    jshell> jf.pack()

Notice that the three buttons act as a rigid group. They flow as a single unit (and they do not seem to want to flow within their group).

Try changing the layout manager of the JPanel container.

    jshell> jp.setLayout(new GridLayout(2,2))
    jshell> jf.pack()

Review the code that has been executed so far.

    jshell> /list

The most interesting idea in this code is that we have constructed a tree data structure, a GUI "scene graph".

            JFrame
            /    \
           /      \
 FlowLayout        List<Component>
                  /    /  |  \    \
            JLabel    /   |   \    \
                 JButton  |    \    \
                        JLabel  \    \
                             JButton  \
                                    JPanel
                                    /    \
                                   /      \
                          GridLayout      List<Component>
                                           /      |     \
                                          /       |      \
                                    JButton    JButton   JButton

Every GUI you see on a screen is represented in the GUI program's memory as a tree (scene graph) data structure. The root of a GUI scene graph must be a "top level container", either a JFrame, a JDialog, or a JWindow. These are the kinds of windows that you can have floating around your computer screen. Every container, including a top level one, contains a reference to a LayoutManager object and a List<Component> object. The members of that list are the components that visually show up inside the container's window. While the program is running the LayoutManager object decides how those components are placed inside the container's window. A JPanel container is also a component object, so we can nest JPanel objects inside a top level window. This allows us to extend our tree to any depth, since the List<Component> for a JPanel can contain more JPanel objects. (But you cannot let a JPanel be the root of a GUI scene graph; a JPanel cannot exist on your computer screen by itself.) Each JPanel groups the components within its list and its LayoutManager object decides how to position those components within the extent of the JPanel.

Here is a diagram of the inheritance tree for components and containers. (Notice how trees keep appearing, over and over again, in Computer Science.)

To see more complex examples of building a GUI, look at the code in the example folder.

In particular, notice the files in the JavaGui/gui-experiments folder that have the filename extension .jsh. These files are Java shell scripts (or "Java scripts", which are not to be confused with JavaScript). A Java shell script is a file meant to be run by the JShell program.

To run a Java shell script, open a command-prompt window in the folder that contains the script file (for example the guiExperiment3.jsh file in the JavaGui/gui-experiments folder`) and on the command-line type the a command like the following.

   > jshell guiExperiment3.jsh

This causes JShell to execute every line of code in the script file. What is really useful is that when JShell gets to the end of the script's code, JShell does not quit, it stops and gives us a prompt from which we can continue entering more Java code. To see what code was executed by the script, start with the JShell list command.

    jshell> /list

Then try adding this line of code.

    jshell> jp.setBackground(Color.yellow)

Putting lines of Java code in a JShell script file, executing the script, and then experimenting with the code using the jshell prompt, is a sophisticated and effective way to learn about the Java language. It works especially well with GUI code.

2.6. Java 2D Graphics

The JPanel class plays two roles in Java's GUI system. As we have seen, it is a container for holding and organizing other components. But JPanel is also a "drawing surface". Java has a library of methods that let you draw two-dimensional geometric shapes on a JPanel. This section explains how to get started with this 2D graphics API.

Open a command-prompt window and on the command-line start a JShell session.

   > jshell

When the jshell prompt appears, copy-and-paste the following block of code into the prompt. JShell allows you to copy several lines of code at a time. It will execute each line of code and then give you another prompt.

import java.awt.*
import javax.swing.*
var jf = new JFrame("2D Graphics")
jf.setLayout(new FlowLayout())
jf.setSize(300, 300)
var jp = new JPanel()
jp.setPreferredSize(new Dimension(200, 200))
jp.setBackground(Color.green)
jf.add(jp)
jf.setVisible(true)
Graphics g = jp.getGraphics()

The last line of code created a "graphics context", called g, that we can use for drawing geometric shapes. A graphics context is essentially a viewport into the framebuffer that holds the pixel data for the GUI window.

At the jshell prompt, type the following lines of Java code, one line at a time so that you can see what each line adds to the GUI. These lines of code update the pixel data in the graphics context (the viewport).

    jshell> g.setColor(Color.red)
    jshell> g.fillRect(10, 20, 60, 30)
    jshell> g.setColor(Color.blue)
    jshell> g.fillRect(100, 10, 30, 80)
    jshell> g.setColor(Color.yellow)
    jshell> g.drawRect(120, 120, 50, 50)
    jshell> g.setColor(Color.black)
    jshell> g.drawLine(0,0, 200,200)
    jshell> g.setColor(Color.magenta)
    jshell> g.drawString("This is graphical.", 20, 180)
    jshell> g.drawOval(150, 10, 30, 80)
    jshell> g.fillOval(50, 100, 50, 50)
    jshell> g.setColor(Color.black)
    jshell> UIManager.getIcon("FileView.computerIcon").paintIcon(jp, g, 30, 70)

Use your mouse to slightly change the size of the JFrame window. Notice that all the graphical shapes disappear. This is because the JFrame and its JPanel "repaint" themselves when you change the size of their window.

When the JPanel object repaints itself, it has no way to remember the code that we used to paint on the JPanel. Our code was not in any way part of that JPanel object. Our code just used a reference to the graphics context from theJPanel object. When the JPanel object repainted itself, it used its background color to redraw all of its pixels. The background color, along with the size, is something we set inside of the JPanel object.

Use your mouse to make the JFrame window very wide, so that the JPanel moves to the right of where it was.. Then copy-and-paste the following block of code into the jshell prompt to repaint the graphical shapes.

g.setColor(Color.red)
g.fillRect(10, 20, 60, 30)
g.setColor(Color.blue)
g.fillRect(100, 10, 30, 80)
g.setColor(Color.yellow)
g.drawRect(120, 120, 50, 50)
g.setColor(Color.black)
g.drawLine(0,0, 200,200)
g.setColor(Color.magenta)
g.drawString("This is graphical.", 20, 180)
g.drawOval(150, 10, 30, 80)
g.fillOval(50, 100, 50, 50)
g.setColor(Color.black)
UIManager.getIcon("FileView.computerIcon").paintIcon(jp, g, 30, 70)

Notice that the graphical shapes are not drawn inside of the JPanel! This is because the "graphics context" g is old and out of date (it represents the graphics context of where the JPanel was). After the JPanel repositions and repaints itself, we need to get a new reference to the graphics context.

    jshell> g = jp.getGraphics()

Do not resize the window yet. After you get the new graphics context, copy-and-paste the above block of code back into your jshell prompt.

This time the graphical elements should be drawn inside of the JPanel (and the misplaced graphical shapes should still be in the JFrame).

Use your mouse to move the JFrame window around your computer screen (but do not change the window's size). The graphical shapes should all move with the window because moving a window does not cause the window to "repaint" itself.

Having our artwork erased every time we even slightly resize (or minimize) our graphics window is very inconvenient. We fix this by using a really interesting concept. We will define our own GUI component, one that knows how to draw itself with the graphical shapes that we want.

We define our own GUI component by defining a subclass of JPanel. The JPanel class contains a lot of code that makes it into a GUI component. We want to reuse as much of that code as we can and modify just enough of the JPanel class so that our subclass of JPanel paints itself the way we want it to. We will leave all the other behaviors of JPanel alone though, if you know what you are doing, you can modify other JPanel behaviors in the subclass.

The minimum amount of code that we need to write to define our new GUI component is a constructor and a paintComponent() method. The paintComponent() method is called by the JVM whenever a GUI component needs to "repaint" itself. The code we put in paintComponent() is what gives our GUI component its own distinct look.

Copy and past this class definition into your jshell prompt.

class MyComponent extends JPanel {
   public MyComponent(){ // constructor
      this.setPreferredSize(new Dimension(200, 200));
      this.setBackground(Color.green);
   }

   @Override public void paintComponent(Graphics g){
      super.paintComponent(g);
      g.setColor(Color.red);
      g.fillRect(10, 20, 60, 30);
      g.setColor(Color.blue);
      g.fillRect(100, 10, 30, 80);
      g.setColor(Color.yellow);
      g.drawRect(120, 120, 50, 50);
      g.setColor(Color.black);
      g.drawLine(0,0, 200,200);
      g.setColor(Color.magenta);
      g.drawString("This is graphical.", 20, 180);
      g.drawOval(150, 10, 30, 80);
      g.fillOval(50, 100, 50, 50);
   }
}

At your jshell prompt enter these two lines of code.

    jshell> jf.add(new MyComponent())
    jshell> jf.pack()

Enter those two lines of code again.

    jshell> jf.add(new MyComponent())
    jshell> jf.pack()

We can create as many instances as we want of our new GUI component.

This idea of creating a new GUI component that knows how to paint itself is the main idea behind how the FrameBufferPanel class manages to paint the pixels from a FrameBuffer object into a JPanel object.

Exercise: Modify the MyComponent class so that the size and background color are constructor parameters.

Notice that MyComponent is not very interactive. It always draws the same shapes and, other than moving it around, it does not respond to any user interactions. We want to learn about Java events so that we can use them to make our GUI components (and our 3D renderer) interactive.

There are more Java 2D code examples in the folder JavaGui/graphics-experiments from the following zip file.

Here are a few references for using Java's 2D graphics API.

The JPanel class is Java's drawing surface but it is also a container for organizing other components. In the world of HTML/CSS/JavaScript they have a canvas element that is their drawing surface (and it has no container like role). It is interesting to compare the design and features of Java's JPanel component with HTML's canvas element.

2.7. Java's Event Handling Model

We need to explain the relationship between three of the categories defined above, components, events, and event listeners. Roughly, a component is an object that a user interacts with, an event listener is a method that handles those interactions, and an event is a message sent by a component to an event handler when the user interacts with the component.

If an event is a message sent by a component to an event handler, then who is the messenger? It turns out that there are two messengers, the operating system (OS) and the Java Virtual Machine (JVM). Every event object originates in the OS. The OS passes them along to the JVM. The JVM calls the appropriate event handler method and passes the event object as a parameter to the method.

For code examples that use Java events, download the following zip file.

Let's look at a simple code example.

Open a command-prompt window and on the command-line start a JShell session.

   > jshell

When the jshell prompt appears, copy-and-paste the following block of code into the prompt. JShell allows you to copy several lines of code at a time. It will execute each line of code and then give you another prompt.

import java.awt.*
import javax.swing.*
var jf = new JFrame("GUI with Events")
jf.setLayout(new FlowLayout())
var jb = new JButton("Press Me")
jf.add(jb)
jf.pack()
jf.setSize(200, 100)
jf.setVisible(true)
jb.addActionListener(e -> System.out.println(e + "\n"))

You should see a Java GUI window appear near the upper left-hand corner of your computer screen and the window should contain a button. Every time you press the button, you should see an output printed in the console window. The output you see is the toString() of an event object (called e in this code) that was delivered by the JVM to an event handler method (which, in this code, is a lambda expression).

The last line of code is new to us. That line of code added an event handler to the GUI code. Let's analyze it in detail. The variable jb refers to a JButton object, which is a GUI component that can be the source of event objects. When a JButton is pressed, the JVM will create an ActionEvnet object that it should deliver to a method that can handle it. In our code, the lambda expression

    e -> System.out.println(e + "\n")

is a method that can handle an event object (called e). The addActionListener() method tells the JButton object to remember this method (the lambda expression) as the method the button's event objects should be delivered to by the JVM.

When we click on the JButton, the JVM creates an appropriate ActionEvent object. Since the JVM knows which pixel we clicked on (the OS told it), the JVM knows which button was clicked. The JVM asks that button to look up which method is registered as its event handler (that's our lambda expression). Then the JVM calls that method and passes the event object as the method's parameter (the variable e). Our method prints the ActionEvent object on the console window.

The above code used a very compact notation, a lambda expression, to represent the event handler method. This compact notation is very nice to use when we are allowed to use it. But the Java language does not always allow this notation. Let us look at a more verbose notation for event handlers. This notation is always allowed.

Copy and past the following code into the jshell prompt. This code adds a second button with its own event handler.

import java.awt.event.*
var jb2 = new JButton("Button 2")
jf.add(jb2)
jf.pack()
class ButtonHandler implements ActionListener {
   @Override public void actionPerformed(ActionEvent e) {
      System.out.println(e + "\n");
   }
}
var buttonHandler = new ButtonHandler()
jb2.addActionListener(buttonHandler)

Like jb, jb2 is a GUI component that can be the source of event objects. So jb2 needs an event handler method. We define a class ButtonHandler which implements an interface, ActionListener. This tells Java that instances of this class are capable of handling ActionEvent objects. The ActionListener interface requires a single method that takes an ActionEvent object as its parameter. We create an instance of our ButtonHandler class. That instance object holds just one thing, a method that is capable of handling ActionEvent objects. The addActionListener() method tells 'jb2' to remember this object as the carrier of the method that is responsible for its events.

We just saw a compact and a verbose syntax for event handlers. Java also has an in-between notation, the "anonymous inner class" syntax.

Copy and past the following code into the jshell prompt. This code adds a third button with its own event handler.

var jb3 = new JButton("Button 3")
jf.add(jb3)
jf.pack()
jb3.addActionListener(new ActionListener(){
   @Override public void actionPerformed(ActionEvent e) {
      System.out.println(e + "\n");
   }
})

Here is a fourth JButton whose event handler uses the most compact notation.

var jb4 = new JButton("Button 4")
jf.add(jb4)
jf.pack()
jb.addActionListener(e -> System.out.println(e + "\n"))

Carefully compare the syntax of the last three button examples.

Here is an example of a GUI component that uses a different kind of event object. A JCheckBox causes ItemEvent objects to be created when it is checked. Notice that we tell the JCheckBox object to remember an ItemListener (instead of an ActionListener).

var cb = new JCheckBox("Check Me");
jf.add(cb)
jf.pack()
jf.setSize(250, 150)
cb.addItemListener(e -> System.out.println(e + "\n"))

An interesting thing with JCheckBox is that we can activate it with a line of code. Type this line into the jshell prompt.

cb.setSelected(true)

Now type this line.

cb.setSelected(false)

These lines change the visual appearance of the checkbox and they also trigger an event. We can call setSelected() on a JButton but it doesn't do anything (try it). We can activate JButton with the doClick() method (try it). What does doClick() do to a JCheckBox?

Here is a JCheckBox handler that uses the anonymous inner class syntax. Notice how we can now see explicit mentions of ItemListener, an itemStateChanged method, and an ItenEvent parameter.

var cb2 = new JCheckBox("CheckBox 2")
jf.add(cb2)
jf.pack()
cb2.addItemListener(new ItemListener(){
   @Override public void itemStateChanged(ItemEvent e) {
      System.out.println(e + "\n");
   }
})

Here is a picture that helps illustrate the relationships between a GUI component, an event handler, and an event. The picture shows how a component object, an event handler object, and an event object are related to each other in the Java heap.

  GUI component object
+----------------------+         List<EventListener>
|                      |         object
| List<EventListener>--|-------->+------+
|                      |         |      |           EventListener object
|                      |         +------+         +----------------------+
| addListener(  ){...} |         | -----|-------->|                      |
|                      |         +------+         | handleEvent(Event e) |
+----------------------+         |      |         | {                    |
An object that initiates         +------+         |    ...               |
events and receives Event                         | }                    |
objects from the JVM.                             |                      |
                                                  +----------------------+
                                                  An object that implements
           Event object                           the EventListener interface
      +-------------------+                       and handles event objects
      | Created by JVM    |                       delivered by the JVM.
      | using information |
      | from the OS about |
      | an event          |
      +-------------------+

Notice that a component holds a reference to a List of event listeners. That means that a component can have multiple event listeners assigned to it (which is actually a common situation). In your JShell session, try adding a second event listener to one of the components in the GUI so that when you click on that component, both of its event handlers respond.

Exercise: Start a new JShell session and create a GUI with two buttons that share a single event listener object. When you click either button, the same event listener method gets called.

Notice that a single component can have multiple event listeners and a single event listener can service multiple components. Try creating a GUI example that demonstrates both of these concepts.

Here are several references for Java's event model.

The Java event model is a special case of the "Observer Design Pattern" which is itself a special case of the "Publish/Subscribe Design Pattern" (or "pub/sub"). Think of our "GUI component object" as a publisher (like a magazine publisher). Our "GUI component object" publishes (sends out) "event objects" (magazine issues) to who ever wants to subscribe to them. The subscribers are our "event listeners". The delivery mechanism between the publisher and its subscribers is the OS/JVM combination. The OS/JVM delivers to the event subscribers the latest event object sent out by the component event source publisher.

2.8. Event-driven GUI Programs

So far we have taken a very limited view of event-driven GUI programming. Our event handlers have done nothing useful. Usually, the GUI of an event-driven program should show some visual change after the user interacts with some component.

We need to add to our examples a GUI object that needs to be updated when the user interacts with one of our components. We need to add a fourth kind of object to our examples. The fourth kind of object, a "GUI object", is kind of a vague reference to something on the screen that the user can see and that changes appearance due to some event from some component in the GUI.

  GUI component object
+----------------------+         List<EventListener>
|                      |         object
| List<EventListener>--|-------->+------+
|                      |         |      |           EventListener object
|                      |         +------+         +----------------------+
| addListener(  ){...} |         | -----|-------->|                      |
|                      |         +------+         | handleEvent(Event e) |
+----------------------+         |      |         | {                    |
An object that initiates         +------+         |    ...               |
events and receives Event                         | }                    |
objects from the JVM.                             |                      |
                                                  +----------------------+
                                                  An object that implements
           Event object                           the EventListener interface
      +-------------------+                       and handles event objects
      | Created by JVM    |                       delivered by the JVM.
      | using information |
      | from the OS about |
      | an event          |                 "GUI object"
      +-------------------+             +------------------+
                                        |                  |
                                        | paintComponent() |
                                        | {                |
                                        |    ...           |
                                        | {                |
                                        |                  |
                                        +------------------+
                                        GUI object that needs
                                        to be repainted after
                                        an event occurs.

One specific kind of "GUI object" that we can use is an instance of Java's JPanel class. The JPanel class has a method paintComponent() that we call to tell the panel to update its drawing surface. That is the method shown in the picture above inside of the "GUI object". In other words, we can use events to make a component, like the MyComponent that we wrote earlier, interactive.

Let's build a simple example of an event-driven GUI program.

Open a command-prompt window and on the command-line start a JShell session.

   > jshell

Copy-and-paste the following block of code into your jshell prompt.

import java.awt.*
import javax.swing.*
import java.awt.event.*

int colorNumber = 0
var jf = new JFrame("Simple GUI Example 1")
jf.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
jf.setLayout(new FlowLayout())
var jb = new JButton("Change Color")
jf.add(jb)
var buttonHandler = new ActionListener(){
   @Override public void actionPerformed(ActionEvent e){
      colorNumber = (colorNumber + 1) % 4;
      System.out.println("The color number is = " + colorNumber);
   }
}
jb.addActionListener(buttonHandler)
jf.pack()
jf.setVisible(true)

Click on the button in the GUI several times. Notice that the GUI prints the value of the colorNumber variable in the console window and it cycles the value between 0 and 3. We want to turn this program into an interactive GUI program that uses the colorNumber variable to change the appearance of the GUI.

We need to add to this example a "GUI object" that we can update on each button click. The following block of code uses an anonymous inner class to define a subclass of JPanel and override the paintComponent() method. This new kind of JPanel uses the colorNumber variable to set its background color. We also need to modify the button handler so that after it updates the value of colorNumber, it tells the "GUI object" that it needs to repaint itself.

Copy-and-paste this block of code into you jshell prompt.

Color[] color = {Color.red, Color.green, Color.blue, Color.yellow};
var jp = new JPanel(){  // The "GUI object".
   @Override public void paintComponent(Graphics g){
      super.paintComponent(g);
      this.setBackground(color[colorNumber]);
   }
}
jp.setPreferredSize(new Dimension(200, 100))
var buttonHandler2 = new ActionListener(){
   @Override public void actionPerformed(ActionEvent e){
      colorNumber = (colorNumber + 1) % color.length;
      System.out.println("The color number is = " + colorNumber);
      jp.repaint();  // Redraw the "GUI object".
   }
}
jb.addActionListener(buttonHandler2)
jf.add(jp)
jf.pack()

The JButten now has two event handlers, but one of then is no longer needed. Here is how we can remove it. Type the following line of code into the JShell prompt.

jb.removeActionListener(buttonHandler)

Notice one very important detail. Even though we override the paintComponent() method in the JPanel class, we do not call it to tell the component to repaint itself. Instead, we call another method in the component, the repaint() method. This seems odd, and it is a common mistake to call paintComponent() instead of repaint(). But calling paintComponent() does not work! And calling it can make a GUI program buggy and act in strange ways. When we call repaint() that tells the JVM to please call 'paintComponent()for us. The JVM will callpaintComponent()`, but it calls it at the correct time and, more importantly, with the correct graphics context.

Now we have a very simple, interactive, event-driven, GUI program. Here is what the complete program looks like.

import java.awt.*
import javax.swing.*
import java.awt.event.*

int colorNumber = 0
Color[] color = {Color.red, Color.green, Color.blue, Color.yellow}

var jf = new JFrame("Simple GUI Example 1")
jf.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
jf.setLayout(new FlowLayout())

var jp = new JPanel(){  // The "GUI object".
   @Override public void paintComponent(Graphics g){
      super.paintComponent(g);
      this.setBackground(color[colorNumber]);
   }
}
jp.setPreferredSize(new Dimension(200, 100))
jf.add(jp)

var jb = new JButton("Change Color");
jf.add(jb);
var buttonHandler = new ActionListener(){
   @Override public void actionPerformed(ActionEvent e){
      colorNumber = (colorNumber + 1) % color.length;
      System.out.println("The color number is = " + colorNumber);
      jp.repaint();  // Redraw the "GUI object".
   }
}
jb.addActionListener(buttonHandler)
jf.pack()
jf.setVisible(true)

Look for the places in the code where each of the four kinds of objects from the above picture get instantiated, the component object that is the source of events, the event handler object, the event object, and the "GUI object".

Let's build a second simple example of an event-driven GUI program.

Start a new JShell session and copy-and-paste this block of code into the jshell prompt.

import java.awt.*
import javax.swing.*
import java.awt.event.*
var jf = new JFrame("Simple GUI Example 2")
jf.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
var mouseHandler = new MouseAdapter(){
   @Override public void mouseClicked(MouseEvent e){
      System.out.println("Mouse click: (x, y) = ("
                         + e.getX() + ", " + e.getY() + ")");
   }
}
jf.addMouseListener(mouseHandler)
jf.setSize(400, 400)
jf.setVisible(true)

This program prints on the console window the coordinates of any mouse clicks that are inside the JFrame window. Play with the program a bit. Notice where the x-coordinate is small and where it is large. Same for the y-coordinate. Change the size of the window and see how this changes the range of x and y coordinate values. Notice that you cannot click on a pixel with coordinate near to (0, 0) (why do you think that is?).

Let's modify this program so that it keeps track of the last five mouse clicks. Give the program two lists, one list to keep track of x-coordinates and the other list for the y-coordinates. When the lengths of the lists get to six, then we remove the an item from each list so we keep just the last five mouse coordinates.

Copy-and-paste the following block of code into your jshell prompt.

var xs = new ArrayList<Integer>()
var ys = new ArrayList<Integer>()
var mouseHandler2 = new MouseAdapter(){
   @Override public void mouseClicked(MouseEvent e){
      xs.add(e.getX());  // Add at the end.
      ys.add(e.getY());
      if (6 == xs.size()){
         xs.remove(0); // Remove from the front.
         ys.remove(0);
      }
      System.out.print("[ ");
      for (int i = 0; i < xs.size(); ++i) {
         System.out.printf("(%d, %d) ", xs.get(i), ys.get(i));
      }
      System.out.println("]");
   }
}
jf.addMouseListener(mouseHandler2)

Again, play with this program a bit. Notice how at the beginning the lists are shorter that five, then they grow to length five, and then stay at length five. Also notice that the JFrame now has two mouse listeners. The first one prints the coordinate of the last mouse click. The second mouse listener prints the list of the last five mouse clicks.

Let's turn this program into an interactive GUI program that draws a disk on the screen wherever one of the last five mouse clicks was. We need to give this program a "GUI object" that we can update the visual appearance of. We will once again use a subclass of JPanel for the "GUI object".

Copy-and-paste the following block of code into your jshell prompt.

var jp = new JPanel(){   // The "GUI object".
   @Override public void paintComponent(Graphics g){
      super.paintComponent(g);
      final Rectangle r = g.getClipBounds();
      g.clearRect(r.x, r.y, r.width, r.height);
      for (int i = 0; i < xs.size(); ++i){
         g.fillOval(xs.get(i), ys.get(i), 10, 10);
      }
   }
}
jp.setPreferredSize(new Dimension(400, 400))
jf.add(jp)
var mouseHandler3 = new MouseAdapter(){
   @Override public void mouseClicked(MouseEvent e){
      System.out.println("Mouse click: (x, y) = ("
                         + e.getX() + ", " + e.getY() + ")");
      xs.add(e.getX());
      ys.add(e.getY());
      if (6 == xs.size()){
         xs.remove(0);
         ys.remove(0);
      }
      jp.repaint(); // Redraw the "GUI object".
   }
}
jp.addMouseListener(mouseHandler3)
jf.pack()

The paintComponent() method uses the lists of x and y coordinates to draw small ovals. Before drawing the ovals the method clears the graphics context of the old ovals.

We need to update the mouse listener so that it tells the JPanel when to repaint itself. One important detail is that we are switching the mouse listener from the JFrame to the new JPanel The old JFrame mouse listeners are still part of the program. Do they have any effect on the program?

Here is what the complete program looks like.

import java.awt.*
import javax.swing.*
import java.awt.event.*
var xs = new ArrayList<Integer>()
var ys = new ArrayList<Integer>()
var jf = new JFrame("Track Last 5 Mouse Clicks")
jf.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
var jp = new JPanel(){   // The "GUI object".
   @Override public void paintComponent(Graphics g){
      super.paintComponent(g);
      final Rectangle r = g.getClipBounds();
      g.clearRect(r.x, r.y, r.width, r.height);
      for (int i = 0; i < xs.size(); ++i){
         g.fillOval(xs.get(i), ys.get(i), 10, 10);
      }
   }
}
jp.setPreferredSize(new Dimension(400, 400))
jf.add(jp)
var mouseHandler = new MouseAdapter(){
   @Override public void mouseClicked(MouseEvent e){
      System.out.println("Mouse click: (x, y) = ("
                         + e.getX() + ", " + e.getY() + ")");
      xs.add(e.getX());
      ys.add(e.getY());
      if (6 == xs.size()){
         xs.remove(0);
         ys.remove(0);
      }
      jp.repaint(); // Redraw the "GUI object".
   }
})
jp.addMouseListener(mouseHandler)
jf.pack()
jf.setVisible(true)

Once again, look for the places in the code where each of the four kinds of objects from the above picture get instantiated, the component object that is the source of events, the event handler object, the event object, and the "GUI object". (Hint: This time the answers are a bit trickier.)

Exercise: What happens if you add the mouse handler to the JFrame instead of the JPanel? Try it.

Let's do a third example.

Start a new JShell session and copy-and-paste this block of code into the jshell prompt.

import java.awt.*
import javax.swing.*
import javax.swing.event.*

int sliderValue = 50

var jf = new JFrame("Simple GUI Example 3");
jf.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
jf.setLayout(new FlowLayout());

var js = new JSlider(JSlider.HORIZONTAL)
jf.add(js)
var sliderHandler = new ChangeListener(){
   @Override public void stateChanged(ChangeEvent e){
      final JSlider slider = (JSlider)e.getSource();
      sliderValue = slider.getValue();
      System.out.println("Slider value = " + sliderValue);
   }
}
js.addChangeListener(sliderHandler)

jf.pack()
jf.setSize(300, 200)
jf.setVisible(true)

This program has a JSlider sitting in a JFrame. When you slide the slider, its current value is printed to the console window. Notice that this (default) slider has integer values between 0 and 100.

Exercise: Try using the your jshell prompt to add a second JSlider to the JFrame but with the second one constructed using JSlider.VERTICAL. Have the second slider share its event listener with the first slider.

We want to use the slider value to change the visual appearance of a "GUI object". We will once again use a subclass of JPanel as our "GUI object".

Let us make the slider move a dot back and forth across the window.

Copy-and-paste the following block of code into your jshell prompt.

var jp = new JPanel(){  // The GUI object.
   @Override public void paintComponent(Graphics g){
      super.paintComponent(g);
      final Rectangle r = g.getClipBounds();
      g.clearRect(r.x, r.y, r.width, r.height);
      g.fillOval((int)((sliderValue/100.0) * r.width) - 10, (r.height/2) - 10, 20, 20);
   }
};
jp.setPreferredSize(new Dimension(200, 100))
jf.add(jp)

var js = new JSlider(JSlider.HORIZONTAL)
jf.add(js)
var sliderHandler = new ChangeListener(){
   @Override public void stateChanged(ChangeEvent e){
      final JSlider slider = (JSlider)e.getSource();
      sliderValue = slider.getValue();
      System.out.println("Slider value = " + sliderValue);
      jp.repaint(); // Redraw the GUI.
   }
}
js.addChangeListener(sliderHandler)

The paintComponent() method first clears its graphics context. Then it uses the slider value as a percentage to determine where across the graphics context the dot should go. Vertically, the dot is placed in the center of the graphics context.

Notice that we needed to redefine the event handler so that it calls the repaint() method on the JPanel.

Here is what the complete program looks like.

import java.awt.*
import javax.swing.*
import javax.swing.event.*

int sliderValue = 50

var jf = new JFrame("Slider GUI Example")
jf.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE)
jf.setLayout(new FlowLayout())

var jp = new JPanel(){  // The GUI object.
   @Override public void paintComponent(Graphics g){
      super.paintComponent(g);
      final Rectangle r = g.getClipBounds();
      g.clearRect(r.x, r.y, r.width, r.height);
      g.fillOval((int)((sliderValue/100.0) * r.width) - 10, (r.height/2) - 10, 20, 20);
   }
};
jp.setPreferredSize(new Dimension(200, 100))
jf.add(jp)

var js = new JSlider(JSlider.HORIZONTAL)
jf.add(js)
var sliderHandler = new ChangeListener(){
   @Override public void stateChanged(ChangeEvent e){
      final JSlider slider = (JSlider)e.getSource();
      sliderValue = slider.getValue();
      System.out.println("Slider value = " + sliderValue);
      jp.repaint(); // Redraw the GUI.
   }
}
js.addChangeListener(sliderHandler)

jf.pack()
jf.setVisible(true)

Look for the places in the code where each of the four kinds of objects from the above picture get instantiated, the component object that is the source of events, the event handler object, the event object, and the "GUI object".

When you look carefully at the last three examples, you will notice that they all have one other ingredient besides the four kinds of objects we talked about (event source object, event handler object, event object, and "GUI object"). All three examples have important data that they keep track of and use for updating the "GUI object" (look at each example and find its data). This data is an important idea, and leads to the next topic, the Model-View-Controller design pattern.

Exercise: In the last example, the disk moves horizontally across the middle of the GUI window. Modify the program so that the disk moves from the lower left-hand corner of the window to the upper right-hand corner as you move the slider to the right. When the disk is in a corner, you should only see one quarter of the disk.

Exercise: Add a second slider to the last example and have that slider move the disk vertically while the original slider moves the disk horizontally.

Exercise: Take the client program renderer_2\clients_r2\ButtonsAndModels_v1.java and turn it into an executable jar file called buttons_and_models.jar. Make the jar file completely self contained. Notice that the program uses an obj file resource. Make that file part of the jar file. You will need to replace the use of the renderer.scene.util.Assets class with a class loader. Look up "this.getClass().getResourceAsStream()". Tryy to put in the jar file only those classes used by the client program. For example, that program doesn't use the DrawSceneGraph.java class, so it shouldn't be in the jar file.

2.9. Model-View-Controler (MVC) Design Pattern

Building event-driven GUIs that do interesting things is not easy. In this section we will discuss a software pattern that is meant to help with the design of complex event-driven GUIs.

As mentioned at the end of the last section, event-driven GUI programs will always have data that needs to updated whenever an event occurs and which needs to be used to update the appearance of the GUI. We call that data the state of the GUI. Much of the complexity in building complicated GUIs comes from managing the state data.

In the last section we emphasized that an event-driven GUI program needed four kinds of objects. Now we will add a fifth object to our designs, a program state object.

Here is a picture showing the five kinds of objects. When an event handling method is called (by the JVM) to handle an event object, the handler should use the information in the event object to update the program state object. Notice that this means that there must be some kind of connection between the event handler object and the program state object. In addition, the "GUI object" should look to the program state object for all the information that it needs to repaint the GUI. This means that the "GUI object" also needs a way to communicate with the program state object.

  GUI component object
+----------------------+         List<EventListener>
|                      |         object
| List<EventListener>--|-------->+------+
|                      |         |      |           EventListener object
|                      |         +------+         +----------------------+
| addListener(  ){...} |         | -----|-------->|                      |
|                      |         +------+         | handleEvent(Event e) |
+----------------------+         |      |         | {                    |
An object that initiates         +------+         |    ...               |
events and receives Event                         | }                    |
objects from the JVM.                             |                      |
                                                  +----------------------+
                                                  An object that implements
     Event object                                 the EventListener interface
 +-------------------+                            and handles event objects
 | Created by JVM    |                            delivered by the JVM.
 | using information |
 | from the OS about |      Program state object
 | an event          |     +----------------------+
 +-------------------+     |  Information about   |        GUI object
                           |  the current state   |     +------------------+
                           |  of the application  |     |                  |
                           |  program. This needs |     | paintComponent() |
                           |  to be updated when  |     | {                |
                           |  an event occurs.    |     |    ...           |
                           +----------------------+     | {                |
                                                        |                  |
                                                        +------------------+
                                                        GUI object that needs
                                                        to be repainted after
                                                        an event occurs.

The Model-View-Controller (MVC) pattern takes the above objects and gives them new names and tries to formalize the role that they play in the GUI program.

In MVC, the program state object is called the Model. What we have been calling the "GUI object" is called the View. The event handler object, the event source component, and the event objects are all together called the Controller.

When programmers think about MVC, they think of the following picture which describes the roles played by these objects. The picture represents a user using a GUI program. The user sees the View. At some point the user interacts with some component and causes an event. The Controller handles the event and uses it to update information in the Model. Then the Controller sends a signal to the View to repaint itself. The View queries the Model to find out how it should update its appearance. The user see the resulting change in the appearance of the GUI and may then initiate another interaction with the Controller.

                       +-----------------+
                       |                 |
                       |      Model      |
                       |                 |
                       +-----------------+
  4. View queries the   /               \
     Model and then    /                 \  2. Controller updates
     repaints itself. /                   \    the Model.
                     /                     \
        +-------------+                  +----------------+
        |             |                  |                |
        |    View     |<-----------------|   Controller   |
        |             |  3. Controller   |                |
        +-------------+     notifies     +----------------+
                    \       the View.      /
       5. User sees  \                    /  1. User interacts
          repainted   \                  /      with the Controller.
          View.        \                /       (The JVM sends an
                        +--------------+         Event object to the
                        |              |         Controller.)
                        |     User     |
                        |              |
                        +--------------+

2.10. Low-level and High-level Events

It's possible to imaging a very simple GUI system that has only one kind of event, mouse events. In this system, when the user clicks the mouse a program gets an event telling it which pixel the user clicked on, and that's it. In such a system, with every mouse click, the program that receives the mouse event would need to figure out, for itself, what kind of component (if any) was under the mouse click and then do the appropriate action. Such a simple GUI system would put a tremendous responsibility on every programmer who used it. It would be tedious and error prone to program with. We would soon wish that the GUI system did some of the work for us and let us know not just where the mouse was clicked, but that the mouse was clicked on a button or a menu. Being told that the mouse was clicked "on a button" is called a high-level event. Being told just that the mouse was clicked "on a pixel" is called a low-level event. All modern GUI systems have both kinds of events.

Let us reconsider the life of a mouse event, from mouse, to OS, to JVM, to some component

As we move our mouse across our desk, the mouse is sending a stream of information to the operating system (through the mouse device drive, which is part of the OS). The mouse is moving physically over our desk but the OS has a way of translating "desk coordinates" to pixel coordinates on the computer's monitor and the OS moves a mouse pointer across the monitor screen. So the OS always knows what pixel the mouse is (virtually) over.

When we click the mouse button, the mouse sends that information to the OS. The OS knows which pixel the mouse is over. The OS builds a "mouse clicked event" message and then the OS needs to decide who (which process) to send that message to.

The OS maintains a data structure of all the open windows on the computer's monitor and which process owns each window. The OS can use this data structure, along with the mouse coordinates, to determine which process owns the pixel the mouse was clicked on. The OS sends a message to that process telling it that there was a mouse click at the given pixel in its window.

In our case, the process that owns the clicked on pixel is the JVM. So the OS tells the JVM that one of its pixels was clicked on. The JVM translates the coordinates of the clicked on pixel from the OS's screen coordinates to the coordinates of the window owned by the JVM. The JVM keeps a data structure that lets it know which GUI component owns each pixel in its window and whether or not that component has an event listener attached to it. If so, the JVM sends a message (an Event object) to the listener object attached to the GUI component that owns the pixel the mouse was clicked on. To our Java program, that GUI component seems to be the source of the event, not the mouse itself (because it is a high-level event).

In general, the JVM receives from the OS what we refer to as "low-level events", like "mouse moved", or "mouse button pressed down", or "mouse button released". The JVM translates these low-level events into "higher-level events" and "high-level event". For example, when the mouse moves into one of the JVM's windows, the JVM will translate the low-level "mouse move" event into a higher-level "mouse enter" event. When the mouse moves out of the JVM's window, the JVM creates a higher-level "mouse exit" event. When the mouse moves over a button, the mouse button is pressed down, and then the mouse button is released, these three low-level events are translated, by the JVM, into a single "button clicked" high-level event.

If the component that was clicked on is, say, a JButton, then our program does not get the low-level "mouse clicked" event from the JVM. Instead, the JVM translates the low-level MouseEvent into a high-level ActionEvent. Most GUI components get high-level events, like ActioEvent, ItemEvent, AdjustmentEvent, ChangeEvent, or TextEvent. These high-level events convey more "meaning" to a program than low-level events like "mouse clicked". However, the JPanel and JFrame containers are components that only receive low-level events from the JVM (or lower-level events like "mouse entered").

When the JVM sends a message (with an Event object) to a clicked on component, the JVM doesn't actually send the message to the GUI component object. Instead, the JVM sends the message to an event listener object that is registered with the GUI component. This is a "separation of concerns" object-oriented-design choice. The GUI component's responsibility is "appearing" in certain way. We do not want to give the GUI component the added responsibility of "acting" in a certain way. We separate the "behavior" part of the GUI component into a separate object (the event listener) that has the responsibility of implementing the GUI component's actions (this uses the Observer Design Pattern). This design allows us to do things like make a button change its behavior as a program executes. Also, we can give a button multiple behaviors or we can add/remove behaviors from a button as a program executes. We often say that the GUI component's behavior has been "delegated" to an event listener object. This terminology is used a lot by Microsoft in the C# language.

When you click the mouse on a GUI component that has a registered MouseListener, one of the methods in that MouseListener object gets called by the JVM. Notice that the methods in the MouseListener object are not called by any of our code. If you look at the code for an event driven program, you will see many methods defined in the code that are never called from within the code. We write these (event handling) methods but we never call them. Normally, that is not how we write software. We tend to write methods in our code that can be called by other methods in our code. In an event driven GUI program we write methods that will be called by the "GUI framework". The idea that we write methods but we don't call them (they get called by someone else) is referred to as "inversion of control" (IoC). When our GUI program is running, it is the Java GUI framework that is really in control, not our code. After our code has instantiated all the necessary objects described above (in the MVC pattern) our code terminates and leaves the Java GUI framework in control. (If you look at any of our GUI program, you will see that the main() method returns after it has instantiated all the needed objects, and yet, our program does not terminate). The Java GUI framework waits for events (from the operating system) and then calls our event listener methods as needed. When an event listener method returns, it is returning to the Java GUI framework, not to any code that we wrote. We say that control has been "inverted" from our program to the Java GUI framework. Another term used to describe the methods in listener objects is "callback functions". This is an older term used in the C language. The idea is that we provide these functions to the GUI framework so that it can "call back" on these functions as events happen.

As was mentioned in the last paragraph, when the main() method of a Java GUI program returns, the program does not terminate. Instead, control shifts to the Java GUI framework. We can be more specific about this shift of control. All Java code runs in some thread. When you launch a Java program, the JVM starts a thread to run your main() method. The thread that runs the main() method is often called the "main thread". Unless you launch more threads, the main thread will be your program's only thread. When main() returns, the JVM terminates the main thread and then your program terminates. But in a Java GUI program, as soon as you make some GUI component visible, the JVM launches a new thread, the "Event Dispatch Thread" (EDT). This thread is how the JVM waits for events from the operating system and it is this thread that calls your event handling methods. So after our main() method returns and terminates, the EDT continues to run and receive events from the OS. So the "inversion of control" happens on the Event Dispatch Thread.

2.11. Focus Events

We started the last section thinking about a GUI system that has only one kind of event, a (low-level) mouse event. We mentioned that such a system can work, but at the cost of much programmer effort. The reason such a simple system can work is that when the operating system detects a mouse click, the OS knows where on the computer screen the mouse is positioned. If the OS keeps a data structure recording which process owns every pixel on the screen, then the OS can translate mouse clicks into processes to which the mouse event should be sent.

Suppose this primitive GUI system adds a second kind of event, a (low-level) keyboard event. This turns out to be a bit harder than the low-level mouse event. Let's analyze why.

When the user taps a key on their keyboard, the operating system knows right way exactly which key was tapped. So the operating system can construct a KeyEvent object, but to which process should the event be delivered? Suppose there are five processes with GUI windows up on the screen. Which one should get the key event? There is no obvious connection between the keyboard and the computer screen.

The operating system needs a way to determine which on screen window gets keyboard events. And if one on screen window is currently receiving keyboard events, how do we switch the key events to another window?

The on screen window that the operating system selects for keyboard events is said to have focus. There are several ways to solve our two problems:

  • How does the OS determine which window has focus?
  • How do we switch focus to another window?

The simplest solution to these two problems is also the oldest solution, called "focus follows mouse". This solution was used in the first GUI systems, but, as we will see, it is not used as much theses days.

The simple solution is that whichever window the mouse is in, that window has focus and gets key events. To switch focus to another window, move the mouse over the other window. If the mouse is not over any window, then key events are not delivered to any process and typing on the keyboard does not have any effect.

Modern GUI systems for the most part no longer use "focus follows mouse" but they still use the mouse to help solve the two focus problems. The window that has focus is the last window the user clicked the mouse in. To switch focus to another window, click the mouse (anywhere) in the other window. So this is "focus follows mouse clicks".

Now we can see why low-level keyboard events are a bit more difficult than low-level mouse events. The keyboard focus system needs the mouse, and its mouse events, in order to work. (There is a solution to keyboard focus that does not rely on the mouse, but it is more complicated.)

The Windows operating system does have one very clever use for "focus follows mouse". Mouse wheel events are delivered to windows using the "focus follows mouse" strategy. You can watch this happen. Put two or more partially overlapping windows on your computer screen and then move the mouse scroll wheel back and forth while moving the mouse from being over one window to being over another window (but do not press any mouse buttons or press down on the scroll wheel).

Every GUI system has a sub-system, called the "focus system", that determines how low-level key events are delivered by the OS to processes.

The Java language has a sophisticated focus system that is well documented.

The HTML/CSS/JavaScript world also has a sophisticated focus system. Since it is used by a lot of people, it is well documented.

Consider a GUI program that has a JFrame that contains a JPanel that contains JButon. Suppose that the user's mouse passes over the JButton. Which of these components should be sent the low-level MouseEvent? One reasonable answer is "all of them". Another reasonable answer is "none of them" (instead, the JButton should be sent a high-level ActionEvent if the Jbutton is clicked on). Questions like this are common in GUI systems and they do not have fixed, uniform answers. The answers vary from system to system.

import java.awt.*
import javax.swing.*
import java.awt.event.*;
var jf = new JFrame("Focus")
jf.setLayout(new FlowLayout())
var jp = new JPanel()
var jb = new JButton("Press Me")
jf.add(jp)
jp.add(jb)
jp.setPreferredSize(new Dimension(200, 200))
jp.setBackground(Color.green)
jf.setSize(300, 300)
jf.setVisible(true)
int x = 0;
jb.addActionListener(e -> System.out.println("Button pressed"))
jp.addMouseMotionListener(new MouseMotionAdapter(){public void mouseMoved(MouseEvent e){System.out.println("JPanel "+x++);}})
jf.addMouseMotionListener(new MouseMotionAdapter(){public void mouseMoved(MouseEvent e){System.out.println("JFrame "+x++);}})

Now enter this additional line of code at the jshell prompt.

jb.addMouseMotionListener(new MouseMotionAdapter(){public void mouseMoved(MouseEvent e){System.out.println("JButton "+x++);}})

A focus event represents a change in who receives low level events. When the focus switches from one window (or component) to another, low level events like mouse and keyboard events change from being delivered from the first window to the second window. Both the window that loses focus and the window that gains focus is sent a focus event object.