mglMetal (mgl version 3.0)

Overview

Metal is here! Cupertino has decided to bring more joy and happiness to the world by deprecating the widely used open standard OpenGL, thus making everyone's code obsolete. The ominous pronouncements have proved true, and OpenGL is no longer supported on MacOS versions after Catalina like Big Sur and Monterey. These now require us to use Apple's proprietary standard Metal. As all the graphics in mgl was written with OpenGL using Cocoa frameworks (a variant of C called Objective-C), we are now rewriting the backend of mgl to make it compatible.

<sigh>

The good news is that since mgl is written using simple, atomic functions with future OS API compatibility in mind, this transition is not such a monumental task. In fact, as of January 2020, I (jlg) already had an alpha version of the mgl code running that is able to do the most important basic functions such as clearing the screen (mglClearScreen), drawing points (mglPoints), drawing lines (mglFixationCross, mglLines), drawing quads (mglQuads) and textures (mglCreateTexture, mglBltTexture).

We (jlg and Ben Heasly) are continuing this work in 2021 and 2022.

As this will be the third major rewrite of the mgl code, it will be version 3.0 and will be written primarily in Swift. The first version was written using the 32-bit Carbon API and the second version was written using the 64-bit Cocoa Frameworks.

Advantages of Metal

There will be advantages to the new 3.0 metal compliant version of mgl.

  • Fast GPU code Metal is much clearer than OpenGL about what happens on the CPU and what happens on the GPU. In fact, one always has to write two functions that run on the GPU, one for doing coordinate transformation of vertices called a vertex shader and another for coloring the vertices (and things in between vertices) called a fragment shader. The upside of this is that we can do some operations that used to be a bit clunky and slow on the GPU in a very elegant and super-fast way.
  • Drifting gratings Changing the phase of a grating inside a gaussian window, can be written in much simpler fashion with the phase offset being coded on the GPU. This is already implemented.
  • White noise stimuli Complex, time critical code can be put on the GPU, so things like white noise stimuli with frame-by-frame accuracy can easily be accommodated. This has not yet been implemented, but could be easily added to the existing structures.
  • Windowing The new version will use a standalone application that will communicate with Matlab through sockets. One advantage of this is that we can make the windowing work more like a typical mac application, thus allowing minimizing, maximizing and changing the size dynamically.
  • Window and full screen display Another advantage is that by having mglMetal work as a standalone app that communicates with Matlab, we can allow it to display both to a 2nd display and a window at the same time, thus allowing the experimenter to see the display at the same time as the subject.
  • iPad display With the current setup, we should be able to display to an iPad. This is done trivially with Apple's new sidecar feature. It also could be done in the feature by making the mglMetal display an iOS application (should be easy given that the frameworks all work both on MacOS and iOS) and talking over a network socket. This could be useful for displays in medical clinics and other places where subjects maybe more comfortable using an iPad than a fixed display.
  • Integration options The new system communicates with sockets so that the controlling side (matlab) has minimal requirements (has to have an open, close, read and write sockets commands implemented). This will have benefits in using this with other systems as functionality can be accessed with a few simple socket read/write commands. It also means that having interfaces to other backends, like Vulkan (Khronos Group's official successor to OpenGL) could be developed as a drop-in replacement for Metal, while preserving all same Matlab code and the domain knowledge it encapsulates. In theory, interfaces for other programming languages like Python could also be written, though these would not benefit from mgl's rich domain knowledge.

Download

The mgl 3.0 work in progress can be retrieved using git by cloning and then switching to the metal branch

git clone https://github.com/justingardner/mgl.git 
cd mgl
git checkout metal

If this worked correctly, then you should see a new directory called metal within the mgl repository:

grumini:/Users/justin/mgl> ls
COPYING     metal/      readme.md   task/
Contents.m  mgllib/     readme.txt  utils/

Note that depending on your version of MacOS, the Gatekeeper may block the system from running mgl, in which case you may see an error like the following pop-up:

If you go to Apple/System Preferences…/Security & Privacy/General you can manually allow it to run. However, it may asks you this for every single compiled function. In which case, you can disable GateKeeper temporarily by going to a terminal and doing (you will need to use a password that has root level access):

sudo spctl --master-disable

After everything is running properly, you may want to enable again by doing:

sudo spctl --master-enable

Testing

We've tested this on a few versions of macOS including Catalina, Big Sur, and Monterey. We're not sure if it will work on Mojave or older OS. Overall, we're focusing on the newer versions and hardware, and trying to move moving forward from there.

Once you've downloaded above, you should be ready to run some tests.

Automated Tests

A quick way to test that Metal rendering is working is to run through a suite of tests located at mgl/mgllib/mglTestMetal/. To execute all the tests in a row, run

mglRunRenderingTests();

This will rapidly start and stop the mglMetal app several times. Each time it will issue some rendering commands to be rendered off-screen, then read back the rendering results and compare to a known good “snapshot” image. If all renderings match the expected snapshots, you should see output like:

mglRunRenderingTests()

Running 21 tests.

... some logging ...

All tests passed!
ans = 
  1×21 struct array with fields:
    snapshot
    renderedImage
    isSuccess
    testName
    snapshotData

If any rendering don't match its expected snapshot, mglRunRenderingTests will raise a figure showing the expected vs actual image.

Interactive Demo

You can run through a demo of MGL Metal rendering with

mglRenderingDemo()

This will go through all the same tests scripts that were used for the automated test above. But this time they will be rendered to screen, and mglRenderingDemo will pause to let you view each one. It will also print a summary of what's expected to appear on the screen. For example:

>> mglRenderingDemo()

Running 21 demos.

1: Hit ENTER for mglTestClearScreen.

... some logging ...

The screen should be cleared to an orange-brown color.

The demo will pause here, and you can compare the expected message, “The screen should be cleared to an orange-brown color,” to what you see in the mglMetal window. It should look like this:

And so on, for several more renderings. Most will run in windowed mode. A few will to fullscreen animations, as well. Here are a few fun examples.

Building

Most users should only need to download the MGL repo and not need to recompile anything. The necessary binaries for Matlab mex-functions and the mglMetal Swift application should be included and ready to go.

In case you do need to build it, there are three places to look these days (as of Spring 2022).

mglMakeMetal.m

This builds most of the mex-functions for MGL, including functions for things like HID inputs and sounds.

mglMakeMetal()

mglMakeSocket.m

This builds mex-functions specifically for socket communications with the new mglMetal swift application. After building you can test the results with mglTestSocket.m. The test will open a pair of sockets and send a bunch of data between them, using various data types that the mglMetal Swift app supports.

mglMakeSocket()
mglTestSocket()

mglMetal.app

To build the standalone mglMetal Swift application, you need to use Xcode. From testing so far, it looks like you'll need macOS 10.15.7 Catalina or later, with Xcode 12.4 or later.

To compile from XCode, open up the mglMetal application which is in mgl/metal/mglMetal.xcodeproj

Then in Xcode click on the sideways triangle, “play” button, at the top left to build the mglMetal application. Xcode should launch the mglMetal app and your desktop should look similar to the following.

You can press the square “stop” button at the top left to close the app. When running from Matlab, Matlab will take care of starting and stopping the mglMetal app.

The Xcode build creates a fresh copy of mglMetal.app into the MGL repo at

mgl/metal/binary/latest/mglMetal.app

When this “latest” version is present, Matlab will use it. Otherwise, Matlab will default to the “stable” version that comes with the repo at

mgl/metal/binary/stable/mglMetal.app

Development status

Development of MGL v3 with Metal is still in progress. Although we're not ready to declare v3 complete, a lot of functionality is in place. You might want to tests it out!

Some things we've tested and seem to be working:

  • macOS Catalina or later, with M1 or Intel
  • Matlab 2021b or later
  • test/demo scripts mentioned above with mglRunRenderingTests and mglRenderingDemo
  • mglRetinotopy task
  • fullscreen vs windowed mode, and getting and setting window position and which display
  • creating, blt-ing, updating, and deleting textures
  • rendering to texture and reading results back to Matlab
  • some initial integration with PsychBench

Some things we know are not done yet:

  • multiple simultaneous displays/windows
  • frame syncing and scheduling
  • documentation for v3 and socket-based integration with the standalone mglMetal.app
  • publish mgl v3 on the ToolboxRegistry
  • movies
  • stencils
  • dig deeper into what Metal offers for things like deep color and linearization 🤘
  • back end for Windows or Linux, using Vulkan instead of Metal 🖖

The tables below goes into more detail about specific mgl functionality and what's in progress.

You can also look for the latest known issues at Git Hub.

Major functionality

Function name Status Notes
mglOpen working Currently starts out as windowed only. Can we pass initial window state as command line arg? Or start out with window hidden?
mglFlush working Works by busy-waiting on the flush within mglMetal - seems like this could be rethought to free up time on the matlab side. Schedule instead of wait?
mglClose working
mglClearScreen working
mglLines2 working
mglPoints2 working
mglPoints2c working
mglQuad working
mglFixationCross working
mglCreateTexture working 2D float textures working. 1D uint textures not yet. What to do about old mglBltTExture gl features? Need to deal with the alignment to 256 byte issue.
mglDeleteTexture working
mglBltTexture working
mglText working Works via Matlab image and mglMetalCreateTexture. Wants some love for backwards colors, color grading, alpha blending, and texture sizing vs natural font size
mglVisualAngleCoordinates working
mglMoveWindow.m working
mglGetWindowPos.m working

Secondary priority functionality

Function name Status Notes
mglFrameGrab.m Not working per se, but mglMetalSetRenderTarget.m and mglMetalReadTexture.m are working
mglMovie.m
mglSwitchDisplay.m
mglStencilCreateBegin.m
mglStencilCreateEnd.m
mglStencilSelect.m
mglHFlip.m working
mglVFlip.m working
mglScreenCoordinates.m working

Minor functionality

Function name Status Notes
mglFillOval.m working
mglFillRect.m working
mglFillRect3D.m working
mglGluAnnulus.m working
mglGluDisk.m working
mglGluPartialDisk.m working
mglPoints3.m working
mglPolygon.m working
mglTextDraw.m Can we deprecate on-the-fly texture creation and instead provide way to update text in an exiting texture?
mglTextSet.m working Setting params to both globals mgl and MGL. Should we pick one?
mglTransform.m working simplified for Metal, no longer supports various OpenGL operations and constants
mglVolume.m (mglPrivateVolume) Volume property does not exist

Functionality not affected by Metal

Function name Status Notes
mglSetGammaTable needs testing
mglGetGammaTable needs testing
mglCharToKeycode.m working Fixed crash from Catalina on
mglDeleteSound.m working
mglDescribeDisplays.m working
mglDisplayCursor.m
mglGetKeyEvent.m working Fixed crash from Catalina on
mglGetKeys.m working
mglGetMouse.m working
mglGetMouseEvent.m working
mglGetParam.m working We have global mgl and MGL, should we pick one?
mglGetSecs.m working
mglInstallSound.m working
mglIsCursorVisible.m
mglKeycodeToChar.m working Fixed crash from Catalina on
mglListener.m working
mglPlaySound.m working
mglPostEvent.m working
mglPrivateListener.m working
mglResolution.m
mglSetMousePosition.m working
mglSetParam.m working We have global mgl and MGL, should we pick one?
mglSetSound.m
mglShowKey.m not working
mglSimulateRun.m working
mglSystemCheck.m working Should add socket and metal tests?
mglWaitSecs.m working

Functionality no longer needed

Function name Notes Done
mglBindTexture.m Fast binding of textures. No need to continue to support n/a
mglStrokeText.m Discontinue n/a
mglNoFlushWait.m Still needed? Check
mglFlushAndWait.m No need to update / or provide as wrapper Check

Brainard Lab functionality

Function name Notes Done
mglBindFrameBufferObject.m Chris Broussard function
mglDrawImage.m Chris Broussard function
mglShader.m Chris Broussard function
mglUnbindFrameBufferObject.m Chris Broussard Function
mglCreateFrameBufferObject.m Chris Broussard function

Matlab - Metal Communications

The v3 Metal version of MGL introduces a standalone app that handles windowing and rendering details. This has some great advantages, for example it lets us take advantage of Apple's shiny integration and tooling for Metal development via Xcode.

It also reduces coupling between the Matlab code and the rendering back end. This gives us a chance to think about how Matlab should integrate with the rendering back end. For v3 we've build the integration around local Unix sockets, giving a clear separation of concerns between task and flow login in Matlab, vs Metal graphics details. From there it's not hard to contemplate rendering to other back ends, like Vulkan or remote sockets.

This section gives an overview and some details about the socket communication.

Overview

The overall life-cycle goes like this:

  • Matlab is generally in control.
  • When we mglOpen(), Matlab starts a new mglMetal.app process.
  • The mglMetal app acts as a server: it binds a socket address and listens for connections. It waits for commands on the connection. For each command, it reads inputs, does some action like rendering, and writes back data as needed. When not processing a command, the mglMetal server is idle.
  • Matlab acts as a client: it connects to the server's socket and initiates all interactions by sending commands over the socket to the server. It decides which commands to send and when.
  • When we mglClose(), Matlab disconnects and terminates the mglMetal.app process.

Command Pattern

Everything the mglMetal app does starts with a command from Matlab. The pattern for each command goes like this:

  • Matlab sends a numeric command code to mglMetal.
  • mglMetal sends back an “ack” timestamp as an acknowledgement that it's ready to process.
  • Matlab sends any additional data for the command, like color, texture, vertex, etc.
  • mglMetal processes the command by rendering and/or sending back requested data.
  • mglMetal sends back a “processed” timestamp to say it's done processing the command.
  • Matlab sits and waits for the “processed” timestamp before proceeding.

This pattern is a work in progress. We might want to make it less synchronous? We probably need a way for mglMetal to “nak” or negative-acknowledge a command in case of error, and return to a predictable state after that.

Our goal is to settle on a straightforward command pattern that gets the job done, and can be expressed in a few bullet points similar to the ones above.

Specific Commands

Here's a listing of specific commands that mglMetal supports as of March 2022. This is a work in progress, but it might be helpful or interesting to see how the interaction works.

The supported commands are expressed in a shared header that us used to build MGL's mex-functions as well as the mglMetal app itself. The v3 rendering functions are implemented by sending these commands, instead of by making OpenGL framework calls.

Several “non-drawing” commands are intended to manage system resources and/or request data. These are executed one at a time, ie once per frame.

Command Data from Matlab Data to Matlab Notes
mglPing echo mglPing command
mglDrainSystemEvents Consume all window events from the OS queue. Do we need this?
mglFullscreen Make mglMetal.app go fullscreen.
mglWindowed Make mglMetal.app go to windowed mode.
mglSetClearColor RGB clear color Set the RGB color to clear to, next time we start a frame.
mglCreateTexture width, height, and rgba float texture data texture number Create a new Metal texture.
mglReadTexture texture number width, height, and rgba float texture data Read an existing Metal texture as an image.
mglSetRenderTarget texture number Set the render target for next time we start a frame – on screen or an existing texture.
mglSetWindowFrameInDisplay display number, x, y, width, height Move the mglMetal window.
mglGetWindowFrameInDisplay display number, x, y, width, height Where is the mglMetal window?
mglDeleteTexture texture number Release an existing Metal texture.

For actual rendering, we send actual drawing commands. When mglMetal receives one of these it enters a tight loop and looks for additional drawing commands to apply in the current frame. It stays in the tight loop until a flush command, which signals the end of a frame and causes mglMetal to present the frame and go back to idle.

Command Data from Matlab Data to Matlab Notes
mglFlush Present the current frame and go back to idle.
mglBltTexture texture number, vertices with texture coordinates Blt the given texture to a polygon surface.
mglSetXform 4×4 matrix Set the vertex coordinate transform going forward.
mglDots vertices with color, size, and shape components Draw the given vertices as points, with various shape options.
mglLine vertices with color components Draw the given vertices as a line.
mglQuad vertices with color components Draw the given vertices as a triangles.
mglPolygon vertices with color components Draw the given vertices as a triangle strip.
mglArcs vertices with color, size, and shape components Draw the given vertices as points, with various curved shape options.
mglUpdateTexture texture number, width, height, and rgba float texture data Replace the given texture's contents, to be displayed during this frame.