Metal is here! Cupertino has decided to bring more joy and happiness to the world by deprecating the widely used open standard OpenGL, thus making everyone's code obsolete. The ominous pronouncements have proved true, and OpenGL is no longer supported on MacOS versions after Catalina (like Big Sur and Monterey). These now require us to use Apple's proprietary standard Metal. As all the graphics in mgl was written with OpenGL using Cocoa frameworks (a variant of C called Objective-C), we are now rewriting the backend of mgl to make it compatible.

<sigh>

The good news is that since mgl is written using simple, atomic functions with future OS API compatibility in mind, this transition is not such a monumental task. In fact, as of January 2020, I (jlg) already had an alpha version of the mgl code running that is able to do the most important basic functions such as clearing the screen (mglClearScreen), drawing points (mglPoints), drawing lines (mglFixationCross, mglLines), drawing quads (mglQuads) and textures (mglCreateTexture, mglBltTexture).

We (jlg and Ben Heasly) are continuing this work in 2021 and 2022.

As this will be the third major rewrite of the mgl code, it will be version 3.0 and will be written primarily in Swift. The first version was written using the 32-bit Carbon API and the second version was written using the 64-bit Cocoa Frameworks.

There will be advantages to the new 3.0 metal compliant version of mgl.

• Fast GPU code Metal is much clearer than OpenGL about what happens on the CPU and what happens on the GPU. In fact, one always has to write two functions that run on the GPU, one for doing coordinate transformation of vertices called a vertex shader and another for coloring the vertices (and things in between vertices) called a fragment shader. The upside of this is that we can do some operations that used to be a bit clunky and slow on the GPU in a very elegant and super-fast way. Some preliminary tests of speed can be found here.
• Drifting gratings Changing the phase of a grating inside a gaussian window, can be written in much simpler fashion with the phase offset being coded on the GPU. This is already implemented.
• White noise stimuliComplex, time critical code can be put on the GPU, so things like white noise stimuli with frame-by-frame accuracy can easily be accommodated. This has not yet been implemented, but could be easily added to the existing structures.
• Windowing The new version will use a standalone application that will communicate with Matlab through sockets. One advantage of this is that we can make the windowing work more like a typical mac application, thus allowing minimizing, maximizing and changing the size dynamically.
• Window and full screen display Another advantage is that by having mglMetal work as a standalone app that communicates with Matlab, we can allow it to display both to a 2nd display and a window at the same time, thus allowing the experimenter to see the display at the same time as the subject.
• iPad display With the current setup, we should be able to display to an iPad. This is done trivially with Apple's new sidecar feature. It also could be done in the feature by making the mglMetal display an iOS application (should be easy given that the frameworks all work both on MacOS and iOS) and talking over a network socket. This could be useful for displays in medical clinics and other places where subjects maybe more comfortable using an iPad than a fixed display.
• Integration options The new system communicates with sockets so that the controlling side (matlab) has minimal requirements (has to have an open, close, read and write sockets commands implemented). This will have benefits in using this with other systems as functionality can be accessed with a few simple socket read/write commands. It also means that having interfaces to other backends, like Vulkan (Khronos Group's official successor to OpenGL) could be developed as a drop-in replacement for Metal, while preserving all same Matlab code and the domain knowledge it encapsulates. In theory, interfaces for other programming languages like Python could also be written, though these would not benefit from mgl's rich domain knowledge.
• Deep color Metal works with high bit depth color modes like those which have 10 or more bits per color channel. See here.

The mgl 3.0 work in in progress can be retrieved using git by cloning and then switching to the metal branch

git clone https://github.com/justingardner/mgl.git 
cd mgl
git checkout metal

If this worked correctly, then you should see a new directory called metal within the mgl repository:

grumini:/Users/justin/mgl> ls
COPYING     metal/      readme.md   task/
Contents.m  mgllib/     readme.txt  utils/

Note that depending on your version of MacOS, the Gatekeeper may block the system from running mgl, in which case you may see an error like the following pop-up:

If you go to Apple/System Preferences…/Security & Privacy/General you can manually allow it to run. However, it may asks you this for every single compiled function. In which case, you can disable GateKeeper temporarily by going to a terminal and doing (you will need to use a password that has root level access):

sudo spctl --master-disable

After everything is running properly, you may want to enable again by doing:

sudo spctl --master-enable

We've tested this on a few versions of macOS including Catalina, Big Sur, and Monterey. We're not sure if it will work on Mojave or older OS. Overall, we're focusing on the newer versions and hardware, and trying to move moving forward from there.

Once you've downloaded above, you should be ready to run some tests.

A quick way to test that Metal rendering is working is to run through a suite of tests located at mgl/mgllib/mglTestMetal/. To execute all the tests in a row, run

>> mglRunRenderingTests();

This will rapidly start and stop the mglMetal app several times. Each time it will issue some rendering commands to be rendered off-screen, then read back the rendering results and compare to a known good “snapshot” image. If all renderings match the expected snapshots, you should see output like:

>> mglRunRenderingTests()

Running 32 tests.


... some logging ...


All tests passed!

ans = 

  1x32 struct array with fields:

    snapshot
    renderedImage
    isSuccess
    testName
    snapshotData

If any rendering don't match its expected snapshot, mglRunRenderingTests will raise a figure showing the expected vs actual image.

You can run through a demo of MGL Metal rendering with

>> mglRenderingDemo

Running 32 demos.

1: Hit any key to continue for mglTestClearScreen.

... some logging ...

The screen should be cleared to an orange-brown color.

The screen should be cleared to an orange-brown color. The demo will pause here, and you can compare the expected message, “The screen should be cleared to an orange-brown color.” to what you see in the mglMetal window. It should look like this:

And so on, for several more renderings. Most will run in windowed mode. A few will to fullscreen animations, as well. Here are a few fun examples.

Most users should only need to download the MGL repo and not need to recompile anything. The necessary binaries for Matlab mex-functions and the mglMetal Swift application should be included and ready to go.

In case you do need to build it, there are three places to look these days (as of Spring 2022).

This builds most of the mex-functions for MGL, including functions for things like HID inputs and sounds.

mglMakeMetal()

This builds mex-functions specifically for socket communications with the new mglMetal swift application. After building you can test the results with mglTestSocket.m. The test will open a pair of sockets and send a bunch of data between them, using various data types that the mglMetal Swift app supports.

mglMakeSocket()
mglTestSocket()

To build the standalone mglMetal Swift application, you need to use Xcode. From testing so far, it looks like you'll need macOS 10.15.7 Catalina or later, with Xcode 12.4 or later.

To compile from XCode, open up the mglMetal application which is in mgl/metal/mglMetal.xcodeproj

Then in Xcode click on the sideways triangle, “play” button, at the top left to build the mglMetal application. Xcode should launch the mglMetal app and your desktop should look similar to the following.

[Xcode app build]

You can press the square “stop” button at the top left to close the app. When running from Matlab, Matlab will take care of starting and stopping the mglMetal app.

The Xcode build creates a fresh copy of mglMetal.app into the MGL repo at

mgl/metal/binary/latest/mglMetal.app

When this “latest” version is present, Matlab will use it. Otherwise, Matlab will default to the “stable” version that comes with the repo at

mgl/metal/binary/stable/mglMetal.app

Note that the default setting for Xcode is to build debug binaries into the directory:

~/Library/Developer/Xcode/DerivedData/Build/Products/Debug

So, mglMetalExecutableName now looks into that directory to see if there is a version of mglMetal that has a newer timestamp then the ones found in the mgl library. If so, it uses that, and will tell you it is doing that when you run mglOpen.

Development of MGL v3 with Metal is still in progress. Although we're not ready to declare v3 complete, a lot of functionality is in place. You might want to test it out!

Some things we've tested and seem to be working:

• macOS Catalina or later, with M1 or Intel
• Matlab 2021b or later
• test/demo scripts mentioned above with mglRunRenderingTests and mglRenderingDemo
• mglRetinotopy task
• fullscreen vs windowed mode, and getting and setting window position and which display
• creating, blt-ing, updating, and deleting textures
• rendering to texture and reading results back to Matlab
• some initial integration with PsychBench

Some things we know are not done yet:

• multiple simultaneous displays/windows
• frame syncing and scheduling
• documentation for v3 and socket-based integration with the standalone mglMetal.app
• publish mgl v3 on the ToolboxRegistry
• movies
• stencils
• dig deeper into what Metal offers for things like deep color and linearization 🤘
• back end for Windows or Linux, using Vulkan instead of Metal 🖖

The tables below goes into more detail about specific mgl functionality and what's in progress.

You can also look for the latest known issues at GitHub.

The v3 Metal version of MGL introduces a standalone app that handles windowing and rendering details. This has some great advantages, for example it lets us take advantage of Apple's shiny integration and tooling for Metal development via Xcode.

It also reduces coupling between the Matlab code and the rendering back end. This gives us a chance to think about how Matlab should integrate with the rendering back end. For v3 we've build the integration around local Unix sockets, giving a clear separation of concerns between task and flow login in Matlab, vs Metal graphics details. From there it's not hard to contemplate rendering to other back ends, like Vulkan or remote sockets.

This section gives an overview and some details about the socket communication.

The overall life-cycle goes like this:

Matlab is generally in control. When we mglOpen(), Matlab starts a new mglMetal.app process. The mglMetal app acts as a server: it binds a socket address and listens for connections. It waits for commands on the connection. For each command, it reads inputs, does some action like rendering, and writes back data as needed. When not processing a command, the mglMetal server is idle. Matlab acts as a client: it connects to the server's socket and initiates all interactions by sending commands over the socket to the server. It decides which commands to send and when. When we mglClose(), Matlab disconnects and terminates the mglMetal.app process.

Everything the mglMetal app does starts with a command from Matlab. The pattern for each command goes like this:

Matlab sends a numeric command code to mglMetal. mglMetal sends back an “ack” timestamp as an acknowledgement that it's ready to process. Matlab sends any additional data for the command, like color, texture, vertex, etc. mglMetal processes the command by rendering and/or sending back requested data. mglMetal sends back a “processed” timestamp to say it's done processing the command. Matlab sits and waits for the “processed” timestamp before proceeding. This pattern is a work in progress. We might want to make it less synchronous? We probably need a way for mglMetal to “nak” or negative-acknowledge a command in case of error, and return to a predictable state after that.

Our goal is to settle on a straightforward command pattern that gets the job done, and can be expressed in a few bullet points similar to the ones above.

Here's a listing of specific commands that mglMetal supports as of March 2022. This is a work in progress, but it might be helpful or interesting to see how the interaction works.

The supported commands are expressed in a shared header that us used to build MGL's mex-functions as well as the mglMetal app itself. The v3 rendering functions are implemented by sending these commands, instead of by making OpenGL framework calls.

Several “non-drawing” commands are intended to manage system resources and/or request data. These are executed one at a time, ie once per frame.

For actual rendering, we send actual drawing commands. When mglMetal receives one of these it enters a tight loop and looks for additional drawing commands to apply in the current frame. It stays in the tight loop until a flush command, which signals the end of a frame and causes mglMetal to present the frame and go back to idle.