Metal is here! Cupertino has decided to bring more joy and happiness to the world by deprecating the widely used open standard OpenGL, thus making everyone's code obsolete. The ominous pronouncements have proved true, and OpenGL is no longer supported on MacOS versions after Catalina (like Big Sur and Monterey). These now require us to use Apple's proprietary standard Metal. As all the graphics in mgl was written with OpenGL using Cocoa frameworks (a variant of C called Objective-C), we are now rewriting the backend of mgl to make it compatible.
The good news is that since mgl is written using simple, atomic functions with future OS API compatibility in mind, this transition is not such a monumental task. In fact, as of January 2020, I (jlg) already had an alpha version of the mgl code running that is able to do the most important basic functions such as clearing the screen (mglClearScreen), drawing points (mglPoints), drawing lines (mglFixationCross, mglLines), drawing quads (mglQuads) and textures (mglCreateTexture, mglBltTexture).
We (jlg and Ben Heasly) are continuing this work in 2021 and 2022.
As this will be the third major rewrite of the mgl code, it will be version 3.0 and will be written primarily in Swift. The first version was written using the 32-bit Carbon API and the second version was written using the 64-bit Cocoa Frameworks.
There will be advantages to the new 3.0 metal compliant version of mgl.
The mgl 3.0 work in in progress can be retrieved using git by cloning and then switching to the metal branch
git clone https://github.com/justingardner/mgl.git cd mgl git checkout metal
If this worked correctly, then you should see a new directory called metal within the mgl repository:
grumini:/Users/justin/mgl> ls COPYING metal/ readme.md task/ Contents.m mgllib/ readme.txt utils/
Note that depending on your version of MacOS, the Gatekeeper may block the system from running mgl, in which case you may see an error like the following pop-up:
If you go to Apple/System Preferences…/Security & Privacy/General you can manually allow it to run. However, it may asks you this for every single compiled function. In which case, you can disable GateKeeper temporarily by going to a terminal and doing (you will need to use a password that has root level access):
sudo spctl --master-disable
After everything is running properly, you may want to enable again by doing:
sudo spctl --master-enable
We've tested this on a few versions of macOS including Catalina, Big Sur, and Monterey. We're not sure if it will work on Mojave or older OS. Overall, we're focusing on the newer versions and hardware, and trying to move moving forward from there.
Once you've downloaded above, you should be ready to run some tests.
A quick way to test that Metal rendering is working is to run through a suite of tests located at mgl/mgllib/mglTestMetal/. To execute all the tests in a row, run
This will rapidly start and stop the mglMetal app several times. Each time it will issue some rendering commands to be rendered off-screen, then read back the rendering results and compare to a known good “snapshot” image. If all renderings match the expected snapshots, you should see output like:
>> mglRunRenderingTests() Running 32 tests. ... some logging ... All tests passed! ans = 1x32 struct array with fields: snapshot renderedImage isSuccess testName snapshotData
If any rendering don't match its expected snapshot, mglRunRenderingTests will raise a figure showing the expected vs actual image.
You can run through a demo of MGL Metal rendering with
>> mglRenderingDemo Running 32 demos. 1: Hit any key to continue for mglTestClearScreen. ... some logging ... The screen should be cleared to an orange-brown color.
The screen should be cleared to an orange-brown color. The demo will pause here, and you can compare the expected message, “The screen should be cleared to an orange-brown color.” to what you see in the mglMetal window. It should look like this:
And so on, for several more renderings. Most will run in windowed mode. A few will to fullscreen animations, as well. Here are a few fun examples.
Most users should only need to download the MGL repo and not need to recompile anything. The necessary binaries for Matlab mex-functions and the mglMetal Swift application should be included and ready to go.
In case you do need to build it, there are three places to look these days (as of Spring 2022).
This builds most of the mex-functions for MGL, including functions for things like HID inputs and sounds.
This builds mex-functions specifically for socket communications with the new mglMetal swift application. After building you can test the results with mglTestSocket.m. The test will open a pair of sockets and send a bunch of data between them, using various data types that the mglMetal Swift app supports.
To build the standalone mglMetal Swift application, you need to use Xcode. From testing so far, it looks like you'll need macOS 10.15.7 Catalina or later, with Xcode 12.4 or later.
To compile from XCode, open up the mglMetal application which is in mgl/metal/mglMetal.xcodeproj
Then in Xcode click on the sideways triangle, “play” button, at the top left to build the mglMetal application. Xcode should launch the mglMetal app and your desktop should look similar to the following.
[Xcode app build]
You can press the square “stop” button at the top left to close the app. When running from Matlab, Matlab will take care of starting and stopping the mglMetal app.
The Xcode build creates a fresh copy of mglMetal.app into the MGL repo at
When this “latest” version is present, Matlab will use it. Otherwise, Matlab will default to the “stable” version that comes with the repo at
Note that the default setting for Xcode is to build debug binaries into the directory:
So, mglMetalExecutableName now looks into that directory to see if there is a version of mglMetal that has a newer timestamp then the ones found in the mgl library. If so, it uses that, and will tell you it is doing that when you run mglOpen.
Development of MGL v3 with Metal is still in progress. Although we're not ready to declare v3 complete, a lot of functionality is in place. You might want to test it out!
Some things we've tested and seem to be working:
Some things we know are not done yet:
The tables below goes into more detail about specific mgl functionality and what's in progress.
You can also look for the latest known issues at GitHub.
|mglOpen||working||Currently starts out as windowed only. Can we pass initial window state as command line arg? Or start out with window hidden?|
|mglFlush||working||Works by busy-waiting on the flush within mglMetal - seems like this could be rethought to free up time on the matlab side. Schedule instead of wait?|
|mglCreateTexture||working||2D float textures working. 1D uint textures not yet. What to do about old mglBltTExture gl features? Need to deal with the alignment to 256 byte issue.|
|mglText||working||Works via Matlab image and mglMetalCreateTexture. Wants some love for backwards colors, color grading, alpha blending, and texture sizing vs natural font size|
|mglFrameGrab.m||Not working per se, but mglMetalSetRenderTarget.m and mglMetalReadTexture.m are working|
|mglTextDraw.m||Can we deprecate on-the-fly texture creation and instead provide way to update text in an exiting texture?|
|mglTextSet.m||working||Setting params to both globals mgl and MGL. Should we pick one?|
|mglTransform.m||working||simplified for Metal, no longer supports various OpenGL operations and constants|
|mglVolume.m||(mglPrivateVolume) Volume property does not exist|
|mglCharToKeycode.m||working||Fixed crash from Catalina on|
|mglGetKeyEvent.m||working||Fixed crash from Catalina on|
|mglGetParam.m||working||We have global mgl and MGL, should we pick one?|
|mglKeycodeToChar.m||working||Fixed crash from Catalina on|
|mglSetParam.m||working||We have global mgl and MGL, should we pick one?|
|mglSystemCheck.m||working||Should add socket and metal tests?|
|mglBindTexture.m||Fast binding of textures. No need to continue to support||n/a|
|mglFlushAndWait.m||No need to update / or provide as wrapper||Check|
|mglBindFrameBufferObject.m||Chris Broussard function|
|mglDrawImage.m||Chris Broussard function|
|mglShader.m||Chris Broussard function|
|mglUnbindFrameBufferObject.m||Chris Broussard function|
|mglCreateFrameBufferObject.m||Chris Broussard function|
The v3 Metal version of MGL introduces a standalone app that handles windowing and rendering details. This has some great advantages, for example it lets us take advantage of Apple's shiny integration and tooling for Metal development via Xcode.
It also reduces coupling between the Matlab code and the rendering back end. This gives us a chance to think about how Matlab should integrate with the rendering back end. For v3 we've build the integration around local Unix sockets, giving a clear separation of concerns between task and flow login in Matlab, vs Metal graphics details. From there it's not hard to contemplate rendering to other back ends, like Vulkan or remote sockets.
This section gives an overview and some details about the socket communication.
The overall life-cycle goes like this:
Matlab is generally in control. When we mglOpen(), Matlab starts a new mglMetal.app process. The mglMetal app acts as a server: it binds a socket address and listens for connections. It waits for commands on the connection. For each command, it reads inputs, does some action like rendering, and writes back data as needed. When not processing a command, the mglMetal server is idle. Matlab acts as a client: it connects to the server's socket and initiates all interactions by sending commands over the socket to the server. It decides which commands to send and when. When we mglClose(), Matlab disconnects and terminates the mglMetal.app process.
Everything the mglMetal app does starts with a command from Matlab. The pattern for each command goes like this:
Matlab sends a numeric command code to mglMetal. mglMetal sends back an “ack” timestamp as an acknowledgement that it's ready to process. Matlab sends any additional data for the command, like color, texture, vertex, etc. mglMetal processes the command by rendering and/or sending back requested data. mglMetal sends back a “processed” timestamp to say it's done processing the command. Matlab sits and waits for the “processed” timestamp before proceeding. This pattern is a work in progress. We might want to make it less synchronous? We probably need a way for mglMetal to “nak” or negative-acknowledge a command in case of error, and return to a predictable state after that.
Our goal is to settle on a straightforward command pattern that gets the job done, and can be expressed in a few bullet points similar to the ones above.
Here's a listing of specific commands that mglMetal supports as of March 2022. This is a work in progress, but it might be helpful or interesting to see how the interaction works.
The supported commands are expressed in a shared header that us used to build MGL's mex-functions as well as the mglMetal app itself. The v3 rendering functions are implemented by sending these commands, instead of by making OpenGL framework calls.
Several “non-drawing” commands are intended to manage system resources and/or request data. These are executed one at a time, ie once per frame.
|Command||Data From Matlab||Data to Matlab||Notes|
|mglPing||echo mglPing command|
|mglDrainSystemEvents||Consume all window events from the OS queue. Do we need this?|
|mglFullscreen||Make mglMetal.app go fullscreen.|
|mglWindowed||Make mglMetal.app go to windowed mode.|
|mglSetClearColor||RGB clear color||Set the RGB color to clear to, next time we start a frame.|
|mglCreateTexture||width, height, and rgba float texture data||texture number||Create a new Metal texture.|
|mglReadTexture||texture number||width, height, and rgba float texture data||Read an existing Metal texture as an image.|
|mglSetRenderTarget||texture number||Set the render target for next time we start a frame – on screen or an existing texture.|
|mglSetWindowFrameInDisplay||display number, x, y, width, height||Move the mglMetal window.|
|mglGetWindowFrameInDisplay||display number, x, y, width, height||Where is the mglMetal window?|
|mglDeleteTexture||texture number||Release an existing Metal texture.|
For actual rendering, we send actual drawing commands. When mglMetal receives one of these it enters a tight loop and looks for additional drawing commands to apply in the current frame. It stays in the tight loop until a flush command, which signals the end of a frame and causes mglMetal to present the frame and go back to idle.
|Command||Data From Matlab||Data to Matlab||Notes|
|mglFlush||Present the current frame and go back to idle.|
|mglBltTexture||texture number, vertices with texture coordinates||Blt the given texture to a polygon surface.|
|mglSetXform||4×4 matrix||Set the vertex coordinate transform going forward.|
|mglDots||vertices with color, size, and shape components||Draw the given vertices as points, with various shape options.|
|mglLine||vertices with color components||Draw the given vertices as a line.|
|mglQuad||vertices with color components||Draw the given vertices as a triangles.|
|mglPolygon||vertices with color components||Draw the given vertices as a triangle strip.|
|mglArcs||vertices with color, size, and shape components||Draw the given vertices as points, with various curved shape options.|
|mglUpdateTexture||texture number, width, height, and rgba float texture data||Replace the given texture's contents, to be displayed during this frame.|