Metal is coming. Cupertino has decided to bring more joy and happiness to the world by deprecating the widely used open standard OpenGL, thus making everyone's code obsolete. There are ominous pronouncements that OpenGL is due to die, most likely in the next major MacOS version after Catalina, to be replaced by their own proprietary standard Metal. As all the graphics in mgl is written with OpenGL using Cocoa frameworks (a variant of C called Objective-C), we will need to rewrite the backend of mgl to make it compatible.

<sigh>

The good news is that since mgl is written using simple, atomic functions with future OS API compatibility in mind, this transition is not such a monumental task. In fact, as of January 2020, I (jlg) have an alpha version of the mgl code running that is able to do the most important basic functions such as clearing the screen (mglClearScreen), drawing points (mglPoints), drawing lines (mglFixationCross, mglLines), drawing quads (mglQuads) and textures (mglCreateTexture, mglBltTexture).

As this will be the third major rewrite of the mgl code, it will be version 3.0. The first version was written using the 32-bit Carbon API and the second version was written using the 64-bit Cocoa Frameworks.

There will be advantages to the new 3.0 metal compliant version of mgl.

• Fast GPU code Metal is much clearer than OpenGL about what happens on the CPU and what happens on the GPU. In fact, one always has to write two functions that run on the GPU, one for doing coordinate transformation of vertices called a vertex shader and another for coloring the vertices (and things in between vertices) called a fragment shader. The upside of this is that we can do some operations that used to be a bit clunky and slow on the GPU in a very elegant and super-fast way.
• Drifting gratings Changing the phase of a grating inside a gaussian window, can be written in much simpler fashion with the phase offset being coded on the GPU. This is already implemented.
• White noise stimuliComplex, time critical code can be put on the GPU, so things like white noise stimuli with frame-by-frame accuracy can easily be accommodated. This has not yet been implemented, but could be easily added to the existing structures.
• Windowing The new version will use a standalone application that will communicate with Matlab through sockets. One advantage of this is that we can make the windowing work more like a typical mac application, thus allowing minimizing, maximizing and changing the size dynamically.
• Window and full screen display Another advantage is that by having mglMetal work as a standalone app that communicates with Matlab, we can allow it to display both to a 2nd display and a window at the same time, thus allowing the experimenter to see the display at the same time as the subject.
• iPad display With the current setup, we should be able to display to an iPad. This is done trivially with Apple's new sidecar feature. It also could be done in the feature by making the mglMetal display an iOS application (should be easy given that the frameworks all work both on MacOS and iOS) and talking over a network socket. This could be useful for displays in medical clinics and other places where subjects maybe more comfortable using an iPad than a fixed display.
• Python bridging The new system communicates with sockets so that the controlling side (matlab) has minimal requirements (has to have an open, close, read and write sockets commands implemented). This will have benefits in using this with other systems as functionality can be accessed with a few simple socket read/write commands. It also means that having interfaces in other programming languages like Python could be easily written.

The alpha mgl 3.0 can be retrieved using git by cloning and then switching to the metal branch

git clone https://github.com/justingardner/mgl.git 
cd mgl
git checkout metal

If this worked correctly, then you should see a new directory called metal within the mgl repository:

grumini:/Users/justin/mgl> ls
COPYING     metal/      readme.md   task/
Contents.m  mgllib/     readme.txt  utils/

I've tested this on the latest version of Catalina, not sure if it will work on Mojave or older OS. You probably will need a current version of XCode because Cupertino wouldn't allow compilation (at least for my setup) w/out updating everything.

First go to matlab and start up mglMetalTest. This will run mglSocketOpen to setup a socket server, which will wait for the mglMetal application to make a connection. Don't hit enter yet, as you need to start the mglMetal application.

>> mglMetalTest
(mglSocketOpen) Opened socket testsocket with socketDescriptor: 298
Hit ENTER to ping: 

From XCode, open up the mglMetal application which is in mgl/mglMetal/mglMetal.xcodeproj

Then click on the sideways triangle, play button, in XCode at the top left to build and start up the mglMetal application. Xcode should look like the following, with the console showing that a socket connection has been opened and that mglRenderer is processing OS events:

You should also see an mglMetal window that has opened. This is where stimuli will be displayed. Note that it may show up as a full screen window somewhere offscreen (still working on that!). If it does you won't see a window, but if you swipe right with three fingers you will get to a gray screen (or if you do command-tab, you will see the mgl icon as one of the running applications).

Now, go back to Matlab and hit enter. This should ping the mglMetal application and return something like this:

Hit ENTER to ping: 
(mglSocketWrite) Waiting for a new connection
(mglSocketWrite) New connection made: 301
(mglSocketWrite) Using connectionDescriptor 301
(mglSocketWrite) Wrote 2 of 2 bytes (Len: 1 (1 x 1), dataSize: 2)
(mglSocketWrite) Using connectionDescriptor 301
(mglSocketWrite) Wrote 2 of 2 bytes (Len: 1 (1 x 1), dataSize: 2)
(mglSocketWrite) Using connectionDescriptor 301
(mglSocketWrite) Wrote 2 of 2 bytes (Len: 1 (1 x 1), dataSize: 2)
(mglSocketDataWaiting) Using connectionDescriptor 301

That means that the connection is working. If that is not what happens, you might want to stop the mglMetal application and hit ctrl-c on the Matlab side and start again (i.e. first run mglMetalTest on Matlab, then rebuild/run the mglMetal application in XCode and hit enter to ping).

If all is well and you got a successful ping, then the communication is setup and you should be in business. You will get a prompt in Matlab like this:

Hit ENTER to clear screen: 

After hitting enter the mglMetal display should turn red:

On Matlab then you get the prompt:

Hit ENTER to test lines: 

And the screen should look like this after you hit ENTER:

Then in Matlab:

Hit ENTER to test quads: 

And a display like this:

It should print out some profiling information, about how long it takes to send and respond to the mglQuad command (first one below) and how long it takes to display. The first one should be way under a ms as it just the communication delay, and the second one should be under one frame refesh (16.7ms for 60Hz).

(mglMetalTest:mglProfile) Profile time for mglProfileOn is: 0.080889 ms
(mglMetalTest:mglProfile) Profile time for mglQuad is: 0.694815 ms

You can now make it flicker

Hit ENTER to flicker: 

Which should just make the checkerboard flicker.

Next we test coordinate xforms (which will be used for setting display coordinates to visual angles in the future). You should get two prompts which will first display three dots offset to the left, and then in the center:

Hit ENTER to test coordinate xform: 
Hit ENTER to test coordinate xform: 

Now, for some (slightly) more fun stuff. Next should display a vertical gabor. Note the profile time tells you how long mglCreate and mglBlt texture take.

Hit ENTER to test texture: 
(mglMetalTest:mglProfile) Profile time for mglProfileOn is: 0.117438 ms
(mglSocketWrite) Using connectionDescriptor 302
(mglSocketWrite) Wrote 1048576 of 1048576 bytes (Len: 262144 (1 x 262144), dataSize: 4)
(mglMetalTest:mglProfile) Profile time for mglCreateTexture is: 3.228833 ms
(mglMetalTest:mglProfile) Profile time for mglBltTexture is: 0.453883 ms

Next it should go full screen, show you the grating drifting to the left and then go back to a windowed context:

Hit ENTER to test blt (drifting): 
(mglMetalTest) Median frame time: 0.0167
(mglMetalTest) Max frame time: 0.0221
(mglMetalTest) Number of frames over 0.0167: 164/271
(mglMetalTest) Number of frames 5% over 0.0167: 17/271
(mglMetalTest) Number of frames 10% over 0.0167: 1/271
(mglMetalTest) Number of frames 20% over 0.0167: 1/271

Next it will do something similar, but show you drifting gratings at different locations and orientations.

Hit ENTER to test blt (multiple rotating and drifting): 
(mglMetalTest) Median frame time: 0.0167
(mglMetalTest) Max frame time: 0.0252
(mglMetalTest) Number of frames over 0.0167: 163/271
(mglMetalTest) Number of frames 5% over 0.0167: 51/271
(mglMetalTest) Number of frames 10% over 0.0167: 1/271
(mglMetalTest) Number of frames 20% over 0.0167: 1/271

Next, a test of dots

Hit ENTER to test dots: 
(mglMetalTest:mglProfile) Profile time for mglProfileOn is: 0.074804 ms
(mglMetalTest:mglProfile) Profile time for mglPoints2 is: 0.422125 ms

Then a test of moving dots, this will also go full screen and then should go back to a windowed context.

Hit ENTER to test dots: 
(mglMetalTest) Median frame time: 0.0167
(mglMetalTest) Max frame time: 0.0924
(mglMetalTest) Number of frames over 0.0167: 554/971
(mglMetalTest) Number of frames 5% over 0.0167: 137/971
(mglMetalTest) Number of frames 10% over 0.0167: 24/971
(mglMetalTest) Number of frames 20% over 0.0167: 5/971

Note that mglMetalTest with no arguments, starts up a new socket which the mglMetal application would have to reconnect with, so if you want to test the sequence again, without restarting mglMetal, you can run it like this:

mglMetalTest(1);

The following table is as of January 2020. The framework for all the key functionality that needs to be ported over to Metal is in place.