Differences

This shows you the differences between two versions of the page.

--- mgl:mglmetal [2020/05/08 12:10]
+++ mgl:mglmetal [2023/08/09 12:19] (current)
@@ Line 1: / Line 1: @@
 ======  mglMetal (mgl version 3.0) ======
-===== Overview =====
+====== Overview ======
-[[https://developer.apple.com/documentation/metal|Metal]] is coming. Cupertino has decided to bring more joy and happiness to the world by deprecating the widely used open standard OpenGL, thus making everyone's code obsolete. There are [[https://developer.apple.com/videos/play/wwdc2019/611/|ominous pronouncements]] that OpenGL is due to die, most likely in the next major MacOS version after Catalina, to be replaced by their own proprietary standard Metal. As all the graphics in mgl is written with OpenGL using Cocoa frameworks (a variant of C called Objective-C), we will need to rewrite the backend of mgl to make it compatible.
+[[https://developer.apple.com/documentation/metal|Metal]] is here! Cupertino has decided to bring more joy and happiness to the world by deprecating the widely used open standard OpenGL, thus making everyone's code obsolete. The [[https://developer.apple.com/videos/play/wwdc2019/611/|ominous pronouncements]] have proved true, and OpenGL is no longer supported on MacOS versions after Catalina (like Big Sur and Monterey). These now require us to use Apple's proprietary standard Metal. As all the graphics in mgl was written with OpenGL using Cocoa frameworks (a variant of C called Objective-C), we are now rewriting the backend of mgl to make it compatible.
  **<sigh>**
-The good news is that since mgl is written using [[:mgl:overview#design_philosophy|simple, atomic functions]] with future OS API compatibility in mind, this transition is not such a monumental task. In fact, as of January 2020, I (jlg) have an alpha version of the mgl code running that is able to do the most important basic functions such as clearing the screen (mglClearScreen), drawing points (mglPoints), drawing lines (mglFixationCross, mglLines), drawing quads (mglQuads) and textures (mglCreateTexture, mglBltTexture).
+The good news is that since mgl is written using [[:mgl:overview#design_philosophy|simple, atomic functions]] with future OS API compatibility in mind, this transition is not such a monumental task. In fact, as of January 2020, I (jlg) already had an alpha version of the mgl code running that is able to do the most important basic functions such as clearing the screen (mglClearScreen), drawing points (mglPoints), drawing lines (mglFixationCross, mglLines), drawing quads (mglQuads) and textures (mglCreateTexture, mglBltTexture).
+We (jlg and Ben Heasly) are continuing this work in 2021 and 2022.
 As this will be the third major rewrite of the mgl code, it will be version 3.0 and will be written primarily in Swift. The first version was written using the 32-bit Carbon API and the [[mgl:mgl2p0|second version]] was written using the 64-bit Cocoa Frameworks.
@@ Line 15: / Line 17: @@
 There will be advantages to the new 3.0 metal compliant version of mgl.
-  * **Fast GPU code** Metal is much clearer than OpenGL about what happens on the CPU and what happens on the GPU. In fact, one always has to write two functions that run on the GPU, one for doing coordinate transformation of vertices called a //vertex shader// and another for coloring the vertices (and things in between vertices) called a //fragment shader//. The upside of this is that we can do some operations that used to be a bit clunky and slow on the GPU in a very elegant and super-fast way.
+  * **Fast GPU code** Metal is much clearer than OpenGL about what happens on the CPU and what happens on the GPU. In fact, one always has to write two functions that run on the GPU, one for doing coordinate transformation of vertices called a //vertex shader// and another for coloring the vertices (and things in between vertices) called a //fragment shader//. The upside of this is that we can do some operations that used to be a bit clunky and slow on the GPU in a very elegant and super-fast way. Some preliminary tests of speed can be found [[mgl:renderpipelinetests | here]].
   * **Drifting gratings** Changing the phase of a grating inside a gaussian window, can be written in much simpler fashion with the phase offset being coded on the GPU. This is already implemented.
   * **White noise stimuli**Complex, time critical code can be put on the GPU, so things like white noise stimuli with frame-by-frame accuracy can easily be accommodated. This has not yet been implemented, but could be easily added to the existing structures.
@@ Line 21: / Line 23: @@
   * **Window and full screen display** Another advantage is that by having mglMetal work as a standalone app that communicates with Matlab, we can allow it to display both to a 2nd display and a window at the same time, thus allowing the experimenter to see the display at the same time as the subject.
   * **iPad display** With the current setup, we should be able to display to an iPad. This is done trivially with Apple's new [[https://www.apple.com/macos/catalina/docs/Sidecar_Tech_Brief_Oct_2019.pdf|sidecar]] feature. It also could be done in the feature by making the mglMetal display an iOS application (should be easy given that the frameworks all work both on MacOS and iOS) and talking over a network socket. This could be useful for displays in medical clinics and other places where subjects maybe more comfortable using an iPad than a fixed display.
-  * **Python bridging** The new system communicates with sockets so that the controlling side (matlab) has minimal requirements (has to have an open, close, read and write sockets commands implemented). This will have benefits in using this with other systems as functionality can be accessed with a few simple socket read/write commands. It also means that having interfaces in other programming languages like Python could be easily written.
+  * **Integration options** The new system communicates with sockets so that the controlling side (matlab) has minimal requirements (has to have an open, close, read and write sockets commands implemented). This will have benefits in using this with other systems as functionality can be accessed with a few simple socket read/write commands. It also means that having interfaces to other backends, like Vulkan (Khronos Group's official successor to OpenGL) could be developed as a drop-in replacement for Metal, while preserving all same Matlab code and the domain knowledge it encapsulates. In theory, interfaces for other programming languages like Python could also be written, though these would not benefit from mgl's rich domain knowledge.
+  * Deep color Metal works with high bit depth color modes like those which have 10 or more bits per color channel. See [[mgl:highbitdepth | here]].
 ====== Download =====
-The alpha mgl 3.0 can be retrieved using git by cloning and then switching to the metal branch
+The mgl 3.0 work in in progress can be retrieved using git by cloning and then switching to the metal branch
 <code>
@@ Line 40: / Line 43: @@
 </code>
-====== Testing =====
+Note that depending on your version of MacOS, the [[https://support.apple.com/en-us/HT202491 | Gatekeeper ]] may block the system from running mgl, in which case you may see an error like the following pop-up:
-I've tested this on the latest version of Catalina, not sure if it will work on Mojave or older OS. You probably will need a current version of XCode because Cupertino wouldn't allow compilation (at least for my setup) w/out updating everything.
+{{ :mgl:macos-big-sur-alert-app-checked.png?200 |}}
-First go to matlab and start up mglMetalTest. This will run mglSocketOpen to setup a socket server, which will wait for the mglMetal application to make a connection. Don't hit enter yet, as you need to start the mglMetal application.
+If you go to Apple/System Preferences…/Security & Privacy/General you can manually allow it to run. However, it may asks you this for every single compiled function. In which case, you can [[https://macpaw.com/how-to/allow-apps-anywhere#:~:text=Enter%20your%20Mac's%20password.,from%20with%20automatically%20selected%20Anywhere.| disable GateKeeper]] temporarily by going to a terminal and doing (you will need to use a password that has root level access):
-<code matlab>
->> mglMetalTest
+<code="csh">
-(mglSocketOpen) Opened socket testsocket with socketDescriptor: 298
+sudo spctl --master-disable
-Hit ENTER to ping:
 </code>
-From XCode, open up the mglMetal application which is in mgl/mglMetal/mglMetal.xcodeproj
+After everything is running properly, you may want to enable again by doing:
-<html><div></html>
+<code="csh">
-{{:mgl:metal_file.png|}}
+sudo spctl --master-enable
-<html></div></html>
+</code>
-Then click on the sideways triangle, play button, in XCode at the top left to build and start up the mglMetal application. Xcode should look like the following, with the console showing that a socket connection has been opened and that mglRenderer is processing OS events:
-<html><div></html>
+====== Testing ======
-{{:mgl:mglmetal_xcode.png|}}
-<html></div></html>
-You should also see an mglMetal window that has opened. This is where stimuli will be displayed. Note that it may show up as a full screen window somewhere offscreen (still working on that!). If it does you won't see a window, but if you swipe right with three fingers you will get to a gray screen (or if you do command-tab, you will see the mgl icon as one of the running applications).
+We've tested this on a few versions of macOS including Catalina, Big Sur, and Monterey. We're not sure if it will work on Mojave or older OS. Overall, we're focusing on the newer versions and hardware, and trying to move moving forward from there.
-<html><div></html>
+Once you've downloaded above, you should be ready to run some tests.
-{{:mgl:mglMetal_init.png|}}
-<html></div></html>
-Now, go back to Matlab and hit enter. This should ping the mglMetal application and return something like this:
+===== Automated Tests =====
+A quick way to test that Metal rendering is working is to run through a suite of tests located at mgl/mgllib/mglTestMetal/. To execute all the tests in a row, run
 <code>
-Hit ENTER to ping:
+>> mglRunRenderingTests();
-(mglSocketWrite) Waiting for a new connection
-(mglSocketWrite) New connection made: 301
-(mglSocketWrite) Using connectionDescriptor 301
-(mglSocketWrite) Wrote 2 of 2 bytes (Len: 1 (1 x 1), dataSize: 2)
-(mglSocketWrite) Using connectionDescriptor 301
-(mglSocketWrite) Wrote 2 of 2 bytes (Len: 1 (1 x 1), dataSize: 2)
-(mglSocketWrite) Using connectionDescriptor 301
-(mglSocketWrite) Wrote 2 of 2 bytes (Len: 1 (1 x 1), dataSize: 2)
-(mglSocketDataWaiting) Using connectionDescriptor 301
 </code>
-That means that the connection is working. If that is not what happens, you might want to stop the mglMetal application and hit ctrl-c on the Matlab side and start again (i.e. first run mglMetalTest on Matlab, then rebuild/run the mglMetal application in XCode and hit enter to ping).
+This will rapidly start and stop the mglMetal app several times. Each time it will issue some rendering commands to be rendered off-screen, then read back the rendering results and compare to a known good “snapshot” image. If all renderings match the expected snapshots, you should see output like:
-If all is well and you got a successful ping, then the communication is setup and you should be in business. You will get a prompt in Matlab like this:
-<code matlab>
+<code>
-Hit ENTER to clear screen:
+>> mglRunRenderingTests()
-</code>
-After hitting enter the mglMetal display should turn red:
+Running 32 tests.
-{{mgl:screen_shot_2020-01-28_at_8.20.12_pm.png}}
-On Matlab then you get the prompt:
+... some logging ...
-<code matlab>
-Hit ENTER to test lines:
-</code>
-And the screen should look like this after you hit ENTER:
+All tests passed!
+ans =
+x32 struct array with fields:
-{{mgl:screen_shot_2020-01-28_at_8.20.25_pm.png}}
+    snapshot
+    renderedImage
+    isSuccess
+    testName
+    snapshotData
-Then in Matlab:
-<code matlab>
-Hit ENTER to test quads:
 </code>
-And a display like this:
+If any rendering don't match its expected snapshot, mglRunRenderingTests will raise a figure showing the expected vs actual image.
-{{mgl:screen_shot_2020-01-28_at_8.20.28_pm.png}}
-It should print out some profiling information, about how long it takes to send and respond to the mglQuad command (first one below) and how long it takes to display. The first one should be way under a ms as it just the communication delay, and the second one should be under one frame refesh (16.7ms for 60Hz).
+===== Interactive Demo =====
-<code matlab>
+You can run through a demo of MGL Metal rendering with
-(mglMetalTest:mglProfile) Profile time for mglProfileOn is: 0.080889 ms
-(mglMetalTest:mglProfile) Profile time for mglQuad is: 0.694815 ms
-</code>
-You can now make it flicker
+<code>
+>> mglRenderingDemo
-<code matlab>
+Running 32 demos.
-Hit ENTER to flicker:
-</code>
+: Hit any key to continue for mglTestClearScreen.
-Which should just make the checkerboard flicker.
+... some logging ...
-Next we test coordinate xforms (which will be used for setting display coordinates to visual angles in the future). You should get two prompts which will first display three dots offset to the left, and then in the center:
+The screen should be cleared to an orange-brown color.
-<code matlab>
-Hit ENTER to test coordinate xform:
-Hit ENTER to test coordinate xform:
 </code>
-{{mgl:screen_shot_2020-01-28_at_8.20.36_pm.png}}{{mgl:screen_shot_2020-01-28_at_8.20.42_pm.png}}
+The screen should be cleared to an orange-brown color.
+The demo will pause here, and you can compare the expected message, “The screen should be cleared to an orange-brown color.” to what you see in the mglMetal window. It should look like this:
-Now, for some (slightly) more fun stuff. Next should display a vertical gabor. Note the profile time tells you how long mglCreate and mglBlt texture take.
+{{ :mgl:mgltestclearscreen_output.png? 400 |}}
-<code matlab>
+And so on, for several more renderings. Most will run in windowed mode. A few will to fullscreen animations, as well. Here are a few fun examples.
-Hit ENTER to test texture:
-(mglMetalTest:mglProfile) Profile time for mglProfileOn is: 0.117438 ms
-(mglSocketWrite) Using connectionDescriptor 302
-(mglSocketWrite) Wrote 1048576 of 1048576 bytes (Len: 262144 (1 x 262144), dataSize: 4)
-(mglMetalTest:mglProfile) Profile time for mglCreateTexture is: 3.228833 ms
-(mglMetalTest:mglProfile) Profile time for mglBltTexture is: 0.453883 ms
-</code>
-{{mgl:screen_shot_2020-01-28_at_8.20.46_pm.png}}
+====== Building ======
+Most users should only need to download the MGL repo and not need to recompile anything. The necessary binaries for Matlab mex-functions and the mglMetal Swift application should be included and ready to go.
-Next it should go full screen, show you the grating drifting to the left and then go back to a windowed context:
+In case you do need to build it, there are three places to look these days (as of Spring 2022).
-<code matlab>
+===== mglMakeMetal.m =====
-Hit ENTER to test blt (drifting):
+This builds most of the mex-functions for MGL, including functions for things like HID inputs and sounds.
-(mglMetalTest) Median frame time: 0.0167
-(mglMetalTest) Max frame time: 0.0221
+<code>
-(mglMetalTest) Number of frames over 0.0167: 164/271
+mglMakeMetal()
-(mglMetalTest) Number of frames 5% over 0.0167: 17/271
-(mglMetalTest) Number of frames 10% over 0.0167: 1/271
-(mglMetalTest) Number of frames 20% over 0.0167: 1/271
 </code>
-Next it will do something similar, but show you drifting gratings at different locations and orientations.
+===== mglMakeSocket.m =====
-<code matlab>
+This builds mex-functions specifically for socket communications with the new mglMetal swift application. After building you can test the results with mglTestSocket.m. The test will open a pair of sockets and send a bunch of data between them, using various data types that the mglMetal Swift app supports.
-Hit ENTER to test blt (multiple rotating and drifting):
-(mglMetalTest) Median frame time: 0.0167
+<code>
-(mglMetalTest) Max frame time: 0.0252
+mglMakeSocket()
-(mglMetalTest) Number of frames over 0.0167: 163/271
+mglTestSocket()
-(mglMetalTest) Number of frames 5% over 0.0167: 51/271
-(mglMetalTest) Number of frames 10% over 0.0167: 1/271
-(mglMetalTest) Number of frames 20% over 0.0167: 1/271
 </code>
-{{mgl:screen_shot_2020-01-28_at_8.20.59_pm.png}}
+===== mglMetal.app =====
+To build the standalone mglMetal Swift application, you need to use Xcode. From testing so far, it looks like you'll need macOS 10.15.7 Catalina or later, with Xcode 12.4 or later.
-Next, a test of dots
+To compile from XCode, open up the mglMetal application which is in mgl/metal/mglMetal.xcodeproj
-<code matlab>
+{{ :mgl:mglmetalapp_folder.png?400 |}}
-Hit ENTER to test dots:
-(mglMetalTest:mglProfile) Profile time for mglProfileOn is: 0.074804 ms
-(mglMetalTest:mglProfile) Profile time for mglPoints2 is: 0.422125 ms
-</code>
-{{mgl:screen_shot_2020-01-28_at_8.21.08_pm.png}}
+Then in Xcode click on the sideways triangle, “play” button, at the top left to build the mglMetal application. Xcode should launch the mglMetal app and your desktop should look similar to the following.
-Then a test of moving dots, this will also go full screen and then should go back to a windowed context.
+[Xcode app build]
-<code matlab>
+You can press the square “stop” button at the top left to close the app. When running from Matlab, Matlab will take care of starting and stopping the mglMetal app.
-Hit ENTER to test dots:
-(mglMetalTest) Median frame time: 0.0167
-(mglMetalTest) Max frame time: 0.0924
-(mglMetalTest) Number of frames over 0.0167: 554/971
-(mglMetalTest) Number of frames 5% over 0.0167: 137/971
-(mglMetalTest) Number of frames 10% over 0.0167: 24/971
-(mglMetalTest) Number of frames 20% over 0.0167: 5/971
-</code>
-{{mgl:screen_shot_2020-01-28_at_8.21.14_pm.png}}
+The Xcode build creates a fresh copy of mglMetal.app into the MGL repo at
-Note that mglMetalTest with no arguments, starts up a new socket which the mglMetal application would have to reconnect with, so if you want to test the sequence again, without restarting mglMetal, you can run it like this:
+<code>
+mgl/metal/binary/latest/mglMetal.app
+</code>
-<code matlab>
+When this “latest” version is present, Matlab will use it. Otherwise, Matlab will default to the “stable” version that comes with the repo at
-mglMetalTest(1);
+<code>
+mgl/metal/binary/stable/mglMetal.app
 </code>
+Note that the default setting for Xcode is to build debug binaries into the directory:
+  ~/Library/Developer/Xcode/DerivedData/Build/Products/Debug
+So, mglMetalExecutableName now looks into that directory to see if there is a version of mglMetal that has a newer timestamp then the ones found in the mgl library. If so, it uses that, and will tell you it is doing that when you run mglOpen.
 ====== Development status =====
-The following table is as of January 2020. The framework for all the key functionality that needs to be ported over to Metal is in place.
+Development of MGL v3 with Metal is still in progress. Although we're not ready to declare v3 complete, a lot of functionality is in place. You might want to test it out!
+Some things we've tested and seem to be working:
+  * macOS Catalina or later, with M1 or Intel
+  * Matlab 2021b or later
+  * test/demo scripts mentioned above with mglRunRenderingTests and mglRenderingDemo
+  * mglRetinotopy task
+  * fullscreen vs windowed mode, and getting and setting window position and which display
+  * creating, blt-ing, updating, and deleting textures
+  * rendering to texture and reading results back to Matlab
+  * some initial integration with [[https://www.psychbench.org/ | PsychBench]]
+Some things we know are not done yet:
+  * multiple simultaneous displays/windows
+  * frame syncing and scheduling
+  * documentation for v3 and socket-based integration with the standalone mglMetal.app
+  * publish mgl v3 on the [[https://github.com/ToolboxHub/ToolboxRegistry | ToolboxRegistry]]
+  * movies
+  * stencils
+  * dig deeper into what Metal offers for things like deep color and linearization 🤘
+  * back end for Windows or Linux, using Vulkan instead of Metal 🖖
+The tables below goes into more detail about specific mgl functionality and what's in progress.
+You can also look for the latest known [[https://github.com/justingardner/mgl/issues\ | issues at GitHub]].
+===== Major functionality =====
+^ Function name ^ Status ^ Notes ^
+| mglOpen | working | Currently starts out as windowed only. Can we pass initial window state as command line arg? Or start out with window hidden?|
+| mglFlush | working | Works by busy-waiting on the flush within mglMetal - seems like this could be rethought to free up time on the matlab side. Schedule instead of wait? |
+| mglClose | working |  |
+| mglClearScreen | working | |
+| mglLines2 | working| |
+| mglPoints2 | working | |
+| mglPoints2c | working | |
+| mglQuad | working | |
+| mglFixationCross | working | |
+| mglCreateTexture | working | 2D float textures working. 1D uint textures not yet. What to do about old mglBltTExture gl features? Need to deal with the alignment to 256 byte issue. |
+| mglBltTexture | working | |
+| mglText | working | Works via Matlab image and mglMetalCreateTexture. Wants some love for backwards colors, color grading, alpha blending, and texture sizing vs natural font size |
+| mglVisualAngleCoordinates | working |  |
+| mglMoveWindow.m | working | |
+| mglGetWindowPos.m | working | |
+===== Secondary priority functionality =====
+^ Function name ^ Status ^ Notes ^
+| mglFrameGrab.m | | Not working per se, but mglMetalSetRenderTarget.m and mglMetalReadTexture.m are working |
+| mglMovie.m | |  |
+| mglSwitchDisplay.m | |  |
+| mglStencilCreateBegin.m | |  |
+| mglStencilCreateEnd.m	 | |  |
+| mglStencilSelect.m | |  |
+| mglHFlip.m | working|  |
+| mglVFlip.m | working|  |
+| mglScreenCoordinates.m | working|  |
+===== Minor functionality =====
+^ Function name ^ Status ^ Notes ^
+| mglFillOval.m | working | |
+|mglFillRect.m	|working | |
+|mglFillRect3D.m | working | |
+|mglGluAnnulus.m | working  | |
+|mglGluDisk.m | working	 | |
+|mglGluPartialDisk.m |working | |
+|mglPoints3.m | working	 | |
+|mglPolygon.m | working	 | |
+|mglTextDraw.m | Can we deprecate on-the-fly texture creation and instead provide way to update text in an exiting texture?  |
+|mglTextSet.m	|working | Setting params to both globals mgl and MGL. Should we pick one?|
+|mglTransform.m	|working | simplified for Metal, no longer supports various OpenGL operations and constants|
+|mglVolume.m | | (mglPrivateVolume) Volume property does not exist|
+===== Functionality not affected by Metal =====
+^ Function name ^ Status ^ Notes ^
+|mglSetGammaTable | | needs testing|
+|mglGetGammaTable | | needs testing|
+|mglCharToKeycode.m | working | Fixed crash from Catalina on|
+|mglDeleteSound.m | working | |
+|mglDescribeDisplays.m | working | |
+|mglDisplayCursor.m | | |
+|mglGetKeyEvent.m | working | Fixed crash from Catalina on |
+|mglGetKeys.m | working	| |
+|mglGetMouse.m | working | |
+|mglGetMouseEvent.m | working | |
+|mglGetParam.m | working | We have global mgl and MGL, should we pick one?|
+|mglGetSecs.m | working	|
+|mglInstallSound.m | working	| |
+|mglIsCursorVisible.m | | |
+|mglKeycodeToChar.m | working | Fixed crash from Catalina on |
+|mglListener.m	| working | |
+|mglPlaySound.m	| working | |
+|mglPostEvent.m	| working | |
+|mglPrivateListener.m |	working	| |
+|mglResolution.m | | |
+|mglSetMousePosition.m	| working | |
+|mglSetParam.m	|working | We have global mgl and MGL, should we pick one?|
+|mglSetSound.m	| | |
+|mglShowKey.m	|not working | |
+|mglSimulateRun.m |working| |
+|mglSystemCheck.m |working | Should add socket and metal tests?|
+|mglWaitSecs.m	| working | |
+===== Functionality no longer needed =====
+^ Function name ^ Notes ^ Done ^
+| mglBindTexture.m | Fast binding of textures. No need to continue to support  | n/a|
+|mglStrokeText.m | Discontinue | n/a|
+|mglNoFlushWait.m | Still needed? | Check|
+|mglFlushAndWait.m |  No need to update / or provide as wrapper | Check |
+===== Brainard Lab functionality =====
+^ Function name ^ Notes ^ Done ^
+| mglBindFrameBufferObject.m | Chris Broussard function | |
+| mglDrawImage.m | Chris Broussard function | |
+| mglShader.m | Chris Broussard function | |
+| mglUnbindFrameBufferObject.m | Chris Broussard function | |
+| mglCreateFrameBufferObject.m | Chris Broussard function | |
+====== Matlab - Metal Communications ======
+The v3 Metal version of MGL introduces a standalone app that handles windowing and rendering details. This has some great advantages, for example it lets us take advantage of Apple's shiny integration and tooling for Metal development via Xcode.
+It also reduces coupling between the Matlab code and the rendering back end. This gives us a chance to think about how Matlab should integrate with the rendering back end. For v3 we've build the integration around local Unix sockets, giving a clear separation of concerns between task and flow login in Matlab, vs Metal graphics details. From there it's not hard to contemplate rendering to other back ends, like Vulkan or remote sockets.
+This section gives an overview and some details about the socket communication.
+===== Overview =====
+The overall life-cycle goes like this:
+Matlab is generally in control.
+When we mglOpen(), Matlab starts a new mglMetal.app process.
+The mglMetal app acts as a server: it binds a socket address and listens for connections. It waits for commands on the connection. For each command, it reads inputs, does some action like rendering, and writes back data as needed. When not processing a command, the mglMetal server is idle.
+Matlab acts as a client: it connects to the server's socket and initiates all interactions by sending commands over the socket to the server. It decides which commands to send and when.
+When we mglClose(), Matlab disconnects and terminates the mglMetal.app process.
+===== Command Pattern =====
+Everything the mglMetal app does starts with a command from Matlab. The pattern for each command goes like this:
+Matlab sends a numeric command code to mglMetal.
+mglMetal sends back an “ack” timestamp as an acknowledgement that it's ready to process.
+Matlab sends any additional data for the command, like color, texture, vertex, etc.
+mglMetal processes the command by rendering and/or sending back requested data.
+mglMetal sends back a “processed” timestamp to say it's done processing the command.
+Matlab sits and waits for the “processed” timestamp before proceeding.
+This pattern is a work in progress. We might want to make it less synchronous? We probably need a way for mglMetal to “nak” or negative-acknowledge a command in case of error, and return to a predictable state after that.
+Our goal is to settle on a straightforward command pattern that gets the job done, and can be expressed in a few bullet points similar to the ones above.
+===== Specific Commands =====
+Here's a listing of specific commands that mglMetal supports as of March 2022. This is a work in progress, but it might be helpful or interesting to see how the interaction works.
+The supported commands are expressed in a [[https://github.com/justingardner/mgl/blob/metal/metal/mglMetal/mglCommandTypes.h | shared header]] that us used to build MGL's mex-functions as well as the mglMetal app itself. The v3 rendering functions are implemented by sending these commands, instead of by making OpenGL framework calls.
+Several “non-drawing” commands are intended to manage system resources and/or request data. These are executed one at a time, ie once per frame.
+^ Command ^ Data From Matlab ^ Data to Matlab ^ Notes ^
+| mglPing | | echo mglPing command | |
+| mglDrainSystemEvents | | | Consume all window events from the OS queue. Do we need this? |
+| mglFullscreen | | | Make mglMetal.app go fullscreen. |
+| mglWindowed | | | Make mglMetal.app go to windowed mode. |
+| mglSetClearColor | RGB clear color | | Set the RGB color to clear to, next time we start a frame. |
+| mglCreateTexture | width, height, and rgba float texture data | texture number | Create a new Metal texture. |
+| mglReadTexture | texture number | width, height, and rgba float texture data  | Read an existing Metal texture as an image. |
+| mglSetRenderTarget | texture number |  | Set the render target for next time we start a frame – on screen or an existing texture. |
+| mglSetWindowFrameInDisplay | display number, x, y, width, height | | Move the mglMetal window. |
+| mglGetWindowFrameInDisplay | | display number, x, y, width, height | Where is the mglMetal window? |
+| mglDeleteTexture | texture number | | Release an existing Metal texture. |
+For actual rendering, we send actual drawing commands. When mglMetal receives one of these it enters a tight loop and looks for additional drawing commands to apply in the current frame. It stays in the tight loop until a flush command, which signals the end of a frame and causes mglMetal to present the frame and go back to idle.
+^ Command ^ Data From Matlab ^ Data to Matlab ^ Notes ^
+| mglFlush | | | Present the current frame and go back to idle.  |
+| mglBltTexture | texture number, vertices with texture coordinates | | Blt the given texture to a polygon surface. |
+| mglSetXform | 4×4 matrix | | Set the vertex coordinate transform going forward. |
+| mglDots | vertices with color, size, and shape components | | Draw the given vertices as points, with various shape options. |
+| mglLine | vertices with color components | | Draw the given vertices as a line. |
+| mglQuad | vertices with color components | | Draw the given vertices as a triangles. |
+| mglPolygon | vertices with color components | | Draw the given vertices as a triangle strip. |
+| mglArcs | vertices with color, size, and shape components | | Draw the given vertices as points, with various curved shape options. |
+| mglUpdateTexture | texture number, width, height, and rgba float texture data	 | | Replace the given texture's contents, to be displayed during this frame.  |
-^ Function name ^ Implementation status ^ Notes ^
-| mglOpen | Working alpha | Code for this is in mglMetalTest, not yet pulled out into a clean function with all the same functionality as the mglOpen function, but should be trivial.|
-| mglFlush | Working alpha | Works by busy-waiting on the flush within mglMetal - seems like this could be rethought to free up time on the matlab side |
-| mglClose | Not yet implemented | Trivial. But, need to decide if there should be a close state in which mglMetal is minimized |
-| mglClearScreen | Working alpha | |
-| mglLine | Working alpha | |
-| mglPoints2 | Working alpha | |
-| mglQuads | Working alpha | |
-| mglFixationCross | Working alpha | |
-| mglCreateTexture | Working alpha | |
-| mglBltTexture | Working alpha | |
-| mglVisualAngleCoordinates | Not yet implemented | Trivial. Already have the shader vertex code setup to accept a transformation matrix from matlab |
-| mglSetGammaTable | No need to update | |
-| mglGetGammaTable | No need to update | |
-| Keyboard / mouse functions | No need to update | |
-| National Instruments digital / analog I/O | NO need to update | |
-| Task code | No need to update | |

Gardner Lab

Stanford University

Differences

Gardner Lab

Stanford University

Site Tools

Differences

Page Tools