Here is a demonstration of how to use stencils using these these functions:

mglOpen;
mglScreenCoordinates;

%Draw an oval stencil
mglStencilCreateBegin(1);
mglFillOval(300,400,[100 100]);
mglStencilCreateEnd;
mglClearScreen;

% now draw some dots, masked by the oval stencil
mglStencilSelect(1);
mglPoints2(rand(1,5000)*500,rand(1,5000)*500);
mglFlush;
mglStencilSelect(0);

purpose: Begin drawing to stencil. Until mglStencilCreateEnd is called, all drawing operations will also draw to the stencil. Check MGL.stencilBits to see how many stencil planes there are. If invert is set to one, then the inverse stencil is made
usage: mglStencilCreateBegin(stencilNumber,invert)

see example above.

purpose: Ends drawing to stencil usage: mglStencilCreateEnd

see example above.

purpose: Sets which stencil to use, 0 for no stencil usage: mglStencilSelect(stencilNumber)

See example above.