Resources for Developers

We have developed a new hardware (camera and pointers) which allows multiple users (10, 80 or later even 500) to interact with a large screen (projection/display). You can see the products here.

But our hardware is nothing without applications which generate a new user experience.

With Raving Bots (greetings to Bartek) we soon found a solution to get the data into Unity. Unfortunately the solution was too complex for us, we are Unity newbies.

We started installing Unity end of September 2022 and learned C# and then we have developed an ultra fast interface for Unity. The basic trick is simple: We setup a block of shared memory and a very simple code within a Unity class opens the shared memory. Our process creates and then fills the shared memory structures with data and within Unity the same physical memory is read through circular FIFOs having enough space for 600 pointing devices. The X and Y co-ordinates as well as the button states and flags (e.g. on/off screen) for e.g. 80 pointing devices is read by the Unity code in less than 1 microsecond.

It is not necessary for the game to talk to the hardware or even worse to wait for the hardware. And it is pretty fast, because we do not ask the operating system to forward messages or draw multiple cursors in different shapes.

Naturally we also implemented the other way round: The Unity game sends some commands to the pointing devices like different vibration sequences or play melodies etc. Writing these commands to e.g. 80 pointing devices also needs less than 1 microsecond. Our process then sends the commands to our hardware.

Multiple Screen Handling:

We support 16 cameras looking to 16 different projection screens, so if one Unity game generates games which span over several screens (next to each other or not) then this will work. This is the reason why there are 16 XYK FIFOs where we feed the pointer data into the matching screen FIFO. This is also the reason why the mobile phones must select a screen number before starting!

If there are different games running simultaneously, each game may attach to the one and only shared memory but must not use the same XYK FIFO which a different game/application is already using!

You can download a Unity game project including the C# source code of the reader/writer class “ILSMemoryInterface.cs” and the command update class “ILSMemoryUpdater.cs”.

But games are not the complete story!

In the meantime also developers outside the game industry recognized the new possibilities having multiple pointing devices associated with the same projection. Microsoft decided to add support for multiple USB mice for classroom interaction to PowerPoint. The download links for the plug-in and the Multi Mouse Windows SDK are dead and the project was stopped (the chapter “Criticism of Mouse Mischief” only tells a small part of the truth).

But teachers around the world liked it anyway. Since we added the reverse channel (feedback to the user) into our pointers and optimized the way how to identify my own cursor (Study of our system at the University of Utrecht, Mark Overmars, you might know him from GameMaker) our system is much better than anything before.

So, our target applications are:

Games based on Unity

      Games based on GameMaker
      Presentations made with LibreOffice Impress
      Video Players with support of User Interactions

Any application for multiple users on the same projection/screen.

But developers of games and presentation software asked us for a solution for somebody wanting to develop a game or prepare a presentation without having our camera and pointers at hand.

Mobile Phones instead of our Pointers and Camera

To be able to support developers not having our hardware, we also developed a mobile phone interface, where the mobile phones can do the same things as our pointers.

The surface of the mobile phone blue area acts as an absolute pointing device (like a digitizer does) and there are 2 buttons (green , red) also on the display acting as the left and right mouse key. If you move the finger, the cursor on the screen (if you decide to show one) also moves.

You can send commands to the individual mobile phones and they start executing vibration sequences. These vibration sequences are single commands lined-up in a FIFO within the mobile phone to execute in sequence. No need to care about the timing!

The mobile phone interface is browser based, open a website, login in and start the interaction!

To simplify this task, please use the also added “QRCodeGenerator.EXE” which allows you to generate the QR code for the Access Point and a QR code for the website (in the above case http://192.168.2.136:8080).

You also get the full source code of the mobile phone interface (it is simple HTML and JavaScript). It is in the file “ILSMobilePhonePointer.TXT” which is read by the webserver at program start. The counterpart is a small webserver “ILSMobilePhonePointer.EXE” we have developed and it is only able to handle the mobile phone requests but very fast since it handles all from memory instead of accessing files or even worse executing CGI scripts on the server side. You get an unlimited license to use our webserver free of charge. Sorry, we only have it for Windows right now, but it is no problem to use any webserver which can handle GET REQUESTS and where you can add code to write into the shared memory.

We appreciate your feedback how to improve the code, remove bugs or if we shall add some functionality. But adding functionality should be easy for you, since you can change our mobile phone code as you like and if you do not like the command “DOUBLE VIBRATE LONG” then replace it with “PLAY MP3” and the phone will do it.

Sorry folks, there is no way for Safari Users on iOS

Moving and clicking should work on iOS (we did not test it), but vibration will not work! Apple decided that vibration is bad and there is a long discussion about that:

“Does this mean that Firefox and Chrome for IOS also do not support navigator.vibrate()?”

Mar 4, 2022

“Yes. Firefox Mobile and Chrome Mobile on iOS aren't the real thing like they are on Android. On iOS, they are Safari-with-a-Firefox-skin and Safari-with-a-Chrome-skin. That means: anything iOS Safari Mobile can't do, they can't do either.

Sorry folks, there is no way to use our tools outside of Windows

We are hardware engineers in Germany, Switzerland and China. We are very happy if we get our code running on Windows.

For the same reason:

We apologize for our bad HTML and JavaScript and C# Coding. We are used to program in C and Assembler for our microcontrollers. We are using pointers, global variables etc. everything which makes code fast and small but hard to read.

If you like, then please contact us to make our tools running on Linux, Android, etc.

Important Advise regarding the mobile phones, some features must be added:

The start page which is sent by our webserver must (!!!) contain an encrypted session ID generated by our PointerNet and each further GET REQUEST of the mobile phone must be preceded by a further always changing (derived from the previous session ID) session ID.

Otherwise anybody knowing the servers IP address can login into the game and even control/disturb other mobile phone users while gaming, clicking, interacting.

Our pointers have internal worldwide unique and always changing serial numbers and we will do the same for the mobile phones so that both of them can be associated with a user account in our PointerNet so that you can download user pictures, user avatars and even payment information for in app purchases.

We will do that for you in our PointerNet! But do not use the current implementation in production environments!

Let us start:

Please download our Quick Start Guide – Mobile Phones.

There is no need to read the introduction, simply read “Making the Mobile Phones Running in 5 Minutes” or “Making the Camera and Pointers Running in 5 Minutes”.

We hope that we did not forget any trouble generator! Add “ILSMobilePhonePointer.EXE” to your firewall exceptions, it is a web server and you can change the port from 8080 to anything you like in the INI file.

A package containing a directory structure with all our mobile phone tools and a working Unity game (compiled for Windows in Windowed Mode) can be downloaded here.

Unity Developers:

If you are a Unity Developer, then please download the complete Unity project:

ProofOfTechnologyGame

We apologize for our Unity code and project settings. We are Unity newbies and it was pretty heavy for us to learn Unity and C#! And this is not a real game, it is a proof of technology game demonstrating painting cursors, detecting clicks or button states within game objects using Colliders and then send command to the pointers or mobile phones.

For drag-and-drop something should be added like “bool bPointerInUse[600]” in the “ILSMemoryInterface” where all other pointer related data resides and the colliders should not get active on those locked (for one frame) pointers.

By the way, we would like to add better cursors with an attached user name (alias) and different cursor shapes per user and the ability to highlight a cursor if e.g. both buttons are held down for a second or so and to display the user pictures (selfies uploaded before), but we are not experienced enough to do this.

There is a special FIFO pair which we call the control FIFOs. One of the pointers or mobile phones we call the master pointer. This master pointer shall be able to control multiple games or video players running outside of Unity or start Impress presentations etc. Although the master pointer is written to the main FIFOs, the master pointer data is additionally copied into the control FIFOs because only one process is allowed to read a FIFO at a time since reading means deleting the read item so no one else can read it again. A Master Control Program (“The Master Control Program (or MCP) is the main antagonist in TRON”) therefore is the only process attached to the control FIFOs but the games still can react on the master pointer. E.g., the MCP pops-up (to foreground) if both master pointer or master phones buttons are held down for e.g. 3 seconds (after 6 seconds, our pointers will enter stand-by, maybe we need a new mechanism for that!). To allow multiple processes to attach to our memory (already possible) we will add thread safe inter process communication (IPC) functions for requesting the focus etc. soon.

Execution Order:

The class members in “ILSMemoryInterface” should be called first within a frame and the class members in “ILSMemoryUpdater” should be called last within a frame and in between all game objects that rely on correct pointer positions and button states and those which send commands to the pointers.

We sometimes had to delete the meta files because we sporadically got the message “Should not be capturing when there is a hotcontrol” and for this deleting the “.meta” files has been proposed as a workaround. But deleting the “.meta” files deleted the execution order in the project settings!

In “stackoverflow.com” we have found that there might be a better way to set the execution order of our classes (still undocumented but a lot of developers use it):

[DefaultExecutionOrder(100)]

unsafe public class ILSMemoryInterface : MonoBehaviour

[DefaultExecutionOrder(400)]

Any other GameObjectsHandler

...

[DefaultExecutionOrder(700)]

Any other GameObjectsHandler

[DefaultExecutionOrder(800)]

public class ILSMemoryUpdater : MonoBehaviour

You can also achieve this within editor scripts. Add class which initialized on load and set the “MonoScripts” Order dynamically.

[InitializeOnLoad]

public class SetExecutionOrder

{

static SetExecutionOrder()

{

MonoScript[] scripts =

(MonoScript[])Resources.FindObjectsOfTypeAll(typeof(MonoScript));

int order = -100; //Set this to whatever order you want

foreach(MonoScript script in scripts)

{

//The type of the MonoScript who's order you want to change

if(script.GetClass() == typeof(MyMonoScriptType))

{

MonoImporter.SetExecutionOrder(script , order);

}

This will also show the script within the script execution order window.

Thank you “ghiboz”, “Zandry” and “Andronicus”.

Any Developer interested in Using Multiple Pointing Devices

You can download a complete directory structure with all of our executables here. You can use and test it without Unity installed. The demo “ProofOfTechnologyGame” is based on Unity but is compiled as stand-alone executable for Windows and works in Windowed Mode so that you can resize like any other program.

Please download the source code of the interface here. It contains all functions to access the shared memory and read or write the different FIFOs.

Please read “There is a special FIFO pair” to understand how multiple running processes share one memory structure.

Please also read “Multiple Screen Handling”. It might not make sense for you but our shared memory supports up to 16 different projection screens or displays and it might happen that you want to create an application spanning over multiple screens.

We are so sorry that we do not have it in native C or C++, we only have it in C# and C# is a strange programming language because it does not like pointers very much (the garbage collector is strange for us at all), especially pointers into unmanaged memory and it does not like macros using “#define”.

If you wonder, why we had to calculate the offsets of our structs in the shared memory, this is the story:

C# does not support struct members which contain arrays of structs. We had no chance to give C# our complete struct of structs, we had to split them and this is quite a long story.

C# is a very strange programming language and we hope to never get in touch with C# again.

Small Improvement to the version of yesterday:

Renamed the Unity project to “ProofOfTechnologyGame” and compiled it for Windows in Windowed Mode because it is not a funny real game :-(

We have moved the general clearing of pointer clicks from class “MagicCirclesHandler” to class “ILSMemoryInterface” and there into function “UpdateCommands()” which is called by “ILSMemoryUpdater.Update()”. The reason: The magic circles are game objects and in this demo game, they are the only game objects using the pointer data but in a real game, this is wrong! Game objects shall only clear pointer click events if their collider has detected an inside state (OverlapPoint is true) and the game object handles the event finally. All (remaining) pointer click events therefore are cleared by “ILSMemoryUpdater.Update()” which is the last script in a frame.

We have changed the code of “ILSMemoryTest.EXE” to also copy the master pointer to the control FIFO XYKCTRL and encode the screen number (0…15) selected on the mobile phone so that the camera calibration can be tested with mobile phones. This does not really make sense, because there is no camera in the mobile phone mode but the project shall be consistent and you shall be able to test the camera calibration (GameMode=2 in the Unity “!MYINI.INI”) also with the mobile phones.

Added Sound to the Mobile Phones

In the directory ILSMobilePhonePointer there is a file ILSMobilePhonePointer.TXT which contains the HTML / JavaScript Code to add sound to the mobile phones. Since we have added the sound data directly into the webpage code, the file size increased dramatically which made it necessary to also increase the internal buffers of the webserver “ILSMobilePhonePointer.EXE”.

Here your can download a new version with a new EXE and a new TXT file:

ILSMobilePhonePointerWithSound.ZIP

You should rename of the old ILSMobilePhonePointer.TXT to ILSMobilePhonePointerNoSound.TXT before. The EXE you can overwrite. The game commands VIBRATE_SHORT_DOUBLE, VIBRATE_NORMAL_DOUBLE and VIBRATE_LONG_DOUBLE now generate sound. We can understand if you want to add your own sounds, especially the 2^nd one. Take your MP3 file binary and feed into a MIME/BASE64 encoder, the your can change the sound. To make live easier for you, we also give you a “Base64Encoder.exe” which does the job for you. It is here:

Base64Encoder.ZIP

A future version will allow you the embed different sounds to each mobile phone individually. And we will increase the number of sounds and assign new commands code to them so that VIBRATE_SHORT_DOUBLE, VIBRATE_NORMAL_DOUBLE and VIBRATE_LONG_DOUBLE are available again.

Improved ILSMobilePhonePointer

2023-10-24

If the requested URL ends with the IP address or the port but nothing else when opening the mobile phone webserver, it sometimes happens that that a browser sends a single slash as HTTP request instead of and empty string.

So this version takes care of the following HTTP request contents

URL in Browser	Internally handled as
x.x.x.x	http://x.x.x.x:80/index.html
x.x.x.x:y	http://x.x.x.x:y/index.html
x.x.x.x:y/	http://x.x.x.x:y/index.html
x.x.x.x:y/index.htm	http://x.x.x.x:y/index.html
x.x.x.x:y/index.html	http://x.x.x.x:y/index.html

So the web server is immune to different URL contents and always sends index.html which is the contents of the file “ILSMobilePhonePointer.TXT”.

Here, you can download this version.

If you want to see it working

You can see 80 cursors (walking in a row from top left to bottom right. There are 5 positions where each pointer does the same:

1. All pointers must disappear

2. All pointers click with the left mouse button (the cursor changes to dark red)

3. All pointers click with the right mouse button (the cursor changes to red)

4. All pointers click with both mouse buttons (the cursor changes to bright red)

5. All pointers must disappear

There is a simulation file (“PointerSimulation09.txt” generated by “PointerSimulation09.xls”), which is written pointer by pointer into the shared memory by “ILSMemoryTest.EXE” in mode 1 (in mode 2 it sends the data from the mobile phone webserver). Whenever a click (left, right, both) happens in one of the 3 magic circles (blue, green, red), then a different command is sent to the pointers (and the mobile phones now). It is not a real game, we have developed it to demonstrate that our system is working. The speed of the circles and the radius you can adjust in the “!MYINI.INI” file of the game. The compiled Unity game is named “ProofOfTechnologyGame”.

Here is the video:

To contact us please send an email to