[TECH FOCUS] Prototype #5

Computer vision algorithm and information management

Goals

As our main goal of Exploratorium is to gather information and try out the technology in a very short time, we decided to develop a simple WebGL application using Unity to get results in a short time. This decision allowed us to implement the various elements of the UI proposed by the design team very quickly.

Prototype #5

During the planning process, an important point has emerged.
We needed to provide both the user and the personal trainer with a space that allowed the first one to see the exercise, and the second to observe the user’s movements.
This space is very similar to a normal video conference, so we explored various video conferencing programs to find the one efficiently aligned to our timing and goals and easy to manage. We decided to use a platform like Google Meet or Zoom, and share the screen via the WebGL application. This would enable the user to watch the virtual trainer, while an operator or personal trainer receives the feed from the user's webcam. In addition, Google Meet is cross-platform and, by sharing a Chrome tab, we are able to avoid any technical or lagging problems.

Software Development process

To proceed, the team approached the development of the prototype in steps.

The first focus was to show the training film on a full screen, with the option to show the icons on the screen when the personal trainer thought it was necessary.
At this time, the personal trainer would give feedback from the keyboard, so that the screen was left free. In this step, we also added two audio effects, related to a positive or negative/warning feedback to allow the personal trainer to judge the users’ movement

Both sounds are very short and similar to app notifications or alarms, to catch the user's attention to the screen.

The second step was about adding a more user-friendly control for the personal trainer. To do this we applied some multiplayer techniques to the build in Unity:

• WebGL build: this was the one shown on the user side. On a full screen, the user was able to follow the training video and see the feedback reported by the personal trainer.

• Local build: this was the one applied to the personal trainer side. Thanks to a simple User Interface, the personal trainer could control the progress of the training video and send feedback to the user.

This prototype is fully usable. To try it out you simply need to set up a call on Google Meet with the user, you launch the WebGL build and the Local build, after which you share the screen of the first one and as soon as the user is connected the session can get started.

Once we got a working prototype, we decided to try and develop a version without using any third-party platforms like Meet or Zoom. In this case, it was enough to create a landing page where the user side displays the WebGL build screen, while the trainer side displays the user's camera feed.

The goal was to upload the WebGL build to a web page, which can behave differently depending on who is using it. The trainer version would act as the actual server, managing all events such as the video start and feedback, while the user's version will be the client, receiving only the signals needed to reproduce locally what the trainer sees from the server. In this way the traffic would be very low, making it easy to manage the interaction between the trainer and the user. The only problem we have had with this solution was sharing the camera.

Accessing the camera feed is easy, but sending this feed from the client to the server was quite problematic, as the amount of data to send is quite high.
WebGL and Unity's High-Level Multiplayer API impose limits on the amount of data you can send and receive. For this week we have decided to leave this development as it is, and instead, focus on the overall improvement of the user experience.

Accedere al feed della camera è un gioco da ragazzi, inviare però questo feed dal client al server è invece abbastanza problematico, dato che la quantità di dati da inviare è elevata e WebGL e le Multiplayer High Level API di Unity impongono dei limiti alla quantità di dati che si possono inviare e ricevere. Per questa settimana abbiamo deciso di lasciar stare questo sviluppo, concentrandoci invece sul miglioramento generale dell'esperienza utente.