[TECH FOCUS] Prototype #3

Recognition of a 3D object and information management

Goals

For this prototype we thought of a process that could be used in the industrial field, applying Computer Vision elements. So we contacted Tecnolaser, a company based in Padua (Italy), that deals with the cutting, bending and painting of metal parts to see whether we could launch a pilot with them.

Talking to them, we have found out that in the painting phase of a certain piece, sometimes the final product presents paint faults and therefore the operator has to perform that step again, causing a slowdown in the process and an economic inefficiency to the company.

The pilot is kicking off!

Prototype #3

The prototype we are building allows the recognition of the shape of the various pieces entering the spray booth using a camera and Computer Vision algorithms, detecting the shape and calculating the critical points of every single piece, in order to inform the operator to which areas he must pay close attention.

Bearing in mind that the software must be used by an operator in a factory, the information must be easily visible and the interaction must be reduced to the bone. Operators most likely wear gloves and using touchscreens may not work, that’s why we need to keep it simple and functional.

Software architecture and technologies used

For the prototype, we have decided to use a 12" iPad Pro, it has a screen big enough to show the required information to the operator, allowing us to use the features of Object Tracking offered by ArKit (ArKit is a software library offered by Apple that offers various types of real-world Object recognition, it is also used for augmented reality applications).

In addition, the iPad pro has a series of sensors on the front side, which makes it possible to detect the distance of one hand from the tablet and thus avoiding to having to use the touchscreen.
In the future version, we could replace the iPad with a computer and connect it to an external camera, such as one of the Mobotix cameras.
The software is then developed in Unity (framework for the development of multi-platform applications).

Software Development process

The software development has started with a visit to Tecnolaser, where we observed the entire painting process and talked to the technical people and directors there.
Direct observation is fundamental, as it forms the basis of all subsequent development.

The visit was followed by a brainstorming session with the MVP team, where we made the first decision of the development process, such as the use of the iPad pro and Unity.
On top of the reasons we already explained pushing to prefer the iPad pro for this prototype, we must say that we were able to use a special stand in our office, where we could place the iPad on it in an ergonomic position for use.

In the meantime, the Uqido Design team has developed two versions of the UI and thanks to an internal test have been able to select the one we thought was best to meet the needs that emerged from the company.

Then, thanks to the availability of Tecnolaser, we obtained 3D models of the different components.
We worked on those models to make them Mobile-Ready adding information that explained how to treat the various surfaces and angles.
For example, the scanning was highlighting the areas to which the operator had to pay close attention to while painting.

The final result is:

We can see when the 3D object, which represents the part to be painted, shows areas highlighted in red and some text. This information allows the operator to easily identify the most critical parts to pay more attention to during the painting process.

Using the button below (Call To Action), the operator can view the information related to the next piece; moreover, as stated before, the system also allows interaction without using the touch screen, placing a hand in front of the tablet itself, as represented in the following image:

Finally, through the Computer Vision we can see the piece that needs to be painted and the information to be communicated to the operator in augmented reality on the real object, as shown below: