How Mixed Reality HoloLens User Experience Can Make Life Easier for Retailers

In this way, the user can access information associated with any product, including the SKU (Stock Keeping Unit), brand and category sales, out-of-stock data, profits per SKU, or any other relevant data that can be pulled from the server. This information can be further compared against the real life state of things to identify cases when, for example, products with certain SKUs are out of stock on the shelf, however, they have significant stock within the system, therefore they are not ordered and supplied to the store frequently.

Actually, there are multiple ways for the manager to work with the visualized analytics. One of the key benefits here is that all the data is available at hand and can be analyzed and compared against the real state of things right on the spot, ensuring reliable input for decision making.

But before we proceed with the implementation process, let’s dive into the details describing the user experience we wanted to provide.

Speaking of HoloLens UX development...let us share some tricks

Now it's time to mention a few interesting findings we discovered while creating our mixed reality HoloLens user experience. You may find these tips useful when working on your own HoloLens-powered UX.

The first interesting thing we’ve noticed associated with HoloLens UI is with regard to the distance from the user’s eyes to the object that needs to be recognized. We created the rectangular frame that defines for the user the virtual area where the QR code should be placed to get read.

Then we set up a distance from a virtual camera (the human eyes) to this frame being 50 cm. As we tested these settings with the device, we’ve found that, because with HoloLens the user sees visuals in a stereo mode, the person naturally recognizes the distance to the frame and understands that the QR code needs to be at the same distance from the eyes as the rectangle is to be captured and recognized by the system. So, the system receives some additional information concerning the space that a QR code should occupy within the frame, as well as the area where this code can be found. So, the computation time is reduced significantly.

Another unexpected finding about HoloLens UI was that if you want to have your 3D models projected the way it is done with regular 3D screens, you, most likely, won’t be able to make it work correctly. The thing is that, in case the model gets projected on the edge of one of the HoloLens screens, the user starts noticing the boundary between the real world and the virtual one. Therefore, you will have to do some post-processing on your side to blur the picture on the edges of the screens.

And how things looked from the code side

Technology-wise, our aim was to make our system perform the following key actions:

1. identify the product;
2. pull relevant information from the server and prepare statistics;
3. visualize the statistics;
4. communicate with the user and respond to interactions.

Speaking about functionality implementation, there was no difficulty associated with points #2 and #3 from our list above. These features are easily enforced with Unity3D, as it offers comprehensive, ready-to-use classes and components that you can employ. The item #4 is also easy to implement using the HoloLensInputModule and applying the business logic of interaction.

And what about #1? Well, this is where we had the most fun. So, I would like to get into the details a bit.

Whether it is about a warehouse or a store, the goods are identified and differentiated with the help of machine-readable codes. For our system, we chose QR code and used ZXing.Net library to enable code reading. As we started, it took up to 1.2 seconds for the system to take a picture, search for the QR code, and have it recognized. Well, things were definitely too slow, even for real-time processing. Since 0.4 seconds out of this time were used to take a picture, and this operation is asynchronous, we could only try to influence the remaining 0.8 seconds.

To speed up performance, we reduced the picture area that the system processed to search for the QR code by 9 times and implemented some optimizations for ZXing.Net, such as removed detection for other types of codes and added a range of the required QR code sizes. As a result, we made our system perform calculations within 0.103 seconds. And to ensure smooth animation, you need to have 30-60 FPS (33-16.7 ms).

Since namespace System.Threading is not available with Unity3D for HoloLens, you won’t be able to establish parallel processing. When looking for ready-to-use solutions, we found that Unity’s coroutines is the best fit. We managed to split method performance into several sub-calls. For example, picture processing was broken into three separate stages:

private void OnPhotoCaptured(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
_photoProcessState.result = result;
_photoProcessState.photoCaptureFrame = photoCaptureFrame;
StartCoroutine(ProcessFrame());
}
private IEnumerator ProcessFrame()
{
yield return null;
PreprocessPhoto(_photoProcessState); // #1
yield return null;
DetectCodes(_photoProcessState); // #2
// return;
}

The first yield was called to avoid the overuse of the frame time, since Unity3D performs some generic actions before it executes the callback function. Method #1 is performed over 0.03 seconds approximately, while #2 takes up to 0.06 seconds. This way, you can split performance into some shorter parts to bypass the exceeded frame time and avoid animation lags.

We can also divide the process of code detection into some smaller chunks and perform “heavy” calculations without the influence on high FPS. To make things even faster, use modules from the unity asset store or WinRT API, which is accessible from C# scripts only.

So, why it is worth bringing mixed reality technology to brick-and-mortar?

In short, with HoloLens managers can effectively combine the digital experience with the benefits of the in-store presence. Mixed reality HoloLens technology gives an immediate possibility to analyze the product-related information right when looking at the shelf. Therefore, retailers can benefit from a great combination of extensive amount of data and visualisation technologies used against the real state of things.