Context
This project was part of the course Virtual Reality in University of Twente. Project teams built virtual or augmented reality (VR/AR) solutions for cases provided by companies. In our iteration of the course, the provider was Nerlandanse Spoorwagen (NS), the main railway operator in the Netherlands.
Define the problem from ill-defined requirements
How can VR/AR enable the digitalization of train operation?
…was the case provided by NS. It was broad so that teams could work on different ideas, but it also meant that we had to specify the problem we wanted to focus on solving.
Our strategy for this is to find the intersection between (1) What problems are NS facing? and (2) What problems can be solved by VR/AR technology? Based on what we learned from the lectures and presentations by NS employees, we decided to focus on the problem of emergency response.
Area of focus: Train emergency response 🚨
The breakdown of a train during operation can lead to serious disruptions to the train network, affecting thousands of passengers and causing financial losses. While drivers can respond to minor problems by themselves, they often need remote assistance for more serious breakdowns.
Area of focus lies between problems of NS and advantages of VR/AR technology
Current assistance process is inefficient
Currently, assistance is provided by engineers from a remote support center. The current process presents the following problems:
- The support center is understaffed, leading to long wait-time
- Communication is conducted through hand-held devices, leading to inefficiency:
- For the driver: handling the devices interferes with the emergency response process.
- For the driver: verbal instructions can be inefficient.
- For the support staff: they cannot observe what is happening, making it more difficult to diagnose the problems and guide the drivers.
How can AR technology increase the efficiency?
Augmented reality (AR) technology possesses advantages that make it a good fit for the aforementioned problems.
Pain points | Advantages of AR |
|---|---|
Handling the devices interferes with the emergency response process. | The AR headset can be operated hands-free. |
Verbal instructions can be inefficient. | Instructions can be embedded into the real world, making it easier to follow. |
Support engineers cannot observe what is happening. | Support engineers can observe what is happening through the AR headset camera. |
General concept
Our solution consists of two main components: an AR application installed on a headset for the train drivers, and an desktop-based application for the support engineers.
Concept of the solution
AR for train drivers
- Automatic failure identification and troubleshooting: the system automatically detects the failure and guide drivers through initial diagnosis and troubleshooting steps.
- Real-time remote assistance: the system allows drivers to communicate with remote support center in real time. Instructions from the support engineers can be embedded into the real world.
Desktop application for the support engineers
- Digital twin of the train: allows support engineers to localize the failure and monitor relevant data of the train. Read more about digital twin.
- AR-based observation and guidance: allows support engineers to observe the situation through the drivers’ POV and give instructions via AR objects.
My main responsibility is designing the AR application for train drivers.
Turning concept into tangible solution
Asking more questions
In order to develop a more concrete solution from the initial concept, we needed to acquire more information about the problem and its context. We did this by interviewing NS staff after our concept presentation and sending follow-up emails. Some questions asked were:
- Describe the whole process of emergency response. What are the steps taken by the train drivers? What information do they receive from the train and support staff?
- Describe a specific scenario of a breakdown.
- What kind of data is available to the support staff? What data/information do they need from the drivers and vice versa?
Learning more about the problem
- Information about the defects and diagnosis code will be shown on a screen in the cabin. Error handling procedure is only available on some newer trains and an app on the drivers’ phone.
- NS staff provide the scenario when a driver has to manually switch between pantographs when the automatic switching fails. The procedure is unfamiliar to most drivers and consists of +/- 20 steps. Performing the wrong actions can have severe consequences.
- The remote helpdesk has more and easier access to the train’s information compared to the driver.
 Display in the train cabin showing diagnosis code and other information. |  When a train transitions from a regular track to a high-speed line, it needs to switch between pantographs. Sometimes this does not happen automatically, requiring manual work by the drivers. |
|---|
Beyond off-the-shelf remote assistance solutions
We also looked at of-the-shelf remote assistance solutions such as Microsoft Dynamic 365 Remote Assist which is based on the Hololens. We made our solution unique by including features tailored to the problems we are solving:
Problems | Solutions |
|---|---|
Remote support staff might not be available immediately | The solution guide drivers through initial diagnosis and troubleshooting steps as they wait for support. |
Train drivers have less access to information. | The solution allows drivers to access different cameras on the train and embed relevant data into the view. |
Designing and Prototyping
To inform my designs, I read through Microsoft Learn materials on designing for Mixed Reality solutions. Some best practices that I applied include placing objects between 1.25 m and 5 m away from users and avoiding using Heads-up displays (HUDs).
Prototype flow
I started by creating the prototype flow to envision how the solution would be used in a specific scenario.
Diagram of the prototype flow
Low-fidelity prototypes
I made various low-fidelity prototypes to discuss the appearance and interactions of the solution with the teams.
Sketches of general working principle of the prototype. | Sketches of UI elements. |
|---|
More detailed sketches of the steps in the functional prototype.
High-fidelity prototype - Using VR to simulate AR
After the whole team was onboard with the design, I created high fidelity prototypes using Figma and ShapesXR. I designed the screens on Figma, imported them into ShapesXR, and added 3D objects and interactivity. The prototype runs on the Meta Quest Pro.
As we do not have a real train cabin available to us, we prototype the application using VR. A team member modeled a virtual train which was used as the environment of the prototype.
Figma prototypes for UI elements.
Starting the troubleshooter
After the driver puts on the AR headset, they can see an application panel alerting them of an error. It shows the diagnosis code, level, description, and a button to start the automatic troubleshooter
Augmented reality instructions
At each step, besides the step description displayed on the application panel, the instruction is also embedded into the real world using AR technology. For instance, an arrow will appear above the button that is supposed to be clicked.
Remote support staff can also add such instructional objects into the surroundings of the train drivers.
Calling the support center
The application also has a control widget for the call to the support center, including staff information and volume control.
Live camera view and embedded data
Below the call widget, the driver can see a camera view of the area where the problem occurs (in this case the pantographs). The driver can click on its button to enter the 360 degree view of the camera.
Within this view, relevant data is also embedded.
Prototype video
Feedback
We presented our solutions at NS headquarter in Utrecht to NS employees from various departments. The feedback was mostly positive. Everyone who tested the prototype were able to use it intuitively.
On the other hand, we also received some feedback on how to further improve the prototype. For example, one senior engineer questioned how the solution ensured that the train drivers followed the steps correctly and who would confirmed that the troubleshooting was successful. To address that concern, we would have to, for example, include some feedback mechanisms after each step, and also a way for remote support engineer to sign off on the completion of the troubleshooting process.
Key learnings
- Define a concrete problem from a general requirement. Finding a good fit between the problem and technology used.
- Design and prototype for a mixed-reality application. Use Figma, ShapesXR and follow design guidelines by Microsoft.
