In order to make Virtual Reality (VR)-based training useful for first responders, it is important to understand the physical and technological limits of current VR setups. Current approaches to immersive virtual environments often involve putting on a heavy and bulky headset with a limited range of external feedback and disconnection with the outside world. First responders need to train with as little obstruction as possible- so why have them train with something they wouldn't be ok with wearing on the job? I have determined that there are a few criteria that must be evaluated when choosing VR equipment to be used for first responder training. It must not restrict the motion of the user, be bulky to the point that it is cumbersome, and must be able to simulate the capabilities of the proposed technology to be developed.
My proposed solution is to use either a commercial-off-the-shelf (COTS) VR setup or a custom VR environment in coordination with a combination of custom and consumer software, as well as using external fixtures to create an environment that mimics a situation first responders could face.
Most promising is the use of surround video displays to create an almost seamless 360-degree space where first responders can view a window into their outside world. This type of simulation is not the first of its kind- it has been gaining steady traction in use for training air-traffic-controllers and boat captains. The reason for this is because it presents a less cumbersome (and more natural) setting than that of a bulky helmet or head-mounted VR setup.
My current vision involves placing multiple 4k television screens (ideally the greater resolution the better) or 4k projectors and projector screens in at least a octagon shape. These television screens would envelop the first responder and give them a physiologically-accurate field of view that they would encounter in the field. These displays would be hooked up to multiple computers either running a video training exercise (captured by a 360-degree camera and post processed) or a real-time rendered simulation that takes input from the first responder and their equipment. However, this system could also be replaced with a single head-mounted VR system (Like the Oculus Rift or HTC Vive) for a more bulky, but more immersive feel.
Although VR setups conventionally take a lot of computing power, I believe that parallel processing and real-time rendering could be used to create a life-like simulated scene. As seen in the past few years, video games are becoming increasingly realistic, and the graphical rendering power afforded to the average consumer is being utilized extremely well. However, an agency with government funding could most likely acquire industrial-grade graphic processing hardware (or a system of consumer graphics cards running in parallel-Nvidia's SLI and Cuda platforms can be used extensively to maximize processing power). Perhaps even a "First Responder Game Engine" could be created that specializes in processing input from decisions to give input to first responders' and the array of technology they happen to be using.
As for making the system immersive and representative of a first responder's environment, I envision the system will have numerous ways to do so. It should be able to recreate the atmosphere through means of creating heat, sound, wind, steam, and rain in specialized compartments overhead/underneath the user. This would allow different simulated environmental factors that the first responder would be subjected to (i.e. high winds -> hard to tell radio from wind noise) in their everyday tasks to be simulated in a repeatable and observable fashion.
The parts for my proposed system(s) can easily be purchased off the shelf, but the real challenge is getting every part of it to work together. I propose having multiple computers networked together hooked up to the system. These computers would each be responsible for one particular task, such as monitoring physiological response of the first responder. This would alleviate the computing load on the overall system.
A good solution would be to have the simulation/training exercise running on a computer connected to multiple projectors (or the control unit for a VR head mounted device, if applicable). This setup can easily be setup by 1 or 2 people, but requires a lot of electrical power. This is not an issue as the overall system is not constrained by electrical power.
The simulation "wall" can be setup by purchasing multiple high-end graphics cards and installing them in the computer running the simulation exercise. The same thing goes for the head-mounted displayed (Oculus Rift), although extra work will need to be done to integrate the custom VR training (whether through SteamVR or a custom game platform).
In terms of safety, virtual reality represents a fairly safe system for training. Although there is risk of discomfort and nausea due to the interpretation of human movements through the computer, the risk of motion sickness is not all that great. As for the risks posed by simulating conditions a first responder would feel, the range of environments capable to be simulated would not reach levels that could be hazardous to the user. It is intended to augment the experience, not make it a real one. The system also presents many opportunities for physiological monitoring, allowing warning signs of any human hazards (if there are any) to be spotted long before they become a problem.
In the virtual world, the ability to realistically portray scenarios is limited only by the power of the designer and the computing hardware. That being said, the current development of high-detail VR graphics is something that requires intense processing power. A pre-rendered scene, however, could be extremely realistic, yet not very computationally taxing.
When you put on a VR headset, it feels real. Combine that with positional audio and interactive environments, and you feel like you're in a whole new world. There are many conditions a first responder might encounter doing their job, and many of those environmental factors can be simulated, including:
-Radio Traffic, wind, Fire, Audible distractions, Visual Distractions, rain, fog, noise, vehicle operation
Of course, there are also plenty opportunities for the user to interact with their environment. Given that custom tools are developed that mimic those used in their job (such as a firearm), their implementation could be used to portray physical items in the virtual world. Interfacing these objects with the VR system is trivial, and could be implemented through protocols like Bluetooth.
When the user is using the system, they are already experiencing a barrage of visual and audible feedback from the system. However, a physical (and noticeable) feedback can help seal the deal.
An array of nozzles can dispense water in a mist or rain pattern, while vents above and underneath create wind currents (hot and /or cold) to simulate windy conditions. Heat lamps can simulate heat and help the user practice not becoming fatigued under stress. These systems in coordination with the VR simulation help to create an extremely realistic environment for the first responder.
Additionally, devices such as electrodes could be placed on the user's body to stimulate muscle contractions, emulating the feel of pain (such as that experienced by a gunshot or blunt injury).
Although my system proposed is very realistic for a training standpoint, it requires considerable effort to adapt physical equipment to be used. However, it is extremely capable of incorporating virtual interfaces or simulated technologies.
The Unity game engine is very adept at VR, and incorporating virtual interfaces is a natural part of the game design process. By designing a virtual model of the interface (such as a heads-up-display), it can be easily integrated into the game engine. Simulated technologies, like augmented reality and real-time risk assessment can also be implemented-their capability is limited only by the skill of their programmer.
1x Oculus Rift/ HTC Vive VR system- US $499/$799
- 2x Gaming/Server Computer 2x ~$1000-Any machine with i5-4690K , 16gb RAM, Nvidia GTX 1090 or equivalent
-Unity Game Engine SDK (Free for community use)
All technology needed to build this system is currently available for consumer-off-the-shelf purchase.
My design allows for many different scenes. The extensibility of the Unity game engine allows for rich and creative game design, and the levels are able to recreate a broad range of settings. This is only limited by the creativity of the designer. The ability to not only import objects into the game environment but also to create custom ones is powerful and allows for an immersive experience.
In a VR game environment, the user's actions can be logged exactly as they were performed. This, in combination with audio/video logging and recording physiological data, gives us a complete picture of every single thing that happened in the training scenario. We can run a simulation, observe what happened (frame by frame) and chart things such as response time and neural activity. Through the use of logging software and data collection, this data will be able to be analyzed and interpreted either after the fact or even in real time. This gives us so much more insight into training data than that provided by conventional training.
Take an example of a police officer under fire: The system could log the officer's response to a gunshot both physically (through monitoring the motion of the user) as well as mentally (monitoring things like heart rate and brain response to stimulus).
This data can be extensively analyzed post-exercise and allows for reinforcing the training often performed under conditions not similar to those of the real environment. By giving the user a simulation where any of its parameters could change at moment's notice, the response behavior is much more representative of that seen in the field.