The Next Generation of Aviation Professionals (NGAPs) who are entering the aviation industry today, represent a new generation of learners. To engage and meet their needs, the aviation community has been harnessing innovative technologies to look beyond tradition training methods and enhance workforce practices.
One technology which is increasing in popularity and aviation use is the integration of mixed reality (MR) using holograms and digital computing headsets. Although virtual and augmented reality (AR) are not new overlay digital content in our real-world environment, and promises to transform the way we train NGAP to operate and maintain aircraft. This can provide a mobile cost-effective solution to enhance real-world environments, create virtual simulations, accelerate learning and increase retention.
The operation and maintenance of modern aircraft calls for an understanding of several interrelated human and machine components that require practice and immersion. This immersive experience can be created or enhanced with augmented reality (AR) or virtual reality (VR). Relevant to the task-at-hand, they both have the ability to engage NGAP, allowing the student to practice, give real-time feedback, improve the efficiency of skills transfer and increase knowledge retention. Where they differ, is the perception of our presence, the ability to work untethered, and the ability to train crews.
Virtual reality (VR) is able to transpose the user through closed visors or goggles, which block out real-world surroundings. VR can be useful for singular operations, such as reviewing a special qualification airport to allow the pilot to experience the terrain and surroundings before actually flying the approach; learning a procedure or checklist; and practicing maintenance or other operational functions. On the other hand, MR blends virtual reality content with the real-world and allows the user to interact with the content using hand gestures or voice commands.
Most significant for aviation training, is the fact that—unlike with VR—the user is not shut away from their surroundings with mixed reality. Whether or not you happen to be sharing a physical space, mixed reality allows you to see, listen, and talk to others while everyone involved sees the same holograms simultaneously. As a result, users can interact with virtual content while continuing to be in touch with the real life around them.
Operational tasks (such as aircraft maintenance) can also be augmented with procedures, checklists and manual information to create a hands free environment.
This experience is achieved by wearing MR headsets like the Microsoft HoloLens. MR adds interactive computer-superimposed holographic enhancements to a user’s real-world environment. This technology also allows for remote instruction for crews and maintenance technicians, which could be a game changer for the entire industry.
MR Technologies are currently widely used in the medical, oil, space and automotive community and have recently been adapted for operational use by the aerospace industry at many companies (Lockheed Martin, Pratt & Whitney, Bell Helicopter, Air New Zealand, TAE Aerospace, and Japan Airlines, to name a few.)
While manufacturers and operators have already experienced the benefits of augmented and mixed reality, the use of MR in aviation training is a recent innovation. Little research has been published on its efficacy when compared to conventional training methods. A study in the medical arena concluded that advanced training methods and 3D environments can be one option for improving performance, reducing errors and enhancing safety (Kluge, et al., 2014). To evaluate the benefits of MR to engage NGAP and enhance aviation training, Western Michigan University (WMU), College of Aviation (USA) has created a HoloLens application called JetXplore. WMU is currently using mixed reality in the classroom to teach aircraft systems subjects.
The application was developed for aviation operations and procedures training (normal and abnormal) for the CRJ-200 regional jet, B787, A380 and SR20 aircraft, to bridge the gap between classroom and flight simulation; engage NGAP; and allow students to practice in a fully immersive environment. The interactive JetXplore application includes customized scenarios, turbofan engines and 360-degree interactive cockpits to teach aircraft systems, flows, checklists and allow students to practice quick reference handbook (QRH) malfunctions. Beyond customizing the JetXplore application for the virtual environment, a significant goal of this project is to explore subjective presence as it affects task performance, to reduce the gap between expensive simulators and the classroom.
The pedagogical material development has been extended to outreach activities and integrated AR micro-simulations in the classroom as interactive 3D knowledge objects. Using Bloom’s Taxonomy in the cognitive domain, 3D learning objectives can be refined to create more meaningful student outcomes, and
mapped to reflect expected assessment and student proficiency in technology-driven training environments.
The Microsoft HoloLens MR devices and immersive headsets are at the forefront of immersive technologies and, rather than replace existing simulators, it can enhance them. Engaging with the holograms by walking around, interacting and even modifying them, can lead to higher motor excitability and increased working muscle memory. We can safely simulate dangerous or difficult to replicate scenarios; require trainees to actively participate in exercises; evaluate based on performance or relevant tasks with data transmitted to a company’s training department or instructor; virtually create new equipment without expense or space-accommodating additions, and allow training anywhere, any time, with MR headsets.
With this in mind, it can be suggested that MR offers the potential for deeper knowledge retention in aviation training, while actively engaging NGAP.
Unlike other advanced technologies, HoloLens is intuitive and offers a natural means of interaction. There’s no mouse, wire or touch-screen. All you need are simple gestures to create and alter holograms, your voice to communicate with apps, and your eyes to navigate and analyze. The JetXplore application allows students to use the real movements in the flight deck when interacting with push buttons, toggles, dials or thrust levers, to ensure no negative training is taking place and improve muscle memory.
Technologies like these bring forth a new medium for aviation training, a new paradigm of mixed reality. For the first time, we have the ability to take the analog world and superimpose digital artifact, creating mixed reality aviation simulations. Instructors can be anywhere and trainees can bring extremely realistic holographic images of say, a giant B787, A380 or turbofan engine directly into their home, training facility, school, university, or anywhere else training is taking place, which allows them to interact with the object to learn and practice procedures, preflight actions and other information needed to operate or maintain equipment.
While not everyone has the luxury of having technologies like the HoloLens in the classroom, most of us do have a smart phone. With image recognition technology we are able enhance our current and future print media and ‘overlay’ our own experiences such as checklist, 3D models, video, procedures or interactive training modules. This is similar to an invisible QR code which is mapped to a corresponding image or URL. Currently, WMU Professor Lori Brown and colleagues at Purdue are creating an aircraft systems textbook with augmented reality overlays to allow students to interact with the images in the textbook. The students simply download the application (similar to a QR code reader) and use the camera in their phone or tablet to see the hidden content.
This technique can also be used by airlines and training providers to overlay content, such as video of procedures or flows over checklists or manual information. Furthermore, the process benefits from the advantage of 3D. Aviation training can break free of 2D limitations, making it easier for people to visualize a finished project, reduce design errors, save time, and allow facilities to open faster.
Companies that rely on engineers and technicians in their workforce also stand to benefit greatly from the immersive potential of AR and MR. As this potential becomes more and more realized, engineers both operationally and from a training perspective may see their entire industries transformed by mixed reality. Where the traditional aviation training model leans heavily on memorization, the educational system is shifting the focus from what students learn to how well students can apply knowledge. As we redefine the aviation training environment through technology and innovation we can prepare NGAP to meet current workplace expectations and prepare for the challenges of tomorrow.