Exploring virtual and augmented reality

Research is a great way to enhance learning and gain practical experience, as computer science majors Dean Schulz and Andrew Johnson are finding out. They have been helping faculty member Rishi Saripalle research the use of virtual reality (VR) and augmented reality (AR) hardware and software solutions in education and healthcare.

In VR, users interact with a virtual environment and object around them designed to emulate the real environment and objects. In AR, virtual objects are placed in the real environment and sometimes even react to real-world events, bringing us closer to making the movie “Minority Report” a reality.

The use of VR and AR for both personal and professional purposes is rapidly increasing due to the availability of accessible hardware (e.g., Oculus Rift, HTC Vive, Google VR) and open-source software tools (e.g., Unity, Unreal) to design virtual environments with ease. Currently VR is used in such disparate areas as healthcare, education, gaming, real estate, and retail. The VR technology and application development initially targeted the gaming sector, which still takes the lion’s share of the application market, but other areas, primarily healthcare, education, automotive, and construction have seen the value VR provides to the consumer. For example, in construction, based on the “about-to-be-built” plans, developers can design a virtual structure and allow users to explore the initial design before starting construction. The virtual structure will enable potential customers to visualize what the actual structure looks like and provide feedback to the engineers.

AR hardware, such as Microsoft Hololens, Magic Leap One, DAQRI Smart Glasses, etc., is available, but not as affordable and accessible as VR hardware. They are still in their infancy and will take time to mature and be adopted.

Right now Saripalle and his students are working on three research projects in these areas:

1. Leveraging VR for human anatomy
The study of human anatomy and simple body movements is crucial for various specializations, but understanding, analyzing, and remembering them is a daunting and difficult task. Most of these concepts are taught using images, lectures, videos, and 3D physical models. It’s sometimes difficult to understand and analyze 3D concepts using 2D content. Students and faculty are limited to observing and cannot interact with the content.

They are researching ways to use VR to illustrate and explain fundamental concepts related to human anatomy and movements. The goal is to design and develop an interactive VR application that accurately renders the structural makeup and basic movements of the human skeletal system.

2. Understanding vestibular disorders using virtual reality
The vestibular system includes the parts of the inner ear and brain that help control balance and eye movements. If the system is damaged by disease, aging, or injury, vestibular disorders can result, associated with vertigo, dizziness, and loss of balance. These symptoms contribute to a 12-times greater risk of falling, increasing the risk of serious injury or death, particularly in the elderly.

Symptoms can be triggered by activities such as driving, navigating in busy environments, navigating in settings with patterned floors/walls, etc. One effective treatment involves reducing symptoms through regular and repeated exposure to stimuli that trigger these symptoms. Cost and scheduling constraints can limit the number of appointments with a healthcare professional. It is also difficult to accurately reproduce the stimuli (e.g., shopping mall, grocery store, patterned walls, etc.) that provoke symptoms in patients.

Virtual reality applications allow creation of the scenarios known to trigger symptoms to be introduced to the patient in a controlled manner. A patient walking down a narrow aisle, or identifying objects in front of a moving checkered background are examples of scenarios that can be simulated in an immersive VR environment, but that are difficult to replicate in a real clinical setting. Furthermore, VR applications could allow for computer-driven gradual, graded, and controlled “practice” sessions that are essential for functional improvements.

3. Use of augmented reality in manufacturing and healthcare
Augmented reality may not have caught the attention of the general public yet, but it has definitely captured industry attention. Researchers are trying to tailor AR hardware and software solutions to improve manufacturing processes across multiple sectors. Saripalle’s preliminary study attempts to understand and analyze various available hardware and software components. They are looking into Microsoft HoloLens and DAQRI glasses and their Worksense software platform and NVidia Holodeck.

Student researcher Dean Schulz summed up his experience thus far, saying, “Working as a VR research assistant under Professor Saripalle was a very rewarding experience that allowed me to test my creativity and coding skills by building VR projects from the ground up using Unity. Through this job I learned new coding languages and how to develop virtual reality projects for clients.”