From Gaming Engines to Global Operating Rooms: The Evolution of VR3S
Virtual reality gaming platforms have evolved significantly in the last 20 years. No longer limited to multiplayer games like Fortnite and Minecraft, VR allows users to fly virtual jets, try on virtual clothes and “visit” the moon. In recent years, a limited number of pediatric heart centers have been using VR to plan complex heart surgeries. At Cincinnati Children’s, heart surgeons use the Virtual Reality Surgical Simulation Suite (VR3S) to plan their most complex cases. Pediatric cardiologist Ryan Moore, MD, MSc, and congenital heart surgeon David Morales, MD, spearheaded the suite’s development nearly a decade ago.
Now, Cincinnati Children’s is taking VR surgical planning to the next level. They are developing a new “Surgical Mediverse” within VR3S that allows medical professionals on opposite sides of the world to plan heart surgeries together.
The goal is to enhance medical outcomes and improve access to advanced health technologies for patients worldwide. In low-resourced countries, tens of thousands of children die from non-complex heart defects. “This technology allows us to partner with local surgeons to help them learn techniques that can save these children,” says Morales, executive co-director of the Heart Institute and director of cardiothoracic surgery at Cincinnati Children’s.
The Digital Experience Technologies development team at Cincinnati Children’s worked closely with Unity Technologies to create VR3S (Unity produces one of the world’s leading game engines).
The VR3S platform has multiuser capabilities to connect surgeons around the globe in real-time or asynchronously (on their own time). It was one of eight innovative 3D software projects to receive a 2024 Unity for Humanity Grant. As a grant winner, Cincinnati Children’s will share $500,000 in funding with other winners. In addition, Unity will provide technical support to optimize VR3S for global impact in technology-resource-limited settings and low-to-middle-income countries.
The VR3S Origin Story
Heart surgeons and tech experts at Cincinnati Children’s began developing virtual surgical planning tools in 2015. Three years later, with the support of a National Institutes of Health grant, they created VR3S. VR3S goes beyond VR visualization, allowing surgeons to place valves, medical devices and complex baffles on a 3D digital twin of a patient’s heart. Surgeons do so while wearing a VR headset and using handheld controllers.
High-resolution screens display the patient’s digital heart in three dimensions. The technology uses data from the patient’s computed tomography and magnetic resonance imaging studies to create the digital heart.
“A few weeks before a complex surgery, I may spend an hour walking through the patient’s heart, gaining a 360-degree perspective that I can’t get in the OR,” says Morales. “I can plan my pathways, create virtual patches, decide where to put baffles and do anything else I need to prepare.
“The day before surgery, I review the video recording of that session and maybe take one more walk around the heart,” Morales adds. “Once in surgery, there are rarely any surprises or deviations from our plan. In fact, the VR session almost always helps us understand the patient’s anatomy better than we could have with traditional medical imaging alone.”
Expanding Opportunities to Offer Complex Biventricular Surgeries
Heart Institute surgeons use VR3S to plan about 10% of all heart surgeries and about 30% of complex surgeries. Morales says the technology has allowed them to offer biventricular (BiV) surgery for patients with complex anatomy who would not be considered candidates elsewhere. This patient group includes children with heterotaxy, or isomerism, who have as many as five or more congenital heart defects.
In addition, this technology has allowed surgeons to perform a single operation on complex heart patients who may previously have been indicated for multiple “staged” surgeries.
“The long-term outcomes for Fontan patients have improved a lot. But the likelihood of going into heart failure and needing a transplant by age 35 is extremely high for these patients,” Morales says. “At Cincinnati Children’s, we are identifying younger and younger patients for biventricular repair, most of whom have isomerism. Other centers may only consider these patients for single-ventricle palliation, but if we get them early enough, we may be able to do a biventricular repair instead. Also, we have reversed many Fontans in patients as old as 25 who previously had a Fontan here or elsewhere. Ultimately, we are in a new era of heart surgery with the opportunity to limit the number of single-ventricle patients requiring the final-stage Fontan.”
What makes this aggressive approach possible? VR3S. “Some of these patients have one-of-a-kind anatomy,” Morales explains. “VR surgical planning helps us sequence each step of surgery, craft different approaches and adjust. We believe many patients with single-ventricular repairs could benefit from a biventricular strategy.”
Exploring Opportunities for the Surgical Mediverse Globally
The Surgical Mediverse within VR3S that Morales, Moore and their team are creating uses a Unity-based gaming network. Gaming capabilities create opportunities for worldwide digital collaboration. “The easiest way to spread knowledge worldwide is to put it on a gaming network, which streams complex, three-dimensional images to remote places in real-time,” says Moore, chief of emerging technologies and director of Heart Institute digital health innovation at Cincinnati Children’s. “With just a VR headset and internet access, even healthcare teams in remote locations can collaborate with an experienced team like the one at Cincinnati Children’s to help them plan an upcoming surgery.”
VR3S allows communication and real-time translation in 10 different languages, including creating voice-to-text annotations. Cincinnati Children’s is partnering with companies like Dell Technologies to increase internet access and bandwidth capacities in lower-resourced areas. In addition, the team can deploy the Surgical Mediverse on multiple devices, including Meta Quest VR headsets and the Apple Vision Pro.
Several other large academic pediatric heart surgery programs are beta testing the VR3S Surgical Mediverse, and the Cincinnati Children’s team is working with several surgical programs in technology-resource-limited countries to help them use the technology.
To learn more about VR3S or discuss a surgical case, please contact ryan.moore@cchmc.org.
Interested in exploring a collaboration with the Cincinnati Children’s Heart Institute? Contact david.cooper@cchmc.org, michael.gaies@cchmc.org or david.morales@cchmc.org.
(Published December 2024)
