What is extended reality & Hologram?
Medical-engineering collaboration with the latest technology will make it possible to perform safer surgery. We summarize new developments in intraoperative support system using extended reality (XR) and Hologram in digestive surgery
3D reconstruction of patients’ own anatomy was made using data from a contrast-enhanced multidetector computed tomography and SYNAPSE VINCENT software (Fuji Film Medical Co Ltd,Tokyo, Japan). Polygon (STL) files were exported from SYNAPSE VINCENT, and these were uploaded into the Holoeyes MD system (Holoeyes Inc, Tokyo, Japan). After uploading the data, 3D images were automatically converted as a case-specific CG for MR, which is called a Hologram. It takes a total of about 10 minutes from the upload of the data until the generation of an application (Fig. 1).
HoloLens 2(Microsoft Corporation, Redmond, WA) is an optical see-trough HMD, which enables optical superposition of virtual content onto the user’s direct view of the physical world. The HoloLens 2 weigh 566 g. The data of the Hologram can be downloaded onto HoloLens. The HMD could not be connected to the internet in the operating rooms at Tokushima University Hospital. Therefore, the HMD for every surgeon who joined the operation was configured in the same Wi-Fi environment by a Wi-Fi router (WSR-2533DHP-CB; Buffalo Inc., Nagoya, Japan). Such a Wi-Fi environment enabled sharing of the hologram by all the surgeons with an HMD.
XR application in hepatobiliary surgery
Liver surgery
Conversion hepatectomy case for more than 20 multiple colorectal liver metastases after systemic chemotherapy. The Hologram contributed to the better imagination of tumor locations, especially tumors located in deeper area and for determining the parenchymal dissection lines. All the surgeons finally could decide on all the tumor locations and the dissection lines for all the tumors just before parenchymal dissection. This final confirmation between preoperative imagination in mind and the intraoperative Hologram might lead to safer and more accurate operations (Fig. 2).
An anatomical liver resection for hepato-cellular carcinoma located in the posterior Segment with a hilar anatomical anomaly. The anatomy of the hepatic hilum was carefully checked just before the hilar procedure using the Hologram. The surgeons could easily switch the real patient’s anatomy with that of the Hologram. As a result, Glissonean pedicle approach with the anatomical anomaly was safely performed (Fig. 3).
Biliary surgery
Intraoperative 3D cholangiography was performed in a hybrid operating room. A Hologram was immediately and successfully made in the operating room using the acquired cholangiography data, and several surgeons wearing the HoloLens 2 succeeded in sharing the same Hologram. Compared with usual two-dimensional cholangiography, this 3D holographic cholangiography technique contributed to more accurate reappearance of the bile ducts, and moving the hologram from the respective operators’ angles by means of easy gesture-handling without any monitors. Furthermore, multiple avatars1 can participate in the same metaverse space by using a function called “virtual session” , which can be applied to remote surgical support (Fig. 4).
XR application in hybrid robotic rectal surgery
In Japan, the standard treatment for stage II/III advanced low rectal cancer is total mesorectal excision plus lateral lymph node dissection (LLND). In our department, hybrid LLND involved simultaneous transabdominal and transanal approaches performed by two surgical teams. However, the transanal anatomy is difficult to understand, and additional support tools are required to improve the surgical safety. Therefore XR is also necessary in this operation as well.
Alignment was performed by superimposing the MDCT and MRI images using the pelvic bone as a landmark. The pelvic bones, arteries, and veins were extracted from the MDCT images, and the lymph nodes and obturator and coccygeal muscles were extracted from the MRI images(Fig. 5).
Intraoperative XR improved the surgical understanding of the lateral lymph node region anatomy. 75% of the surgeons answered that the Hologram accurately reflected the anatomy, and 92% of the surgeons agreed that intraoperative Hologram was a useful support tool for improving the surgical safety.
- avatars: A character that serves as an alter ego of oneself (the user) used in a virtual space ↩︎