Under-the-Skin Multimodal Vision for Real-Time Surgical Tool Tracking in High Fidelity Medical Simulation Training

restricted
Restricted
Author:
Khalil, Jasmine
Area of Honors:
Electrical Engineering
Degree:
Bachelor of Science
Document Type:
Thesis
Thesis Supervisors:
Jason Zachary Moore, Thesis Supervisor
Julio Urbina, Thesis Honors Advisor
Keywords:
Central Venous Catheterization
Computer Vision
Deep Learning
Medical Simulation Training
Stereo Calibration
Epipolar Geometry
Under-the-Skin
Surgical Tool Tracking
High Fidelity
Multimodal Vision
Abstract:
Central Venous Catheterization (CVC) is a high-risk clinical procedure performed for long-term blood access. It is among the most frequently performed procedures in medicine, with approximately 27 million CVCs inserted worldwide annually. Despite its prevalence, the procedure carries substantial risk, with mechanical complications occurring in 5% - 19% of patients. Simulation training has emerged as a proven intervention for improving CVC outcomes. However, existing training simulators are limited in their capacity to deliver objective, real-time spatial feedback and cost over $5,000. This gap between clinical need and accessible, real-time feedback motivates this work. This thesis presents a multimodal computer vision framework for realtime tracking of surgical tools within a high-fidelity medical simulation environment. The proposed system uses a novel Under-the-Skin camera architecture combined with deep-learning detection and classical computer vision tracking to estimate the positions of surgical tools during simulated CVC. The system integrates a dual camera network; one stereo pair captures the procedural cross-section of a custom phantom skin, and a second stereo pair is oriented upward beneath the phantom skin. The system is enclosed, creating a controlled workspace that enables occlusion-resistant tracking without expensive infrastructure. Color-coded instruments, a solid white interior, and controlled illumination collectively reduce detector complexity and improve tracking reliability. Tool tracking is performed to identify 3 of the main instruments from the Seldinger technique: needle, dilator, and catheter. The system tracks these tools as they are inserted through a custom phantom skin model using the cross-section-facing USB stereo cameras. A ChArUco board visible in both camera views enables automatic stereo calibration at the start of each training session, computing the intrinsic and extrinsic parameters and generating rectification maps that align both camera views along a common epipolar plane. The tracking pipeline uses a hybrid approach. An HSV color filter system is the primary tracker, exploiting the distinct colors of each tool to detect and localize the tool tip and base in each frame. A YOLO11n Oriented Bounding Box (OBB) detector, trained on 894 rectified stereo images of the tools, serves as a periodic correction mechanism, firing every 50 frames to verify and correct the HSV-derived tip and base image coordinates. This design keeps the YOLO detector dormant 92.5% of the time, preserving computational resources while maintaining tracking accuracy. The system was evaluated across seven sessions totaling over 5000 frames on a CPU-only Apple M2, with no GPU. When tools were in view, the HSV tracker achieved 98.2%, 97.9%, and 98.1% reliability for the needle, dilator, and catheter, respectively, with average tip confidence scores of 0.950, 0.677, and 0.779, respectively. The combined system operated at a median frame time of 54ms (18 FPS equivalent during HSV-only frames) and 12.8 FPS overall. These results demonstrate that a hybrid HSV-YOLO vision pipeline can reliably track multiple surgical tools in real time on commodity hardware, offering a practical and low-cost alternative to electromagnetic sensor tracking for surgical simulation training. This work advances the accessibility of CVC simulation training, with direct implications for clinical competency assessments and ultimately, patient safety.