• Endoscopic surgery videos (e.g. tested with the EndoNeRF, EndoVis'17, and SurgToolLoc datasets), which are usually captured while performing surgery in vivo using an endoscope.

• Microsurgery videos, such as those taken using a microscope during ear surgery, such as cochlear implant surgery.

Features of SAM-2

• Zero-shot segmentation: Without pre-labeled data, the SAM 2 model can identify and segment surgical tools directly from surgical videos. By using cues (e.g., points, boxes, masks), the model is able to take information in the first frame of the video and apply it to the entire video for tool tracking and segmentation.

• Multi-scenario applicability: The tool is suitable for different types of surgical videos, including endoscopic surgery and microsurgery. It can handle scenes containing multiple surgical tools, as well as surgical procedures with different video lengths.

• Efficient Segmentation: The SAM 2 model was trained using the extensive Segment Anything Video (SA-V) dataset, which enables efficient segmentation of surgical tools in videos, reducing reliance on manual annotation and improving segmentation accuracy.

• Built-in memory: The model integrates a memory to propagate initial cues between frames of the video, thus ensuring the continuity and accuracy of the segmentation.

• Coping with complex environments: Despite challenges such as blur, occlusion, and bleeding, SAM 2 can still provide reliable tool segmentation results in these cases and further improve the segmentation results through additional cues.

• Suitable for real-time applications: Due to its combination of zero-shot segmentation capabilities and memory bank, the tool has the potential to be used in real-time surgical scenarios to help surgeons identify and track surgical tools more accurately.

Main Uses of SAM-2

Improving surgical precision and safety:

Reduce the workload of manual annotation:

Support diverse surgical scenarios:

Helping medical research and training:

Promoting the development of surgical robots:

Laying the foundation for future surgical AI systems:

Experimental Results of SAM-2

1. Endoscopic Surgery Dataset

Dataset used:

• EndoNeRF: Contains two surgical video clips, containing 63 and 156 frames respectively.

• EndoVis'17: Contains 8 robotic surgery videos, each containing 255 frames, and the corresponding ground truth segmentation masks.

• SurgToolLoc: Contains 24,695 video clips, each lasting 30 seconds and captured at 60 frames per second, all from the da Vinci robotic surgery system.

Evaluation results:

• In the endoscopic surgery dataset, SAM 2 demonstrates strong tool segmentation capabilities, especially in multi-tool scenarios and with varying video lengths.

• Quantitative evaluation using the EndoVis'17 dataset shows that SAM 2 outperforms other mainstream segmentation methods such as U-Net, UNet++, and TransUNet in terms of Dice score, IoU (Intersection over Union), and MAE (Mean Absolute Error).

Quantitative evaluation data :

• Dice score: SAM 2 reaches 0.937, which is significantly higher than U-Net (0.894), UNet++ (0.909) and TransUNet (0.904).

• IoU: The IoU value of SAM 2 is 0.890, which is also better than other methods.

• MAE: SAM 2 has the best performance with a mean absolute error (MAE) of 0.018.

2. Microsurgery Dataset

Dataset used :

• Our own dataset: Cochlear implant surgery videos collected from Vanderbilt University Medical Center and the Medical University of South Carolina, including surgery clips of varying lengths (2 to 10 seconds) covering different stages of the surgery (such as drilling and implantation).

Evaluation results :

• SAM 2 performs well in microsurgery videos, especially providing reliable segmentation results in both single-tool and multi-tool scenarios.

• SAM 2 segmentation results are best when the surgical scene is well lit and the tool motion quality is high.

Conclusion of SAM-2

Model performance:

• SAM 2 demonstrates significant performance advantages in zero-shot video segmentation tasks, especially in accurately segmenting tools in surgical videos under good lighting conditions and high-quality tool motion.

• In various surgical scenarios (e.g., endoscopic and microsurgery), SAM 2 is able to generate reliable tool segmentation results by providing point cues only in the first frame of the video, demonstrating its strong versatility and adaptability.

Limitations of the model:

• Segmentation Challenges for Long Video Sequences: As the video sequence gets longer, the segmentation accuracy of SAM 2 decreases, especially in the later stages of the video, where the accuracy of detail segmentation decreases. This performance degradation is a significant challenge for real-time surgical video applications and requires further improvement.

• Impact of complex surgical environments: Complex factors in the surgical environment, such as blur, bleeding, and tool occlusion, significantly affect the segmentation accuracy of SAM 2. Especially in microsurgery, the model is prone to losing segmentation fineness due to the limitations of the microscope camera and the interaction of tools with the surgical surface.

Coping strategies:

• Introducing additional cues (such as when a new tool enters the scene) can improve the segmentation accuracy to a certain extent, especially when dealing with complex or dynamically changing surgical scenes.

• To address these challenges, future research directions should focus on how to improve the performance of the model in long video sequences and how to enhance its robustness in complex environments by fine-tuning the model.

💡

Scale Up, Scale Down. Your Server, Your Rules. Highlighting flexibility and control.