Multi-modal Registration
Registration of Ultrasound to MRI and CT
Image registration generally refers to the process of aligning two or more images, establishing a common coordinate system and thus achieving correspondence between the information contained within the images. For many modern applications of computer-assisted diagnosis and therapy, it has evolved into a key technology and continues to be actively investigated in research. This project is basically a meta-project – a topic I have been working on for more than 10 years now.
Multi-modal image registration is highly challenging when ultrasound is involved:
-
Going from 2D to 3D: Typically, ultrasound images are 2D and arbitrarily oriented in space. There is no standard grid such as in CT or MRI, an external optical, electro-magnetic or robotic tracking system is needed for accurately measuring the ultrasound transducer’s position in space at any time, and complex ultrasound reconstruction (“compounding”) is needed to be even able to compare 3D ultrasound against anything else. Often, there are lots of areas between US frames where there is no information available.
-
Anisotropic Point-spread Function: The point-spread function of ultrasound transducers is quite anisotropic. In-plane, especially around the focal spot, we can achieve a good effective resolution, but out of plane, objects several millimeters away from the imaging plane may also generate echo.
-
Artifacts: Ultrasound imaging suffers from all kinds of artifacts, including shadowing and reverberation. Ultrasound cannot penetrate beyond an air gap (lungs), and seeing anything behind a bone surface is really hard (dedicated transducers) to impossible.
-
Deformations: For the acquisition, the probe needs to be pressed against the skin to achieve good acoustic coupling, sometimes creating large deformations. Image registration algorithms need to take that into account, either by undoing the deformation on the ultrasound frames to produce a deformation-free 3D ultrasound reconstruction, or by also applying the deformation onto the other, tomographic image modality.
At ImFusion, the ultrasound group is dedicated to overcoming these challenges in a product-grade way, and enabling medical devices that utilize 3D ultrasound for diagnostics, interventional guidance, and even treatment.
A common issue is the initialization of a registration, find more details there.
The objective of image registration can be achieved in two distinct ways, either by extracting and aligning a set of features from the input images (feature-based registration), or by employing the image intensities directly (intensity-based registration). Over the years, I have used most approaches to tackle hard registration problems, see a overview of related publications below.
For intensity-based registration, a similarity metric is needed during the optimization. Its purpose is to quantify how well a fixed image A and a transformed moving image B correspond in the overlapping area. At ImFusion, the proprietary similarity metric LC2 for multi-modal registration was developed and refined over the years. Recently, we found that complex, multi-modal similarity metrics such as LC2 can be approximated with a small CNN, which can be trained in a completely unsupervised fashion. As a result, gradient-based optimizers can be used (LC2 itself is not differentiable), which greatly speeds up the registration, while at the same time retraining meaningful representations across modalities to ensure a good registration result:
