Back to Works
Back to Works

Project

Interactive Machine Learning

Computer VisionActive LearningSegmentationHuman-in-the-Loop
Interactive Machine Learning

Outlining an organ on a CT or MRI scan means tracing it on a hundred slices or more, one at a time. We built a way to make that far less painful.

The problem

Think of a CT or MRI scan as a flip book. Each page is a thin slice of the body, and a scan can have a hundred pages or more. If a doctor wants a liver measured, or a radiotherapy plan drawn up, someone has to outline the liver on every single page. That takes hours of skilled time, and it is part of the reason why carefully labelled medical data is so scarce.

So we let models do a first pass. The trouble is that a first pass is rarely good enough. Scanners differ from one hospital to the next, patients differ from one another, and organs have awkward shapes whose edges can almost vanish on low-contrast slices. A model trained in one place often stumbles in another.

The usual remedy is interactive segmentation. The model draws a mask, you fix the worst parts with a few clicks, boxes or scribbles, and it tries again. This helps, but almost every method has the same blind spot: each slice starts from zero. Whatever you taught the model on slice 54 is gone by slice 55, even though the two images are nearly twins, so you end up correcting the same mistake again and again. Foundation models such as SAM have their own limits here. They work on many kinds of images without retraining, but low-contrast medical scans are hard for them, they want precise clicks or boxes, and they don't carry anything over from one slice to the next either.

The obvious idea is to let the model keep learning from your corrections as it goes. That's harder than it sounds. A model that adapts too eagerly overfits to the slice in front of it and gets worse on the next one. A model that adapts too cautiously never learns anything useful. The real problem sits between the two: how do you learn fast from a handful of scribbles and still remember everything the model knew before?

Our approach

We use two copies of the same model, a student and a teacher. The student is quick. When you scribble on a slice, it updates itself to match. The teacher is steady. Its weights are a slow running average of the student's, so it gradually collects what the student learns across the whole volume, and one odd slice can't pull it off course. The mask you see comes from the teacher.

Overview of the framework: a pre-trained model initialises a student and a teacher; the student learns from scribbles and a consistency loss, and the teacher is a moving average of the student

The framework. The student adapts to each slice using your scribbles, and the teacher, a moving average of the student, keeps what has been learned so far. Figure 1 of the paper.

Two signals guide the student. One pulls it toward your scribbles. The other keeps it consistent with the teacher's prediction for that slice. To make that prediction more reliable, we show the model several slightly altered copies of the slice (a little noise, some blur, inverted intensities), average the outputs, and trust only the pixels where the copies agree. When the model is unsure, we use more copies: 4, 16 or 32, depending on how uncertain it is.

Two practical points matter to us. The method works with a model you already have, and UNet, UNet++ and DeepLabV3 all worked, without touching its architecture, so better backbones can be dropped in later. It also doesn't need the original training data at test time, which in medicine is often unavailable for privacy reasons.

What we found

We tested on five public settings: abdominal CT and MRI from CHAOS, liver tumours from LiTS, 15 organs from AMOS, and microscopy cell images from the Data Science Bowl.

  • User annotation time dropped by a factor of 3.05 to 6.72 compared with outlining every slice by hand.
  • Compared with other interactive segmentation methods, it cut the time by a factor of 2.64.
  • Against a prompted foundation model, it reached a Dice score of 0.9 in 3 to 4 interactions, where the foundation model needed 5 to 8.
  • Against the best previous interactive method, it needed 2 to 3 fewer rounds of interaction, a 20 to 36 percent reduction in user effort.

Dice score against number of interactions on three datasets, with our method rising fastest

Dice score after each round of interaction, on CHAOS CT (top), LiTS CT (middle) and Data Science Bowl cell images (bottom). Our method, shown in purple and green, climbs fastest and needs the fewest interactions. Figure 2 of the paper.

Here is one concrete case: the time a person spends to reach a Dice score of 0.95 on a CHAOS CT volume of 84 slices.

MethodUser time (minutes)
Outlining every slice by hand63.00
Copying the previous slice's mask and editing it32.00
GrabCut66.10
f-BRS52.18
RiTM48.82
iSegFormer35.23
PseudoClick30.44
FocalClick25.33
Ours (UNet backbone)9.24

And here is what it looks like on two neighbouring slices of a liver scan. Our method gets both right after two interactions on the first slice and one on the second. The other methods need between 3 and 12 interactions on each.

Two consecutive LiTS slices segmented by GrabCut, f-BRS, iSegFormer, FocalClick and our method, shown after each interaction

Two consecutive LiTS slices, shown after each round of interaction, from the initial mask to the final result next to the ground truth. Figure 3 of the paper.

Read the paper

No Prompting Frozen Foundation Models: Interactive Medical Volume Segmentation using Continual Test Time Adaptation of Compact Models was written with Abhilaksh Singh Reen, C. V. Jawahar and Chetan Arora, and presented at ICVGIP 2024, where it received the Best Paper Runner-Up award. The work grew out of my MS by Research at IIIT Hyderabad. My thesis goes into more depth.

The figures above are reproduced from the paper, which is licensed under CC BY-NC-ND 4.0. Copyright remains with the authors.