Project

Outlining an organ on a CT or MRI scan means tracing it on a hundred slices or more, one at a time. We built a way to make that far less painful.
Think of a CT or MRI scan as a flip book. Each page is a thin slice of the body, and a scan can have a hundred pages or more. If a doctor wants a liver measured, or a radiotherapy plan drawn up, someone has to outline the liver on every single page. That takes hours of skilled time, and it is part of the reason why carefully labelled medical data is so scarce.
So we let models do a first pass. The trouble is that a first pass is rarely good enough. Scanners differ from one hospital to the next, patients differ from one another, and organs have awkward shapes whose edges can almost vanish on low-contrast slices. A model trained in one place often stumbles in another.
The usual remedy is interactive segmentation. The model draws a mask, you fix the worst parts with a few clicks, boxes or scribbles, and it tries again. This helps, but almost every method has the same blind spot: each slice starts from zero. Whatever you taught the model on slice 54 is gone by slice 55, even though the two images are nearly twins, so you end up correcting the same mistake again and again. Foundation models such as SAM have their own limits here. They work on many kinds of images without retraining, but low-contrast medical scans are hard for them, they want precise clicks or boxes, and they don't carry anything over from one slice to the next either.
The obvious idea is to let the model keep learning from your corrections as it goes. That's harder than it sounds. A model that adapts too eagerly overfits to the slice in front of it and gets worse on the next one. A model that adapts too cautiously never learns anything useful. The real problem sits between the two: how do you learn fast from a handful of scribbles and still remember everything the model knew before?
We use two copies of the same model, a student and a teacher. The student is quick. When you scribble on a slice, it updates itself to match. The teacher is steady. Its weights are a slow running average of the student's, so it gradually collects what the student learns across the whole volume, and one odd slice can't pull it off course. The mask you see comes from the teacher.

The framework. The student adapts to each slice using your scribbles, and the teacher, a moving average of the student, keeps what has been learned so far. Figure 1 of the paper.
Two signals guide the student. One pulls it toward your scribbles. The other keeps it consistent with the teacher's prediction for that slice. To make that prediction more reliable, we show the model several slightly altered copies of the slice (a little noise, some blur, inverted intensities), average the outputs, and trust only the pixels where the copies agree. When the model is unsure, we use more copies: 4, 16 or 32, depending on how uncertain it is.
Two practical points matter to us. The method works with a model you already have, and UNet, UNet++ and DeepLabV3 all worked, without touching its architecture, so better backbones can be dropped in later. It also doesn't need the original training data at test time, which in medicine is often unavailable for privacy reasons.
We tested on five public settings: abdominal CT and MRI from CHAOS, liver tumours from LiTS, 15 organs from AMOS, and microscopy cell images from the Data Science Bowl.

Dice score after each round of interaction, on CHAOS CT (top), LiTS CT (middle) and Data Science Bowl cell images (bottom). Our method, shown in purple and green, climbs fastest and needs the fewest interactions. Figure 2 of the paper.
Here is one concrete case: the time a person spends to reach a Dice score of 0.95 on a CHAOS CT volume of 84 slices.
| Method | User time (minutes) |
|---|---|
| Outlining every slice by hand | 63.00 |
| Copying the previous slice's mask and editing it | 32.00 |
| GrabCut | 66.10 |
| f-BRS | 52.18 |
| RiTM | 48.82 |
| iSegFormer | 35.23 |
| PseudoClick | 30.44 |
| FocalClick | 25.33 |
| Ours (UNet backbone) | 9.24 |
And here is what it looks like on two neighbouring slices of a liver scan. Our method gets both right after two interactions on the first slice and one on the second. The other methods need between 3 and 12 interactions on each.

Two consecutive LiTS slices, shown after each round of interaction, from the initial mask to the final result next to the ground truth. Figure 3 of the paper.
No Prompting Frozen Foundation Models: Interactive Medical Volume Segmentation using Continual Test Time Adaptation of Compact Models was written with Abhilaksh Singh Reen, C. V. Jawahar and Chetan Arora, and presented at ICVGIP 2024, where it received the Best Paper Runner-Up award. The work grew out of my MS by Research at IIIT Hyderabad. My thesis goes into more depth.
The figures above are reproduced from the paper, which is licensed under CC BY-NC-ND 4.0. Copyright remains with the authors.
More work
Back to Works
Ayu estimates a person's age from a resting-state fMRI scan, then uses the model's attention to show which brain connections change as we get older.

A COVID-19 screening tool that listens to a short cough recording and predicts whether it's positive. Our team placed 7th in the Microsoft Azure AI Challenge.