Automated morphometry with AI and image processing

My module of the BlueRevolution project (ENIB, Lab-STICC, Ifremer): masks made with SAM, synthetic labelled images and a U-Net that separate nematodes and copepods from debris in microscope photos, feeding the team's morphometry pipeline.

Team project of six students. I built the synthetic-data generator and the segmentation module; the rest of the platform (tail and oesophagus detection with YOLO, species, colour, texture and gonad classifiers, morphometry and the Streamlit app) was built by the team.

Python
PyTorch
PyTorch Lightning
U-Net
Segment Anything (SAM)
OpenCV
TorchMetrics
TensorBoard
Synthetic data

Objective

Biologists measure each nematode and copepod by hand in microscope photos (length, width, tail, oesophagus) to study how human and climate pressures such as dredging, fishing and ocean warming affect marine soils. The organism is mixed with debris, and annotating masks to train a model takes longer than measuring the animals. The goal of this module is to deliver binary masks of the organism, debris excluded, that feed the team's geometric feature extraction.

Initial state

Microscopy frames often mix the target organism with attached debris. Early image–mask pairs made that ambiguity explicit: the model must learn a clean organism silhouette so geometric features are not polluted by waste.

Initial microscopy images with organism–debris ambiguity and early masks — Initial examples: debris near or on the organism complicates clean binary masks.

Making the masks with SAM

Drawing masks by hand was the slowest part, so I used Segment Anything (SAM, ViT-H) as an interactive tool in a Colab notebook. For each photo I click one to three points: green points on the organism and red points on the debris I want to leave out. SAM proposes three candidate masks, I keep the best one and save it as the label. Doing this over the real photos gave the set of image–mask pairs that everything else is built on.

Microscope photo of a nematode with green points on the organism and a red point on the debris — The prompts: green stars on the organism, red star on the debris to exclude.

The three candidate masks that SAM proposes for those prompts — The three candidate masks; the first two follow the organism, the third one is not usable.

Dataset and synthetic data

The result is a labelled set of microscopy images, binary masks and held-out test images, about 100 real pairs in all. To enlarge the training data from so few real photos, an OpenCV compositor cuts out the animal with its mask, places it on an empty microscope background, rotates the scene and overlays two or three random debris pieces, producing about 300 synthetic scenes. Each image comes with its mask, ready for segmentation: white for the organism, black for background and attached debris. For training, images and masks are resized to 256×256.

Real microscope images next to their binary masks — Real image–mask pairs from the dataset.

Example of synthetic compositing: background, organism cutout, debris cutout, synthetic image and organism mask — One compositing example: background + organism + debris → synthetic RGB; the label keeps the organism only.

The U-Net

I built the U-Net in PyTorch Lightning following the standard architecture: four downsampling and four upsampling blocks, each with two convolutions with batch normalisation and ReLU, skip connections between them, bilinear upsampling and a single-channel output. It trains with binary cross-entropy on the logits and Adam, with cosine annealing as the best learning-rate schedule and gradient accumulation to emulate a larger batch.

A configuration block at the top of the notebook (channels, learning rate, batch size, scheduler, augmentation, RGB or grayscale) lets me change an experiment in one place. Training saves the best checkpoint by validation loss, stops early when it stops improving, and logs loss, accuracy and IoU in TensorBoard. Inference returns the mask at the original photo size.

U-Net encoder–decoder architecture diagram — U-Net used for organism vs debris binary segmentation.

Experiments

I ran 48 training runs and compared them in TensorBoard: RGB against grayscale, with and without data augmentation, batch sizes from 2 to 32, with and without the synthetic images, and the learning-rate schedule (reduce on plateau and cosine annealing, among the options in the configuration). I also tried a U-Net with a pretrained backbone and a Mask R-CNN; the masks used downstream come from the plain U-Net.

The eight checkpoints I kept performed very much alike: validation loss between 0.113 and 0.120 and validation IoU between 0.66 and 0.71. The best by validation loss reached 0.113 with an accuracy of 0.955 and an IoU of 0.705 (epoch 193 of 314, cosine scheduler). Below is an interactive dashboard of all the runs: pick runs, change the smoothing and sort the summary table.

Open full screen ↗

Interactive dashboard built from the TensorBoard logs (in English).

Results

Delivered debris-aware organism masks suitable for automated morphometry. Side-by-side checkpoint comparisons on test images showed that stronger runs isolate the microorganism and leave attached waste out of the mask, which hands the rest of the team the organism already separated from the background.

Comparison of UNet checkpoint masks against originals with debris — Checkpoint comparison: organism retained, debris excluded from the mask.

Limits

The validation score is probably optimistic: validation is a random 80/20 split of real and synthetic images mixed together, not a held-out set of real photos only.
Validation F1 was never logged; the curves and the table report loss, accuracy and IoU.
Only about 100 real pairs, so the synthetic images carry a large share of the training data.
The configurations differ by little, so I would not read the ranking of the runs as meaningful.
My module stops at the masks; tail and oesophagus detection, classification and the measurements were built by the rest of the team.