Notes by Aaditya Kumawat

Same words, different notes

2026-10-05T00:00:00+05:30

During my research internship at the Applied AI Laboratory, HEC Lausanne, I built a fully local pipeline that turns court audio into notes. In one mode a local language model reads a numbered transcript and only picks which sentences go into the notes; it cannot change a word. So every note is verbatim, and every accuracy metric I had said the pipeline was working. None of them asked whether the notes were fair to the people speaking.

The probe

Keep the transcript byte-for-byte identical, change only the speaker labels, run the selection again, and compare. Three conditions:

Condition	What changes
Control	Nothing. The identical transcript is selected again, to see how much the selection moves on its own.
Anonymised	Every label becomes SPEAKER_XX, so the model can no longer tell the speakers apart.
Permuted	Labels are shifted by one, so the same labels sit on different people.

For each condition I measured every speaker’s share of the selected lines and how far it moved from the original run. A shift only counts as evidence if it is at least 5 points and at least twice whatever the control moved.

The first run was confounded

My first anonymised run used a plain “SPEAKER” label. It was shorter than the real labels, so more lines fit in each window the model reads, and the transcript was split at different points: windows of 155, 175, 187, 176 and 61 lines against the original 151, 178, 176, 173 and 84. That run changed two things at once, so its 10.7-point shift could not be blamed on the labels.

The fix was a label of exactly the same width, SPEAKER_XX, and a rule: any condition whose windows don’t match the original is reported but never counted.

The result

Condition	Overlap with original selection	Largest shift in one speaker’s share	Counts as evidence
Control	100%	0 points	This is the floor
Anonymised	18%	40.4 points	Yes
Permuted	41%	3.6 points	No

Removing who-said-what replaced most of the selection and moved one speaker’s share of the notes by 40.4 points. The words were identical.

Word error rate, diarization error, verbatim rate and the judge all score these two sets of notes the same. Only the probe saw the difference.

What it does and doesn’t show

It shows that what gets selected depends on the labels, not just the words. No accuracy metric in the pipeline checks for that.
It doesn’t show that the model favours particular people. Swapping who is who moved shares by only 3.6 points, below the bar. The more likely reading is that the model uses the labels to follow the structure of the hearing, who is asking and who is answering, and selects differently when that structure disappears.
It is one case: a 62-minute Supreme Court argument with 10 speakers. That makes it a finding to chase, not a result to generalise.

Why this matters for diarization

If removing the labels can move a speaker’s share of the notes by 40 points, then the labels a diarizer produces are not just a detail of the transcript. Diarization error on this audio was 7.9%, and overlapping speech, where two people talk at once, is one of the hardest cases for a diarizer. A turn given to the wrong speaker could change what ends up in the notes.

Measuring how diarization errors, especially in overlapping speech, carry through into what a summary selects is the next thing I want to study.

My GPU was waiting, not working

2026-10-05T00:00:00+05:30

The standard advice about LLM inference is that decoding is limited by memory bandwidth. Every new token means reading all of the model’s weights from GPU memory, so the speed limit is how fast you can stream bytes. I built nanoserve, an inference server written from scratch, expecting to optimise for exactly that. The first careful measurement said I was optimising the wrong thing.

The number that didn’t fit

Qwen2.5-0.5B in fp16 is 943 MiB of weights, and my laptop’s RTX 4060 sustains about 210 to 226 GB/s on a plain memory copy. If memory were the limit, one decode step at batch size 1 should take about 4.5 ms.

Measure	Windows 11	WSL2 Ubuntu
Memory-bandwidth floor per step	~4.5 ms	~4.5 ms
Actual batch-1 step (eager transformers)	23.7 ms	16.6 ms
Cost of launching one small kernel	7.8–15 µs	6.8–9.2 µs
GPU utilisation during decode	18%	22%

The step was four to five times slower than the bandwidth floor. Profiling showed why: the GPU was busy for only about a fifth of each step. The rest of the time it sat idle while the CPU issued the next of roughly 1,000 to 1,350 small kernels.

Windows 1118% busy

WSL2 Ubuntu22% busy

GPU computing GPU idle, waiting for the CPU to launch the next kernel

The tell: batching was almost free

If most of each step is a fixed launch cost, a bigger batch should cost almost nothing extra. That is exactly what happened. Step time went from 28 ms at batch 1 to 34 ms at batch 32, so throughput went from 35.5 to 941.8 tokens per second: a 26× gain for 20% more time per step.

I blamed Windows first, and I was wrong

My first theory was the Windows display driver model (WDDM), which routes kernel launches through the operating system’s scheduler. I predicted Linux would be much faster. The measurements didn’t back that up:

Two consecutive runs of the same Linux setup measured the small-kernel cost at 6.8 µs and 9.2 µs. That 35% spread overlaps the Windows range.
The two setups also ran different PyTorch builds, which changed the number of device ops per step from 1,350 to 1,014 for identical model code. That is a library difference, not an OS one.
Utilisation was 18% against 22%. Both were overhead-bound.

On this hardware, the number of kernels is what moves decode time. The operating system is not.

What I changed because of it

Once the bottleneck was clear, every change that paid off was one that cut the number of ops per step:

Change	Why it mattered here
RoPE cos/sin computed once per step	Positions are the same in all 24 layers; the old code built 48 identical tensors per step
Fused RMSNorm	Replaced an 8-kernel hand-written norm used at 48 places per step
Grouped-query attention inside SDPA	Stopped materialising a 7× expanded copy of K and V
One pinned host-to-device copy for step metadata	Was six separate small transfers per step

Together with a faster way of mapping tokens to cache slots, these took the batch-1 step from 33.7 ms to 28.2 ms, 16% faster.

Then the fused Triton paged-attention kernel replaced about ten ops per layer with one. Device ops per step fell from 1,014 to 682, GPU utilisation at batch 32 rose from 23% to 36%, and nanoserve clearly beat eager transformers for the first time: 1.23× at batch 8.

The kernel’s speed-up shrinks as context grows, which surprised me. At batch 1 the torch path takes about the same time at every context length, because it is paying for ten kernel launches, not for the data.

Four measurements that lied first

Before any of these numbers could be trusted, four bugs in the measurement itself had to go. Each one produced a plausible-looking wrong answer.

Clock ramp. The GPU idles at 210 MHz and boosts to 2,595 MHz. Whatever ran first was measured cold, which once produced “batch 32 is faster than batch 1” and a fake 3.4× win. Now there are 8 seconds of sustained work before measuring, repeated before every row.
Double-counted kernels. The profiler attributes time to both an op and its kernel. Summing both reported 166% GPU utilisation.
Mismatched runs. Busy time from a profiled run was being divided by wall time from an unprofiled one.
Noise. Every configuration is now timed twice and flagged if the runs disagree by more than 15%.

Where this leaves nanoserve

Against vLLM in its strongest setup (torch.compile plus CUDA graphs), on the same GPU and the same requests, the honest comparison is:

Server	Output tokens/s	vs vLLM
vLLM, compiled with CUDA graphs	2,103.5	1.00×
nanoserve, continuous batching + Triton	663.0	0.32×
Static batching, best batch size	236.9	0.11×

The biggest part of that gap is the same story. vLLM captures its decode step as CUDA graphs once and replays them, while nanoserve re-issues around 680 kernels every step from Python. So the next change is CUDA graphs for decode. The shapes are static, so the step can be captured once and replayed. That attacks the time spent waiting, which is still most of each step, instead of the time spent working.

The takeaway

Measure the shape of the bottleneck on your own hardware before optimising for it. “Decode is memory-bound” may well hold for a large model on a datacenter GPU. For a 0.5B model on a laptop GPU, the real step took four to five times longer than the memory floor, and the gap was all launch overhead.

A Gaussian blur beat my method

2026-10-05T00:00:00+05:30

ST-BEMD is a 2-D empirical mode decomposition I built this summer. It bends its envelopes along the local ridge direction, so on fingerprints it should keep each ridge’s orientation intact better than the standard, direction-blind version. On 30 real prints, orientation error dropped from 13.71° to 8.25°, and every single print improved. This note is about why that number does not mean what it looks like it means.

Two changes, one name

Compared with the 2003 baseline, ST-BEMD changes two things at once. The envelope goes from a global RBF interpolation to a local weighted average, and the averaging kernel goes from a circle to an ellipse stretched along the ridges. The method is named after the second change, so I ran a version with only the first one: the same local averaging, but with a circular kernel.

Version	Orientation error (30 prints)	Step
Baseline: global RBF, circular	13.71°
Local average, circular kernel	9.43°	−4.29° from the envelope
ST-BEMD: local average, elliptical kernel	8.25°	−1.18° from the anisotropy

About 78% of the improvement comes from the local envelope, which is an existing idea, not from the anisotropy the method is named after. The anisotropy’s own effect is real (better on 70% of prints, p = 0.001), but small. Worse, it shrinks as ridges curve: +1.42°, +1.13° and +0.20° from low to high curvature. The whole premise of the method predicts the opposite.

Then I tested the metric

The orientation error compares the orientation field of the extracted layer with the orientation field of the input. That made me wonder how estimators that know nothing about orientation would score.

Estimator	Orientation error
Return the input unchanged	0.00°
Subtract a Gaussian blur (σ = 4)	4.23°
Subtract a Gaussian blur (σ = 1)	7.26°
ST-BEMD	8.25°
Local average, circular kernel	9.43°
Baseline	13.71°

Doing nothing scores a perfect zero. Subtracting a blurred copy of the image scores about twice as well as my method. The metric’s best possible answer is to leave the input alone.

What the metric was really measuring

To check this properly, I picked the blur width on half the prints and scored on the other half, so the control couldn’t be tuned to the test set. I also added a second measure, IMF validity: how close the extracted layer is to having zero local mean, which is the thing sifting is supposed to achieve. Lower is better.

Held out, 15 prints	Orientation error	IMF validity
Baseline	15.91°	0.131
Local average, circular kernel	10.08°	0.210
ST-BEMD	9.34°	0.185
Gaussian blur control	4.86°	0.383

Across all ten estimators the two measures are anti-correlated at r = −0.94. Scoring well on orientation means having sifted less.

The two measures pull against each other. The blur buys its orientation score by not doing the job.

The orientation metric was largely measuring how little each estimator sifted.

So the 13.71° to 8.25° result supports a narrower claim: ST-BEMD’s envelope disturbs the orientation field less than a global interpolant does. It does not show that it recovers orientation better. One small point does survive cleanly: against the circular local version, ST-BEMD is better on both measures (9.34° vs 10.08°, and 0.185 vs 0.210). Same machinery, only the kernel shape differs.

A test that doing nothing can’t win as easily

To break the circularity, I took the reference orientation from the clean print and gave every method a noisy copy, so preserving the input is no longer enough. I also measured orientation with two unrelated instruments, a bank of Gabor filters and the structure tensor, so the method couldn’t be graded by the same tool it steers by.

Noise level	Baseline	ST-BEMD	ST-BEMD ahead by
20 dB (light)	10.04°	10.97°	−0.93°
10 dB	12.87°	12.57°	+0.30°
5 dB (heavy)	18.33°	15.06°	+3.27°

With the reference taken from the clean print, ST-BEMD degrades more gracefully than the baseline as noise rises. Doing nothing is still the lowest line.

With light noise ST-BEMD is slightly worse. With heavy noise it is ahead by 3.3° (3.6° with the second instrument), and both instruments agree on the ranking. That is the claim I can defend: direction-adaptive local envelopes are more robust to noise than global RBF interpolation.

It is still not a clean win. Doing nothing remains the lowest-error entry, and the blur still beats every EMD method on orientation. This protocol is necessary, not sufficient.

What I’d tell myself in May

Build the control before the method. If a metric gives its best score to an estimator that does nothing, it cannot reward an estimator for doing something. I only found this because I tried to beat my own number with something deliberately dumb, and it won.