feat(nb): group transcripts into paragraphs and label speakers #254

Open

dominik.polakovics wants to merge 1 commit from feat/nb-transcribe-speakers into main

dominik.polakovics commented

2026-08-02 19:51:27 +02:00

Owner

Transcribing a meeting produced one unbroken line of text. The cause is in whisper-cli, not the recording: -nt suppresses the timestamp prefix and the trailing newline (cli.cpp:447 only emits \n when timestamps or speakers are on). Switching to -oj JSON output fixes it at the root and supplies the millisecond offsets everything else here needs.

What changed

Paragraphs. whisper segments are breath groups of 20-40 characters. They are now regrouped on a 2 s pause, or at a sentence end once a paragraph passes 700 characters.
--speakers=N labels who is talking, via sherpa-onnx (pyannote segmentation-3.0 + NeMo TitaNet-large embeddings). ONNX only, so ~200 MB and no gated HuggingFace token, versus the multi-gigabyte PyTorch closure pyannote.audio would need. whisper.cpp's own --diarize is stereo-channel guessing and --tinydiarize is English-only.
--names A,B,C names speakers in order of first appearance and implies the count.
--timestamps now stamps each paragraph, not each segment.

The count is required, deliberately

Auto-detection was measured against a real 4-person meeting and is not usable:

cluster threshold	speakers found
0.5	85
0.7	28
0.8	22
0.9	15

An auto mode emitting 22 labels reads as broken; guessing N and re-running converges. Bare --speakers therefore errors and points at --speakers=N.

Cost

Diarization is CPU-only, roughly 40 minutes per hour of audio, so it is opt-in. It also forces VAD off — the speaker join needs timestamps on the real timeline — which carries the same slowdown and occasional silence hallucination already documented for --timestamps.

Verification

Built standalone and exercised against a real 69-minute German meeting recording (slices of it). Turn-taking is coherent:

[00:00:00] Anna: Läuft bei dir.
[00:00:02] Bernd: Gut, also es freut mich, dass alle da sind.
[00:00:05] Anna: Ebenso.
[00:00:06] Bernd: Und wir starten durch. Ich habe einige Punkte, ...

Checked by hand: default mode, --timestamps, --speakers=N, --names, name-count inference, --speakers overriding --names, and the rejection paths (bare --speakers, --speakers=1, --speakers=abc, --names with no value, missing file, unknown flag). nb dry-build passes.

Transcribing a meeting produced one unbroken line of text. The cause is in whisper-cli, not the recording: `-nt` suppresses the timestamp prefix *and* the trailing newline (`cli.cpp:447` only emits `\n` when timestamps or speakers are on). Switching to `-oj` JSON output fixes it at the root and supplies the millisecond offsets everything else here needs. ## What changed - **Paragraphs.** whisper segments are breath groups of 20-40 characters. They are now regrouped on a 2 s pause, or at a sentence end once a paragraph passes 700 characters. - **`--speakers=N`** labels who is talking, via `sherpa-onnx` (pyannote segmentation-3.0 + NeMo TitaNet-large embeddings). ONNX only, so ~200 MB and no gated HuggingFace token, versus the multi-gigabyte PyTorch closure `pyannote.audio` would need. whisper.cpp's own `--diarize` is stereo-channel guessing and `--tinydiarize` is English-only. - **`--names A,B,C`** names speakers in order of first appearance and implies the count. - **`--timestamps`** now stamps each paragraph, not each segment. ## The count is required, deliberately Auto-detection was measured against a real 4-person meeting and is not usable: | cluster threshold | speakers found | |---|---| | 0.5 | 85 | | 0.7 | 28 | | 0.8 | 22 | | 0.9 | 15 | An auto mode emitting 22 labels reads as broken; guessing N and re-running converges. Bare `--speakers` therefore errors and points at `--speakers=N`. ## Cost Diarization is CPU-only, roughly 40 minutes per hour of audio, so it is opt-in. It also forces VAD off — the speaker join needs timestamps on the real timeline — which carries the same slowdown and occasional silence hallucination already documented for `--timestamps`. ## Verification Built standalone and exercised against a real 69-minute German meeting recording (slices of it). Turn-taking is coherent: ``` [00:00:00] Anna: Läuft bei dir. [00:00:02] Bernd: Gut, also es freut mich, dass alle da sind. [00:00:05] Anna: Ebenso. [00:00:06] Bernd: Und wir starten durch. Ich habe einige Punkte, ... ``` Checked by hand: default mode, `--timestamps`, `--speakers=N`, `--names`, name-count inference, `--speakers` overriding `--names`, and the rejection paths (bare `--speakers`, `--speakers=1`, `--speakers=abc`, `--names` with no value, missing file, unknown flag). `nb` dry-build passes.

dominik.polakovics added 1 commit

2026-08-02 19:51:27 +02:00

feat(nb): group transcripts into paragraphs and label speakers 327a958917

The transcript arrived as a single unbroken line. That was not the recording:
whisper-cli's -nt only suppresses the timestamp prefix, and cli.cpp then skips
the trailing newline too, so every segment ran together. Reading -oj JSON instead
of scraping stdout sidesteps that and hands over millisecond offsets, which the
paragraph grouping and the speaker join both need anyway.

Segments are whisper's breath groups, 20-40 characters each, so they are
regrouped into paragraphs on a 2 s pause or a sentence end past 700 characters.
--timestamps now stamps the paragraph rather than every segment; per-segment
[from --> to] spans were half of what made the output unreadable.

Speaker labels come from sherpa-onnx: pyannote segmentation-3.0 finds the turns,
NeMo TitaNet-large embeddings cluster them into people. Both are ONNX, so this
costs ~200 MB rather than the multi-gigabyte PyTorch closure pyannote.audio
proper would pull, and needs no gated HuggingFace token. whisper.cpp's own
--diarize is stereo-channel guessing and --tinydiarize is English-only, so
neither was an option.

--speakers requires the count. The clustering cannot infer it: on a real
4-person meeting it estimated 85, 28, 22 and 15 speakers at distance thresholds
0.5, 0.7, 0.8 and 0.9. Shipping an auto mode that answers 22 would look broken,
whereas guessing N and re-running converges. --names implies the count, since
listing the room is easier than counting it.

Diarization needs true timestamps to join against, so like --timestamps it turns
VAD off, with the same hallucination and runtime cost that implies. It runs
~40 minutes of CPU per hour of audio, hence opt-in.

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/nb-transcribe-speakers:feat/nb-transcribe-speakers

git switch feat/nb-transcribe-speakers

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff feat/nb-transcribe-speakers

git switch feat/nb-transcribe-speakers

git rebase main

git switch main

git merge --ff-only feat/nb-transcribe-speakers

git switch feat/nb-transcribe-speakers

git rebase main

git switch main

git merge --no-ff feat/nb-transcribe-speakers

git switch main

git merge --squash feat/nb-transcribe-speakers