TRAC FM runs radio polls in Uganda. Listeners call in and answer in their own languages, often switching between them mid-sentence, over noisy phone and radio lines. The partner wanted those answers as English text, grouped by theme for each question.
The problem with choosing a model
Every stage had several candidates. Cleaning: FFmpeg loudness, voice activity detection, noise suppression. Transcription: Whisper, or models built for low-resource languages. Translation: an LLM or dedicated MT. Grouping: embeddings with clustering, or an LLM. New models in this space ship every few months, so any choice made once would be stale by launch.
The design: a harness first
The anthropologists already had the most valuable thing: original audio, transcripts in the original languages, and English translations they had validated. That became the gold set.
The harness takes a provider and a model, runs the gold set, and appends a column: word error rate for transcripts, translations side by side for human rating, and a notes column for failure modes. Cleaning is judged the same way, by what it does to the error rate downstream. The pipeline itself is containerised stage by stage, so swapping in the winner is a configuration change.
Where it ended
The partnership ended at the design stage, before we built it. What I would keep is the order: agree on the gold set and the metric first, and make the model the part you expect to replace.