Speech to text

What it is

Modern ASR is dominated by one family. Whisper-style encoder-decoders listen to a 30-second window of audio and generate text, which makes them robust to accents and noise but means they can also hallucinate fluent text during silence.

The thing to understand early is that transcription alone is rarely the deliverable. Who spoke which line — diarisation — is a separate model. Timestamps, redaction and downstream classification are separate steps again. Budget for the pipeline, not the transcriber.

Model families 4

Sub-categories within this task. The distinction that matters is not which is best, but which fits your constraints.

Whisper family

39M (tiny) to 1.5B (large-v3).

Use when: The default. Multilingual, robust to accents and noise, handles punctuation and casing.
Avoid when: Real-time streaming — it processes 30-second chunks.
Examples: openai/whisper-large-v3openai/whisper-smalldistil-whisper/distil-large-v3

Distilled Whisper variants

Around half, with most of the quality.

Use when: Batch transcription at volume where cost per hour of audio matters more than the last point of accuracy.
Avoid when: Difficult audio, unusual accents, or low-resource languages.
Examples: distil-whisper/distil-large-v3

CTC models (wav2vec2 family)

95M–1B.

Use when: Streaming or genuinely low latency. Frame-by-frame decoding means partial results as audio arrives.
Avoid when: You need punctuation and casing — these typically produce unpunctuated lowercase output.
Examples: facebook/wav2vec2-large-960h

Managed services

Not applicable.

Use when: You want diarisation, redaction and custom vocabulary without assembling them.
Avoid when: Audio cannot leave your boundary.
Examples: Amazon Transcribe (Call Analytics)

In a bank 4

The third line in each is the part that makes it harder than the textbook version.

Contact centre quality and compliance monitoring

Transcribe calls, then check whether required disclosures were read and prohibited statements avoided.

The wrinkleCall recording obligations under conduct rules mean the audio already exists and is retained. The transcript inherits the same retention and discoverability, so it needs the same controls.

Vulnerable customer and complaint detection

Flag calls where a customer expresses distress or where the conversation should have been escalated.

The wrinkleTranscription errors concentrate exactly where emotion is high — raised voices, crying, interruptions. The audio that matters most transcribes worst.

Trader voice surveillance

Monitor recorded lines for market abuse indicators.

The wrinkleTrading floor audio is noisy, fast, jargon-dense and often multilingual within a single call. Custom vocabulary and domain adaptation are not optional here.

Meeting and advice-session records

Transcribe advisory meetings into the client file as a record of what was discussed.

The wrinkleDiarisation accuracy determines whether the record is usable — 'the adviser said' versus 'the client said' is the entire evidential value.

Code 2

Named functions and the arguments that actually change the outcome. Comments flag the traps.

Transcribe long audio with timestamps

chunk_length_s is what lets Whisper handle files longer than its 30-second window.

import torch
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="openai/whisper-large-v3",
    torch_dtype=torch.float16,
    device="cuda:0",
    chunk_length_s=30,      # required for audio > 30s
    stride_length_s=5,      # overlap, so words on a boundary survive
)

out = asr("call_recording.wav",
          return_timestamps=True,
          generate_kwargs={"language": "en",     # pin it; auto-detect drifts
                           "task": "transcribe"} )  # not "translate"

for seg in out["chunks"]:
    start, end = seg["timestamp"]
    print(f"[{start:7.2f}-{end:7.2f}] {seg['text'].strip()}")

The realistic pipeline: transcribe, redact, classify

Transcription is step one of four. This is what a compliance monitoring job actually looks like.

from transformers import pipeline

asr = pipeline("automatic-speech-recognition",
               model="openai/whisper-large-v3", chunk_length_s=30, device=0)
ner = pipeline("token-classification", model="dslim/bert-base-NER",
               aggregation_strategy="simple")
clf = pipeline("zero-shot-classification",
               model="facebook/bart-large-mnli", device=0)

RISK = ["customer expressed financial hardship",
        "customer disputed a transaction",
        "adviser gave a guarantee about returns",
        "routine service enquiry"]

def process(path):
    text = asr(path, return_timestamps=True)

    # redact before anything leaves the controlled environment
    clean = text["text"]
    for e in ner(clean):
        if e["entity_group"] == "PER":
            clean = clean.replace(e["word"], "[NAME]")

    flags = clf(clean, candidate_labels=RISK, multi_label=True)
    return {
        "transcript": clean,
        "segments": text["chunks"],
        "flags": [(l, s) for l, s in zip(flags["labels"], flags["scores"])
                  if s > 0.5],
    }

What catches people out 4

Whisper hallucinates on silence

Given long silence or non-speech audio it can generate plausible sentences that were never spoken — commonly repeated phrases or subtitle boilerplate from its training data. Run voice activity detection first and skip non-speech regions.
Word error rate hides what matters

A 5% WER sounds excellent until the errors land on account numbers and amounts. Measure error rate separately on the entities you actually care about.
Diarisation is a different model

Whisper gives you words, not speakers. Speaker attribution needs something like pyannote layered on top, and its errors compound with the transcription's.
Pin the language

Auto-detection can switch mid-file on accented speech or code-switching, producing a transcript that quietly becomes a translation. If you know the language, set it explicitly.