Document QA

What it is

A statement, invoice or form is not a flat string. Its meaning depends on layout: the number to the right of 'Total Due' is the total due, and a column header applies to everything under it. Flatten it to text and that structure is destroyed.

Document QA models take word positions as an input alongside the words. And because most such documents arrive as scans, an OCR step sits in front — which is usually where the real error budget goes. Measure it separately before blaming the model.

Model families 4

Sub-categories within this task. The distinction that matters is not which is best, but which fits your constraints.

OCR then text QA

OCR engine plus 66M–350M.

Use when: Simple single-column documents where reading order is unambiguous.
Avoid when: Tables, multi-column layouts or forms — reading order breaks and the answer becomes unfindable.
Examples: Tesseract / Textract + roberta-base-squad2

Layout-aware models

125M–400M.

Use when: Forms and structured documents. The model sees the bounding box of each token, so spatial relationships survive.
Avoid when: You have no reliable OCR with coordinates — these need positions, not just text.
Examples: microsoft/layoutlmv3-baseimpira/layoutlm-document-qa

OCR-free end-to-end

200M–1B.

Use when: OCR is the bottleneck. These read the image pixels directly and output structured text.
Avoid when: You need word-level provenance back — there are no coordinates to return.
Examples: naver-clova-ix/donut-base-finetuned-docvqa

Vision language models

2B–90B.

Use when: Complex, varied documents where you want to ask arbitrary questions rather than extract fixed fields.
Avoid when: Volume or cost matters, or you need exact provenance.
Examples: Qwen2-VLLlama 3.2 Vision

In a bank 4

The third line in each is the part that makes it harder than the textbook version.

Mortgage and loan document processing

Extract income, employment and asset details from payslips, statements and tax documents.

The wrinkleEnormous format variety — every employer's payslip differs. Fixed templates do not scale; layout-aware models generalise far better, but need a confidence threshold routing uncertain extractions to a human.

Trade finance document checking

Check bills of lading, invoices and letters of credit for consistency against each other.

The wrinkleThe task is cross-document comparison, not single-document extraction. Extract to a structured schema first, then compare deterministically — do not ask a model to do both.

KYC and onboarding evidence

Read identity documents, proof of address and incorporation papers.

The wrinklePhotographs taken on phones, at angles, in poor light. Image quality pre-checks reject more cases than the model ever will, and rejecting early is better than extracting wrongly.

Legacy correspondence digitisation

Make decades of scanned customer letters searchable.

The wrinkleHandwriting and degraded scans. Set expectations that a meaningful fraction will not be readable, and design the search to degrade gracefully.

Code 2

Named functions and the arguments that actually change the outcome. Comments flag the traps.

Ask questions of a scanned document

The pipeline runs OCR internally if you have tesseract installed, and returns the coordinates of the answer.

from transformers import pipeline
from PIL import Image

doc_qa = pipeline("document-question-answering",
                  model="impira/layoutlm-document-qa")

image = Image.open("statement.png")

for q in ["What is the total amount due?",
          "What is the statement date?",
          "What is the account number?"]:
    r = doc_qa(image=image, question=q)[0]
    print(f"{q:32} -> {r['answer']:20} ({r['score']:.3f})")
    # word_ids / bounding boxes give you provenance on the page,
    # which is what makes the extraction auditable

Production shape: OCR with coordinates, then extract

Separating OCR from extraction lets you measure each. When accuracy drops you will know which half moved.

import boto3

textract = boto3.client("textract")

def ocr_with_layout(png_bytes):
    """Textract returns text WITH geometry, which layout models need."""
    r = textract.analyze_document(
        Document={"Bytes": png_bytes},
        FeatureTypes=["FORMS", "TABLES"],
    )
    words, boxes = [], []
    for b in r["Blocks"]:
        if b["BlockType"] == "WORD":
            g = b["Geometry"]["BoundingBox"]
            words.append(b["Text"])
            boxes.append([int(g["Left"] * 1000), int(g["Top"] * 1000),
                          int((g["Left"] + g["Width"]) * 1000),
                          int((g["Top"] + g["Height"]) * 1000)])
    return words, boxes

# Route on confidence rather than accepting everything
def extract(image_bytes, questions, floor=0.80):
    out, review = {}, []
    for q in questions:
        r = doc_qa(image=image, question=q)[0]
        if r["score"] >= floor:
            out[q] = r["answer"]
        else:
            review.append({"question": q, "guess": r["answer"],
                           "score": r["score"]})
    return out, review

What catches people out 3

Measure OCR separately

When extraction accuracy is poor, it is usually OCR, not the model. Keep a small set of pages with hand-typed ground truth text so you can attribute failures correctly.
Tables remain genuinely hard

Merged cells, multi-row headers and continuation across pages defeat most general models. Dedicated table extraction, then querying the extracted structure, works better than asking a document model directly.
Confidence thresholds are the operating control

Full automation on documents this varied is rarely achievable. Design for a review queue from the start and treat the automation rate as the metric — not accuracy on the cases you chose to automate.