Text generation and LLMs

What it is

Mechanically these models do one thing: given a sequence, predict the most likely next token, append it, repeat. Summarising, answering, translating and writing code are all the same operation with different prompts.

Two distinctions matter before you pick one. A base model only continues text and will not follow instructions — it is raw material for fine-tuning. An instruct or chat model has been further trained to follow directions, and is what you want for almost any application. And size buys capability at a steeply rising cost, so the practical question is always the smallest model that clears your quality bar, not the best model available.

Model families 4

Sub-categories within this task. The distinction that matters is not which is best, but which fits your constraints.

Small instruct models (1B–8B)

Single GPU, often a 16GB card.

Use when: Focused tasks with a clear format — classification, extraction, rewriting. Cheap enough to run at volume and small enough to self-host inside your own boundary.
Avoid when: Multi-step reasoning or broad world knowledge.
Examples: Llama 3.x 8B InstructMistral 7B InstructQwen2.5 7B Instruct

Mid-size (13B–70B)

One large GPU to a small multi-GPU node.

Use when: The quality-to-cost sweet spot for most real applications.
Avoid when: Latency-critical paths, or when a smaller model already passes your evaluation.
Examples: Llama 3.x 70B InstructQwen2.5 72B Instruct

Mixture of experts

Large in memory, cheaper per token.

Use when: You want big-model quality with lower inference cost. Only a fraction of the parameters activate per token — but all of them must be resident in memory.
Avoid when: Memory-constrained. The full weight has to load regardless of what activates.
Examples: Mixtral 8x7BQwen MoE variants

Hosted API models

Someone else's problem.

Use when: Best quality with no infrastructure, and you are able to send the data outside your perimeter.
Avoid when: Data residency, confidentiality or vendor concentration rules prevent it — which in a bank is often.
Examples: Claude, GPT, Gemini via API or Bedrock

In a bank 4

The third line in each is the part that makes it harder than the textbook version.

Drafting responses for human review

Complaint responses and customer correspondence drafted from the case file, then edited and sent by a person.

The wrinkleThe human-in-the-loop is not a temporary safety measure here — it is the control that makes the whole thing permissible. Design the workflow so the reviewer must actively approve rather than passively accept.

Code and query generation for analysts

Turn a plain-English question into SQL against a governed warehouse.

The wrinkleGenerated SQL must run under the analyst's own permissions, not a service account. Otherwise you have built a very convenient way around your access controls.

Synthetic data for testing

Generate realistic but fictitious customer records and narratives for testing without exposing real data.

The wrinkleVerify that generated records do not accidentally match real customers, and be careful about training models on synthetic data generated from real data — the privacy properties are weaker than they appear.

Knowledge assistants over internal documentation

Grounded question answering across policy, procedure and product material.

The wrinkleRetrieval quality dominates. Also: whatever the assistant can retrieve, every user of it can effectively read, so the index must respect the same entitlements as the source systems.

Code 3

Named functions and the arguments that actually change the outcome. Comments flag the traps.

Local generation with the arguments that matter

Chat templates are not cosmetic. Using the wrong format for a model degrades output badly and silently.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "Qwen/Qwen2.5-7B-Instruct"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL,
    torch_dtype=torch.bfloat16,   # half the memory of fp32, no real quality loss
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You classify banking complaints. "
                                  "Reply with exactly one word."},
    {"role": "user", "content": "I was charged twice for the same payment."},
]

# Each model family has its own special tokens. Let the tokenizer
# build the prompt rather than concatenating strings yourself.
prompt = tok.apply_chat_template(messages, tokenize=False,
                                 add_generation_prompt=True)

inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs,
                     max_new_tokens=16,
                     do_sample=False,      # deterministic - required if you
                                           # need reproducible decisions
                     temperature=None,
                     pad_token_id=tok.eos_token_id)

# Slice off the prompt; otherwise you get it echoed back
print(tok.decode(out[0][inputs["input_ids"].shape[1]:],
                 skip_special_tokens=True))

Forcing structured output you can actually parse

Free text from an LLM is not an interface. Constrain it and validate it.

import json
from pydantic import BaseModel, ValidationError
from typing import Literal

class Triage(BaseModel):
    category: Literal["fees", "fraud", "service", "technical"]
    urgency:  Literal["low", "medium", "high"]
    rationale: str

SYSTEM = ("Return ONLY a JSON object with keys category, urgency, rationale. "
          "category must be one of: fees, fraud, service, technical. "
          "urgency must be one of: low, medium, high. No prose, no fences.")

def triage(text, retries=2):
    for _ in range(retries + 1):
        raw = generate(SYSTEM, text)
        raw = raw.strip().removeprefix("```json").removesuffix("```").strip()
        try:
            return Triage(**json.loads(raw))
        except (json.JSONDecodeError, ValidationError):
            continue
    return None      # fail closed - route to a human, never guess

Deploy an open model to a SageMaker endpoint

Text Generation Inference is the serving container AWS provides; you are renting a GPU by the hour.

from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

model = HuggingFaceModel(
    image_uri=get_huggingface_llm_image_uri("huggingface", version="2.0.0"),
    env={
        "HF_MODEL_ID": "Qwen/Qwen2.5-7B-Instruct",
        "SM_NUM_GPUS": "1",
        "MAX_INPUT_LENGTH": "4096",
        "MAX_TOTAL_TOKENS": "8192",
    },
    role=role,
)

predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge",       # ~$1.50/hr, billed while it exists
    container_startup_health_check_timeout=600,   # LLMs load slowly
)

print(predictor.predict({"inputs": prompt,
                         "parameters": {"max_new_tokens": 128,
                                        "do_sample": False}}))

predictor.delete_endpoint()   # do not forget this one

What catches people out 4

Memory is roughly 2 bytes per parameter, plus overhead

A 7B model in bfloat16 needs about 14GB for weights before any context. Add the KV cache, which grows with sequence length and batch size. Fine-tuning needs three to four times more again, which is why a model that infers on one card often will not train on it.
Temperature zero is not fully deterministic

Batching and GPU floating-point non-associativity mean identical inputs can still produce different outputs. If you need reproducibility for audit, log the exact output rather than assuming you can regenerate it.
Prompt injection is an unsolved problem

If your prompt includes retrieved documents or customer text, that content can contain instructions the model follows. Never let model output trigger a privileged action without a human or a deterministic check in between.
Benchmark scores do not transfer to your task

Leaderboard performance says little about how a model handles your complaint categories. Build a small evaluation set of your own — a hundred real examples with agreed answers is worth more than any public benchmark.