Demos are easy
Anyone can produce a compelling LLM demo in an afternoon. The hard part starts when real users arrive with inputs nobody anticipated, and when the feature has to keep working after someone upgrades the model.
Build the evaluation set before the feature
Collect fifty to two hundred realistic inputs with agreed-good outputs. Not synthetic examples - real questions from real support tickets or documents. This set becomes your regression test. Every prompt change, model upgrade or retrieval tweak runs against it, and you get a number instead of an opinion.
Without this, teams end up changing prompts based on the last complaint they heard, with no way to know what they broke.
Retrieval quality beats model choice
For document-grounded features, most failures trace to retrieval, not generation. The model answered faithfully from the wrong chunk. Invest in chunking strategy, metadata filtering and reranking before reaching for a larger model - it is cheaper and the improvement is usually bigger.
Guardrails at the boundary
Validate structure on the way out. If the feature must return JSON matching a schema, enforce it and retry on mismatch rather than shipping malformed data downstream. Constrain what the feature is allowed to say, and give it an explicit way to decline: a model that answers "I don't have that information" is far more useful than one that invents an answer.
Watch cost and latency like any other budget
Set a per-request cost ceiling early. Cache aggressively - identical or near-identical requests are more common than teams expect. Route simple requests to smaller models and reserve the expensive one for cases that need it. We routinely cut inference spend by half with no measurable quality change.
Instrument everything
Log inputs, outputs, retrieved context, latency and cost for every call, with a sampling policy that respects privacy. When a customer reports a bad answer three weeks later, you want to be able to reconstruct exactly what happened.