Context Engineering on Messy Enterprise Data

Context Engineering Is Bigger Than Prompting

Prompt engineering asks "how do I phrase the instruction." Context engineering asks a broader question: what does the model actually see before it answers, and is that the right information, correctly formatted, free of noise, and scoped to what this specific user is allowed to know? In enterprise deployments, the second question usually matters more. A beautifully worded prompt fed the wrong retrieved chunk still produces a wrong or hallucinated answer.

Chunking Strategy for Inconsistent Documents

Fixed-size chunking — splitting every document into, say, 500-token blocks — is the default starting point in tutorials and the wrong choice for most enterprise document sets, because it routinely cuts a table in half or separates a heading from the paragraph it introduces. A few patterns that hold up better in the field:

• Structure-aware splitting — chunk along headings, list boundaries, and table edges where the document format allows it, rather than at a fixed character count. Most modern document parsers preserve enough structure to do this.
• Overlap on prose, not on tables — a small overlap between adjacent prose chunks helps preserve context across a boundary; overlapping a table just duplicates rows and confuses retrieval scoring.
• Metadata-first for spreadsheets — treat spreadsheet-as-database files as structured data with a query layer, not as prose to chunk. Converting a 10,000-row pricing sheet into text chunks is almost always the wrong move; a SQL-style lookup tool the model can call is usually right.
• Per-source-type pipelines, not one universal pipeline — resist building a single ingestion pipeline for every document type. A PDF policy manual, a Confluence export, and a ticketing system's transcripts each deserve their own chunking logic, even if it means more upfront engineering.

When the Parser Isn't Enough: Verification Over Perfection

Some document sets resist clean parsing no matter how much effort goes in — scanned forms, inconsistent legacy exports, documents that mix languages mid-paragraph. Chasing a perfect parser for these is usually a losing bet on time. A more durable approach: accept imperfect chunking and add a cheap verification step downstream — a fast check, using a small model or a rules-based filter, that confirms a retrieved chunk plausibly answers the query before it's passed to the main model. Log every rejection; the log becomes a prioritized list of which document types actually need manual attention.

Retrieval Must Respect Real Access Boundaries

This is the constraint that trips up more enterprise deployments than any technical retrieval-quality problem. If a support agent cannot see legal's internal notes in the source system, the retrieval layer must not surface those notes to that agent through the AI system either — not even summarized, not even indirectly. Building this correctly means the retrieval query itself needs to carry the requesting user's identity and permissions, and the permission check needs to happen at retrieval time, not as an afterthought filter on the final answer.

Building Evaluation Sets From Real Failures, Not Benchmarks

Public benchmarks tell you almost nothing about how a model will perform on this customer's specific documents and questions. The evaluation set that matters is built from the customer's own real query patterns and real documents — ideally starting from the 20–50 examples gathered during discovery (Article 3) and growing continuously as the system runs.

A practical evaluation loop: every time a human reviewer flags an output as wrong, add the input and the correct answer to the evaluation set, tagged with why it failed — missing context, wrong chunk retrieved, correct context but poor reasoning, or an ambiguous question with no single right answer. Over a few weeks this produces a evaluation set that's directly representative of the failure modes that actually occur, which is far more valuable than any generic benchmark for deciding whether a change to the pipeline is actually an improvement.

Iterating Like an Engineer, Not Like a Prompt Tinkerer

Treat prompts, chunking parameters, and retrieval configuration as versioned artifacts, same as code. Change one variable at a time, run it against the evaluation set, and record the result before making the next change. It's tempting, under deadline pressure, to make several changes at once and eyeball whether things "feel better" — resist it. The discipline pays for itself the first time a change that looked like an improvement turns out to have quietly regressed a category of query nobody was watching.

Frequently Asked Questions

Is context engineering the same as prompt engineering?
Prompt engineering is a subset of it. Context engineering also covers what data reaches the model in the first place — chunking, retrieval, access filtering, and tool outputs — which in enterprise deployments usually matters more than prompt wording. A perfect prompt fed the wrong context still fails.

How do you handle documents with no consistent structure at all?
Accept that automated chunking will be imperfect and add a cheap verification layer instead of chasing a perfect parser. Have the model or a lightweight check confirm that a retrieved chunk actually addresses the query before it's used, and log the misses so you can see which document types need manual handling.

Should retrieval permissions mirror the source system's permissions exactly?
Yes, and this is non-negotiable in enterprise deployments. If a user cannot see a document in the source system, the retrieval layer must not surface it to them through the AI system either, even indirectly through a summary. Treat this as a security requirement, not a nice-to-have, and test it explicitly.