Enterprise RAG at Fortune Global 500 Scale

The challenge

Enterprise RAG becomes much harder once the system has to work across real company data rather than a small demonstration dataset.

Documents varied substantially in structure and quality: conventional text, long PDFs, scans, complex tables, charts, diagrams, and other visually rich material. The system also operated at meaningful production scale.

• Which retrieval and reranking configurations actually surface the right evidence?
• Does search quality remain strong across different languages?
• How much does document extraction quality affect downstream retrieval?
• Can tables be reconstructed accurately enough to answer questions about them?
• How should charts, diagrams, and other visual content become searchable?
• When one component changes, does the overall user experience actually improve?
• What accuracy, cost, latency, and operational trade-offs are acceptable in production?

The system

The platform is an agentic enterprise RAG environment where employees can upload or access large multilingual document collections and ask questions over company knowledge.

Because errors early in the chain propagate downstream, improving the final answer requires evaluating individual components as well as the complete system.

Workstream 1 — Building an evaluation framework for RAG

Representative evaluation datasets and experiment pipelines were created to compare retrieval and document-processing configurations.

Retrieval metrics such as Hit@K, Recall@K, MRR, and nDCG reveal whether the necessary evidence was found and whether the most useful evidence ranked early enough.

Generated answers were evaluated for correctness, completeness, faithfulness to retrieved evidence, citation quality, and behavior when information was missing or ambiguous.

Workstream 2 — Comparing multilingual retrieval components

Different retrieval approaches, rerankers, and multilingual embedding configurations were evaluated using representative enterprise data across seven languages.

The relevant question was whether a component performed well on the document types, languages, and queries employees actually use—not whether it ranked highly on a public leaderboard.

Workstream 3 — Improving document intelligence

RAG quality depends heavily on what enters the index. If parsing destroys a table, misses important text, or ignores a diagram, even a strong retrieval model cannot recover the missing information.

Evaluation included character and word error rates plus structural checks for tables: rows, columns, empty cells, table structure, and downstream usability.

GLM-OCR performed better on the document types and table structures that mattered most while also reducing dependence on an expensive external API.

Workstream 4 — Making visual information searchable

Charts, graphs, diagrams, and other visual elements can contain information that conventional OCR misses or represents poorly.

Relevant visual elements were processed with a vision-language model, producing descriptions that could be added to the structured document representation and made available to retrieval.

How improvements were evaluated

One of the most important principles was isolating changes whenever possible. A component change was judged by downstream behavior, not merely cleaner-looking output.

• Process the same documents with each candidate extraction pipeline
• Index both versions using the same retrieval configuration
• Run the same evaluation queries
• Compare retrieval metrics
• Compare answer correctness, completeness, faithfulness, and citation quality
• Compare cost, latency, and processing failures

Production reality

This was not a small proof of concept. At this scale, reliability, latency, cost, and regression testing become operational concerns—not theoretical ones.

• Hundreds of thousands of documents
• Seven languages
• Hundreds of employees
• Thousands of daily queries

What this case study demonstrates

A production RAG system is not one model or one API call. Reliability depends on the interaction between ingestion, document understanding, retrieval, ranking, generation, and production constraints.

At enterprise scale, the most valuable capability is often the measurement framework that shows which part should be improved next and whether the change actually helped.