Document intelligence — extracting structured information from unstructured or semi-structured documents — was a difficult problem for a long time. Traditional OCR extracted text; humans then interpreted meaning. Templated extraction worked on fixed-format documents but broke when format varied. Modern multimodal models changed the equation: a model that sees both the visual layout and the text of a document simultaneously can handle the ambiguity that makes traditional extraction brittle.
This article covers the architecture and engineering decisions for production document intelligence pipelines: from document ingestion through visual extraction, structured output generation, quality validation, and downstream consumption. The patterns are drawn from client deployments processing invoices, contracts, technical specifications, and medical documents.
Pipeline Architecture
A document intelligence pipeline has five stages: ingestion, preprocessing, extraction, validation, and delivery. Ingestion handles receiving documents in multiple formats — PDF, DOCX, XLSX, scanned images — and normalising them to a common internal representation. Preprocessing handles quality improvement: deskewing scanned images, enhancing contrast for low-quality scans, splitting multi-page documents into processable chunks.
Extraction is the core stage where the multimodal model processes document content and produces structured outputs. The output schema must be defined before the extraction stage — a model asked to "extract all the relevant information" will produce different structures on different documents. A model given an explicit JSON schema target produces consistent, parseable outputs that downstream systems can process reliably.
Validation checks that extraction outputs conform to expected constraints: required fields are present, date formats are valid, numerical values fall within plausible ranges, extracted entity names match reference lists where applicable. Validation failures should not silently pass — they should route to a human review queue where a human either corrects the extraction or flags the document type as a new category requiring extraction model retraining.
Document Intelligence Pipeline Stages
- Ingestion: format normalisation, de-duplication, routing by document type
- Preprocessing: deskew, contrast enhance, chunk large documents
- Extraction: multimodal model inference with explicit output schema
- Validation: schema conformance, business rule checks, confidence thresholds
- Delivery: structured output to downstream system or human review queue
Extraction Strategies for Different Document Types
Fixed-layout documents — invoices from known vendors, standard form types, regulated document formats — are best handled by template-anchored extraction: identify the document type first, then apply a type-specific extraction prompt with few-shot examples drawn from that document type. This two-stage approach (classify then extract) produces higher accuracy than a single general-purpose extraction prompt, because the extraction prompt can be tuned specifically for the layout and vocabulary of each document type.
Variable-layout documents — contracts, correspondence, technical reports — require a different approach. A general-purpose multimodal model (GPT-4V, Claude's vision capability, or Gemini Vision) with a well-specified output schema handles the full layout variation space better than templated approaches. Provide the model with the document image and an explicit JSON schema of the information to extract, with field descriptions that explain what each field means in business terms, not just data type.
Table extraction deserves special attention. Tables in PDFs frequently survive conversion as text without spatial relationship information — the column headers are separated from the data by the linearisation of the PDF text stream. Multimodal models that see the document as an image preserve the visual table structure, making table extraction significantly more reliable than text-only approaches. For high-volume table extraction, consider fine-tuning a smaller multimodal model specifically on your document types rather than relying on general-purpose models.
Quality and Validation
Confidence-based routing is the most effective quality architecture for document intelligence pipelines. The extraction model provides a confidence score for each extracted field. High-confidence extractions route directly to the downstream system. Low-confidence extractions route to a human review queue. Set the confidence threshold based on the cost of a downstream error — a mismatch in a payment amount has a higher cost than a mismatch in a reference number, so the thresholds should reflect the risk profile of each field.
Build a feedback loop from human review back to the extraction model. When a human reviewer corrects an extraction, that correction — the document, the incorrect extraction, and the correct answer — is a training example for improving the model. Over time, a well-maintained feedback pipeline reduces the human review queue as the model learns from corrections and improves on document types it previously handled poorly.
Audit trails are mandatory in regulated document processing. Every extraction must be logged with: the source document, the model version that performed the extraction, the extracted output, any human corrections, and the final output delivered downstream. The extraction record must be retained for the duration of any applicable regulatory requirement. For financial documents, this is typically seven years.
Scaling Document Processing
Document processing is naturally batch-workload shaped: documents arrive in bursts (end-of-month invoice batches, overnight scan uploads, bulk import from legacy systems) and need to be processed within a time window without blocking interactive application use cases. Queue-based architecture — SQS, RabbitMQ, or Celery — separates document arrival from document processing and handles burst without overwhelming the inference backend.
Multi-page document processing creates a choice: process the whole document as a context-window-limited prompt, or split into pages and aggregate results. The optimal choice depends on document structure. Documents where each page is independent (an invoice batch, a report with independent sections) benefit from page-level processing that can be parallelised. Documents where context spans pages (contracts with cross-references, reports with summary pages) require whole-document processing or a multi-pass approach.
Caching is valuable for document intelligence at scale. Documents that are re-submitted (a common pattern when extraction fails and the document is re-queued) should not trigger a new inference call if the document has not changed. Content-hash-based caching with a TTL matched to the expected resubmission window eliminates redundant inference compute for repeated documents.
Production Patterns by Document Type
Invoice processing is the most common document intelligence deployment and the most directly measurable. The extraction targets are well-defined (vendor, date, line items, amounts, tax, total, payment terms), the downstream system is typically an ERP or accounts payable platform, and the cost of an extraction error is quantifiable. The business case for automation is straightforward when processing volumes exceed a few hundred invoices per month.
Contract intelligence has a different profile: the extraction targets are more complex (parties, obligations, dates, termination conditions, special terms), the documents are highly variable in structure, and the cost of a missed term can be significant. Contract intelligence pipelines typically operate at lower automation rates — extracting the structured metadata while leaving the full contract text for human review — with automation focused on reducing search and triage time rather than replacing human review entirely.
Technical specification extraction — BOM tables, component parameters, dimensional data from engineering drawings — is a growing use case for manufacturing and procurement teams. The challenge is vocabulary: technical documents use domain-specific terminology and abbreviated representations that general-purpose models handle poorly. Fine-tuning on domain-specific examples or using retrieval-augmented approaches that give the model access to reference databases significantly improves accuracy in technical document processing.
