Case Study 6: PDF Invoice Data Extraction Pipeline

Scenario

The finance department receives invoices from multiple vendors in PDF format. Information must be extracted and transformed into structured data for downstream systems. Your task is to automate invoice processing and generate validated structured output.

Suggested Libraries

• pdfplumber
• pydantic
• re
• json

Sample Files You May Create

• invoices/ — a folder of vendor invoice PDFs
• invoice_extractor.py — your solution script
• invoices_extracted.json — the structured output

Requirements

• Read PDF invoices.
• Extract key invoice information (vendor, invoice number, date, total amount).
• Validate extracted data against a schema.
• Normalize dates and numeric values.
• Process invoices in batches.
• Record extraction failures rather than letting one bad file kill the run.
• Generate structured JSON output suitable for APIs or databases.

Have a genuine attempt at this before revealing the solution below. Since vendor invoices don't share a common layout, decide up front how much you'll rely on regex versus a more structural approach — and what you'll do when a pattern doesn't match at all.

💡 Show Answer

import pdfplumber
import re
import json
from pathlib import Path
from pydantic import BaseModel, ValidationError
from datetime import date

class Invoice(BaseModel):
    vendor: str
    invoice_number: str
    invoice_date: date
    total_amount: float
    source_file: str

PATTERNS = {
    "invoice_number": re.compile(r"Invoice\s*#?:?\s*([A-Z0-9\-]+)", re.I),
    "total_amount": re.compile(r"Total\s*(?:Due|Amount)?:?\s*\$?([\d,]+\.\d{2})", re.I),
    "invoice_date": re.compile(r"Date:?\s*(\d{1,2}[/-]\d{1,2}[/-]\d{2,4})"),
}

def extract_text(pdf_path):
    with pdfplumber.open(pdf_path) as pdf:
        return "\n".join(page.extract_text() or "" for page in pdf.pages)

def parse_invoice(pdf_path):
    text = extract_text(pdf_path)
    fields = {}
    for key, pattern in PATTERNS.items():
        match = pattern.search(text)
        if not match:
            raise ValueError(f"Could not find {key} in {pdf_path}")
        fields[key] = match.group(1)

    fields["total_amount"] = float(fields["total_amount"].replace(",", ""))
    fields["vendor"] = text.strip().splitlines()[0]  # simplistic: first line as vendor
    fields["source_file"] = str(pdf_path)

    # normalize date formats like 12/31/2024 or 12-31-24
    d, m, y = re.split(r"[/-]", fields["invoice_date"])
    fields["invoice_date"] = date(int(y) if len(y) == 4 else 2000 + int(y), int(m), int(d))

    return Invoice(**fields)

def process_folder(folder):
    results, errors = [], []
    for pdf_file in Path(folder).glob("*.pdf"):
        try:
            results.append(parse_invoice(pdf_file).model_dump(mode="json"))
        except (ValueError, ValidationError) as e:
            errors.append({"file": str(pdf_file), "error": str(e)})
    return results, errors

if __name__ == "__main__":
    results, errors = process_folder("./invoices")
    Path("invoices_extracted.json").write_text(json.dumps(results, indent=2, default=str))
    print(f"Extracted {len(results)} invoices, {len(errors)} failed")

How it scales: Regex-per-vendor breaks down past a handful of templates. In production you'd evolve this into a vendor-specific parser registry (a dict mapping vendor name to parser function) or swap regex for a layout-aware model. Pydantic gives you a fail-fast schema contract before the data hits your database.