Prolo — Ingestion Data Flow (Stages 1–4)

Step-by-step journey of a quote: Gmail poll → fetch → classify → extract → persist to RDS Postgres. Ingestion ends when the extracted quote is durable in the pipeline store; matching + submission (stages 5–6) pick up from there. This is the logical flow (Deliverable F2); the AWS services behind it are in aws-architecture.html (F1). Yellow box is the AI-bearing stage. Dotted paths escalate to Human Review. 

flowchart TB
    G[Gmail inboxes] -->|"scheduled poll"| N

    N["1 · NOTIFICATION
poll · list since watermark"]
    F["2 · FETCH
pull body + attachments from Gmail"]
    C["3 · CLASSIFICATION
is this a quote? (rules-first)"]
    X["4 · EXTRACTION + PERSIST (P1)
pull out header + line items → own store"]

    N <-->|"advance watermark"| TRK[("Email tracker
watermark
+ status per messageId")]
    N -->|new messageIds| F
    F <-->|"dedupe (messageId)"| TRK
    F -->|"store raw body + attachments"| S3[("S3 bucket
raw email + files")]
    F -->|"RawMessage (refs)"| C
    C -->|QuoteCandidate| X
    S3 -->|"attachment bytes"| X
    X -->|"ExtractedQuote (P1 write)"| DB[("Pipeline store
(RDS Postgres)")]

    C -->|"mark classified / discarded"| TRK

    DB -.->|"handoff → stages 5–6"| NEXT[RFQ matching]

    C -.discard.-> D[Dropped]
    C -.unsure.-> H
    X -.low confidence.-> H

    H["HUMAN REVIEW
cross-cutting"]

    classDef ai fill:#fde047,stroke:#ca8a04,color:#000
    class X ai

    %% node/zone colours
    style G fill:#f2f0ff,stroke:#8250df,stroke-width:1.5px,color:#1f2328
    style N fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style F fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style C fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style TRK fill:#eef1f4,stroke:#57606a,stroke-width:1.5px,color:#1f2328
    style S3 fill:#eef1f4,stroke:#57606a,stroke-width:1.5px,color:#1f2328
    style DB fill:#eef1f4,stroke:#57606a,stroke-width:1.5px,color:#1f2328
    style H fill:#fdeef0,stroke:#d1242f,stroke-width:1.5px,color:#1f2328
    style D fill:#f6f8fa,stroke:#8c959f,stroke-width:1.5px,color:#57606a
    style NEXT fill:#e6fbf0,stroke:#1a7f37,stroke-width:1.5px,color:#1f2328
Pipeline stage (1–4) Yellow = AI-bearing stage (Extraction) Data store (tracker · S3 · RDS) Human Review (cross-cutting) Solid = data flow · Dotted = escalation / handoff

Stage 4 — Extraction (decomposed sub-pipeline)

How a QuoteCandidate becomes a structured ExtractedQuote durable in RDS Postgres. Extraction is not one LLM call — it's a small sub-pipeline (download → parse (±OCR) → select context → LLM extract → ground → store), with three deliberate design choices: two content sources, one structured call, and added confidence scoring. Yellow = AI-bearing. Diamonds are decisions. 

flowchart TB
    IN["QuoteCandidate
(RawMessage refs)"]
    BODY["email body text
(inline — no parse)"]
    ATT["attachment bytes
(read from S3)"]
    PARSE["PARSE
PyMuPDF text layer"]
    OCR{"scanned / broken /
watermark page?"}
    TESS["Tesseract OCR
fallback"]
    PW{"password
protected?"}
    ASM["ASSEMBLE context
body + parsed text + provenance tags"]
    SEL{"fits token
budget?"}
    CHUNK["CHUNK (table-aware)
+ merge page-by-page"]
    LLM["LLM EXTRACT (Bedrock/Claude)
schema-constrained → SupplierQuote"]
    CONF["CONFIDENCE
score per field"]
    GRND["GROUND / provenance
value → source·page·line"]
    EMBED["EMBED line-item descriptions
Bedrock/local → vector"]
    STORE[("P1 · RDS Postgres
status = extracted")]
    H["HUMAN REVIEW"]

    IN --> BODY
    IN --> ATT
    ATT --> PW
    PW -- yes --> H
    PW -- no --> PARSE --> OCR
    OCR -- yes --> TESS --> ASM
    OCR -- no --> ASM
    BODY --> ASM
    ASM --> SEL
    SEL -- "yes (whole doc)" --> LLM
    SEL -- no --> CHUNK --> LLM
    LLM --> CONF --> GRND --> EMBED --> STORE
    CONF -.low confidence.-> H

    classDef ai fill:#fde047,stroke:#ca8a04,color:#000
    class LLM ai
    class EMBED ai

    %% node colours
    style IN fill:#f2f0ff,stroke:#8250df,stroke-width:1.5px,color:#1f2328
    style BODY fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style ATT fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style PARSE fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style TESS fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style ASM fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style CHUNK fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style CONF fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style GRND fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style OCR fill:#fff8e6,stroke:#bf8700,stroke-width:1.5px,color:#1f2328
    style PW fill:#fff8e6,stroke:#bf8700,stroke-width:1.5px,color:#1f2328
    style SEL fill:#fff8e6,stroke:#bf8700,stroke-width:1.5px,color:#1f2328
    style STORE fill:#eef1f4,stroke:#57606a,stroke-width:1.5px,color:#1f2328
    style H fill:#fdeef0,stroke:#d1242f,stroke-width:1.5px,color:#1f2328
Process step Decision (branch) Yellow = AI-bearing (LLM extract) Data store (RDS) Human Review