Prolo — AI Strategy (Deliverable F4)

Prompt-engineering over hosted Bedrock Claude — not fine-tuning. Versioned prompt templates · schema-constrained output · deterministic validation on top.

flowchart LR
    T["Load versioned
template (YAML)"]
    I["Interpolate
{{CONTEXT}} + {{OUTPUT_SCHEMA}}"]
    B["Call Bedrock (Claude)
via private VPC endpoint
model per task · tiering"]
    V["Parse JSON
+ validate: schema & math"]
    E["Embed line-items
2nd cheap model → vector"]
    P[("Persist to RDS (P1)
quote + confidence + provenance + vectors")]
    H["HUMAN REVIEW"]

    T --> I --> B --> V
    V -- ok --> E --> P
    V -.fail / low conf.-> H

    classDef ai fill:#fde047,stroke:#ca8a04,color:#000
    class B ai
    class E ai

    style T fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style I fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style V fill:#fff8e6,stroke:#bf8700,stroke-width:1.5px,color:#1f2328
    style P fill:#eef1f4,stroke:#57606a,stroke-width:1.5px,color:#1f2328
    style H fill:#fdeef0,stroke:#d1242f,stroke-width:1.5px,color:#1f2328
    

Prompts as versioned templates

Prompts live in versioned YAML (not code), interpolated per call ({{CONTEXT}} / {{OUTPUT_SCHEMA}}) with an assistant prefill that locks output to JSON. A large quote can't be extracted in one call (a middle chunk has no header/totals), so it's a small family sharing the same principles + schema, differing only in scope:

Fits → one supplier_quote.yaml call. Oversized → the 1 + N parallel pattern: 1 header/totals call on first + last page, plus N line-item calls (one per table-aware chunk) — no data dependency, so they run in parallel; merged in Python with overlap-dedupe, then the math check validates the assembled quote. Whole-doc is the default; chunking is fallback-only (more tokens/min, more merge complexity).

Example A — whole document (fits) → fields/supplier_quote.yaml

prompts/fields/supplier_quote.yaml

prompt: |
  You extract structured data from construction-materials supplier quotes
  (unstructured supplier emails and their PDF attachments).

  EXTRACTION PRINCIPLES
  - Ground every value in text actually present in the document. If a field is
    not stated, return null — a confident null is better than a guessed value.
  - Never invent or "tidy" numbers. Copy prices, quantities, and totals exactly
    as written, preserving currency codes and units of measure.

  FIELDS
  - Header: supplierName, quoteNumber, quoteDate (ISO-8601 YYYY-MM-DD).
  - Line items — one object per row of the pricing table: description,
    quantity (number), unitOfMeasure, unitPrice (number).
  - Totals: tax, shipping, net (numbers).

  CONFIDENCE
  - Rate each field 0.0–1.0 by how directly the document states it: 0.9+ when
    literally stated, 0.6–0.8 when reasonably inferred, <0.5 when weak. 0.0 for null.

  OUTPUT FORMAT — return valid JSON only, no prose:
  {{OUTPUT_SCHEMA}}

  <document>
  {{CONTEXT}}
  </document>

# response prefill: forces the model straight into the JSON object
assistant_prompt: |
  {"supplierName":

{{OUTPUT_SCHEMA}} interpolates the shared contract:

{
  "supplierName": "string|null", "quoteNumber": "string|null", "quoteDate": "YYYY-MM-DD|null",
  "lineItems": [{"description": "string", "quantity": 0, "unitOfMeasure": "string",
                 "unitPrice": 0, "confidence": 0.0}],
  "totals": {"tax": 0, "shipping": 0, "net": 0},
  "fieldConfidence": {"supplierName": 0.0, "quoteNumber": 0.0, "quoteDate": 0.0, "totals": 0.0}
}

Example B — one chunk (fallback) → fields/quote_line_items.yaml

Scoped to a fragment: same grounding rules, but asks only for the rows it can see and expects no header/totals. One call per chunk; results are merged in code.

prompts/fields/quote_line_items.yaml

prompt: |
  You extract pricing-table LINE ITEMS from ONE FRAGMENT of a construction-
  materials supplier quote. This is a partial view — header and totals may be
  absent here, and that is expected. Do not invent them.

  PRINCIPLES
  - Ground every value in text present in THIS fragment; copy quantities and
    prices exactly, never tidy or infer.
  - Extract ONLY complete rows. Skip any row cut off at the top/bottom edge —
    the overlapping fragment will capture it (dedup happens on merge).

  FIELDS — one object per pricing row:
    description, quantity (number), unitOfMeasure, unitPrice (number)

  OUTPUT FORMAT — valid JSON only:
  {"lineItems": [{"description": "string", "quantity": 0,
                  "unitOfMeasure": "string", "unitPrice": 0, "confidence": 0.0}]}

  <fragment>
  {{CONTEXT}}
  </fragment>
assistant_prompt: |
  {"lineItems":

(Header + totals come from a single quote_header_totals.yaml call on the first/last page. The classifier reuses the same machinery — classifier/is_quote.yaml returning {"isQuote", "confidence", "reason"} — as the Stage-3 AI fallback.)

Schema Design — Supplier Quote Object

The brief requires a SupplierQuoteObject that must include: vendor identifier, financial totals (tax / shipping / net), and an array of line items (description, quantity, unit of measure, price). The schema maps 1:1 to those three requirements — nothing more in the core contract.

{
  "$id": "…/supplier-quote.json",
  "title": "SupplierQuoteObject",
  "type": "object",
  "required": ["vendorIdentifier", "totals", "lineItems"],
  "additionalProperties": false,
  "properties": {

    "vendorIdentifier": { "type": "string" },   // ← Vendor identifier

    "totals": {                              // ← Financial totals
      "type": "object",
      "required": ["tax", "shipping", "net"],
      "properties": {
        "tax":      { "type": "number", "minimum": 0 },
        "shipping": { "type": "number", "minimum": 0 },
        "net":      { "type": "number", "minimum": 0 }
      }
    },

    "lineItems": {                           // ← Array of Line Items
      "type": "array", "minItems": 1,
      "items": {
        "type": "object",
        "required": ["description", "quantity", "unitOfMeasure", "price"],
        "properties": {
          "description":   { "type": "string" },
          "quantity":      { "type": "number", "exclusiveMinimum": 0 },
          "unitOfMeasure": { "type": "string" },
          "price":         { "type": "number", "minimum": 0 }
        }
      }
    }
  }
}

{
  "vendorIdentifier": "Acme Steel Co.",
  "totals": { "tax": 56.00, "shipping": 40.00, "net": 796.00 },
  "lineItems": [
    { "description": "Rebar 12mm", "quantity": 200,
      "unitOfMeasure": "m", "price": 3.50 }
  ]
}

Embeddings for matching — a second, small model

At persist, a separate embedding model (Bedrock Titan / Cohere — not the extraction LLM) turns each line-item description into a vector, so Stage-5 RFQ matching is a fast semantic kNN (pgvector in RDS).

Evaluation — how do we know it's good?

Offline, engineer-run — a different question from the runtime confidence score: "is the extractor any good overall, and did my last change help or hurt?" A human builds an answer key once; deterministic text-matching grades the extractor's output — no AI judge, no human watching live.

flowchart LR
    DOCS["~40 real documents"]
    HUMAN["human labels them once
(the answer key)"]
    LABELS[("labels
correct answers")]
    RUN["run the real extractor
on the same docs"]
    PRED[("predictions
extractor's answers")]
    EVAL["metric evaluation"]

    DOCS --> HUMAN --> LABELS
    DOCS --> RUN --> PRED
    LABELS --> EVAL
    PRED --> EVAL

    style DOCS fill:#f2f0ff,stroke:#8250df,stroke-width:1.5px,color:#1f2328
    style HUMAN fill:#fdeef0,stroke:#d1242f,stroke-width:1.5px,color:#1f2328
    style RUN fill:#eef4ff,stroke:#3b7ddd,stroke-width:1.5px,color:#1f2328
    style EVAL fill:#fff8e6,stroke:#bf8700,stroke-width:1.5px,color:#1f2328
    style LABELS fill:#eef1f4,stroke:#57606a,stroke-width:1.5px,color:#1f2328
    style PRED fill:#eef1f4,stroke:#57606a,stroke-width:1.5px,color:#1f2328