Ingest: Structured Vision Extraction from EOB Screenshots

ingest() in lib/pipeline/ingest.ts accepts a PNG image as a Uint8Array and calls GPT-5.6 Sol via the OpenAI Responses API to extract structured remittance data. The image is base64-encoded and sent as an input_image content block with detail: "original". The model response is parsed and validated against IngestOutputSchema before the function returns. To avoid redundant API calls during development and testing, results are cached to disk by a SHA-256 content hash combined with the active schema version.

Function signature

export async function ingest(
  imageBytes: Uint8Array,
  dependencies: IngestDependencies = {},
): Promise<IngestOutput>

imageBytes is the raw binary content of the PNG file — the same bytes produced by file.arrayBuffer() in the route handler. dependencies is an optional bag of injectable collaborators; when omitted, the function uses the real filesystem cache and the live OpenAI vision client.

Caching

Before making a model call, ingest() computes a cache key by SHA-256 hashing the image bytes and prepending the schema version constant "v2":

const CACHE_SCHEMA_VERSION = "v2";

function cacheKey(imageBytes: Uint8Array): string {
  const imageHash = createHash("sha256").update(imageBytes).digest("hex");
  return `${CACHE_SCHEMA_VERSION}-${imageHash}`;
}

The key is used to look up a JSON file under samples/parsed/. On a cache hit the file is parsed through IngestOutputSchema and returned immediately. On a cache miss the model is called, the result is written to disk, and the validated output is returned. The v2 prefix ensures that a schema change automatically invalidates all existing cache entries without touching the files.

Output schema

IngestOutput is z.array(ClaimSchema).min(1) — an array with at least one Claim object per claim shown on the remittance. Each Claim has three top-level fields: remittance — header fields shared across every claim on the same remittance:

Field	Type	Description
`payer`	`string`	Payer name as printed
`provider`	`string`	Provider name as printed
`provider_npi`	`string`	National Provider Identifier
`notice_date`	`string` (ISO 8601)	Date the remittance was issued
`payment_date`	`string` (ISO 8601)	Date payment was made
`eft_trace`	`string`	EFT trace number
`code_legend`	`CodeLegendEntry[]`	Every CARC and RARC entry from the printed legend

claim — claim-level identifiers:

Field	Type	Description
`claim_id`	`string`	Payer claim identifier
`patient_name`	`string`	Patient name as printed
`service_date`	`string` (ISO 8601)	Date of service
`claim_received_date`	`string` (ISO 8601)	Date claim was received

lines — one entry per service line (ServiceLine[], minimum one):

Field	Type	Description
`line_number`	`number`	Positive integer line identifier
`service_date`	`string` (ISO 8601)	Service date for this line
`procedure_code`	`string`	CPT or HCPCS code
`description`	`string`	Service description
`billed_amount`	`number`	Amount billed (nonnegative)
`allowed_amount`	`number`	Amount allowed (nonnegative)
`paid_amount`	`number`	Amount paid (nonnegative)
`patient_responsibility`	`number`	Patient’s share (nonnegative)
`group_code`	`string`	Claim adjustment group code
`carc`	`string`	Claim Adjustment Reason Code
`rarc`	`string \| null`	Remittance Advice Remark Code, or `null` if absent
`notes`	`string \| null`	Claim remark or line-level note, or `null` if absent

Dependency injection

export interface IngestCache {
  read(key: string): Promise<unknown | null>;
  write(key: string, value: IngestOutput): Promise<void>;
}

export interface IngestVisionClient {
  extract(imageBytes: Uint8Array, prompt: string): Promise<IngestResponse>;
}

export interface IngestDependencies {
  cache?: IngestCache;
  visionClient?: IngestVisionClient;
}

Replacing cache with an in-memory implementation lets tests skip filesystem I/O entirely. Replacing visionClient with a fixture that returns a pre-built response lets tests run without an OpenAI API key. Both substitutions are independent — you can inject only the one you need.

Prompt design

The ingest prompt at prompts/ingest.md gives the model a strictly bounded transcription role. The key constraints are:

Transcribe only — copy values visible on the page; never infer, normalize, or complete a code from context or model knowledge.
Exact code strings — CARC and RARC are strings, not numbers; preserve leading zeroes and alphanumeric codes exactly as printed.
Verbatim legend definitions — transcribe every code_legend entry verbatim from the printed legend; never use the model’s internal knowledge of what a code means.
Always emit rarc and notes — use null when the page provides no value for either field.
ISO 8601 dates — convert page dates only when all date components are visible.
Nonnegative amounts — no currency symbols or thousand separators.

The ingest prompt explicitly forbids inferring missing values. If a required schema value is absent or illegible on the image, the model must omit the affected claim or line entirely rather than guessing. Similarly, if a printed legend entry or definition is illegible, extraction fails rather than completing it from the model’s memory. This means IngestOutput reflects only what is actually visible on the remittance.

The reasoning effort level is "high", set in the responseWithImageSchema() call inside ingest.ts. This gives the model maximum deliberation time to transcribe complex multi-claim remittance layouts accurately.

Get Started

Core Concepts

Pipeline

Configuration & Deployment

Ingest: Structured Vision Extraction from EOB Screenshots

Function signature

Caching

Output schema

Dependency injection

Prompt design

​Function signature

​Caching

​Output schema

​Dependency injection

​Prompt design

Function signature

Caching

Output schema

Dependency injection

Prompt design