PDF Ingestion Tutorial

PDF Ingestion Tutorial

Product: v0.19.0 · Contract: openapi.snapshot.json · Spec ops: Ingestion cancel & fairness

EdgeQuake converts PDFs to Markdown (vision LLM or EdgeParse), then ingests the result into the knowledge graph. This tutorial covers upload, progress, cancel, and query using the lawful API (OpenAPI snapshot is SSOT).

Prerequisites: EdgeQuake running (API :8080, WebUI :3000). See Quick Start.

Time: ~20 minutes

---

Endpoint cheat sheet

Goal	Method	Endpoint	Body
Upload PDF (preferred)	POST	/api/v1/documents/pdf	multipart/form-data
Upload any file (incl. PDF)	POST	/api/v1/documents/upload	multipart/form-data
Upload plain text	POST	/api/v1/documents	application/json only
Poll progress	GET	/api/v1/documents/pdf/progress/{task_id}	—
Cancel	POST	/api/v1/tasks/{task_id}/cancel	—
Query	POST	/api/v1/query	JSON

Do not send multipart/form-data to POST /api/v1/documents — that route accepts JSON text content only.

---

Two phases: convert ≠ ingest (SPEC-057)

PDF admission enqueues convert (TaskType::PdfProcessing). After durable markdown is stored and the PDF row is Completed, the worker enqueues a separate ingest task (TaskType::Insert).

┌───────────────────────────────────────────────────────┐
│ Convert then ingest (SPEC-057)                        │
│                                                       │
│  POST /documents/pdf  -->  admit task_id              │
│              |                                        │
│              v                                        │
│  [1] PdfProcessing (convert only)                     │
│      vision / edgeparse --> markdown                  │
│      PDF row --> Completed (artifact)                 │
│              |                                        │
│              v  markdown barrier                      │
│  [2] Insert (KG ingest, new lease)                    │
│      chunk --> extract --> embed --> store            │
│              |                                        │
│              v                                        │
│  document display_status = completed                  │
└───────────────────────────────────────────────────────┘

• PDF Completed means convert finished — the document may still be extracting or embedding.
• Terminal success for querying: document display_status = completed (not indexed).
• Cancel during convert or ingest cancels both linked tasks for the same pdf_id. See Ingestion cancel & fairness.

---

Step 1: Upload a PDF

Default (vision backend)

curl -X POST http://localhost:8080/api/v1/documents/pdf \
  -H "X-Workspace-ID: default" \
  -F "file=@/path/to/paper.pdf" \
  -F "title=Research Paper"

Equivalent generic upload endpoint:

curl -X POST http://localhost:8080/api/v1/documents/upload \
  -H "X-Workspace-ID: default" \
  -F "file=@/path/to/paper.pdf" \
  -F "title=Research Paper"

Response (fields that matter):

{
  "pdf_id": "abc-123",
  "document_id": null,
  "status": "processing",
  "task_id": "pdf-550e8400-e29b-41d4-a716-446655440000",
  "track_id": null,
  "message": "PDF uploaded and processing started",
  "estimated_time_seconds": 120
}

Field	Use
task_id	Authoritative progress/cancel identity — subscribe and cancel with this
track_id	Optional client correlation echo only — not the progress-store key
pdf_id	PDF row; use for content/cancel-via-PDF routes
document_id	Populated after ingest creates the doc row

---

Step 2: Parser backend (pdf_parser_backend)

Runtime backends (see PdfParserBackend in edgequake-pdf):

Value	Behavior
vision (default)	Render pages → vision LLM markdown (EDGEQUAKE_VISION_PROVIDER / EDGEQUAKE_VISION_MODEL)
edgeparse	CPU EdgeParse fallback when vision is unavailable or for cost control

Per-upload override:

curl -X POST http://localhost:8080/api/v1/documents/pdf \
  -H "X-Workspace-ID: default" \
  -F "file=@scanned.pdf" \
  -F "title=Scanned Book" \
  -F "pdf_parser_backend=vision" \
  -F "enable_vision=true" \
  -F "vision_provider=ollama" \
  -F "vision_model=gemma4:latest"

Global default: EDGEQUAKE_PDF_PARSER_BACKEND=vision|edgeparse.

Vision provider resolution chain: per-request fields → EDGEQUAKE_VISION_* env → LLM defaults. Mismatch diagnostics: GET /api/v1/config/effective. Details: FAQ — vision configuration.

---

Step 3: Track progress

HTTP poll

TASK_ID="pdf-550e8400-e29b-41d4-a716-446655440000"

curl -s "http://localhost:8080/api/v1/documents/pdf/progress/${TASK_ID}" \
  -H "X-Workspace-ID: default" | jq .

SSE variant: GET /api/v1/documents/pdf/progress/stream/{task_id}.

WebSocket

Connect to the pipeline WebSocket (see OpenAPI / Pipeline Progress). Use task_id from the upload response — not a client-supplied track_id unless you only need batch correlation.

Document status (UI badges)

List/detail JSON includes SPEC-057 presentation fields:

Field	Meaning
display_status	Badge key: converting, extracting, embedding, completed, failed, cancelled, …
ui_phase	idle | running | stopping | terminal — show Stopping… when stopping

Prefer display_status over re-deriving from raw status / current_stage.

curl -s "http://localhost:8080/api/v1/documents?workspace_id=default" \
  -H "X-Workspace-ID: default" | jq '.documents[] | {id, display_status, ui_phase}'

Ready to query when display_status is completed.

---

Step 4: Cancel (optional)

Canonical cancel:

curl -X POST "http://localhost:8080/api/v1/tasks/${TASK_ID}/cancel" \
  -H "X-Workspace-ID: default"

Also supported: DELETE /api/v1/documents/pdf/{pdf_id}/cancel, WebSocket { "type": "cancel", "track_id": "..." } (uses same SSOT). Cancel is cooperative — expect a short delay until the in-flight LLM/vision call aborts.

Terminal cancel: display_status=cancelled, ui_phase=terminal. PDF cancel maps to PdfProcessingStatus::Cancelled (not Failed).

Full semantics: Ingestion cancel & fairness.

---

Step 5: Query the content

After display_status: completed:

curl -X POST http://localhost:8080/api/v1/query \
  -H "Content-Type: application/json" \
  -H "X-Workspace-ID: default" \
  -d '{
    "query": "What are the key findings?",
    "mode": "hybrid",
    "top_k": 10
  }'

QueryResponse (no top-level chunks / entities):

{
  "answer": "The key findings show that…",
  "sources": [
    {
      "document_id": "doc-uuid",
      "snippet": "The results demonstrate…",
      "score": 0.94,
      "file_path": "Research Paper.pdf"
    }
  ],
  "mode": "hybrid",
  "stats": {
    "total_time_ms": 1200,
    "retrieval_time_ms": 400,
    "generation_time_ms": 800
  }
}

Prefer the official SDK: pip install edgequake-sdk — see Python SDK.

---

Configuration reference (multipart fields)

Field	Type	Description
file	file	Required PDF bytes
title	string	Display title
metadata	JSON string	Custom metadata object
enable_vision	bool	Default true for vision path
vision_provider	string	Override vision LLM provider
vision_model	string	Override vision model
pdf_parser_backend	vision | edgeparse	Parser backend
process_options	string	Multimodal process options tag
force_reindex	bool	Re-process duplicate checksum
track_id	string	Client batch correlation only

Legacy config={"mode":"Vision",…} on /documents/upload may still appear in older examples; v0.19.0 PDF path uses pdf_parser_backend + vision env/per-request fields as SSOT.

---

Troubleshooting

Symptom	Check	Fix
400 on upload to /documents	Wrong content-type	Use /documents/pdf or /documents/upload with multipart
Stuck on converting	Vision provider down	curl http://localhost:11434/api/tags or set pdf_parser_backend=edgeparse
Vision errors / empty markdown	Provider/model mismatch	GET /api/v1/config/effective → Vision area
display_status: failed	Backend logs	/tmp/edgequake-backend.log; re-upload or retry
Cancel shows stopping long	Cooperative abort	Normal — wait for terminal cancelled
Query returns generic answer	Doc not completed	Poll until display_status=completed

More: PDF Processing Deep Dive · Common Issues

---

Next steps

1. Document Ingestion — text upload and pipeline stages
2. Pipeline Progress — WebSocket/SSE details
3. Document Upload Quick Reference — all upload endpoints
4. REST API Reference — full contract