Troubleshooting Guide

Troubleshooting Guide

Product: v0.19.0 · Ingestion SSOT: Ingestion cancel & fairness

Diagnosing and Resolving Common EdgeQuake Issues

This guide helps you identify and fix common problems when running EdgeQuake.

---

Quick Diagnostics

Health Check

# Check basic health
curl http://localhost:8080/health

# Check readiness with dependencies
curl http://localhost:8080/ready

# Check if backend is responding
curl -I http://localhost:8080/api/v1/workspaces

Service Status

# Check all services (if using make)
make status

# Check PostgreSQL reachability
pg_isready -h localhost -p 5432

# Check Ollama
curl http://localhost:11434/api/tags

---

Common Issues

1. Document Upload Errors

Symptom: “Expected request with Content-Type: application/json”

Cause: Using multipart form data (-F "file=@...") with the wrong endpoint.

Solution: EdgeQuake has two different endpoints for document upload:

Option A - Upload Text as JSON (/api/v1/documents):

curl -X POST http://localhost:8080/api/v1/documents \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Your text content here...",
    "title": "Document Title"
  }'

Option B - Upload Files (PDF, TXT, MD, etc.) (/api/v1/documents/upload):

curl -X POST http://localhost:8080/api/v1/documents/upload \
  -F "file=@your-document.pdf" \
  -F "title=My Document"

Symptom: “Failed to parse the request body as JSON”

Cause: Using -F flag (multipart) with a JSON endpoint, or mixing content types.

Solution: Choose the correct endpoint and format:

Upload Type	Endpoint	Content-Type	Format
Text/JSON	/api/v1/documents	application/json	-d '{...}'
Files	/api/v1/documents/upload	multipart/form-data	-F "file=@"
Batch Files	/api/v1/documents/upload/batch	multipart/form-data	-F "files=@"

Examples:

# ❌ WRONG - multipart to JSON endpoint
curl -X POST http://localhost:8080/api/v1/documents \
  -F "file=@doc.pdf"

# ✅ CORRECT - multipart to upload endpoint
curl -X POST http://localhost:8080/api/v1/documents/upload \
  -F "file=@doc.pdf"

# ✅ CORRECT - JSON to documents endpoint
curl -X POST http://localhost:8080/api/v1/documents \
  -H "Content-Type: application/json" \
  -d '{"content": "Text here", "title": "My Doc"}'

---

2. Server Won’t Start

Symptom: “Address already in use”

Cause: Port 8080 is already in use.

Solution:

# Find what's using port 8080
lsof -i :8080

# Kill the process
kill -9 <PID>

# Or use different port
PORT=9090 cargo run

Symptom: “DATABASE_URL is not valid” or a startup log says PostgreSQL storage failed to initialize

Cause: Invalid PostgreSQL connection string, unreachable database, or a database daemon that is not running.

Solution:

# Check format
DATABASE_URL="postgresql://user:password@host:port/database"

# Test connection directly
psql "$DATABASE_URL" -c "SELECT 1"

# Or use a lightweight readiness probe
pg_isready -h localhost -p 5432

Why this matters: EdgeQuake now fails fast and clearly at startup when PostgreSQL is unavailable. This is intentional and prevents flaky half-started server states.

Symptom: OrbStack stops unexpectedly or make dev says Docker daemon is unavailable

Cause: this is usually a host network route conflict, not an EdgeQuake crash. On the affected machine, the route table already claims broad private ranges such as 10/8, 172.16/12, or 192.168/16 through another interface (for example a VPN, Homebridge, or router helper). OrbStack/Docker then cannot create its bridge network and reports errors such as “failed to add network” or “conflict with existing route”.

What changed: make dev and make dev-bg now use incremental startup. They reuse healthy services, avoid blind stop/start cycles, and fail with a clear diagnosis instead of hammering Docker repeatedly.

Solution:

# Show the targeted diagnosis
make db-start

# If OrbStack still reports route conflicts:
# 1. stop the conflicting VPN / bridge helper
# 2. restart OrbStack
# 3. rerun the stack
make dev-bg

Symptom: “Extension ‘vector’ not found”

Cause: pgvector extension not installed.

Solution:

-- As superuser in PostgreSQL
CREATE EXTENSION IF NOT EXISTS vector;

Or rebuild Docker container:

docker compose down -v
docker compose up -d

---

3. Document Processing Stuck

Symptom: Documents stay in “processing” status

Diagnosis:

# Queue pressure + store contention (v0.19 SSOT)
curl -s http://localhost:8080/api/v1/pipeline/queue-metrics | jq

# Check pending tasks
curl http://localhost:8080/api/v1/tasks?status=pending

# Check backend logs
docker compose logs -f edgequake

# Or if running locally
tail -f /tmp/edgequake-backend.log

Common Causes:

Cause	Solution
LLM rate limit	Wait and retry
Invalid API key	Check OPENAI_API_KEY
Ollama not running	Start Ollama: ollama serve
Worker crash	Restart backend
Lease expired	See Interrupted / Reprocess below
Tenant fairness park	Normal under local LLM clamp (ingest≤2, lifecycle≤4); check tenant_park_waiters / per-lane waiters in queue-metrics. Deletes use the lifecycle lane so a new PDF should not stay Queued behind deletes.

Solution:

# Restart workers cleanly
make stop
make dev

# Or manually retry document (POST body, not path id)
curl -X POST "http://localhost:8080/api/v1/documents/reprocess" \
  -H "Content-Type: application/json" \
  -d "{\"document_id\":\"$DOC_ID\",\"force\":true,\"mode\":\"full\"}"

Reliability note: delete, reprocess, and recovery flows are workspace-scoped and restart-safe. A deleted document should not be resurrected by stale task recovery after a backend restart.

Full cancel, fairness, lease, and multi-replica semantics: Ingestion cancel & fairness.

---

3.1 Interrupted / Reprocess (v0.19)

Symptom: Document shows Failed with message containing “Interrupted — use Reprocess”

Cause: A task was in Processing when the server restarted or its lease expired. With EDGEQUAKE_STARTUP_AUTO_RESUME=0 (opt-out), stale Processing rows become Failed with an Interrupted message. The code default when the env var is unset is ON (auto-reclaim to Pending).

Solution:

# Reprocess (real route — body carries document_id)
curl -X POST "http://localhost:8080/api/v1/documents/reprocess" \
  -H "Content-Type: application/json" \
  -d "{\"document_id\":\"$DOC_ID\",\"force\":true,\"mode\":\"full\"}"

Default auto-resume (unset env): reclaim stale Processing → Pending on boot. Opt out:

export EDGEQUAKE_STARTUP_AUTO_RESUME=0

Pending tasks always survive restart — workers claim via FOR UPDATE SKIP LOCKED whether or not auto-resume is on.

---

3.2 Lease stuck in Processing

Symptom: Task row stays Processing for longer than expected; no progress updates

Diagnosis:

# Inspect queue depth and pressure
curl -s http://localhost:8080/api/v1/pipeline/queue-metrics | jq '{pending, processing, pressure, store_contention}'

# Check for lease reaper messages
grep -i "lease expired\|heartbeat lost\|Interrupted" /tmp/edgequake-backend.log | tail -20

Cause: Worker died without releasing lease, or LLM call blocked past the lease TTL. Leases refresh every ~60s; TTL defaults to 120s (EDGEQUAKE_TASK_LEASE_TTL_SECS).

Solution:

1. Wait for the lease reaper to mark the task Failed (Interrupted) — then Reprocess
2. Or restart backend to trigger orphan recovery
3. If recurring, reduce ingest concurrency or raise lease TTL for slow local LLMs

Fairness park releases the claim before waiting on tenant capacity — a parked task should not hold a lease indefinitely.

---

3.3 Cancel is not Failed (v0.19)

Symptom: User cancelled ingestion but UI or API still shows Failed instead of Cancelled

Cause (pre-v0.19): Cancel paths sometimes mapped to Failed. v0.19 adds PdfProcessingStatus::Cancelled, display_status=cancelled, and ui_phase=terminal.

Solution:

# Canonical cancel
curl -X POST "http://localhost:8080/api/v1/tasks/$TRACK_ID/cancel"

Check presentation fields (prefer over raw status):

curl -s "http://localhost:8080/api/v1/documents/$DOC_ID" | jq '{display_status, ui_phase, status, failure_class}'

Expected after cancel: display_status=cancelled, ui_phase=terminal (or stopping briefly while cooperative abort completes). Cancelled documents are excluded from failed-count chips in the WebUI.

See Ingestion cancel & fairness — Status SSOT.

---

3.4 Multi-replica boot failure (EDGEQUAKE_REPLICAS>1)

Symptom: Server exits at startup with a task-delivery validation error

Cause: EDGEQUAKE_REPLICAS is set above 1 but EDGEQUAKE_TASK_DELIVERY=local (default). Local delivery is single-process only.

Solution:

export EDGEQUAKE_REPLICAS=2
export EDGEQUAKE_TASK_DELIVERY=bridged   # or notify_only
make dev-bg

Correctness always comes from Postgres claim_next + lease — bridged / notify_only are wake signals only. Never process a task from a channel payload without claiming it in the database.

Details: Multi-replica delivery.

---

3. CI or local runs feel flaky

Symptom: repeated pushes trigger too many redundant CI runs

Cause: multiple in-progress jobs for the same branch create stale signal and wasted runner time.

Solution:

• use the repository workflow concurrency settings so newer runs cancel older ones
• keep heavy coverage and full-E2E flows out of the fastest blocking loop
• prefer readiness probes over fixed sleeps when starting backend and frontend services
• keep the pinned Rust toolchain active locally so clippy and formatting results match CI

---

4. PDF Extraction Issues

PDF extraction can fail or produce poor quality results due to PDF structure, encoding, or layout complexity. This section covers the most common PDF-specific problems.

Issue 3.1: No Text Extracted or Low-Content Warning

Symptom: After PDF upload, chunk_count = 0, chunks are empty, or the UI shows “Low text content - consider using Vision extraction”

Diagnosis:

# Check document details
curl http://localhost:8080/api/v1/documents/doc-uuid

# Response shows:
{
  "chunk_count": 0,
  "metadata": {"pages": 50, "pdf_extraction_method": "edgeparse"}
}

Cause:

• PDF is image-based (scanned document, no embedded text layer)
• edgeparse was used on a scan or image-only PDF

Solution 1 - Retry with Vision backend:

# Re-upload with explicit Vision backend
curl -X POST http://localhost:8080/api/v1/documents/upload \
  -F "file=@scanned_book.pdf" \
  -F "title=Scanned Book" \
  -F "pdf_parser_backend=vision"

Solution 2 - Set Vision as the workspace default for scan-heavy corpora:

curl -X PUT http://localhost:8080/api/v1/workspaces/$WORKSPACE_ID \
  -H "Content-Type: application/json" \
  -d '{"pdf_parser_backend":"vision"}'

When to use EdgeParse:

• Digital-native reports
• Invoices and statements with embedded text
• PDFs where speed and deterministic output matter more than OCR-like robustness

When to use Vision:

• Scanned documents
• Image-heavy PDFs
• Layouts with poor embedded text quality

Cost Warning: Vision mode consumes LLM calls and is slower than EdgeParse.

Verification:

• Check chunk_count > 0 in response
• Check document lineage shows pdf_extraction_method = "vision"
• Download chunks to verify content extracted

Related: See PDF Ingestion Tutorial for backend selection guidance.

---

Issue 3.2: Tables Not Detected or Malformed

Symptom: Tables appear as scrambled text or not detected at all

Before (raw extraction):

Header1 Header2 Header3 Data1a Data1b
Data1c Data2a Data2b Data2c Data3a

Diagnosis:

# Check table detection
curl http://localhost:8080/api/v1/documents/doc-uuid

# Response:
{
  "metadata": {
    "tables_detected": 0  // Should be > 0 if tables exist
  }
}

Cause: Complex table layout (merged cells, nested structures, no clear borders)

Solution 1 - Enable Table Enhancement:

# LLM-based table refinement
curl -X POST http://localhost:8080/api/v1/documents \
  -F "file=@financial_report.pdf" \
  -F "title=Financial Report" \
  -F 'config={"enhance_tables": true, "ai_temperature": 0.1}'

Solution 2 - Combine with Multi-Column Detection:

# For academic papers with tables in columns
curl -X POST http://localhost:8080/api/v1/documents \
  -F "file=@research_paper.pdf" \
  -F 'config={
    "enhance_tables": true,
    "layout": {"detect_columns": true}
  }'

After (enhanced):

| Header 1 | Header 2 | Header 3 |
| -------- | -------- | -------- |
| Data 1a  | Data 1b  | Data 1c  |
| Data 2a  | Data 2b  | Data 2c  |
| Data 3a  | Data 3b  | Data 3c  |

Trade-offs:

• Processing time: 2x slower
• Cost: ~$0.0001 per table
• Accuracy: Significantly improved for complex tables

Limitations:

• Very complex tables (5+ levels of merged cells) may still fail
• Tables without any borders are harder to detect

Verification:

• Check tables_detected > 0 in response
• Inspect chunk content for proper markdown table format
• Verify cell alignments correct

Related: See PDF Processing Deep Dive for algorithm details.

---

Issue 3.3: Wrong Text Order (Multi-Column Layout)

Symptom: Text from different columns interleaved incorrectly

Example Problem:

# PDF has 2 columns:
Column 1: "The experiment showed that X leads to Y..."
Column 2: "In conclusion, we recommend Z..."

# Extracted (wrong order):
"The experiment In showed conclusion, that we X recommend leads Z... to Y..."

Diagnosis:

# Download chunks and check text order
curl http://localhost:8080/api/v1/documents/doc-uuid/chunks

# Look for interleaved text from different sections

Cause: PDF has multi-column layout (academic papers, newspapers, magazines)

Solution - Enable Column Detection:

# Detect and respect column boundaries
curl -X POST http://localhost:8080/api/v1/documents \
  -F "file=@research_paper.pdf" \
  -F "title=Research Paper" \
  -F 'config={
    "layout": {
      "detect_columns": true,
      "column_gap_threshold": 20.0
    }
  }'

Configuration Options:

• detect_columns: Enable multi-column detection (default: true)
• column_gap_threshold: Minimum gap in points between columns (default: 20.0)
  • Increase for wider column gaps
  • Decrease for narrower column gaps

Verification:

• Read first few chunks - text should flow naturally
• Check column boundaries respected
• Verify reading order left-to-right within each column

Tip: Academic papers almost always need column detection enabled.

Related: See PDF Processing Deep Dive for XY-Cut algorithm.

---

Issue 3.4: Encoding Errors (Special Characters)

Symptom: � or ? characters appear instead of actual text

Examples:

• "Caf�" instead of "Café"
• "Na�ve" instead of "Naïve"
• "????" instead of Chinese/Arabic text

Diagnosis:

# Check extracted content for garbled characters
curl http://localhost:8080/api/v1/documents/doc-uuid/chunks | jq -r '.chunks[0].content'

# Look for � or ? characters

Cause: PDF uses custom fonts or non-standard encoding not supported by text extraction

Solution 1 - Retry with the Vision backend:

# LLM vision reads the actual glyphs
curl -X POST http://localhost:8080/api/v1/documents/upload \
  -F "file=@custom_fonts.pdf" \
  -F "pdf_parser_backend=vision"

Solution 2 - Check PDF Font Embedding:

# Use pdffonts to check font embedding (requires poppler-utils)
pdffonts document.pdf

# Look for fonts marked "no" in "emb" column
# These fonts may cause encoding issues

Workaround: If Vision is too expensive, consider:

1. Re-generate PDF with embedded fonts
2. Convert PDF to another format (DOCX) then back to PDF
3. Use OCR tool (Tesseract) to create new PDF with text layer

Verification:

• Check special characters render correctly
• Verify non-English text (if applicable)
• Compare extracted text with PDF visual

Related: Vision mode uses LLM to read actual rendered text, avoiding encoding issues entirely.

---

Issue 3.5: Low Chunk Quality or Empty Chunks

Symptom: Some chunks are very short, empty, or contain garbage

Example:

{
  "chunks": [
    { "content": "Page 1", "token_count": 2 }, // Too short
    { "content": "", "token_count": 0 }, // Empty
    { "content": "||||||||", "token_count": 8 } // Garbage
  ]
}

Diagnosis:

# Count empty or short chunks
curl http://localhost:8080/api/v1/documents/doc-uuid/chunks | \
  jq '[.chunks[] | select(.token_count < 10)] | length'

# If many short chunks → extraction quality issue

Causes:

1. PDF has headers/footers (page numbers, logos)
2. Complex layout confuses chunking
3. Embedded images without captions
4. Poor quality scan

Solution 1 - Enable Readability Enhancement:

# LLM cleans up extracted text
curl -X POST http://localhost:8080/api/v1/documents \
  -F "file=@complex_layout.pdf" \
  -F 'config={"enhance_readability": true}'

Solution 2 - Normalize Spacing:

# Fix concatenated words and spacing issues
curl -X POST http://localhost:8080/api/v1/documents \
  -F "file=@poor_spacing.pdf" \
  -F 'config={"normalize_spacing": true, "consolidate_headers": true}'

Solution 3 - Adjust Chunking:

# Increase chunk size to merge small fragments
curl -X POST http://localhost:8080/api/v1/documents \
  -F "file=@fragmented.pdf" \
  -F "chunk_size=1024" \
  -F "chunk_overlap=100"

Verification:

• Check average chunk token count (should be 100-500 tokens)
• Verify no empty chunks
• Inspect chunks for coherent content

---

Issue 3.6: Upload Fails or Times Out

Symptom: PDF upload returns 500 error or times out

Common Errors:

{
  "error": "Request timeout",
  "status": 408
}

{
  "error": "File too large",
  "status": 413
}

Diagnosis:

# Check file size
ls -lh document.pdf

# Check backend logs
docker compose logs -f edgequake

# Or local logs
tail -f /tmp/edgequake-backend.log

Common Causes:

Error	Cause	Solution
413 Payload Too Large	File > 50MB	Split PDF or increase limit
408 Timeout	Processing > 60s	Use vision mode or increase timeout
500 Internal Error	Corrupted PDF	Repair with pdftk or ghostscript
500 Memory Error	Large PDF + Vision	Process in batches with max_pages

Solution 1 - Test with Page Limit:

# Process first 10 pages to verify config
curl -X POST http://localhost:8080/api/v1/documents \
  -F "file=@large_report.pdf" \
  -F 'config={"max_pages": 10}'

# If successful, process full document

Solution 2 - Increase Server Timeouts:

# Increase timeout in config (if you control server)
REQUEST_TIMEOUT=300  # 5 minutes

# Or split PDF into smaller files
pdftk large.pdf cat 1-50 output part1.pdf
pdftk large.pdf cat 51-100 output part2.pdf

Solution 3 - Repair Corrupted PDF:

# Using ghostscript to repair
gs -o repaired.pdf -sDEVICE=pdfwrite -dPDFSETTINGS=/prepress original.pdf

# Using pdftk
pdftk original.pdf output repaired.pdf

Verification:

• Check upload returns 200 OK
• Verify status: "completed" in response
• Check no error logs in backend

---

Issue 3.7: Vision Extraction Timeout — Provider/Model Mismatch

Symptom: PDF processing fails with:

• “Processing failed permanently after 3 attempts”
• “Circuit breaker tripped after 3 consecutive timeouts”
• “Vision extraction timed out after 480s”
• “Provider ‘ollama’ may be unresponsive”

Root Cause: The vision model does not belong to the vision provider. For example, EDGEQUAKE_VISION_MODEL=gpt-4.1-nano paired with provider ollama will always time out because Ollama cannot serve OpenAI models.

This commonly happens when:

• A Makefile or Docker Compose file sets EDGEQUAKE_VISION_MODEL to an OpenAI model but .env overrides the provider to ollama
• Env vars are inherited from a parent process (e.g. Makefile export)

Diagnosis:

# Check the effective configuration and mismatch status
curl -s http://localhost:8080/api/v1/config/effective | jq '.areas[] | select(.name == "Vision")'

# Expected response when mismatched:
# {
#   "name": "Vision",
#   "has_mismatch": true,
#   "mismatch_description": "Model 'gpt-4.1-nano' is incompatible with provider 'ollama'.\nHow to fix:\n..."
# }

You can also check in the Settings UI → Configuration Explainability panel. Sections with mismatches are auto-expanded and show remediation options.

Fix — choose one:

Option	Command	When to use
A	unset EDGEQUAKE_VISION_MODEL	You want to use the default model for your provider
B	EDGEQUAKE_VISION_PROVIDER=openai + set OPENAI_API_KEY	You want to use the OpenAI model
C	EDGEQUAKE_VISION_MODEL=gemma4:latest	You want to stay on Ollama

Then restart the backend.

Verification:

# Confirm no mismatch
curl -s http://localhost:8080/api/v1/config/effective | jq '.areas[] | select(.name == "Vision") | .has_mismatch'
# Should return: false

# Re-upload the failed PDF
curl -X POST http://localhost:8080/api/v1/documents/upload -F "file=@document.pdf"

Prevention: EdgeQuake now auto-corrects mismatches at runtime — if a model is incompatible with the resolved provider, it is skipped and the next candidate in the resolution chain is used (with a WARN log). The explainability endpoint helps you fix the env vars permanently.

---

PDF Troubleshooting Decision Tree

Use this ASCII decision tree to diagnose PDF issues:

┌───────────────────────────────────────────────────┐
│ PDF troubleshooting decision tree                 │
│                                                   │
│ PDF upload issue?                                 │
│   |                                               │
│   +-- chunk_count = 0                             │
│   |     +-- Retry with pdf_parser_backend=vision  │
│   |     +-- Still 0? Check encrypted/protected PDF│
│   |     +-- Still 0? File GitHub issue with sample│
│   |                                               │
│   +-- Tables malformed                            │
│   |     +-- enhance_tables=true                   │
│   |     +-- vision + enhance_tables               │
│   |     +-- Complex table? known limitation       │
│   |                                               │
│   +-- Text order wrong                            │
│   |     +-- layout.detect_columns=true            │
│   |     +-- adjust column_gap_threshold           │
│   |                                               │
│   +-- Encoding errors                             │
│   |     +-- Retry vision                          │
│   |     +-- Check fonts (pdffonts)                │
│   |                                               │
│   +-- Upload fails / timeout                      │
│   |     +-- Split if >50MB / max_pages=10         │
│   |     +-- Check vision provider mismatch        │
│   |     +-- Repair PDF (ghostscript/pdftk)        │
│   |                                               │
│   +-- Poor quality chunks                         │
│         +-- enhance_readability=true              │
│         +-- normalize_spacing=true                │
│         +-- chunk_size=1024                       │
└───────────────────────────────────────────────────┘

---

PDF Configuration Quick Reference

Common configurations for different PDF types:

Digital PDF (good quality):

# Default settings - no config needed
curl -X POST http://localhost:8080/api/v1/documents/upload -F "file=@digital.pdf" http://localhost:8080/api/v1/documents/upload

Scanned Document:

curl -X POST http://localhost:8080/api/v1/documents/upload -F "file=@scanned.pdf" \
     -F "pdf_parser_backend=vision" \
     http://localhost:8080/api/v1/documents/upload

Academic Paper (multi-column):

curl -X POST http://localhost:8080/api/v1/documents/upload -F "file=@paper.pdf" \
     -F 'config={"layout": {"detect_columns": true}}' \
     http://localhost:8080/api/v1/documents/upload

Financial Report (complex tables):

curl -X POST http://localhost:8080/api/v1/documents/upload -F "file=@financials.pdf" \
     -F 'config={"enhance_tables": true}' \
     http://localhost:8080/api/v1/documents/upload

Unknown Quality:

curl -X POST http://localhost:8080/api/v1/documents/upload -F "file=@unknown.pdf" \
     -F "pdf_parser_backend=edgeparse" \
     http://localhost:8080/api/v1/documents/upload

Critical Document (maximum accuracy):

curl -X POST http://localhost:8080/api/v1/documents/upload -F "file=@critical.pdf" \
     -F 'config={
       "pdf_parser_backend": "vision",
       "enhance_tables": true,
       "enhance_readability": true,
       "vision_dpi": 200
     }' \
     http://localhost:8080/api/v1/documents/upload

---

When to Seek Further Help

If PDF extraction still fails after trying these solutions:

1. Read Full Documentation:

   • PDF Ingestion Tutorial - Configuration details
   • PDF Processing Deep Dive - Algorithm internals

2. Check GitHub Issues:

   • Search existing issues: https://github.com/org/edgequake/issues
   • Look for similar PDF problems
   • Check if limitation is already known

3. File New Issue:

   • Include PDF metadata (pages, file size, type)
   • Include error messages / logs
   • Attach problematic PDF (if not confidential)
   • Describe expected vs actual behavior

4. Community Support:

   • Discord: #pdf-extraction channel
   • Stack Overflow: Tag edgequake pdf

---

3b. Knowledge Graph shows UUID/GUID entity names

Symptom: Organization/Concept nodes labeled like 84b69e27-E38b-444a-…

Cause (067): The extractor treated opaque machine/resource IDs in the document as entity names. Those strings were stored as graph identity. Display was faithful — this was not a UI-only bug. (Drawing im-… labels are a separate case; see improvement 066.)

After upgrade (067+):

• New ingest rejects UUID/GUID/ULID/ObjectId/hex-hash/ARN-shaped names at EntityId normalization.
• Prompts instruct the model not to emit opaque IDs as entity_name.
• Legacy opaque nodes get a soft label (description snippet or Opaque ID · {type}) without re-ingest.

Cleanup for a clean graph:

1. Re-ingest UUID-heavy documents after upgrading, or
2. Manually delete low-value opaque nodes from the Graph UI / API, or
3. Prune nodes whose bare id matches an opaque identifier pattern and have low degree.

See: specs/001-benchmark/001-edgquake-improvements/067-opaque-entity-name-reject.md.

---

4. Empty Query Results

Symptom: Query returns empty answer

Diagnosis:

# Check document count
curl "http://localhost:8080/api/v1/documents?workspace_id=$WORKSPACE_ID"

# Check entity count
curl "http://localhost:8080/api/v1/graph/entities?workspace_id=$WORKSPACE_ID"

# Check chunk count
curl "http://localhost:8080/api/v1/workspaces/$WORKSPACE_ID/stats"

Common Causes:

Symptom	Cause	Solution
0 documents	No uploads	Upload documents
Documents but 0 entities	Processing failed	Reprocess documents
Entities but no results	Query not matching	Try different mode

Debug Query:

# Try naive mode (vector only) to verify basic retrieval
curl -X POST "http://localhost:8080/api/v1/query?workspace_id=$WORKSPACE_ID" \
  -H "Content-Type: application/json" \
  -d '{"query": "test", "mode": "naive", "max_chunks": 20}'

---

5. LLM Errors

Symptom: “OpenAI API error: Rate limit exceeded”

Solution:

# Wait and retry
sleep 60

# Or switch to different provider
export EDGEQUAKE_LLM_PROVIDER=ollama

Symptom: “OpenAI API error: Invalid API key”

Solution:

# Check key is set
echo $OPENAI_API_KEY

# Check key starts with sk-
# Keys should look like: sk-proj-abc123...

# Test key directly
curl https://api.openai.com/v1/models \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Symptom: “Connection refused to Ollama”

Solution:

# Start Ollama
ollama serve

# Check it's running
curl http://localhost:11434/api/tags

# Pull required model
ollama pull gemma4:latest
ollama pull nomic-embed-text

---

6. Slow Performance

Symptom: Queries taking > 5 seconds

Diagnosis:

# Enable debug logging
RUST_LOG="edgequake=debug" cargo run

# Check query timing in logs
# Look for: "query completed in Xms"

Common Causes:

Cause	Diagnosis	Solution
Cold LLM	First query slow	Warm up with test query
Large context	Too many chunks	Reduce max_chunks
Slow embedding	Ollama on CPU	Use GPU or OpenAI
DB connection	Pool exhausted	Check connections

Quick Fixes:

# Use faster model
curl -X POST "http://localhost:8080/api/v1/query" \
  -d '{"query": "test", "llm_model": "gpt-4.1-nano"}'

# Reduce context size
curl -X POST "http://localhost:8080/api/v1/query" \
  -d '{"query": "test", "max_chunks": 5, "max_entities": 5}'

---

Symptom: “Timeout after 180s (attempt X/3)” during document ingestion

Root Cause: The default 180-second per-chunk timeout fires before your local LLM finishes processing. This is common with Ollama on CPU or a slow GPU, or when ingesting large documents with many text chunks.

Two timeout layers exist and both need to be configured:

Layer	Variable	Default	Description
1 (fires first)	EDGEQUAKE_CHUNK_TIMEOUT_SECS	180s	Per-chunk pipeline timeout
2 (HTTP cap)	EDGEQUAKE_LLM_TIMEOUT_SECS	600s	HTTP-level safety cap (max 3600)

Diagnosis:

# Check backend logs for the pattern
grep "Timeout after\|attempt.*/" /tmp/edgequake-backend.log | tail -20

# Measure how long a single LLM call actually takes on your hardware
time curl -s http://localhost:11434/api/chat \
  -d '{"model":"gemma4:latest","messages":[{"role":"user","content":"extract entities from: Alice works at Acme"}]}'

Fix — raise both timeout layers to match your hardware:

# Single-GPU Ollama workstation
export EDGEQUAKE_CHUNK_TIMEOUT_SECS=600        # 10 minutes per chunk
export EDGEQUAKE_MAX_CONCURRENT_EXTRACTIONS=4   # avoid GPU overload
export EDGEQUAKE_LLM_TIMEOUT_SECS=3600         # 1-hour HTTP cap

# CPU-only (slow — be patient)
export EDGEQUAKE_CHUNK_TIMEOUT_SECS=600
export EDGEQUAKE_MAX_CONCURRENT_EXTRACTIONS=2
export EDGEQUAKE_CHUNK_RETRY_DELAY_MS=5000
export EDGEQUAKE_LLM_TIMEOUT_SECS=3600

Key insight: Reducing EDGEQUAKE_MAX_CONCURRENT_EXTRACTIONS often fixes timeouts even without raising the timeout limit. 16 parallel LLM calls on a single GPU creates contention where each request queues behind 15 others — all 16 then time out together.

See Performance Tuning — Ingestion Pipeline for profile-based recommendations.

---

7. Database Issues

Symptom: “Connection pool exhausted”

Solution:

# Check active connections
psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity WHERE datname='edgequake'"

# Increase pool size in DATABASE_URL
DATABASE_URL="postgresql://user:pass@host:5432/db?max_connections=50"

# Or use PgBouncer for pooling

Symptom: “relation ‘documents’ does not exist”

Cause: Migrations haven’t run.

Solution:

# Migrations run automatically on startup
# Restart the backend
cargo run

# Or run manually if needed
psql $DATABASE_URL -f migrations/001_initial.sql

Symptom: “disk full” or slow inserts

Solution:

# Check disk usage
df -h

# Check table sizes
psql $DATABASE_URL -c "
SELECT schemaname, relname,
       pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;
"

# Vacuum database
psql $DATABASE_URL -c "VACUUM ANALYZE"

---

8. Graph Issues

Symptom: “AGE extension not loaded”

Solution:

-- Load AGE extension
LOAD 'age';
SET search_path = ag_catalog, "$user", public;

-- Create graph if not exists
SELECT create_graph('edgequake_graph');

Symptom: Entities not connected

Diagnosis:

# Check relationship count
curl "http://localhost:8080/api/v1/graph/relationships?workspace_id=$WORKSPACE_ID"

# If 0 relationships, check extraction logs

Solution:

# Reprocess with verbose logging
RUST_LOG="edgequake_pipeline=debug" cargo run

# Then reprocess document
curl -X POST "http://localhost:8080/api/v1/documents/reprocess" \
  -H "Content-Type: application/json" \
  -d "{\"document_id\":\"$DOC_ID\",\"force\":true,\"mode\":\"full\"}"

---

9. Frontend Issues

Symptom: Frontend can’t connect to backend

Check:

# Is backend running?
curl http://localhost:8080/health

# CORS enabled?
# Backend should return Access-Control-Allow-Origin header
curl -I http://localhost:8080/health

Solution:

# Start backend first, then frontend
make backend-dev &
sleep 5
make frontend-dev

Symptom: Graph visualization empty

Causes:

1. No entities extracted
2. WebSocket connection failed
3. Sigma.js not loading

Solution:

# Check entities exist via API
curl "http://localhost:8080/api/v1/graph/entities?workspace_id=$WORKSPACE_ID"

# Check browser console for errors
# Open DevTools → Console

---

Diagnostic Commands

Logs

# Backend logs
tail -f /tmp/edgequake-backend.log

# Frontend logs
tail -f /tmp/edgequake-frontend.log

# Docker logs
docker compose logs -f

# Specific component
docker compose logs -f edgequake
docker compose logs -f postgres

Database Queries

-- Check document status
SELECT status, count(*) FROM documents GROUP BY status;

-- Find failed documents
SELECT id, title, error_message FROM documents WHERE status = 'failed';

-- Check entity counts by workspace
SELECT workspace_id, count(*) FROM entities GROUP BY workspace_id;

-- Check embedding dimensions
SELECT embedding_dimension, count(*) FROM embeddings GROUP BY embedding_dimension;

API Debugging

# Verbose curl output
curl -v http://localhost:8080/api/v1/workspaces

# Pretty print JSON
curl http://localhost:8080/api/v1/workspaces | jq

# Check response headers
curl -I http://localhost:8080/health

---

Error Reference

Error Code	Meaning	Solution
400	Bad request	Check request format
401	Unauthorized	Add API key
404	Not found	Check workspace_id exists
422	Validation error	Check required fields
429	Rate limited	Wait and retry
500	Server error	Check logs
503	Service unavailable	Check DB/LLM connection

---

Getting Help

Before Asking for Help

1. Check this troubleshooting guide
2. Check logs for specific error messages
3. Verify environment variables are set
4. Try with minimal configuration (PostgreSQL + mock LLM)

Debug Mode

Start with maximum logging:

RUST_LOG="edgequake=trace,sqlx=debug,tower_http=debug" cargo run

Report an Issue

Include in your report:

1. EdgeQuake version (cargo run --version)
2. Storage mode (PostgreSQL or Memory)
3. LLM provider and model
4. Steps to reproduce
5. Relevant log output
6. Expected vs actual behavior

---

See Also

• Ingestion cancel & fairness — cancel SSOT, leases, multi-replica, queue-metrics
• Observability — Prometheus metrics, GenAI spans, queue pressure
• Configuration Reference - All settings
• Monitoring Guide - Observability setup
• Deployment Guide - Production setup