Feature: AI Document Intelligence Foundation

Metadata

Issue ID: FEAT-60
Status: Done
Owner: danrave1234
Related PRs: 60-ai-document-intelligence-foundation -> dev

Overview

AI Document Intelligence Foundation gives authenticated users a suite of AI-powered tools to extract value from their uploaded documents. Users can generate flashcard sets, summaries, quizzes, concept explanations, and typing tests from document content, and ask natural-language questions answered by semantic search over document embeddings. The backend provides a unified multi-provider AI routing layer, a document extraction/chunking/embedding pipeline, OCR support for scanned PDFs and images, and a set of AI tool endpoints gated by subscription plan. The frontend provides an AI Tools Hub and individual tool workspace pages.

Frontend Behavior

Users open /ai to see the AI Tools Hub listing all available tools with name, description, and icon; cards link to individual tool pages.
Individual tool pages: /ai/document-analyzer, /ai/flashcard-builder, /ai/quiz-generator, /ai/concept-explainer, /ai/typing-test, /ai/ask-documents.
Each tool page renders an AiToolWorkspace component handling document selection (from the user's library or a new upload), tool-specific options (e.g. number of flashcards, difficulty), and inline result display with copy/export controls.
Unauthenticated users are redirected to /login?next=<target> following FEAT-31 behavior.
Users without a qualifying subscription plan see an upgrade prompt instead of tool output.
Empty, loading, and retryable error states are handled per tool workspace; design follows the StudyBoost design system.
#278 (slim hardening): AI-tool calls go through useAiTool (POST /ai/tools/:toolId), which now wraps the request in a 45s client-side AbortController timeout. The happy path resolves well under this (no added latency); on timeout the hook sets a retryable error ("This is taking longer than expected. Please try again.") that the workspace renders with its existing "Try again" affordance, instead of an indefinite spinner. The reported processing "stuck/loading forever" did not reproduce on preview — this is defensive coverage of the slow/hung path only.

Backend Behavior

The ai module (backend/src/ai/) exposes AiToolsController with guarded AI tool endpoints, GeminiGenerationProvider for generation routing, and tools under tools/ (flashcards.tool.ts, summary.tool.ts). The documents module adds DocumentExtractionService, DocumentChunkingService, DocumentEmbeddingService, embedding providers (GeminiEmbeddingProvider, LocalDeterministicEmbeddingProvider), and OCR (LocalTesseractOcrProvider).

Endpoints (all require FEAT-31 JWT cookie auth + plan-access guard):

Method	Path	Description
`POST`	`/ai/tools/flashcards`	Generate a flashcard set from document content
`POST`	`/ai/tools/summary`	Generate a summary from document content
`POST`	`/documents/:id/ingest`	Trigger the extraction + chunking + embedding pipeline
`GET`	`/documents/:id/ask`	Semantic search / ask-documents query

AI provider routing: AI_PROVIDER env var selects the default provider (gemini); per-capability overrides via AI_MODEL_CHAT, AI_MODEL_REASONING, AI_MODEL_SUMMARY, AI_MODEL_CLASSIFICATION, AI_MODEL_EMBEDDING. Embeddings fall back to local deterministic embeddings in dev when Gemini quota is exceeded (GEMINI_EMBEDDING_DEV_QUOTA_FALLBACK_TO_LOCAL=true).

Ingestion pipeline: (1) DocumentExtractionService reads bytes via pdfjs-dist (PDF), mammoth (DOCX), or UTF-8 decode (TXT/MD/JSON/XML). Bytes are resolved in this order: in-memory upload buffer → local source_reference path → cloud_storage_url fetch. PDFs are scored per-page; if >40% of pages have <40 non-whitespace chars (likely scanned), native text is dropped and the document falls through to OCR. (2) LocalTesseractOcrProvider rasterizes PDF pages to PNG via pdftoppm (poppler-utils) at 200 DPI, runs tesseract per page, and concatenates output. Standalone images go directly through tesseract. (3) DocumentChunkingService splits text into overlapping semantic chunks. (4) DocumentEmbeddingService generates and persists vector embeddings per chunk. Ephemeral/sync uploads run extraction + OCR inline so AI tools can call the document immediately; non-ephemeral async uploads route through OcrProcessor on the FEAT-33 queue, which fetches bytes from cloud_storage_url.

Failure modes: PlanAccessGuard returns 403 with an upgrade-prompt payload for free-plan users; unauthenticated requests return 401; upstream AI/provider failures surface a retryable error rather than a raw 500 leak.

Data model — migration 202505030300_add_ai_tools_foundation: document_chunks (text chunks with position metadata), document_embeddings (vectors with provider/model metadata), ai_tool_results (cached AI outputs per document per user).

QA Test Scenarios

Scenario ID	Description	Steps	Input	Expected Result
FEAT-60-01	Happy path: flashcard generation	Login as a qualifying-plan user, select an ingested document, run flashcard tool	Valid `documentId`, qualifying plan	Returns flashcard set JSON; result displayed inline
FEAT-60-02	Happy path: ask-documents query	Run an ask-documents query against an ingested document	Valid `documentId`, natural-language question	Top-k semantically relevant chunks returned with similarity scores
FEAT-60-03	Validation: free-plan user blocked	Free-plan user requests flashcard generation	Valid `documentId`, free plan	`403` with upgrade-prompt payload; no AI output returned
FEAT-60-04	Validation: malformed tool request	Submit a tool request with a missing/invalid `documentId`	Empty or non-UUID `documentId`	Backend rejects with `400 Bad Request`; no AI call made
FEAT-60-05	Authorization: unauthenticated request	Call any AI tool endpoint without auth cookies	No auth cookies	Backend returns `401`
FEAT-60-06	Authorization: unowned document	Run a tool against a document owned by another user	Foreign `documentId`	Backend rejects without leaking document content
FEAT-60-07	Backend failure: AI provider error	Run a tool while the upstream AI provider returns an error	Valid input, provider unavailable	UI shows a retryable error state; no broken or blank screen
FEAT-60-08	Backend failure: Gemini quota exceeded in dev	Trigger ingestion when Gemini embedding quota is exhausted	Valid document, quota exceeded	Falls back to local deterministic embeddings without error
FEAT-60-09	Edge case: scanned-image PDF ingestion (sync)	Upload a scanned image-only PDF via `/ai/document-analyzer` (ephemeral)	PDF with no embedded text	`pdftoppm` rasterizes pages; `tesseract` OCRs; `text_content` populated with `extraction_method=ocr`; `ocr_used=true`; document chunked before request returns
FEAT-60-10	Edge case: text-PDF ingestion	Trigger ingestion for a PDF with embedded text	PDF with selectable text	`pdfjs-dist` extracts text inline; `extraction_method=native`; OCR skipped; `document_chunks` and `document_embeddings` rows created
FEAT-60-13	Edge case: mixed-content PDF	Ingest a PDF where some pages have text and others are scanned images	PDF with >40% sparse pages	Per-page density check drops native text; OCR runs on all pages; full text recovered
FEAT-60-14	Edge case: plain-text upload	Upload a `.txt` or `.md` file	UTF-8 text file	`extractTextFromBuffer` decodes buffer as UTF-8; `text_content` populated immediately; no OCR invoked
FEAT-60-15	Edge case: DOCX upload	Upload a Word `.docx` file	DOCX with body text	`mammoth.extractRawText` parses XML; `text_content` populated; no OCR invoked
FEAT-60-11	Rate limiting / abuse control	Rapidly fire repeated AI tool requests beyond the throttle window	Many requests from one user	Throttled requests are rejected; no unbounded provider spend
FEAT-60-12	AI Tools Hub loads	Open `/ai` as an authenticated user	Valid session	All six tool cards render with correct links and icons

Edge Cases

Ephemeral uploads (AI Course, document-analyzer) live only as a multer buffer; the extractor + OCR provider both accept an inputBuffer so OCR succeeds without disk or cloud storage.
Mixed-content PDFs (text covers + scanned body) trigger OCR via the per-page sparsity check, recovering text from pages pdfjs-dist cannot reach.
If pdftoppm is missing on PATH, OCR throws pdftoppm produced no page images — install poppler-utils (Linux) or set PDFTOPPM_EXECUTABLE to its absolute path.
Documents with too little extracted text (and OCR yields nothing) degrade gracefully rather than producing empty AI output.
Gemini quota exhaustion in dev falls back to local deterministic embeddings; production surfaces a clear error.
Concurrent ingestion requests for the same document should not produce duplicate chunk/embedding rows.
Upstream AI provider timeouts surface a retryable error state instead of a blank screen.
Document content is never returned raw in AI tool responses — only derived outputs.

Notes

Feature flags: AI tool access is gated by subscription plan via PlanAccessGuard rather than a static flag.
Dependencies: FEAT-31 cookie auth, FEAT-33 job queue (async OCR for non-ephemeral uploads), Prisma models (document_chunks, document_embeddings, ai_tool_results), Gemini provider SDK, pdfjs-dist (native PDF text extraction), mammoth (DOCX), tesseract-ocr + poppler-utils system packages (OCR). The backend Dockerfile installs tesseract-ocr poppler-utils so production parity matches local dev.
OCR configuration env vars (all optional): DOCUMENT_OCR_ENABLED (default true), TESSERACT_EXECUTABLE (default tesseract on PATH), PDFTOPPM_EXECUTABLE (default pdftoppm on PATH), TESSERACT_LANG (default eng), OCR_PDF_DPI (default 200), OCR_MAX_PAGES (default 50).
Known limitations: provider routing currently defaults to Gemini; embedding fallback to local deterministic embeddings is a dev-only convenience, not a production path. OCR latency is roughly 3–6s per page — sync uploads of long scanned PDFs may exceed reasonable request timeouts; reduce OCR_MAX_PAGES or move to async queue mode.

Security Notes

No AI endpoint accepts arbitrary user_id in request bodies; identity is always resolved from the secure cookie auth context.
All AI tool endpoints require the auth guard and plan-access guard.
Document content is never returned raw — only derived outputs (flashcards, summaries, etc.).
OCR processing is async and isolated in the job queue to avoid blocking the request thread.
Embedding provider API keys are server-side only and never exposed to the frontend.

Canonical Feature Document

Implementation and QA requirements are tracked in: docs/features/FEAT-60-AI-Document-Intelligence-Foundation.md