Feature: SEO Pipeline (Sitemaps + AI Visibility)

Metadata

• Issue ID: FEAT-145
• Status: In Progress
• Owner: danrave1234
• Related PRs: feature/145-seo-pipeline -> dev (StudyBoostIO/StudyboostV2#145)
• Related issues: builds on the API-side robots/UA hardening from #143; complementary, not a duplicate (different host, different policy).

---

Overview

The public-catalog host studyboost.com carries the SEO surface area. As of 2026-05-24 the catalog covers 1,200 institutions / 101,423 courses / 356,742 documents (~458k+ entity URLs). Pre-existing SEO posture exposed only the 5 catalog roots in sitemap.ts, so the millions of high-value detail URLs were invisible to Google/Bing crawlers and AI answer-engines.

FEAT-145 lands a single layered SEO + AI-visibility pipeline:

1. Frontend SEO foundation — root metadata (metadataBase, OG defaults, search-engine verification env hooks), AI/answer-engine robots policy, paginated-list canonical fixes.
2. JSON-LD across the site — Organization + WebSite (root), Course / CollegeOrUniversity / CreativeWork / LearningResource (detail pages, already existed; kept as-is), BreadcrumbList (every detail page, new), CollectionPage (combination route), SoftwareApplication + HowTo (12 AI tools).
3. Chunked dynamic sitemaps — backend SitemapService streams slug + updated_at chunked by 50,000 URLs (Google's per-file cap). Frontend /sitemap.xml is a sitemap index pointing at children.
4. GCS bucket cache + Vercel Cron — SitemapBucketService uploads every chunk to gs://<bucket>/v2/sitemaps/*.xml with Cache-Control: public, max-age=604800 (7 days). Vercel Cron triggers full regeneration weekly; @nestjs/schedule backstop covers Vercel outages.
5. AI tool server shells — per-tool layout.tsx for all 12 /ai/* pages emits SEO metadata + JSON-LD without touching the existing "use client" widgets.
6. llms.txt — markdown index per llmstxt.org. Live catalog counts + AI tool index + sitemap pointers.
7. Admin "Regenerate now" button — /admin/seo page hits a cookie-authed admin endpoint so ops can rebuild without secrets in the browser.
8. Combination route — /institutions/[slug]/courses is a dedicated SEO landing for "{University} courses" queries.

Canonical feature doc: docs/src/app/docs/features/145-seo-pipeline/page.md (this file).

---

Frontend Behavior

Root layout — frontend/app/layout.tsx

Sets metadataBase, default OG and Twitter cards (including a /og-default.png placeholder), alternates.canonical: "/", search-engine verification meta (read from GOOGLE_SITE_VERIFICATION, BING_SITE_VERIFICATION, YANDEX_SITE_VERIFICATION env vars), and Googlebot directives (max-image-preview: large, max-snippet: -1). Injects two server-only JSON-LD payloads via components/seo/json-ld.tsx:

• Organization — name, logo, sameAs (Twitter/LinkedIn placeholders)
• WebSite — inLanguage: en-US, publisher, SearchAction pointing at /documents?q={search_term_string}

List pages — frontend/app/(features)/{institutions|courses|documents}/page.tsx

Each is now a generateMetadata-driven page calling the shared buildListMetadata helper. Canonical strategy:

• ?sort=... and ?q=... are stripped from the canonical (sort/filter views are duplicate content; search results are noindex'd via robots: { index: false, follow: true }).
• ?page=N (N > 1) self-canonicalizes to /path?page=N. Previously it self-canonicalized to page 1, which caused Google to de-index deep pages.
• Title is augmented with "— Page N of M" when on a deep page.
• Each list page emits a BreadcrumbList (e.g. Home > Institutions).
• Visible breadcrumb in FeatureHeader is unchanged in behavior but now consults a BreadcrumbLabelProvider so detail-page slugs render as real entity names.

Detail pages — frontend/app/(features)/{institutions|courses|documents}/[slug]/page.tsx

generateMetadata derives the description from real content where available:

• Institutions: prefer institution.description; fallback uses courseCount + documentCount + location to avoid templated copy.
• Courses: prefer course.description; fallback uses documentCount + institution name.
• Documents: prefer document.description then document.seoContent; final fallback uses document.topicTags ("Covers: linear algebra, eigenvalues, …").
• 404 metadata returns robots: { index: false, follow: false } so the "Not Found" title isn't indexed.

Detail pages also render BreadcrumbJsonLd with the full trail (e.g., document → its course → its institution → root). The documents page additionally RegisterBreadcrumbLabels the parent course + institution slugs so the visible breadcrumb shows real names all the way up.

Combination route — frontend/app/(features)/institutions/[slug]/courses/page.tsx

New SEO-targeted route for "{University} courses" queries. Server-renders cards listing every course at one institution, paginated. Emits BreadcrumbList + CollectionPage JSON-LD with each course as an ItemListElement of type Course.

AI tool pages — frontend/app/(features)/ai/{slug}/layout.tsx

A layout.tsx per tool (12 total). The page.tsx files stay "use client" for the interactive widget. The layout owns:

• metadata from lib/seo/ai-tools.ts (unique title, description, keywords per tool).
• SoftwareApplication + HowTo JSON-LD with 3-5 step recipes. HowTo is rich-result eligible — earns a step-by-step card in Google.
• The /ai hub layout adds an ItemList of all 12 tools.
• Removed /ai from frontend/proxy.ts's PROTECTED_PREFIXES so the landing pages render for unauthenticated visitors (and crawlers). API calls inside the widgets still require auth.

Pricing — frontend/app/pricing/layout.tsx

Split into a server layout (metadata + JSON-LD) wrapping the existing "use client" shell. JSON-LD emits BreadcrumbList + one Product + Offer per plan from lib/plans.ts.

Frontend sitemap proxy — frontend/app/sitemap.xml/route.ts

Replaces the deleted frontend/app/sitemap.ts. The XML is built backend-side now (so the bucket cache, cron, and admin UI all converge on one source of truth). The route fetches from backend /sitemap/xml/{index|static|kind/chunk} via lib/seo/sitemap-source.ts. Cache headers: public, max-age=300, s-maxage=3600, stale-while-revalidate=86400.

Three frontend route handlers wire through:

• /sitemap.xml → backend /sitemap/xml/index
• /sitemaps/static → backend /sitemap/xml/static
• /sitemaps/{kind}/{chunk} → backend /sitemap/xml/{kind}/{chunk}

robots.txt — frontend/app/robots.ts

5 rule blocks:

1. User-Agent: * — default fall-through.
2. Googlebot family (Googlebot, Googlebot-Image, Googlebot-News).
3. Bingbot family.
4. Secondary search engines (DuckDuckBot, YandexBot, Slurp, Baiduspider).
5. 18 named AI/answer-engine bots: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot, Applebot-Extended, CCBot, cohere-ai, Bytespider, Diffbot, YouBot, Meta-ExternalAgent, Meta-ExternalFetcher.

Every block shares the same allow-list (/, /institutions, /courses, /documents, /pricing, /ai) and disallow-list (/api/, /login, /signup, /onboarding, /dashboard, /my-courses, /my-documents, /notes, /settings, /ai-courses, /verify-email, /forgot-password, /reset-password, /unauthorized, /warning).

llms.txt — frontend/app/llms.txt/route.ts

Markdown index per llmstxt.org. Sections: H1 + tagline, Catalog scale (live counts from backend), Catalog (3 directory links with descriptions), AI study tools (all 12 tools with one-liner each), Pricing & company, Sitemaps, Crawling policy. Cache-Control: public, max-age=300, s-maxage=3600, stale-while-revalidate=86400.

Pagination 403 fix

frontend/lib/api/catalog.ts now sets user-agent: "StudyBoost-Frontend-SSR/1.0" on every SSR fetch. Required because backend/src/main.ts's FORBIDDEN_USER_AGENT middleware (from #143) was 403'ing the frontend's own fetches when Node's undici fetch sent the default User-Agent: node. Backend middleware also tightened: only enforced when NODE_ENV === 'production' so local dev never 403s its own SSR.

---

Backend Behavior

SitemapService — backend/src/sitemap/sitemap.service.ts

Tight findMany({ select: { slug, updated_at } }) chunked by 50,000 rows. Three reads per kind:

• GET /sitemap/counts — total + chunk math per kind. Used by frontend sitemap index.
• GET /sitemap/{kind}/{chunk} — JSON SitemapChunkResponse (legacy path; kept for live-render fallback).
• GET /sitemap/xml/{index|static|kind/chunk} — XML (bucket-first; see below).

All endpoints set Cache-Control: public, max-age=300, s-maxage=3600, stale-while-revalidate=86400.

SitemapBucketService — backend/src/sitemap/sitemap-bucket.service.ts

Owns the full sitemap XML pipeline:

• Builds XML strings (sitemap index, static sections sitemap, per-chunk urlsets) with namespace http://www.sitemaps.org/schemas/sitemap/0.9.
• readOrBuild(objectPath, build) — bucket fast-path. Downloads gs://<bucket>/<objectPath> if it exists; otherwise live-generates and opportunistically uploads to the bucket (fire-and-forget) so the next read is fast.
• regenerateAll() — iterates every chunk across all three kinds + static + index. Uploads each with Cache-Control: public, max-age=604800 (7 days). Returns { durationMs, counts, uploaded, skipped }.
• @Cron(SITEMAP_CRON_SCHEDULE || '0 4 * * 0') — env-configurable in-process backstop. Default Sunday 04:00 UTC.

Bucket layout:

gs://<GCP_BUCKET_NAME>/v2/sitemaps/index.xml
gs://<GCP_BUCKET_NAME>/v2/sitemaps/static.xml
gs://<GCP_BUCKET_NAME>/v2/sitemaps/institutions-{N}.xml
gs://<GCP_BUCKET_NAME>/v2/sitemaps/courses-{N}.xml
gs://<GCP_BUCKET_NAME>/v2/sitemaps/documents-{N}.xml

If GCS env (GCP_PROJECT_ID, GCP_BUCKET_NAME, GCP_CREDENTIALS_JSON) is not configured, the service falls back to live-render-every-time. regenerateAll() no-ops with a log line.

GcsStorageService — backend/src/storage/gcs-storage.service.ts

Extended with three helpers for the sitemap pipeline:

• isConfigured() — true when the bucket client is ready.
• downloadBuffer(objectPath) — returns Buffer | null. Existence-checks first to avoid throwing on miss.
• uploadBufferOpportunistic(buffer, objectPath, contentType, cacheControl) — like uploadBuffer but swallows errors and returns null rather than throwing. Used by the lazy-cache path so an opportunistic write failure doesn't fail the request.

POST /sitemap/regenerate — shared-secret endpoint

Headers: x-sitemap-secret: ${SITEMAP_REGENERATE_SECRET}. Used by the Vercel Cron handler at frontend/app/api/cron/sitemap/route.ts. Returns 403 if the env var is unset or the header doesn't match.

POST /admin/sitemap/regenerate — cookie-authed endpoint

Lives on AdminController so it inherits the existing @UseGuards(CookieAuthGuard, AdminGuard) chain. Calls the same SitemapBucketService.regenerateAll(). No secrets in the browser; the admin dashboard button at /admin/seo uses credentials: "include".

Middleware change — backend/src/main.ts

The bare-node UA filter from #143 is now gated on NODE_ENV === 'production'. In dev/test the filter does not run, so a Node fetch missing a UA header no longer 403s its own SSR. Production behavior unchanged.

---

Vercel Cron — frontend/vercel.json + frontend/app/api/cron/sitemap/route.ts

vercel.json:

{
  "crons": [
    { "path": "/api/cron/sitemap", "schedule": "0 3 * * 0" }
  ]
}

Sundays 03:00 UTC. The route handler:

1. Validates Authorization: Bearer ${CRON_SECRET} (Vercel auto-injects on scheduled hits).
2. Forwards a POST to ${BACKEND_URL}/sitemap/regenerate with x-sitemap-secret: ${SITEMAP_REGENERATE_SECRET}.
3. Returns the backend's { durationMs, counts, uploaded, skipped } as JSON.

Vercel Pro is required — full regeneration takes ~70 seconds at current scale, which exceeds Hobby tier's 10-second function timeout (Pro is 5 minutes).

---

Admin UI — admin/app/seo/page.tsx

New /admin/seo route with:

• <Globe2>-iconed "Regenerate now" button.
• Live <Loader2> spinner during execution (typical ~70s).
• Result panel showing durationMs, per-kind counts, uploaded/skipped tallies.
• Error banner that distinguishes 401 (re-auth required), 403 (revoked access), 429 (throttle), and 5xx.
• Documentation panel summarizing the pipeline.

Dashboard quick-action card added at admin/app/page.tsx.

---

Environment variables

Per-environment templates (all gitignored): backend/.env.{preview,prod}, frontend/.env.{preview,prod}, admin/.env.{preview,prod}.

---

Error codes the frontend may observe

---

QA scenarios

See PR description for SEO-1..SEO-20 (all green, verified live against localhost:3000/3001 with the same Neon DB the deployed preview uses).

Key checks worth repeating after any future refactor:

1. GET /sitemap.xml parses as XML, has <sitemapindex> root, references chunked children with <lastmod>.
2. Each chunk parses as XML, has <urlset> root, every <url> has <loc>, <lastmod>, <changefreq>, <priority>.
3. GET /llms.txt returns text/plain; charset=utf-8, has H1 # StudyBoost and 6 sections.
4. GET /robots.txt includes all 18 AI/answer-engine UAs and Sitemap: https://www.studyboost.com/sitemap.xml.
5. GET /ai/quiz-generator (and the other 11 tool pages) → 200, no /login redirect, JSON-LD includes SoftwareApplication + HowTo.
6. GET /institutions?page=N>1 → canonical is /institutions?page=N, NOT /institutions.
7. Admin button at /admin/seo → "Regenerate now" → ~70s later → success banner with non-zero uploaded.
8. After regeneration, https://storage.googleapis.com/<bucket>/v2/sitemaps/index.xml (HEAD) returns 200 with Cache-Control: public, max-age=604800.

---

Edge cases

• Cold start, both caches empty — first crawler hitting a chunk after Cloud Run scale-to-zero gets a live-render response (~8s for a 9 MB chunk), and the response triggers a fire-and-forget bucket upload so the next request is fast. stale-while-revalidate=86400 on the HTTP layer means most crawlers see a stale-but-instant response while the upload completes.
• Catalog grows past 50k per chunk — SitemapService.CHUNK_SIZE = 50_000 (the protocol max). Adding a 9th documents chunk just adds one more URL to the sitemap index automatically — no code change needed.
• GCS not configured — SitemapBucketService checks gcs.isConfigured() and falls back to live-render-every-time. Dev without GCP_* env still works; cron skips with a log line.
• Vercel Cron times out — backend regeneration keeps running to completion regardless. Next cron tick simply uses the now-cached bucket files. On Vercel Pro (5-min function timeout) full regen at current scale fits in ~75s.
• Secret rotation — CRON_SECRET and SITEMAP_REGENERATE_SECRET are independent. Either can be rotated without touching the other. Backend SITEMAP_REGENERATE_SECRET must match the value Vercel forwards.
• AI tool pages still client-only inside the shell — page.tsx stays "use client" for the interactive widget; layout.tsx provides server-rendered metadata + JSON-LD. Best of both.
• Breadcrumb override hydration — visible breadcrumb shows the URL-derived slug for the first render, then swaps to the real entity name once RegisterBreadcrumbLabel's useEffect runs. JSON-LD always carries the real name, so crawlers are unaffected.
• Combination route at scale — /institutions/[slug]/courses reuses the existing findByInstitutionSlug backend method; pagination is honored. For institutions with thousands of courses, deep pages stay indexable (canonical fix from list-page metadata helper applies here too).

---

References

• Issue: StudyBoostIO/StudyboostV2#145
• Builds on #143 (API crawler hardening) and #137 (webhook reliability)
• llmstxt.org convention for AI crawler discovery
• Google Search Central — Sitemaps overview, Robots.txt overview, Google's crawlers
• Vercel Cron Jobs