StudyBoost Docs

Feature: Noindex non-production deployments

Metadata

  • Issue ID: FEAT-148
  • Status: In Progress
  • Owner: danrave1234
  • Related PR: fix/148-noindex-non-production-deployments → dev
  • Related Issues: #148; follow-up regression fix for #145; complements #143 on the API side.

Overview

After FEAT-145's SEO pipeline deployed, preview.studyboost.com started receiving GPTBot/ClaudeBot/etc. crawl traffic to /documents/* because preview ships the same robots.ts + sitemap.xml as production. Verified via Vercel runtime logs at 2026-05-24 16:53 UTC — one example:

GET /documents/case-study-wagner-sided-too-fast-1552927 → 200
User-Agent: ... GPTBot/1.4 ...
Referer: https://preview.studyboost.com/courses/ethics-in-public-administration-382

Multiple requests per second to document detail pages. Same failure mode FEAT-143 fixed for the API host, just re-introduced on the frontend by the FEAT-145 work.

This change gates every SEO-positive surface on VERCEL_ENV === 'production'. On non-production deployments the site returns hard-disallow robots, 404s on sitemap routes, a stub llms.txt, and X-Robots-Tag: noindex, nofollow, noarchive on every page. Production behavior on studyboost.com is unchanged from FEAT-145.

Frontend Behavior

frontend/lib/seo/is-production.ts (new)

Single source of truth. Returns true only when:

  • process.env.VERCEL_ENV === 'production', OR
  • VERCEL_ENV is undefined AND NEXT_PUBLIC_SITE_URL === 'https://www.studyboost.com'

The fallback handles non-Vercel callers (Cloud Run frontend, self-hosted, CI). Defaults closed — anything ambiguous resolves to non-production.

frontend/app/robots.ts

If !isProduction(), returns one rule block: User-Agent: *\nDisallow: /. No allow lines, no AI bot list, no Sitemap: directive. Otherwise the full FEAT-145 ruleset applies unchanged.

frontend/proxy.ts (middleware)

A new helper applyNoindexIfPreview(res) stamps X-Robots-Tag: noindex, nofollow, noarchive on every NextResponse returned by the middleware when !isProduction(). Applies to all three exit paths: protected-route login redirect, auth-page redirect for logged-in users, and the pass-through NextResponse.next().

Belt-and-suspenders: any crawler that fetches a page before reading robots.txt still sees the noindex directive in the response header.

frontend/app/sitemap.xml/route.ts + /sitemaps/static + /sitemaps/[kind]/[chunk]

All three return 404 (new Response('Not Found', { status: 404 }) for the top-level, notFound() for the dynamic chunk route) on non-production. Crawlers that probe these URLs on preview get a clear "nothing here" signal and don't walk into the chunked tree.

frontend/app/llms.txt/route.ts

On non-production returns a short markdown stub:

# StudyBoost (non-production deployment)

This deployment is for preview/staging only and should not be indexed.
See the canonical llms.txt at https://www.studyboost.com/llms.txt for the
production catalog map.

Content-Type: text/plain; charset=utf-8, Cache-Control: public, max-age=3600, X-Robots-Tag: noindex, nofollow, noarchive. The stub explicitly redirects LLM crawlers to the production URL.

Backend Behavior

No backend changes. FEAT-143 already locked down preview.api.studyboost.com with Disallow: / robots.txt and X-Robots-Tag on every response. Together with this PR, the entire preview hostname chain is closed to crawlers.

Vercel Cron behavior

The cron entry in frontend/vercel.json is unaffected. Vercel only schedules cron jobs on production deployments by contract — preview deployments don't run scheduled functions. Manual invocations of /api/cron/sitemap continue to work on preview if needed (Bearer auth still required).

Error codes the frontend may observe

None. Existing user flows are unchanged on production. On non-production, requests to /sitemap.xml, /sitemaps/static, /sitemaps/{kind}/{chunk} return 404 — but human users never visit those URLs.

QA scenarios

See docs/features/FEAT-148-noindex-non-production-deployments.md for FEAT-148-01..FEAT-148-11. Highlights:

  • FEAT-148-01: preview.studyboost.com/robots.txt body is exactly User-Agent: *\nDisallow: /\n
  • FEAT-148-02: studyboost.com/robots.txt still contains all 18 AI-bot rules (regression check on production)
  • FEAT-148-03: preview.studyboost.com/sitemap.xml returns 404
  • FEAT-148-04: studyboost.com/sitemap.xml still serves the sitemap index
  • FEAT-148-07: every preview response carries X-Robots-Tag: noindex, nofollow, noarchive
  • FEAT-148-08: production responses do NOT carry X-Robots-Tag (the SEO-positive headers stay clean)
  • FEAT-148-09: 24h after deploy, GPTBot request volume to preview.studyboost.com drops to ~0
  • FEAT-148-11: Vercel Cron on production still runs the full sitemap regeneration

Edge cases

  • Local next dev — VERCEL_ENV undefined, NEXT_PUBLIC_SITE_URL typically http://localhost:3000 → fallback returns non-production → noindex output. Intentional; nothing on localhost should be indexed.
  • CI / integration tests — same as local dev. Tests that need to assert sitemap or robots content must explicitly set VERCEL_ENV=production (or stub isProduction()).
  • Vercel Deployment Protection — if enabled, Vercel returns 401 from the edge before our code runs. Env gating becomes redundant but stays safe.
  • Future staging.studyboost.com alias — Vercel reports VERCEL_ENV=preview for non-production aliases, so the gate continues to apply.
  • Rolling deploy transition — for a few minutes after the merge, preview will still expose stale ISR'd sitemap. Acceptable; AI crawlers don't re-fetch faster than minutes.
  • Per-PR preview URLs (e.g. studyboostv2-git-feature-xyz.vercel.app) — also gets VERCEL_ENV=preview, also covered.

References

  • Builds on #145 (SEO pipeline) and complements #143 (API crawler hardening)
  • Vercel docs: System Environment Variables — the VERCEL_ENV contract
  • Vercel runtime logs at 2026-05-24 16:52..16:53 UTC showing GPTBot crawl pattern