Feature: API Crawler Hardening

Metadata

• Issue ID: FEAT-143
• Status: In Progress
• Owner: danrave1234
• Related PRs: fix/143-api-crawler-hardening -> dev (StudyBoostIO/StudyboostV2#143)

---

Overview

The backend API host (preview.api.studyboost.com, and by extension production api.studyboost.com) was being indexed and crawled by AI bots and unbranded scrapers despite serving zero human users. Verified via gcloud logging read on 2026-05-24: roughly 500 requests per hour against /documents/:slug and /documents/:slug/related, none from real users. Cloud Run instances never scaled to zero, memory utilization climbed steadily, and preview-only data was being slurped into third-party crawl corpora.

FEAT-143 applies three layered, infrastructure-free defenses at the NestJS HTTP boundary:

1. GET /robots.txt returning User-agent: *\nDisallow: / — APIs should never be indexed.
2. X-Robots-Tag: noindex, nofollow on every response — covers crawlers that fetch resources without consulting robots.txt first.
3. Reject requests with User-Agent: node (case-insensitive, trimmed, exact match) with 403 FORBIDDEN_USER_AGENT — no legitimate caller in this codebase sends a bare node UA. Cuts the unbranded scraper traffic instantly.

The frontend app/robots.ts (which governs studyboost.com) is intentionally unchanged — public catalog pages there should remain indexable. The API and the frontend live on different hosts and have different SEO policies.

Canonical feature doc: docs/features/FEAT-143-api-crawler-hardening.md.

---

Frontend Behavior

No frontend changes. Browser requests carry real User-Agent headers; nothing on the client side sent bare node. Existing flows render identically.

---

Backend Behavior

GET /robots.txt

Defined in backend/src/app.controller.ts. Returns plain-text body, cached at the edge for 24h.

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Cache-Control: public, max-age=86400

User-agent: *
Disallow: /

X-Robots-Tag header on every response

Express middleware registered in backend/src/main.ts, before Helmet:

app.use((_req, res, next) => {
  res.setHeader('X-Robots-Tag', 'noindex, nofollow');
  next();
});

Applied uniformly across success and error responses.

Bare-node UA reject

Express middleware in backend/src/main.ts, registered before the X-Robots-Tag middleware so blocked requests short-circuit early:

app.use((req, res, next) => {
  if ((req.headers['user-agent'] ?? '').trim().toLowerCase() === 'node') {
    res.status(403).json({ code: 'FORBIDDEN_USER_AGENT' });
    return;
  }
  next();
});

The match is intentionally narrow: exact equality, case-insensitive, trimmed. node-fetch/x.y.z, Next.js/<version>, and any string that merely contains node pass through. Only callers with a totally unconfigured Node default UA (node) are blocked. The error body uses the same { code, ... } shape as AppExceptionFilter.

Middleware order

In main.ts, middleware registers in this deliberate order:

1. JSON / urlencoded body parsers with raw-body capture for Stripe webhooks.
2. trust proxy = 1.
3. Bare-node UA reject (cheap, short-circuits hostile traffic immediately).
4. X-Robots-Tag setter (runs for every accepted request).
5. Helmet security headers.
6. WebSocket adapter, CORS, NestJS routes.

---

Error codes the frontend may observe

---

QA scenarios

Full scenario table lives in docs/features/FEAT-143-api-crawler-hardening.md (FEAT-143-01 .. FEAT-143-13). Key checks:

• /robots.txt returns the disallow-all body with text/plain and a 24h cache header.
• X-Robots-Tag: noindex, nofollow appears on success and error responses.
• User-Agent: node is rejected; User-Agent: node-fetch/x.y.z is allowed.
• Browser flows and Stripe webhooks are unaffected.
• 24h post-deploy: httpRequest.userAgent="node" request count drops to near zero.
• 48h post-deploy: ClaudeBot/GPTBot/OAI-SearchBot requests to /documents/* drop materially.

---

Edge cases

• Cloud Run probes and Google Front End never use bare node — confirmed by current logs.
• Future ops scripts using fetch() MUST set a descriptive User-Agent (e.g. studyboost-ops/<purpose>).
• Third-party webhooks (Stripe, GitHub, Discord) all identify themselves; none are at risk.
• Hostile crawlers may rotate UAs; if the current scraper returns as Mozilla/5.0, the escalation path is Cloud Armor IP rules and per-route rate limiting — out of scope for this PR.
• Memory growth was likely a side-effect of traffic preventing scale-to-zero rather than a real leak. If memory still climbs a week after cleaner traffic, file a separate issue for heap analysis.

---

Notes

• The bare-node reject is a behavioural firewall, not a security boundary. The repo-wide grep shows no legitimate caller currently sends that UA; future callers must set one explicitly rather than relax the filter.
• Frontend app/robots.ts (host: studyboost.com) intentionally remains permissive for public catalog pages. The API host is the only one disallowing crawlers.