Scalix Shield

Control which AI agents and crawlers can access your apps

Overview

Scalix Shield gives every app deployed on Scalix Cloud visibility and control over AI crawlers, bots, and scrapers. It is a control plane, not a wall: you decide which agents are welcome — search engines that drive SEO, AI assistants that cite your content in answers, or training scrapers that download it for model datasets — each with its own policy.

Shield runs inside the Scalix gateway, so enforcement happens before a single byte reaches your app. Blocked crawlers get a 403 at the network layer and never consume your compute.

Agent Categories

Shield classifies known crawlers into six categories:

Scalix's own coding agent (Scalix Coder) is in the database and policed like every other agent — Shield does not exempt our own bots.

Policies

Every crawler can be set to Allow, Block, or Rate Limit (requests per minute), per project. Unset crawlers fall back to your per-category defaults, and ultimately to allow. New crawler activity triggers a console notification so you can decide with data, not guesswork.

# List crawlers with policies and traffic stats
curl https://api.scalix.world/v1/shield/crawlers \
  -H "Authorization: Bearer $SCALIX_API_KEY"
 
# Block GPTBot
curl -X PUT https://api.scalix.world/v1/shield/crawlers/gptbot/policy \
  -H "Authorization: Bearer $SCALIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action": "block"}'
 
# Rate-limit Bytespider to 30 requests/minute
curl -X PUT https://api.scalix.world/v1/shield/crawlers/bytespider/policy \
  -H "Authorization: Bearer $SCALIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action": "rate_limit", "rate_limit_per_min": 30}'

Impersonator Detection

Anyone can fake a User-Agent string. Operators of major crawlers publish their IP ranges (OpenAI's gptbot.json, Google's googlebot.json), and Shield refreshes these daily. A request claiming a verified identity from outside its published ranges is flagged as an impersonator in your analytics — and blocked outright if you enable Block impersonators.

You can also set your own per-crawler IP ranges — the IPs action on each crawler row in the console (or PUT /v1/shield/crawlers/{id}/ip-ranges) accepts one IP or CIDR per line and replaces the verification list for that crawler.

robots.txt

Shield answers /robots.txt for your apps at the edge, so this is worth reading before you deploy a site that has one.

Keep writing your own robots.txt. Whatever your app serves is passed through unchanged and Shield appends its crawler rules underneath. Your Disallow, Allow, Crawl-delay and Sitemap lines are preserved exactly as written, and you do not need to copy anything out of the console into your project.

Given an app serving this:

User-agent: *
Disallow: /admin
Disallow: /api/
 
Sitemap: https://example.com/sitemap.xml

with GPTBot blocked in Shield, the served file becomes:

User-agent: *
Disallow: /admin
Disallow: /api/
 
Sitemap: https://example.com/sitemap.xml
 
# ---- appended by Scalix Shield (your AI crawler policies) ----
# GPTBot (OpenAI)
User-agent: gptbot
Disallow: /

Three details that follow from this:

• Your app owns the User-agent: * group. Shield never appends its own wildcard group when merging, because a second permissive group after your Disallow: /admin could be read as re-allowing it. Shield only ever adds crawler-specific rules.
• If your app has no robots.txt, Shield serves its generated file instead, which lists your blocked crawlers and ends with User-agent: * / Allow: /.
• Turning Shield off hands robots.txt back entirely. With enforcement disabled your app's file is served untouched and Shield adds nothing.

Your app's robots.txt is read from your service and cached for six hours. The read wakes a scaled-to-zero service, and crawlers request robots.txt frequently, so a shorter cache would keep your service awake and billing. Expect a change to your robots.txt to take up to six hours to appear.

Custom domains do not get the merge yet. A custom domain resolves to a project rather than to a single Run service, so there is no one app file to merge into; those requests still receive Shield's generated robots.txt. Apps on *.run.scalix.world get the merge today.

Shield enforces at the network layer regardless of robots.txt, which matters because some crawlers ignore it.

Compliance Headers

• X-Robots-Tag: noai, noimageai signals that content must not be used for AI training.
• tdm-reservation: 1 is the EU DSM Directive text-and-data-mining rights reservation.

Both are appended to your responses, never substituted for headers you already set. If your app sends X-Robots-Tag: noindex on a staging or admin surface, that directive survives and Shield's is added alongside it. Multiple X-Robots-Tag headers are valid and combine. Like robots.txt, neither header is injected when Shield enforcement is off.

Detection Scope

Shield v1 identifies crawlers by User-Agent with optional IP-range verification. This reliably covers crawlers that identify themselves — the large majority of AI crawler traffic, since major operators are legal entities that announce their bots. Detecting disguised scrapers (fake browser user-agents) requires fingerprinting, which is on the roadmap and not part of v1. We say this plainly so you can plan accordingly.

Plan Availability