Scalix Shield

Control which AI agents and crawlers can access your apps

Overview

Scalix Shield gives every app deployed on Scalix Cloud visibility and control over AI crawlers, bots, and scrapers. It is a control plane, not a wall: you decide which agents are welcome — search engines that drive SEO, AI assistants that cite your content in answers, or training scrapers that download it for model datasets — each with its own policy.

Shield runs inside the Scalix gateway, so enforcement happens before a single byte reaches your app. Blocked crawlers get a 403 at the network layer and never consume your compute.

Agent Categories

Shield classifies known crawlers into six categories:

Scalix's own coding agent (Scalix Coder) is in the database and policed like every other agent — Shield does not exempt our own bots.

Policies

Every crawler can be set to Allow, Block, or Rate Limit (requests per minute), per project. Unset crawlers fall back to your per-category defaults, and ultimately to allow. New crawler activity triggers a console notification so you can decide with data, not guesswork.

# List crawlers with policies and traffic stats
curl https://api.scalix.world/v1/shield/crawlers \
  -H "Authorization: Bearer $SCALIX_API_KEY"
 
# Block GPTBot
curl -X PUT https://api.scalix.world/v1/shield/crawlers/gptbot/policy \
  -H "Authorization: Bearer $SCALIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action": "block"}'
 
# Rate-limit Bytespider to 30 requests/minute
curl -X PUT https://api.scalix.world/v1/shield/crawlers/bytespider/policy \
  -H "Authorization: Bearer $SCALIX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"action": "rate_limit", "rate_limit_per_min": 30}'

Impersonator Detection

Anyone can fake a User-Agent string. Operators of major crawlers publish their IP ranges (OpenAI's gptbot.json, Google's googlebot.json), and Shield refreshes these daily. A request claiming a verified identity from outside its published ranges is flagged as an impersonator in your analytics — and blocked outright if you enable Block impersonators.

You can also set your own per-crawler IP ranges — the IPs action on each crawler row in the console (or PUT /v1/shield/crawlers/{id}/ip-ranges) accepts one IP or CIDR per line and replaces the verification list for that crawler.

Compliance Tools

• robots.txt generator — always in sync with your enforcement policies; serve it at your domain root as the courtesy layer. Shield enforces at the network layer regardless, which matters because some crawlers ignore robots.txt.
• X-Robots-Tag: noai, noimageai — response headers signalling content must not be used for AI training.
• tdm-reservation: 1 — EU DSM Directive text-and-data-mining rights reservation.

Detection Scope

Shield v1 identifies crawlers by User-Agent with optional IP-range verification. This reliably covers crawlers that identify themselves — the large majority of AI crawler traffic, since major operators are legal entities that announce their bots. Detecting disguised scrapers (fake browser user-agents) requires fingerprinting, which is on the roadmap and not part of v1. We say this plainly so you can plan accordingly.

Plan Availability