HomeBook a Demo

Unblock the Inline crawler in Cloudflare

Step-by-step: create a Cloudflare WAF custom rule so Cloudflare stops blocking or challenging Inline AI's crawler.

If your site sits behind Cloudflare, its bot protection can block or challenge Inline's crawler before the request ever reaches your server. When that happens, Inline cannot ingest your content, and the assistant on your site has less to answer with, or nothing at all.

This page walks you through one Cloudflare WAF custom rule that fixes it. The rule matches Inline's user agent and Inline's published IP addresses, then tells Cloudflare to skip the security features that would otherwise stop the crawler.

It takes about five minutes. You do not need to change anything on your origin server.

Before you start

  • You need access to the Cloudflare account for your domain, with the Firewall, Super Administrator, or Administrator role. If your host or an agency manages Cloudflare for you, send them this page.
  • Custom rules exist on every Cloudflare plan, including Free. Free gives you 5 custom rules, Pro gives you 20, and Business gives you 100. This uses one.
  • Adding Inline to robots.txt does not fix a Cloudflare block. Cloudflare stops the request before your server ever serves robots.txt.

1. Confirm Cloudflare is what blocks us

Do this first. It takes a minute, and it tells you which of the steps below you actually need.

  1. In the Cloudflare dashboard, select your account, then your domain.
  2. Go to Security > Events (newer navigation: Security > Analytics, then the Events tab).
  3. Set the time range to the last 24 hours.
  4. Add a filter: field User Agent, operator contains, value Inline-AI-Crawler.

What you see tells you what to do:

What the events showWhat it means
No events at allCloudflare is not blocking us. Contact Inline support.
Action Block, Managed Challenge, or JS ChallengeCloudflare is blocking us. Continue with this page.
Service Bot Fight ModeGo to step 5 as well. A rule alone will not fix this.
Service Custom rules or User Agent BlockingOne of your own rules blocks us. The rule below overrides it once it sits first.

Inline's crawler also reports what it sees. A Cloudflare block usually reaches us as an HTTP 403 with a Cloudflare page, a challenge page, or error 1010 or 1020. If you email us, include the Ray ID from that page. It lets you find the exact request in your own logs.

2. Create the custom rule

  1. Go to Security > WAF > Custom rules. In Cloudflare's newer navigation, this is Security > Security rules.
  2. Click Create rule, then select Custom rules.
  3. In Rule name, enter Allow Inline AI Crawler.

3. Set the match

Under When incoming requests match, click Edit expression to switch from the visual builder to the expression editor. Paste this in:

(http.user_agent contains "Inline-AI-Crawler" and ip.src in {98.82.101.115 3.232.159.3 3.92.246.252 34.202.184.50 98.91.52.16})

Two things about that expression are deliberate:

  • Match Inline-AI-Crawler, not the full string. Our full user agent is Inline-AI-Crawler/0.1 (+https://getinline.io; support@getinline.io). The version number changes, so a rule that matches the whole string stops working after we upgrade.
  • Keep the IP condition. Those five addresses are the only addresses our crawler fetches from, across production and staging.

Never allow a user agent on its own. Anyone can send the header Inline-AI-Crawler and walk straight through a rule that checks nothing else. The IP list is what makes this rule safe to deploy. The current list is always at https://www.getinline.io/crawler/ips.json. Treat that file as the source of truth, and check it if the crawler stops reaching you.

4. Set the action to Skip

Under Then take action, open the Choose action dropdown and select Skip.

Cloudflare then shows you what the rule can skip. Select these:

SectionSelect
WAF components to skipAll remaining custom rules
WAF components to skipAll managed rules
WAF components to skipAll rate limiting rules
WAF components to skipAll Super Bot Fight Mode rules
Skip products / More components to skipBrowser Integrity Check
Skip products / More components to skipUser Agent Blocking
Skip products / More components to skipSecurity Level

The last three appear in a separate list, because those products do not run on Cloudflare's rules engine. Security Level matters more than it looks: it also covers I'm Under Attack mode, which challenges every visitor, our crawler included.

You do not need Zone Lockdown or Hotlink Protection unless you use them.

Click Deploy.

5. Check the bot settings for your plan

Free plan: you must turn off Bot Fight Mode

Bot Fight Mode is the one thing this rule cannot fix. Cloudflare's own documentation says it plainly: you cannot bypass or skip Bot Fight Mode with WAF custom rules, because it does not run on the rules engine, so Skip has no effect on it.

You have two options:

  • Turn it off. Go to Security > Settings, filter by Bot traffic, and switch Bot fight mode off. Your custom rules, managed rules, and rate limits all keep working, so you are not left unprotected.
  • Upgrade to Pro. Pro replaces it with Super Bot Fight Mode, which the rule you just built does cover.

Pro, Business, Enterprise: Super Bot Fight Mode

Super Bot Fight Mode usually challenges traffic it labels "Definitely automated", which includes our crawler until Cloudflare verifies it. The All Super Bot Fight Mode rules skip you selected in step 4 handles this, as long as the rule sits first. See the next step.

Every plan: check IP Access Rules and legacy Firewall Rules

These run before custom rules, so they can still block us on their own. Go to Security > WAF > Tools and look for any block, challenge, or country rule that covers the five addresses above. Remove it, or set those five addresses to Allow.

6. Move the rule to the top

Go back to Custom rules and drag Allow Inline AI Crawler to position 1.

Custom rules run in order, and the first match wins. A skip rule below a block rule does nothing.

7. Confirm it worked

Wait a few minutes, then go back to Security > Events and apply the same Inline-AI-Crawler filter you used in step 1.

You want to see either no new blocks, or events whose service is your new rule and whose action is Skip. Any remaining Block or Challenge event names the feature that still stops us, so check that feature against step 5.

You can also send a test request from your own machine:

curl -sSI -A "Inline-AI-Crawler/0.1 (+https://getinline.io; support@getinline.io)" https://yourdomain.com/

A 403 from that curl does not mean the rule failed. The request leaves your IP address, not ours, so the IP half of the expression cannot match and the skip correctly does not apply. Cloudflare's Security Events log is the real test.

8. Tell us, so we can re-crawl

Email support@getinline.io once the rule is live. We re-queue your site right away instead of waiting for the next scheduled crawl, and we confirm from our side that the pages now come back 200.

Still blocked?

Send us:

  • Your domain.
  • The Ray ID and timestamp of a blocked request, from Security > Events.
  • Your Cloudflare plan.

We can tell you exactly what our crawler received, which usually names the feature that is still in the way.

An alternative, soon: verified bots

Inline's crawler signs every request with Web Bot Auth, the HTTP Message Signatures scheme defined in RFC 9421. Cloudflare can verify that signature itself, without any IP list.

Once Inline is listed in Cloudflare's Verified Bots directory, you will be able to allow us as a verified bot and drop the IP condition. Until then, use the rule on this page. Nothing about it breaks when that happens.

  • Inline AI Crawler: the user agent, the IP list, the Web Bot Auth signature, and how the crawler treats robots.txt.