Inline AI Crawler
How Inline's crawler identifies itself: user agent, published IP addresses, Web Bot Auth signature, and robots.txt behaviour.
What the crawler does
Inline's crawler ingests a publisher's own content via direct integration, sitemap, RSS feed, or submitted URLs. Once a publisher has allowlisted Inline AI's crawler user agent, it only fetches pages that are explicitly permitted. That indexed content is used solely to ground the assistant's answers to readers on the publisher's own site, not to train models or build a cross-publisher corpus. More information can be found on Inline AI's terms of service and privacy policy.
User agent
Every request the crawler makes carries this user agent string:
Inline-AI-Crawler/0.1 (+https://getinline.io; support@getinline.io)The product token to match in robots.txt is Inline-AI-Crawler. To allow the crawler:
User-agent: Inline-AI-Crawler
Allow: /IP addresses
The crawler makes all of its requests from these addresses:
| Address | Environment |
|---|---|
98.82.101.115 | Production |
3.232.159.3 | Production |
3.92.246.252 | Production |
34.202.184.50 | Staging |
98.91.52.16 | Staging |
Both environments are listed. Our staging crawler sends the same user agent, so treat every address above as ours.
A machine-readable copy of this list is published at https://www.getinline.io/crawler/ips.json. We keep that file current, so treat it as the source of truth rather than copying the addresses by hand.
Web Bot Auth signature
The crawler signs its requests with Web Bot Auth, the HTTP Message Signatures scheme defined in RFC 9421. Signed requests carry three extra headers:
| Header | Value |
|---|---|
Signature-Agent | "https://www.getinline.io", the URL of our key directory, as a quoted string. |
Signature-Input | The covered components ("@authority" "signature-agent"), our key identifier, alg="ed25519", the created and expires timestamps, and tag="web-bot-auth". |
Signature | The Ed25519 signature itself. |
Our Ed25519 public keys are published as a JSON Web Key Set at:
https://www.getinline.io/.well-known/http-message-signatures-directoryThe directory response is itself signed, so a mirror of it cannot pass as ours. To verify a request:
- Fetch the key directory.
- Select the key whose RFC 7638 thumbprint matches the
keyidinSignature-Input. - Check the signature over the request authority.
How the crawler treats robots.txt
The crawler ingests content from two kinds of source, and it treats them differently. We state both here so that our disclosure matches our traffic.
Publisher-supplied sources
A sitemap, an RSS feed, a URL list, or a page submitted from the publisher dashboard is a direct instruction from the site owner to ingest that page. The crawler fetches those URLs. A publisher who wants a page excluded removes it from the source they gave us, or asks us to remove it.
Automatically discovered sources
When the crawler discovers a URL on its own rather than receiving it from the publisher, it reads
that host's robots.txt first and skips any URL that is disallowed for Inline-AI-Crawler. It
caches robots.txt for at most six hours, so a new Disallow rule takes effect quickly.
Request behaviour
- The crawler fetches only pages on domains that the publisher has registered with us.
- It fetches over HTTP and HTTPS only, follows at most five redirects, and stops reading a response at 32 MiB.
- It fetches on a schedule set by the publisher, and it does not crawl a site continuously.
- It does not attempt to bypass a paywall, a login, or a bot challenge.
Blocking the crawler
Add a Disallow rule for Inline-AI-Crawler in robots.txt, or email us. If you are an Inline
publisher, blocking the crawler stops the assistant on your site from answering with your content.
Contact
For any question about this crawler, including a request to slow it down or stop it, email support@getinline.io. We answer crawler reports first.