LinkupBot

LinkupBot is Linkup’s web crawler. It discovers and retrieves publicly accessible webpages to build and maintain the Linkup web search index, which powers search and web grounding for AI applications.

LinkupBot identifies itself in its HTTP User-Agent header and respects applicable robots.txt rules.

Identifying LinkupBot

Requests from LinkupBot contain the following user-agent token:

LinkupBot

An example complete user-agent is:

LinkupBot/1.0 (LinkupBot for web indexing; https://www.linkup.so/bot; bot@linkup.so)

LinkupBot requests originate from the CIDR ranges published at: https://www.linkup.so/linkupbot-ips.txt

Because user-agent headers can be spoofed, verify LinkupBot traffic using both:

• A source IP contained in the published CIDR list.
• A user-agent containing the LinkupBot token.

The IP list may change. Systems that automate allowlisting should retrieve it periodically.

Controlling LinkupBot with robots.txt

Place robots.txt at the root of each host you want to control, for example:

https://example.com/robots.txt

Rules apply only to the corresponding host, protocol, and port. Separate subdomains may need their own robots.txt.

Block the entire site

User-agent: LinkupBot
Disallow: /

Block specific content

User-agent: LinkupBot
Disallow: /private/
Disallow: /drafts/
Disallow: /internal-document.pdf

Block everything except selected content

User-agent: LinkupBot
Disallow: /
Allow: /public/

Explicitly allow LinkupBot

Crawling is allowed by default when no applicable rule prohibits it. To allow LinkupBot while blocking other crawlers:

User-agent: *
Disallow: /
User-agent: LinkupBot
Allow: /

Allowing LinkupBot through a firewall or WAF does not override robots.txt. Both the network policy and the applicable robots rules must permit access.

robots.txt is not an access-control mechanism. Protect confidential content with authentication or other server-side controls.

Allowlisting LinkupBot

To allow LinkupBot through a firewall, CDN, or WAF:

1. Allow source addresses from linkupbot-ips.txt.
2. Require the user-agent to contain LinkupBot.
3. Refresh the CIDR list periodically.
4. Avoid matching the complete user-agent string, as its version may change.

For the strongest verification, require both the published source IP and user-agent conditions.

Contact

Questions or concerns about LinkupBot can be sent to bot@linkup.so.

Please include the requested URL, timestamp and timezone, source IP, complete user-agent, and relevant server-log entries.