LimitFactsBot

By LimitFacts. Last updated: 2026-10-18. Editorial policy.

LimitFactsBot is the research crawler for LimitFacts. It collects prices and plan limits from official vendor pricing pages for our comparison pages. The numbers on this page are our own operating rules. They are not vendor data. This page describes what the crawler code does.

Crawler code hash 426abbe3

User agent

LimitFactsBot always sends this user agent:

Mozilla/5.0 (compatible; LimitFactsBot/1.0; +https://limitfacts.com/bot)

It sends the same user agent for robots.txt, for plain requests, in its headless browser and for every request that the browser makes.

Which sites and pages

LimitFactsBot requests pages only on these 9 domains and their subdomains:

• surferseo.com
• hubspot.com (this includes legal.hubspot.com)
• elevenlabs.io
• kinsta.com
• digitalocean.com
• hostinger.com
• hosting.com
• seranking.com
• liquidweb.com

It refuses every other domain before it sends a request.

• It requests only the pricing pages on our fixed list. The list has at most 5 pages for each domain.
• It runs each listed page at most once per day (Dubai time). One run is one plain request and, only if needed, one headless browser load of the same page. The only exception is one plain retry after a server error (see below).
• It also reads robots.txt on each host that it visits.
• It uses HTTPS only. It does not crawl whole sites and it does not follow links from page to page.

Plain request first, headless browser only when needed

• LimitFactsBot first requests the page as plain HTML. It does not run scripts for this request.
• It uses a headless browser only when the plain request returned 200, the page is not a bot challenge, and the plain HTML shows fewer than two prices. Our check looks for a currency sign or code next to a number.
• HubSpot pages are fetched as static pages only. LimitFactsBot never uses the headless browser on hubspot.com or its subdomains.

When a site says no or has a problem

• A challenge page, captcha, or a 401, 403 or 429 response parks a domain until a person reviews it.
• While a domain is parked, LimitFactsBot sends no request of any kind to it, robots.txt included. The park does not end with time. Only a person removes it, after a review.
• This applies to pages, robots.txt, redirect steps and browser requests. A challenge page is a response that matches our list of bot challenge and CAPTCHA markers, with any status code.
• On a server error (5xx) for a page, LimitFactsBot does not use a browser. It does not request the other pages on that site for the rest of the day. It tries the same page one more time at most, as a plain request, after at least one hour. Then it skips the site until the next day.
• If robots.txt cannot be read because of a server error or a network error, LimitFactsBot fetches nothing from the site for the rest of the day.
• A site is one domain with all its subdomains together.

Redirects

• LimitFactsBot never follows redirects automatically. It follows them one step at a time, up to 5 steps.
• Each step goes through all the checks on this page before it is requested: domain list, URL rules, robots.txt, the delay and the daily limit.
• It refuses a redirect to another domain.

Inside the headless browser

• Each page loads in a new, empty browser session.
• One filter checks every request before it is sent.
• It sends only GET requests.
• The filter blocks all requests to other domains.
• The filter blocks requests that match our list of trackers, analytics, ads, tag managers, pixels, affiliate scripts, chat widgets and consent tools, also when they come from the vendor's own domain.
• The filter blocks all images, fonts, media and websockets. Service workers are turned off.
• Only these first-party requests can load: the page itself, scripts, stylesheets and data requests from the page.
• Each of these requests is checked against robots.txt and the URL rules, waits for the delay, and counts toward the daily limit.

Cookies and headers

• LimitFactsBot sends no cookies and does not keep cookies, in plain requests and in the browser.
• The browser sends no client hint headers.

How it uses robots.txt

• It reads robots.txt before the first request to each host. It keeps a copy for at most 24 hours, then reads it again.
• It reads robots.txt under the robots.txt standard (RFC 9309), including the wildcard characters * and $. The longest matching rule wins, and Allow wins a tie.
• It follows the group for LimitFactsBot. If there is no such group, it follows the group for all user agents.
• If robots.txt sets a Crawl-delay longer than the default delay, LimitFactsBot waits that long between requests.
• If robots.txt answers another client error, for example not found (404), LimitFactsBot treats the host as allowed.

Rate limits

• At least 3 seconds between requests to the same site, or longer if Crawl-delay asks for it.
• At most 50 requests to the same site per day (Dubai time). This includes robots.txt reads, redirect steps and browser requests.
• All subdomains of a site share these limits.

What it does not do

• It does not log in and it does not create accounts.
• It does not fill in or send forms.
• It does not click links, buttons or anything else on a page.
• It refuses URLs that match our list of affiliate, referral, coupon, tracking, signup, login, checkout, cart, billing and account patterns. The crawler code enforces this.
• This check runs on every request, also on robots.txt, scripts, data requests and each redirect step.
• It does not try to get around robots.txt, rate limits or other access controls.

Logs

LimitFactsBot writes a log line with a reason for every request, every blocked request and every decision.

How to opt out

To stop LimitFactsBot from visiting your site, add these lines to your robots.txt:

• User-agent: LimitFactsBot
• Disallow: /

LimitFactsBot reads the new rule the next time its copy of your robots.txt expires. To block only some pages, disallow those paths.

Contact

Write to hello@limitfacts.com if LimitFactsBot causes a problem on your site, or if you want us to correct data about your product.