Crawler documentation
GeoWebBot
GeoWebBot is the crawler GeoWeb (geoweb.tw) uses to analyze websites. If it shows up in your server logs, someone analyzed your site with geoweb.tw.
Last updated: 2026-11-15
When it visits
GeoWebBot only visits a site when someone analyzes it with geoweb.tw. That includes audits run by our own team. Unlike a search engine crawler, it doesn't roam the web; it only fetches the site being analyzed.
The content it fetches is used only to produce GEO audit reports and aggregate statistics for our research. It is never used to train AI models.
How to identify it
GeoWebBot sends one of these two User-Agent strings:
GeoWebBot/1.0 (+https://geoweb.tw/bot)
Used to fetch pages, robots.txt and sitemaps.
GeoWebBot/1.0 (+https://geoweb.tw/bot; agentic-web-check)
Used for the Agent Friendliness check.
Some pages only show their content after JavaScript runs. For those, GeoWebBot loads the page again in a headless browser (Chromium), and it does the same once if the first request gets a 401 or 403. It still sends the first User-Agent above; the page's own images, CSS and JavaScript load the way they would in a normal browser.
Verifying it's really GeoWebBot
Anyone can fake a User-Agent, so the requests GeoWebBot sends to your site also carry a Web Bot Auth signature (HTTP Message Signatures, RFC 9421). The signature's Signature-Agent is "https://geoweb.tw", and the matching public key is published at:
https://geoweb.tw/.well-known/http-message-signatures-directory
Bot protection services that support Web Bot Auth can use this key to verify that a request really comes from GeoWebBot.
What it fetches
- The page someone asked to analyze, plus robots.txt, sitemaps and the IndexNow key file.
- For a single-page analysis, up to 20 other pages on the same site.
- For a site-wide analysis, up to 300 pages on the same site.
- For the Agent Friendliness check, a few public files at fixed locations (such as
llms.txtand the files under/.well-known/), a few common pages (such as the about, contact, privacy and terms pages), and one URL that deliberately doesn't exist, to check whether the site correctly returns a 404.
GeoWebBot never logs in or submits forms.
Crawl rate
When reading pages, GeoWebBot sends at most 2 concurrent requests to your site and pauses 0.5 seconds after each page. The Agent Friendliness check mostly reads small text files, with up to 4 concurrent requests.
Blocking or limiting it with robots.txt
GeoWebBot respects robots.txt. To opt out entirely, add this to your robots.txt:
User-agent: GeoWebBot
Disallow: /
- If robots.txt doesn't allow GeoWebBot to fetch the page being analyzed, the analysis stops there and nothing else on your site is fetched.
- You can also block specific paths, for example by adding
Disallow: /members/underUser-agent: GeoWebBot. When fetching those other pages for a single-page analysis, or during a site-wide analysis, GeoWebBot checks robots.txt page by page and skips anything disallowed. The fixed files and pages read by the Agent Friendliness check are not covered by this per-page check. - GeoWebBot caches robots.txt for up to an hour, so a change can take up to an hour to take effect.
- If robots.txt can't be read (it doesn't exist or the request fails), GeoWebBot treats the site as having no restrictions.
Allow-listing it in your firewall
If you want to analyze your own site with geoweb.tw but a firewall, CDN or your host's bot protection blocks GeoWebBot, the analysis will fail, or GeoWebBot will only see a challenge page and won't be able to score your site properly. In whichever of these is blocking it, add an allow-list rule for requests whose User-Agent contains GeoWebBot. If your bot protection supports Web Bot Auth, verifying the signature is more reliable than matching the User-Agent.
Contact
Questions about GeoWebBot? Email contact@geoweb.tw.