Crawler documentation

GeoWebBot

GeoWebBot is the crawler GeoWeb (geoweb.tw) uses to analyze websites. If it shows up in your server logs, someone analyzed your site with geoweb.tw.

Last updated: 2026-11-15

When it visits

GeoWebBot only visits a site when someone analyzes it with geoweb.tw. That includes audits run by our own team. Unlike a search engine crawler, it doesn't roam the web; it only fetches the site being analyzed.

The content it fetches is used only to produce GEO audit reports and aggregate statistics for our research. It is never used to train AI models.

How to identify it

GeoWebBot sends one of these two User-Agent strings:

GeoWebBot/1.0 (+https://geoweb.tw/bot)

Used to fetch pages, robots.txt and sitemaps.

GeoWebBot/1.0 (+https://geoweb.tw/bot; agentic-web-check)

Used for the Agent Friendliness check.

Some pages only show their content after JavaScript runs. For those, GeoWebBot loads the page again in a headless browser (Chromium), and it does the same once if the first request gets a 401 or 403. It still sends the first User-Agent above; the page's own images, CSS and JavaScript load the way they would in a normal browser.

Verifying it's really GeoWebBot

Anyone can fake a User-Agent, so the requests GeoWebBot sends to your site also carry a Web Bot Auth signature (HTTP Message Signatures, RFC 9421). The signature's Signature-Agent is "https://geoweb.tw", and the matching public key is published at:

https://geoweb.tw/.well-known/http-message-signatures-directory

Bot protection services that support Web Bot Auth can use this key to verify that a request really comes from GeoWebBot.

What it fetches

GeoWebBot never logs in or submits forms.

Crawl rate

When reading pages, GeoWebBot sends at most 2 concurrent requests to your site and pauses 0.5 seconds after each page. The Agent Friendliness check mostly reads small text files, with up to 4 concurrent requests.

Blocking or limiting it with robots.txt

GeoWebBot respects robots.txt. To opt out entirely, add this to your robots.txt:

User-agent: GeoWebBot
Disallow: /

Allow-listing it in your firewall

If you want to analyze your own site with geoweb.tw but a firewall, CDN or your host's bot protection blocks GeoWebBot, the analysis will fail, or GeoWebBot will only see a challenge page and won't be able to score your site properly. In whichever of these is blocking it, add an allow-list rule for requests whose User-Agent contains GeoWebBot. If your bot protection supports Web Bot Auth, verifying the signature is more reliable than matching the User-Agent.

Contact

Questions about GeoWebBot? Email contact@geoweb.tw.