Why do you need free datasets?
To assess the external-perception strength of a website (your own, a competitor’s, or a partner’s), paid tools like Ahrefs / SimilarWeb / Semrush start at $129+ per month and come with strict ToS restrictions.
But academia and the open-data world actually offer several high-quality open datasets whose content overlaps heavily with the core signals of paid tools — they just don’t come wrapped in a pretty dashboard. This article introduces the three most practical ones, and how to combine them to make an assessment.
1. Tranco — Global Top 1M domain ranking (updated daily)
What it is
Tranco is a research project from KU Leuven and Stanford that combines 4 sources to compute a ranking of the world’s most popular websites:
- Cisco Umbrella (DNS query volume, anonymized statistics from millions of enterprise endpoints)
- Cloudflare Radar
- Chrome User Experience Report (CrUX)
- Majestic Million
All four sources are updated daily, avoiding the bias of any single source (this is the most commonly used successor since Alexa rank was retired in 2022).
What it can tell you
“What rank does this website hold within the global Top 1M?”
The higher the rank = the larger the DNS query volume = the higher the real traffic. This is more credible than SimilarWeb’s “estimated traffic” — because it reflects real DNS behavior, not a statistical estimate.
How to obtain it
https://tranco-list.eu/api/lists/date/today
Or just download the daily CSV (~30 MB, containing 1M entries). Licensed under CC BY 4.0, with no ToS restrictions.
How to interpret it
- Rank < 1,000: Top-tier global websites (Wikipedia, Google, Reddit, the major platforms)
- Rank 1K–10K: Industry leaders
- Rank 10K–100K: Well-known brands or active communities
- Rank 100K–1M: The lower edge of what AI training corpora can cover
- Not in the 1M: “Nonexistent” as far as LLM training corpora are concerned
Limitations
- Only the Top 1M. New sites / small sites are completely invisible
- The rank is a relative position, not an absolute traffic figure
- No categorization (you can’t tell “well-known vs. academic vs. government”)
2. Common Crawl — The source of LLM training corpora
What it is
Common Crawl is an open web archive that has been continuously crawled monthly from 2007 to today. It adds roughly 2.5 billion pages each month, all published in WARC format on AWS S3.
ChatGPT, the major AI labs, Google, and Meta all use Common Crawl as one of the primary corpus sources when training their LLMs. In other words: a web page that makes it into Common Crawl = an entry ticket into LLM training data.
What it can tell you
“How many pages of this website have been indexed by the crawler? How many other websites link to it?”
How to obtain it
Two main query methods:
1. CDX API (single queries, free)
https://index.commoncrawl.org/CC-MAIN-2026-01-index?url=geoweb.tw&output=json
Returns all URLs and timestamps for that website indexed in that month’s crawl.
2. cc-webgraph derived data (aggregate analysis)
https://commoncrawl.org/web-graphs
Download the domain-level webgraph released each quarter (compressed ~1.5 GB), which includes, for each domain:
- Harmonic centrality (≈ Domain Rating)
- In-degree (= number of Referring Domains)
- Out-degree (= how many other sites you link to)
Download it once and query entirely locally — no API call limits.
How to interpret it
- Indexed page count < 100: The website contributes very little to LLM training corpora
- 100–10K: A typical small-to-mid-sized website; some content makes it into the training pool
- 10K+: Large amounts of content enter the training corpus, and AI’s “implicit knowledge” baseline about your brand is solid
Limitations
- Monthly + quarterly updates, not suitable for tracking real-time changes
- WARC files are enormous and require streaming processing
- The crawl timing of individual pages is uncontrollable (you can’t ask Common Crawl to come crawl your site)
3. Wayback Machine — Credible evidence of a website’s age
What it is
The web archive that the Internet Archive has been storing from 1996 to today. Every archived web page’s timestamp is an externally verifiable time anchor.
When LLMs judge a website’s trustworthiness, they consider “how long has this website existed” — but the founding date a website claims for itself can be falsified. The Wayback Machine’s first-archived time is a third-party time anchor that AI vendors recognize.
What it can tell you
“When was this website first archived? And when was the most recent archive?”
How to obtain it
The most useful is the CDX API:
https://web.archive.org/cdx/search/cdx?url=geoweb.tw&limit=1&from=19960101&output=json&filter=statuscode:200
Returns the timestamp of the domain’s first archive with a 200 status.
GeoWeb’s own “off-site visibility” dimension uses this API (it originally used archive.org/wayback/available, but that returns empty for popular sites; switching to CDX stabilized it).
How to interpret it
- First archive < 3 years ago: A new site whose external perception is still accumulating
- 3–10 years: A stable brand; LLMs have a basic awareness of you
- 10+ years: An established website; the LLM training-corpus baseline runs deep
Limitations
- It only reflects “having been archived,” not “actual content quality”
- The Wayback Machine tends to archive popular / well-liked websites more frequently, creating a slight bias
- Some websites actively block Wayback archiving (via robots.txt’s
User-agent: ia_archiver)
How do you use the three signals together?
Reading a single signal in isolation is prone to misjudgment; only cross-referencing all three yields a conclusion:
Scenario A: High Tranco + High Common Crawl + High Wayback
A genuine authority brand. High probability of AI citation, with no special GEO effort needed (but you still need structure).
Scenario B: Low Tranco + High Common Crawl + High Wayback
An established website whose traffic has declined. It needs its content-update cadence reactivated.
Scenario C: High Tranco + Low Common Crawl + Low Wayback
A newly viral website. The AI training corpus hasn’t caught up yet; you’ll need to wait 1–2 months / a quarterly update cycle. In the short term, rely on real-time citation crawlers (ChatGPT-User, PerplexityBot).
Scenario D: Low Tranco + Low Common Crawl + Low Wayback
An extremely low starting point. It requires comprehensive GEO planning + long-term accumulation through media and PR.
How to obtain these signals quickly
GeoWeb’s “off-site visibility” dimension automatically integrates the above three + the DuckDuckGo knowledge graph + AI platform backtest; run a single health check and you’ll see your own website’s combined assessment across these 5 signals.
If you want deeper competitor analysis (a full-signal comparison of your site vs. 5–10 peers + cross-quarter trend tracking), that part requires custom tooling and long-term data collection — it falls within the scope of GEO consulting services: [email protected]
GEO advanced series. Previous article: “Why is Wikipedia inclusion one of the strongest signals for GEO?”