What happened

If you use a Facebook Page or Instagram for your business, then once Meta’s updated Commercial Terms take effect, you are in effect agreeing to let Meta send software to read your website. That’s a new passage in section 1 of the updated terms. Meta is the company behind Facebook and Instagram, and every business that uses its products for business has to follow these terms.

The terms were due to take effect on 30 October. According to Reuters on 23 October, Meta has pushed that back to 20 November, saying it wants more time to explain to partners what is changing and what isn’t; before that, publishers and advertisers had complained about the terms to EU and UK antitrust authorities. It’s the date that has moved: the report doesn’t say the wording has changed. The official Traditional Chinese preview still gave 30 October on 24 October; when we checked again on 30 October, it showed 20 November, and the crawling passage was unchanged.

The passage says you authorise Meta to use automated software such as crawlers to access, analyse and scrape websites and pages related to your products or services, and to come back regularly for the latest information. A crawler is a program that automatically reads its way around the web, page by page. In the Traditional Chinese version we read on 24 October, the stated purpose is to “provide, develop, improve, research and maintain advertising and business products”, such as helping you advertise and telling users about your products and services.

Who does this cover? The business uses the terms list include running ads and listing products, and also managing a Page, so businesses that have never bought an ad and only run a Page are covered too. If you don’t agree, the route the terms offer is to stop using Meta’s products for business; INSIDE, a Taiwanese tech publication, also noted that as of 19 October, business accounts had no way to opt out in advance.

Whether any of this goes towards training AI is unsettled. According to Search Engine Journal, a US search industry publication, the first version listed “train” among the purposes; after criticism, Meta changed the wording in mid-October, and the Traditional Chinese version we read no longer has that word. Meta says the passage only applies to its advertising and business products. But according to INSIDE’s summary, some legal observers think that if Meta launches AI business services, the word “develop”, which stayed in, could still cover training. The wording could also change again before the terms take effect.

Why it matters to you

Say you run a bike shop in Douliu, in Yunlin county. Your Facebook Page posts photos of group rides every week, and your website has your bikes, prices and servicing plans, plus write-ups of cycling routes around Yunlin that you’ve ridden yourself. Once the terms take effect, they allow Meta to come and read those pages regularly. Taken at face value, what’s on your website could become material Meta later uses to tell users about you. Being allowed to isn’t the same as doing it, though: we haven’t found any public information on whether Meta will actually come, or which pages it would read.

When you hear Meta is going to crawl, your first instinct may be to block the lot. The way a website tells crawlers what it wants is a small file called robots.txt, a bit like a notice on the shop door saying which crawlers may come in. The new terms don’t mention this notice; according to Search Engine Journal, Meta says website owners can still use it to manage Meta’s crawlers. You can’t pick and choose within the terms, but the notice on your website’s door is still yours to decide.

But Meta doesn’t send just one crawler. The documentation it publishes for website owners lists five, each doing its own job. These three matter most to a bike shop:

  • One gathers information for Meta AI. Meta AI is the AI assistant you can ask questions inside Facebook, Instagram and Messenger; Meta says allowing this crawler helps Meta AI cite you and link to you in its answers.
  • One, when someone shares a link to your page on Facebook, Instagram or Messenger, fetches the page’s title and thumbnail to build the preview card people see.
  • One is used for training AI models, and also indexes content directly to improve Meta’s products.

Going by that division of labour, block everything and you may lose both the chance of Meta AI linking to you and the photo preview card that appears when a customer sends your bike page to a friend on Messenger; allow everything and the cycling routes you put time into writing may be used to train Meta’s AI models; going by Meta’s documentation, the crawler that helps cite you and link to you in answers is the one that works for Meta AI, and the training crawler isn’t described that way. These crawlers can be set separately; which to let in and which to keep out depends on what matters more to you.

The notice isn’t all-powerful, either. Meta’s documentation says that in two situations its crawlers may not follow the notice: when a user asks Meta’s AI to open one of your pages, and when the preview-card crawler is running security checks. In other words, even if your notice says keep out, Meta’s crawlers may still come in those two cases.

What to do now

Act now. The later date gives you three more weeks; use them to find out whether your website currently lets each of these crawlers in or keeps it out. You can start by asking whoever built your website, or hand it straight to us. When you ask, you can put it like this: are Meta’s crawlers Meta-WebIndexer, FacebookExternalHit and Meta-ExternalAgent each allowed or blocked on our website right now? Plenty of owners don’t know what their notice says. You won’t see the file when you browse your own site; if your website is built on an e-commerce platform, or sits behind a protection service such as Cloudflare, that service may have its own crawler-blocking switches, and some block a share of AI crawlers by default. Even once it’s set up, which crawlers actually came and whether they followed your settings is only recorded in your web server’s own access logs, which owners don’t normally look at.

Making the trade-offs and changing the settings is something we can take on. GEO stands for generative engine optimisation; it deals with whether AI tools that write their own answers, such as ChatGPT and Meta AI, can read your website and how they describe you, and which of Meta’s crawlers to let in or keep out is part of that. With our managed GEO service, our team fixes your site’s content, brings what’s said about you elsewhere into line, and regularly checks how each AI engine describes you; you just review the results.

You let in the crawler that works for Meta AI so that AI can bring you up when it answers customers. How Meta AI will describe you from your website is hard to say yet; ChatGPT may also go online and read businesses’ websites before answering customers, and that side you can see right now. To see it for yourself, line up a demo: a session with one of our consultants where they show you live. The consultant takes something a customer would ask, such as “Where in Douliu can I buy an entry-level road bike, and can I test-ride one first?”, puts it to ChatGPT there and then, and checks whether you’re in the answer and whether what it says about your bikes and prices matches your website. In the same session, the consultant will also show you your website’s GEO audit score. The GEO audit is geoweb.tw’s free website audit, which checks how well AI can understand your site and scores it across 12 areas.

Further reading