The card network Visa has set AI an exam on how well it shops for people: 22 AI models took on simulated shopping scenarios, buying things like TVs and laptops for a customer, to see whether they could first find out what the customer wanted and then decide whether to buy. Visa Taiwan announced the first-round results in a press release on 12 November, and according to a report that day by Taiwan’s Central News Agency (CNA), the average pass rate was just 56.6%; among the tests that the AI failed, about 53% ended with the AI still deciding to buy. So letting AI do the shopping is something that a major payments company is already testing question by question, and what the tests show is that AI often fails, and in about half of those failures it still goes ahead and buys. Our position is this: when AI chooses and decides for a customer, it can only go on the information that it reads, and the vaguer the information about you, the more likely it is to get you wrong, so that information needs to be clear enough to leave no room for misunderstanding, and the work needs to start now.
What happened
The benchmark is called the Visa Agentic Commerce Benchmark. Visa first launched it in the US on 5 November, and in Chinese it goes by a name that translates as “AI smart shopping benchmark”. It works like this: customers often can’t say exactly what they want at the outset, so the AI has to draw out their budget and needs through conversation, buy only if it finds a product that fits, and not buy at all if nothing does. Visa’s example is choosing a TV: the customer mentions only that a relative has recommended a brand and that they hold a membership with a third-party supplier, and leaves out the other details, so the AI has to ask about the budget on its own, along with whether certain picture formats need to be supported and whether the TV is up to gaming.
Besides the low average, the results were inconsistent: each AI took every scenario three times, and a model that passed a scenario one time often failed when it took the same one again. The failed runs that still ended in a purchase were either recommendations that didn’t meet the customer’s requirements, or purchases made before all of the customer’s conditions had been found out. Visa said in the press release that the key to consumers trusting AI to shop for them is fewer errors and better consistency, and that both still have room for improvement; Visa also says that it will extend the benchmark to checkout, payment and placing orders.
The benchmark was run and published by Visa itself, and Visa has its own range of products and services designed for AI that shops on people’s behalf. Also, a few of the figures in the English press release differ slightly from the ones in the Taiwan release; neither the English release nor the Taiwan coverage of the Chinese release explains why, and this article goes by Visa Taiwan’s Chinese version.
Why it matters to you
What Visa tested is the AI’s own ability: whether it asks the right questions and whether it should buy. Improving that is down to the companies that build AI, and it’s out of your hands. But when AI makes a decision, the other half of what it checks the customer’s conditions against is the information that it reads about products and businesses, and the part of that half on your own website is in your hands.
Say, for example, that you run a shop in Gangshan, Kaohsiung, that sells water purifiers, and you have your own website. A customer tells AI: “There’s no power socket under my kitchen sink, so I’m looking for a water purifier that doesn’t need to be plugged in, ideally one where I can change the filter myself.” This customer has spelled out the conditions clearly, so for the AI to choose for them, it next needs to know which of your models don’t need power and whether changing the filter takes a technician. You could answer all of this with your eyes closed, yet your website may only list model numbers and prices.
If the AI can’t find the answers, it has nothing to go on in judging whether your models fit, and at that point, whether it counts you in or rules you out, it isn’t going on what your models actually do. If it wrongly decides that you don’t fit, the sale goes to someone else; if it wrongly decides that you do, it may pick one of your models that actually needs power, the customer buys it on that advice and then can’t install it at home, and the returns and complaints land on your shop.
What to do about it
You need to act, and it’s precisely because AI still gets these decisions wrong that your information needs to be made clear now: the clearer it is, the less is left for the AI to guess. Customers can already use this. On 9 November, Google launched Chinese-language shopping in Gemini in Taiwan: customers describe what they need, and it recommends products and lists prices, reviews and where to buy them. What you need to do is set out in writing on your website which conditions each product and service meets and which it doesn’t, and make sure that the product pages on shopping platforms and the articles that other people have written about you match your website.
Yet when AI gets you wrong, a single answer won’t tell you which side the problem is on: whether the AI failed to find out what the customer needed, or whether what it read about you wasn’t clear enough. Only the second is something that you can fix, and if you start changing your website without knowing which it is, you could spend a long time on changes and never touch the spot that the AI misread. Telling the two apart means checking the AI’s answers line by line against what each of your products actually does, and asking with the same set of conditions over several rounds, because in the benchmark, a model that passed a scenario often failed when it took the same one again, so asking once won’t show whether it answers that way every time.
To see how AI reads you when it’s choosing for a customer, you can set up a demo with one of our consultants. The consultant will give ChatGPT or Gemini the kind of conditions that customers in your line of business set, like “no socket, has to work without power” above, and have the AI choose for a customer while you watch: who it picks, whether you’re among them and whether it gets your products right are all there on the screen. The same session also brings up your website’s GEO audit score. GEO means making sure that when AI chooses and recommends for customers, it counts you in and describes you correctly; the GEO audit is a free website check on geoweb.tw that scores, across 12 dimensions, whether AI can make sense of your website.
We take on the long-term work that follows through Managed GEO: we rewrite your website so that it states clearly which conditions each product meets; for shopping platforms and other websites that mention you, we list what needs changing and hand it to you; and at regular intervals we put the conditions that customers set to ChatGPT, Gemini and other leading AI engines, keep each set of answers and check them against what each of your products actually does, to see whether the AI is reading you correctly when it chooses.
Further reading
- Big companies are laying the rails for AI agents to pay — Visa’s other track in Taiwan: tests of letting AI agents pay on people’s behalf, with 9 card-issuing banks in Taiwan taking part
- Gemini launches Chinese-language shopping in Taiwan — customers in Taiwan can already tell Gemini their budget and needs in Chinese and have it pick products and build a comparison table
- AI shopping agents drop brands whose product data doesn’t add up — which facts AI agents compare when they shop around for customers, and where they read them from