Visa launched the Visa Agentic Commerce Benchmark on 5 November 2026, testing 22 AI models from 12 labs on 15 shopping tasks, and the models failed 42.8% of runs, according to Visa’s announcement. Fintechbits reported on 23 October that Visa, BILL and Billtrust were already piloting AI agents that pay approved supplier invoices, so Visa is grading AI shopping agents while it puts agents in charge of real payments. In 54% of failed runs the AI still made a purchase.

What the Visa Agentic Commerce Benchmark measured

Each model had to work out a shopper’s needs through conversation, then buy or decline. In one scenario a customer wants a TV, mentions a brand a relative recommended and a membership with a third-party supplier, and leaves the budget and technical needs unsaid. To pass, the model has to find every requirement the task defines and make the right purchase or no-purchase decision, which Visa says no earlier agentic commerce benchmark required. Across the Visa Agentic Commerce Benchmark the overall pass rate was 57.2%, ranging from 82.2% for the best models to 28.9% for the worst. Consistency was weak: every model got three attempts per scenario, and all three passed only 34.2% of the time. The five most expensive models passed 67.1% of runs against 50.2% for the five cheapest.

The buy-anyway figure is the one that matters for payments. An agent that fails and stops costs nothing. An agent that fails and buys creates a refund, a dispute or a chargeback, and Fintechbits has argued that the liability rules for agentic payments are not ready for that. Visa says it will extend the benchmark to checkout, payment and order placement, which is where its own network carries the risk. Visa’s September Trust Index found 72% of US consumers had used an AI assistant but only 23% trusted AI to handle payments for them. The Visa Agentic Commerce Benchmark results give those consumers little reason to change their minds.