A study by the technology company Saturn revealed that popular AI chatbots often make mistakes when dealing with financial questions. The testing involved ChatGPT, Claude, Copilot, Grok, and Gemini: a total of 18 models received more than 10,000 requests, each of which could be repeated up to five times if necessary. Researchers prepared over 100 finance-related questions and used both free and paid versions of the services. On average, the models gave incorrect answers in 57% of cases. When tasks became more complex and required several consecutive calculations, the error rate increased to 88%. In such scenarios, some models reached a 99% error rate. The most accurate result was shown by Claude Opus 5 in reasoning mode, yet it still made mistakes in 39% of answers. Paid versions performed better than free ones, and newer models were more accurate than older ones.
The authors identified several types of problems: arithmetic errors, ignoring future changes in tax legislation, references to non-existent regulations, incorrect answers to multi-step requests. One example was a request to the free version of Claude Haiku 4.5 about the taxation of pension savings: according to researchers' calculations, the model's error could potentially lead to an additional HMRC charge of £17,500, or about $23,430. In another case, Claude formulated a non-existent rule about student loans and stated that a borrower could stop payments after moving abroad.
"Millions of people trust AI models for financial advice but receive incorrect answers that could cause them to lose money," said Saturn CEO Amal Jolly.
Sarah Coles, head of personal finance at investment platform AJ Bell, noted that AI can still be useful for budgeting and finding information. "This is a good example of why people who need support require professional advice rather than relying on AI," she said.
The study was compared with data from the UK's Financial Conduct Authority (FCA): one in five adults in the country is willing to trust AI with financial decisions, with the greatest interest seen in complex areas such as debt, pensions, and investments. According to Jolly, AI financial recommendations are not yet regulated, so users do not have the same level of protection and compensation as when consulting a human specialist.
Related articles
