Strolio

Accuracy

Do AI tour guides make things up?

By Strolio · Published · Updated

Yes. AI guides can invent details, especially about less-documented places. Checking sources helps, but errors remain possible.

What our test found

We wrote stories for 12 places in Barcelona in two ways. Both versions had map data and could search the web. One also got written source passages about the place. We then checked each story’s claims against those passages.

Across 12 places Without source passages With source passages
Claims contradicted by the sources 7 0
Claims supported by the sources 49 114
Precise claims the sources don't cover 44 12

A small internal test of two Strolio versions, not a comparison with other apps or a guarantee for other places. How we ran it.

Claims the sources don’t cover are not necessarily false. The sources simply don’t confirm or deny them. The version with sources still made 12 such claims.

Of the seven contradictions, four were wrong dates or history. The other three were the story wrongly saying reliable details didn’t exist, such as the architect, when the source named them.

Examples from the test

Place Without sources, the story said The source says With sources, the story said
Casa Bloc Built between 1932 and 1936 Commissioned in 1932; project dated 1933–1939 Commissioned in 1932; developed between 1933 and 1939
Pavelló de la República Became a university library in 1994, after a City Council transfer The library opened in 1996; no transfer is mentioned Has housed the university library since 1996
Can Framis Began at the end of the 18th century First half of the 19th century Dates to the first half of the 19th century
Lleialtat Santsenca Reliable details don’t establish a precise architect 2017 project by HArquitectes HArquitectes carried out a project there in 2017

Sources didn’t fix everything. For Masia Torre Llobeta, the map search matched a library 183 m away. The version with sources told the farmhouse’s history correctly, but gave the library’s address. Making sure the source and the map point describe the same building is a separate problem we’re still working on.

How we ran the test

  • Places. Twelve Barcelona buildings from El Globus Vermell’s architecture guides. Five had detailed source text, four had a short text, and three had only a basic record: dates, architects, and addresses.
  • Two versions. Both used the same model and settings: English, medium length, facts only. Both got map data and could search the web. One also got the guide’s text for that place, with publisher, licence, and retrieval date. We matched each text to its place by hand. The version without that text searched the web for 7 of the 12 places; the other never did.
  • Scoring. A separate AI review saw each story and the source text, without knowing which version wrote it. It listed checkable claims, such as dates, people, uses, materials, and protection status. It then marked each as supported, contradicted, or not covered by the source. Word counts and web-search use came from the run logs.
  • Date. Run on 14 September 2026.

This test has limits. It covers one publisher, one city, and mostly architecture. The reviewer is the same kind of model as the writer. Twelve places show a clear difference, but can’t give a reliable error rate. The contradiction count is the strongest result, because it compares each story against fixed text. The web search also found real facts, so “not covered” measures what Strolio can cite, not what is wrong.

How we check stories

  1. Collect evidence. Gather documented facts from archives, local publishers, and open data, recording each source’s publisher, licence, and retrieval date.
  2. Write from those facts. Keep the story short when reliable evidence is limited.
  3. Label and credit. Distinguish facts, local stories, and legends, and list sources when available.

Why mistakes happen

Language models generate plausible text; fluency does not establish whether a claim is true. Research by Mallen and colleagues and by Kandpal and colleagues found that models struggle more with less-documented facts. That matters for a guide explaining the small building on your street.

Travel tools have already illustrated the problem. BBC Travel reported tourists searching for an invented canyon in Peru and receiving incorrect transport times in Japan. ABC News reported an AI-written article directing visitors to nonexistent hot springs in Tasmania. These involved travel tools, not walking-tour apps, but show why practical information needs checking.

What sources cannot guarantee

  • No errors. Source-based systems can still misread or invent details. A Stanford study found errors remained in AI legal research tools using retrieval; its rates do not measure walking guides.
  • Complete coverage. Some places have little reliable documentation, so there may be only a short story or none.
  • Perfect evidence. Archives and databases can themselves be wrong or out of date. Twelve places in one city cannot establish a universal accuracy rate.

What to check on your walk

Look for sources and clearly labelled legends. Check opening hours and transport times on official websites. If Strolio gets something wrong, send us feedback.

Walk with a guide that shows its sources

Strolio is free during the beta, in any city.

Try on iOS Try on Android

Sources

  1. Lynn Brown, "The perils of letting AI plan your next trip", BBC Travel, 29 September 2025. bbc.com
  2. Josh Duggan, "Tour website's AI sends visitors to Tasmanian sites that do not exist", ABC News, 22 January 2026. abc.net.au
  3. "When AI Gets It Wrong: Addressing AI Hallucinations and Bias", MIT Sloan Teaching & Learning Technologies. mitsloanedtech.mit.edu
  4. Alex Mallen et al., "When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories", ACL 2023. arXiv:2212.10511
  5. Nikhil Kandpal et al., "Large Language Models Struggle to Learn Long-Tail Knowledge", ICML 2023. arXiv:2211.08411
  6. Patrick Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020. arXiv:2005.11401
  7. Varun Magesh et al., "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", Stanford, 2024. arXiv:2405.20362
  8. El Globus Vermell, Guies d’arquitectura de Barcelona, pages for the 12 test places, retrieved 14 September 2026. guiesbarcelona.elglobusvermell.org
  9. Strolio internal test, 12 places in Barcelona, 14 September 2026. Method and examples are described above.