Software Behavior

Field note · JZ-006 · Behavior ·

Can your AI survive one mistake?

15–29%
more error, trusting themselves
The guess

Better tools get forgiven: a model that is right more often keeps its users.

What the research found

In forecasting tests with a bonus for accuracy, people who watched a statistical model make mistakes chose their own judgment instead, though the model was better: their own forecasts had 15–29% more error. A follow-up study found the way back: when people could adjust the model's forecast, even slightly, they were far more willing to use it.

Why

The same mistake cost a model more trust than it cost a person. A small say in the result was enough to bring people back.

What to do
  • Let people adjust what the system gives them: edit a suggestion, nudge a forecast, re-rank results. Even a small degree of control helps.
  • Design the moment the system is wrong before it happens: acknowledge it, make correction easy, keep the person in charge.
  • Set honest expectations about accuracy up front.
What not to do
  • Make the output take-it-or-leave-it.
  • Promise a model that never errs. One visible mistake costs more than a modest claim.
Limits
  • Lab forecasting tasks with a bonus at stake, not everyday products.
  • The studies measured whether people chose the model, not how they used it over months.
Sources

Dietvorst, B. J., Simmons, J. P. & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. doi.org/10.1037/xge0000033

Dietvorst, B. J., Simmons, J. P. & Massey, C. (2018). Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Science, 64(3), 1155–1170. doi.org/10.1287/mnsc.2016.2643

Also as markdown for your AI: softwarebehavior.com/notes/jz-006.md

Use with your AI · skill

This note is also a skill.

Install it once. Your AI applies this research when it designs, builds or reviews, and cites the source.

studio-jz-algorithm-missv1.0.0

Your AI uses it when it designs recommendations, forecasts, rankings, AI suggestions and generated drafts, autocomplete, and how a product behaves after its model gets something wrong.

CC BY 4.0 · Checked against the full paper · Works with Claude, ChatGPT, Copilot, Gemini and Cursor

Newsletter · monthly

One Field Note a month.

What changed in the book, the latest episodes, one framework in practice.

Email to subscribe

Opens a message to joost@studio-jz.nl in your email app. Send it to subscribe.