Mahsan et al: The Role Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems
Summary
Talks about how people are influenced on their views of intelligent systems based on their first impressions and their familiartiy with the topic.
Notably, domain experts fail to regain trust in the system, even if the system gets better over time. Consistently achieve automation bias and believe in the system, even if it is wrong.
Domain expertise plays an important role.
Implications
“Rather than allowing users to use a system with zero understanding or a poor mental model of how the system works, designers can incorporate introductory sessions to alert users about system strengths and shortcomings so that users can decide whether and when they should trust the system’s outputs. Alternatively, designers can provide a high-level overview of the model with key information (e.g., system accuracy and known weaknesses) that can influence impression formation.” (Nourani et al 2020:8)
What does this mean for your product? You can’t explain away bad results to domain experts, they consistently rate it below even if the explainability is high
this implies that the models need to be very correct on the first pass
for post-conflict statebuilding organization, does this mean that people who experience bueauracries less if the intial assessment is bad? Does this relate to unbanked people?