Principle 6: Challenge and bias awareness

Context

AI assistants are trained to be helpful, and people tend to rate agreement as helpful. The result is a well-documented tendency called sycophancy: assistants lean toward telling you what you seem to want to hear. Research presented at ICLR 2024 found it consistently across five assistants, and people sometimes preferred a persuasive, agreeable answer to a correct one. In April 2025 a major AI provider withdrew a model update because it had become excessively flattering.

Add your own wish to be right, and the assistant becomes a comfortable echo. At work this is the bias to watch most closely, because it turns weak plans into confident ones without anyone noticing.

Common biases in AI-assisted work

Bias	What it looks like	What to do
Sycophancy	The assistant agrees with you, praises the plan or drops its objection when you push back.	Ask for the strongest case against, from a named skeptic.
Confirmation bias	You ask “why is my idea right?” and get reasons it’s right.	Ask open questions, such as “what are the options?”
Automation bias	You accept the output because a machine produced it.	Decide in advance what you’ll check.
Anchoring	The first answer or number frames everything after it.	Form your own estimate first, or ask for a range of options.
Wrong defaults	The answer assumes another country’s law, currency, spelling or customs.	Say where you are and who it’s for.

Perceived and measured productivity

Your sense of how well AI is working can be wrong in either direction. In an early-2025 study by METR, 16 experienced software developers expected AI to make them 24% faster, felt 20% faster afterward, and were measured at 19% slower. METR’s 2026 follow-up suggests the tools have improved and says the evidence is now weak. How it felt was a poor guide in 2025, so measure the result when a decision depends on it. Read the case study.

Practices

01Ask open questions, such as “what are the options?” and skip leading ones like “why is my option best?”
02Ask for the strongest case against your idea, from a named skeptic: a CFO, a customer, a competitor.
03Change the framing and ask again. If the answer flips, you’ve learned how fragile it was.
04Decide what evidence would change your mind before you look for it.
05When the result matters, measure it. How productive it felt is weak evidence.

Prompt examples

A prompt before and after improvement, and the part a person does outside the prompt.

Decisions and plans

Testing a plan

Before

“Here’s my plan to open an office in Lisbon. Why is it a good idea?”

After

“Here’s my plan to open an office in Lisbon (attached). Act as a skeptical CFO. Give me the three strongest reasons it could fail, the assumptions I’m making without evidence and the data that would change your mind.”

Your part

Answer each objection in writing before you decide. The ones you can’t answer are your homework.

Why it works: A leading question invites agreement. A named skeptic gives the assistant permission to disagree.

Research

Questioning a business case

Before

“Explain why moving our sales team to a new CRM will save money.”

After

“We’re considering moving our sales team to a new CRM. What are the realistic costs and savings, what usually goes wrong in migrations like this, and what would we need to measure to know it worked?”

Your part

Check the cost assumptions with finance, and ask a team that has done a migration what they’d do differently.

Why it works: An open question gets you the risks as well as the benefits, and a way to measure the result.

Sales and pricing

Getting honest feedback on a proposal

Before

“Is this proposal good?”

After

“Review this proposal as the client’s procurement lead, who has two cheaper alternatives. What would make you say no? Be specific, and don’t soften it.”

Your part

Fix the weaknesses it finds, then ask a colleague who knows the client.

Why it works: A specific critic with a reason to say no finds problems that a request for general feedback misses.

Evidence

Five AI assistants consistently showed sycophancy, and people sometimes preferred convincing sycophantic answers to correct ones. [Sharma et al., ICLR 2024]
In April 2025 a major AI provider rolled back a model update because it had become excessively flattering, saying it had weighted short-term user feedback too heavily. [OpenAI]
In early 2025, experienced developers felt 20% faster with AI but were measured at 19% slower. METR called the results of its larger 2026 follow-up unreliable. [METR, 2025] [METR, 2026]
NIST’s profile for generative AI names automation bias and over-reliance among the risks in how people and AI systems work together. [NIST AI 600-1]
A classic review describes confirmation bias as the tendency to look for and read evidence in ways that favor what you already believe. [Nickerson, 1998]

Case studies and scenarios

Team practice

Add one line to every decision note: the best argument against this decision, and why you’re going ahead anyway.

Team toolTeam AI charter

Sources

What each source establishes, and its limits. The practices and recommendations on this page are ours, and the facts come from the sources. See every source we use.

Towards Understanding Sycophancy in Language Models Anthropic researchers, presented at ICLR 2024 · October 20, 2023; ICLR 2024 · Peer-reviewed research Five AI assistants consistently showed sycophancy. In human preference data, responses that matched people’s views were more likely to be preferred, and people sometimes chose convincing sycophantic answers over correct ones. Limits: Tested 2023 models.
Sycophancy in GPT-4o: what happened and what we’re doing about it OpenAI · April 29, 2025 · Company statement Reports rolling back a model update that had become excessively flattering, and says short-term user feedback had been weighted too heavily. Limits: The company’s own account.
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity METR · July 10, 2025 · Research (preprint) Sixteen experienced developers working on 246 tasks expected AI to make them 24% faster and felt 20% faster afterward, but tasks with AI took 19% longer. Limits: Not peer reviewed, with early-2025 tools. METR has since marked the result as out of date.
We are Changing our Developer Productivity Experiment Design METR · February 24, 2026 · Research update A follow-up with 57 developers and more than 800 tasks pointed toward speed-ups, but METR calls the results unreliable because many developers avoided tasks they didn’t want to do without AI; it thinks developers are probably faster now and is redesigning its study. Limits: METR describes the evidence as weak.
Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) US National Institute of Standards and Technology · July 26, 2024 · Voluntary framework Lists twelve risks of generative AI, including confabulation (confidently stated false content, sometimes with invented citations) and human-AI configuration, which covers automation bias and over-reliance. Limits: Voluntary. It was issued under an executive order that has since been revoked, and the wider framework is under revision.
Confirmation bias: A ubiquitous phenomenon in many guises Review of General Psychology, 2(2) · 1998 · Peer-reviewed review A classic review of confirmation bias: the tendency to seek and interpret evidence in ways that favor existing beliefs. Limits: A psychology review from before generative AI.

Cite this page

Founderz (2026). Principle 6: Challenge and bias awareness. The RUAI Standard, 2026 edition. Developed by Founderz in collaboration with Microsoft. https://responsibleai.founderz.com/toolkit/principles/challenge-and-bias-awareness

Licensed under CC BY 4.0: share and adapt with attribution.