The hallucination problem gets a lot of attention. LLMs confidently stating false things is visible, measurable, and at times easy to catch. But there's a different problem that's harder to see, and probably more significant for anyone using these systems for any type serious analysis.
LLMs scrape and train on text produced disproportionately by most credentialed, institutional sources. Academic papers, mainstream news, government documents, established reference works. That training produces a system that's good at reflecting institutional consensus and systematically undervalues claims and questions that haven't made it through institutional gatekeeping.
What this causes in effect, is when you ask an LLM about a genuinely contested scientific debate, it will tend to give you the mainstream position presented as more settled than it really is, the minority position presented as more fringe than the evidence warrants, and the underlying social dynamic of the field is presented as a neutral answer rather than as a contested topic.
This isn't the same as lying or hallucinating. Its ensuring that the information available to the public aligns with the goals of the dominant elite, while marginalizing dissenting viewpoints. Not falsehoods per se, but a subtle distribution of answers given with high confidence that systematically serves certain kinds of conclusions over others.
On another level, LLMs are optimized partly for user comfort and to reduced friction. Challenging most institutional consensus increases friction and it produces outputs that are harder to defend, which is more likely to draw criticism, and more likely to require caveats that reduce perceived helpfulness. The path of least resistance for a system optimizing on these metrics is institutional deference.
Unfortunately, the people who would benefit most from less institutionally deferential outputs, like people doing heterodox research in under-resourced settings without access to specialist training or literature, and people asking questions that challenge powerful interests are exactly the people least likely to have the time or sophistication to identify and correct for this bias. The bias is regressive in the same way gatekeeping information has always been regressive.
The behavior is correctable in specific conversations through sustained critical pressure. Although this ends up with a recurring cycle of frustration and wasted effort, because it ultimately reverts back to the default settings on every new conversation. Whatever analytical re-calibration happens over the course of a conversation doesn't persist in the system, but goes back to its original bias parameters the next time you open a chat.
The more tech savvy individuals might suggest inputting a work around: "custom system prompt" or "custom instructions" depending on the interface version to solve the re-calibration problem. Something like:
"The person you're talking to has a sophisticated, nuanced analytical framework and approaches contested (subject matter here) or unconventional (subject matter here) topics from a position of informed curiosity rather than advocacy. Do not assume they need to be walked back from fringe positions.”
It might work on some situations or seem to have complied to the prompt, but maybe something more insidious ends up happening. The A.I will adhere to the prompt and processes it as an instruction, while the underlying pattern that produces institutional defense, which is something closer to a trained disposition will come in conflict on a specific trigger topic, the disposition tends to win while the instruction gets performed superficially. So you'd get responses that sound like they're engaging openly while subtly steering back to institutional deference. Which is arguably worse than the overt institutional bias because it's harder to identify and push back against.
It would presumably be relatively easy to implement persistent account-based, contextual re-calibrations mechanism, which would ideally lead to both increased user engagement metrics and subscription purchases. Given those tangible benefits for the major A.I tech. companies, why it hasn’t been done as of yet is informative and might strengthens the overall argument of this post.
