Do you have a dumb question that you're kind of embarrassed to ask in the main thread? Is there something you're just not sure about?
This is your opportunity to ask questions. No question too simple or too silly.
Culture war topics are accepted, and proposals for a better intro post are appreciated.
Jump in the discussion.
No email address required.
Notes -
How do I stop LLMs from lying to me?
Every single AI out there, without fail, produces completely different and contradictory interpretations of literally the exact same data just because I phrased the prompt slightly differently. Adding or removing a single word, or changing the order of sentence, or making a spelling mistake will give you a different output.
This is frustrating. I cannot believe or trust anything these things say. I know that it can see the data. And it obviously can "correctly" analyze the data, as well. But this requires a nudge. Why? Why does the LLM react to my nudge and not the actual data?
Is there any way for me to verify that I'm actually getting an honest response from an LLM, and that it's not just telegraphing back whatever I already implied when prompting it?
Which LLM, what prompt, and what data, and how are you presenting it?
There’s a big difference between copy pasting a document in the free ChatGPT interface, making an API call directly to a frontier model with a custom system prompt (instead of putting instructions in a user message), or using agents with custom tool calling abilities. You might be better off having it write code to analyse the data, versus having it analyse the data itself directly.
I’ve had good results with verifying outputs by using another LLM API call with its own, different system instructions, but you still have to learn how to write good prompts to get the best results. Because they write in a (reasonably) human-like way, it’s easy to fool yourself into believing they also think in a human-like way.
Not the best example, but still... Today, in Google's AI Studio, I gave Gemini 3.1 Pro (paid) my Discord logs. I was arguing with a maintainer of a website who just released a rewritten front end. The website is now an ugly mess, and the developer is utterly incompetent. He lacks a basic understanding of UI/UX, and it took him 2000 lines of code to implement a fucking table. The values in the table overlap and bleed off the screen when you resize it. He blamed Mantine for this. It took me three lines of CSS to fix the tables.
Anyway, the developer's code is indefensible. But despite this and the developer being very clearly fucking r-slurred, the AI's initial interpretation of events was fence sitting. Muh both sides. I probed and tried to word my follow-up prompt in as neutral a way as possible, but based on the response I received, the AI clearly understood who I was. It apologized and started to ruthlessly shit on the dev. I probed further, and then it flipped back around to shitting on me.
The code is objectively bad. The AI should not be flip-flopping. It can and does recognize the code as bad, but politeness filters reject draft responses that are too harsh and critical, if it believes that the user might not want to hear that. And so, it must base its interpretations entirely on the wording of my prompt and whatever implications it can pick up from how I phrase things.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link