Do you have a dumb question that you're kind of embarrassed to ask in the main thread? Is there something you're just not sure about?
This is your opportunity to ask questions. No question too simple or too silly.
Culture war topics are accepted, and proposals for a better intro post are appreciated.
Jump in the discussion.
No email address required.
Notes -
I have a few vibes benchmarks:
Asking a model to rewrite an essay (usually mine) in the style of an author I am familiar with. Can it redo a few chapters from novel in the voice of Banks, or Morgan? Most models flanderize them, or settle for a shallow pastiche. Some get them, and you'd be fucking surprised which models those are. Some aren't even SOTA, but beat the best reasoning models.
Throw in as much of my profile as patience and context windows allow, and then ask it to mine it for insight.
Do the above, and then ask it to do an intentional emulation. Call that fine-tuning on a budget. Can it capture my voice? Can it write something I see myself writing? This is a hard problem, most of them suck ass. Gemini 2.5 Pro flanderizes me, Sonnet 4 did a decent job (after a lot of prompting), but paradoxically, Sonnet 4.5 often refuses, gets confused, or simply does poorly.
Ask it to solve physics or maths problems (where I have access to ground truth). I'd use medicine, but models are already so competent that my ability to evaluate them there is limited.
That's it for semi-formal assessment. For the rest, I build up impressions through sustained usage, till I have a firm grasp on model capabilities and personality.
For reasoning tasks:
GPT-5T is the best, almost matched by Gemini 2.5 Pro.
Quick answers, where I don't want to wait around:
Claude Sonnet (4.5 is too new for me to know how good it is)
Diversity candidates who have interesting capabilities in one domain or the other:
Kimi K2, GPT 4.1
(I refuse to use GPT-5 Instant. It's ass, and is dominated on the Pareto Frontier by other models)
More options
Context Copy link