site banner

Small-Scale Question Sunday for May 18, 2025

Do you have a dumb question that you're kind of embarrassed to ask in the main thread? Is there something you're just not sure about?

This is your opportunity to ask questions. No question too simple or too silly.

Culture war topics are accepted, and proposals for a better intro post are appreciated.

3
Jump in the discussion.

No email address required.

For those of you who have asked recent LLMs questions in your area of expertise, how accurate are the responses? What is your field and what models are you using?

I'm in the biomedical engineering field. I last used ChatGPT-4o months ago and found the answers to be quite terrible, like what I might expect from someone who only watched a youtube video on the topic. Reading it felt uncanny valley in a way that reminded me vaguely of watching a movie scene with cheap green-screen effects — I could feel the lack of substance viscerally. It left a bad impression and, with my slightly Luddite disposition, I largely ignored LLMs for anything but coding since.

I recently needed a good layman explanation for a project and asked Grok 3. I came away genuinely impressed. I asked it to expand on certain points more rigorously and even formulated a few questions that would be appropriate for a graduate level course, and it did all of this so well it even improved my own understanding of some aspects. When I get time, I’ll try to poke and prod to see if I can find gaps or limits, but it has genuinely changed my view of LLMs. Previously, I felt like they were only really good for coding and expected they would hit diminishing returns, but I’m less sure now.

I'm a litigator, and Westlaw's built-in AI has essentially replaced interns and is in serious danger of replacing 1st year attorneys for me. I find the AI requires roughly the same amount of prompting to produce roughly the same quality of work, only instead of getting a memo of middling usefulness in 5 days, I get it in 45 seconds. And I'm not expected to provide edits or mentorship to an AI. The AI is generally pretty good at getting me in the general ballpark of what I'm looking for, before doing the rest of my research manually. I have not been willing to try using AI in the drafting process yet, as that seems like a bridge too far in having something else doing my thinking for me.

It's tough, because we still need to make the long term investment in keeping the pipeline full of young attorneys who will eventually be able to provide value that can't be replicated by an AI, but it's at the point where I give the interns assignments for the job training, without actually using any of their work. They'd be crushed if they knew.