This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
What a sterling example of making the dream of perfection the sworn enemy of the merely better. As others have pointed out before, the most likely alternative, in the absence of ChatGPT, would have been this poor fellow resorting to Google Translate or other, far simpler ML solutions. There isn't an abundance of fluent English and Japanese speakers willing to proof read random YouTube comments.
I don't speak Japanese, but I see nothing particularly objectionable in the translation. It might not capture all nuance, but it gets the gist of it across. Learning language takes time, probably years, and by the time this gentleman gets good enough that he needs or appreciates the nuance, LLMs will be even better at the job.
This paper: https://arxiv.org/html/2504.18221v1 grades gpt-4 versus other translators with actual human grading (not bs like ROUGE which is useless) and finds that gpt-4 doesn't seriously outperform deepl, and google translate, while worse, isn't even that far off.
This test is actually also unfair in favor of ChatGPT, as since the test text is a story by a famous author, ChatGPT has likely already taken a peek at human translations of the work during training.
I'm reading the paper, but initial issues that caught my eye:
They're not evaluating GPT-4. They're using 4o. The exact implementation details of 4o are still opaque, it might be a distill of 4, but they're not the same model. As far as I can tell, that's a point of confusion on your part, not the authors.
4o, even at the time of publication, was not the best model available. Very far from it. It is a decent generalist model, but not SOTA. It isn't the best model, the second best model, or even the third... best model, on almost any metric one opts for.
I have, as far as I'm aware, never claimed that LLMs match or outperform professional human translators. My core contention was that even basic bitch LLMs are good enough, and an improvement over previous SOTA, including DeepL, which this paper supports.
This would hold true even if the authors had chosen to use something like o3, Opus 4, Gemini 2.5 Pro etc. It is unclear to me if they were aware that better options were unavailable, there's little reason to use 4o if one wants to know what the best possible output is.
And even if it is true, it doesn't matter. The models are consistently getting better. We have a new SOTA every few months these days.
4o vs gpt4 is my mistake, but gpt4 is generally considered obsolete and nobody uses it. It's true that 4o is a mixed bag and underperforms gpt-4 in some aspects, but we have no reason to believe that it's significantly worse than gpt-4 at translation.
4o is also what powers chatgpt.com so it's the model that most casual users will get the output from.
4o was released well before gemini 2.0 or claude 3.5, so it likely was the best model at the time, along with the original gpt-4. I agree that right now 4o is not good.My core contention is that deepl is good enough, as it's within spitting distance of chatgpt. But on the other hand ChatGPT has given people ways to do much much worse when they use it wrong.
The paper seems to have been published on April 2025.
Gemini 2.0 Pro and 3.7 Sonnet came out in February 2025. Claude 3.5 Sonnet came out in June 2024 and was better than the version of 4o out then.
At the very least, the authors should have made a note that they weren't using the SOTA, or that the SOTA would have moved significantly by the time of publication. To do less is mild dishonesty. This isn't 2022, the pace of progress is evident.
True, but that's OAI being cheap, and not an indictment of the utility of LLMs for translation. It's akin to claiming TVs suck, and then only using a cheap and cheerful $300 model from Walmart as the standard.
My criticisms stand, namely that LLMs only get better, they're "good enough", and that this is a net improvement over the status quo. It remains to be seen how much better the SOTA is over 4o or DeepL.
oh oops, I misread your comment, I thought you said that 4o was not sota when it was released. Yes it was obsolete when the paper came out.
Won't change the fact that people who use them wrong will still do worse than not using LLM at all.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I replied to your OP a few weeks ago expressing skepticism that using ChatGPT was actually improving your writing. This comment reinforces my skepticism. Yes, the ChatGPT output has fewer "errors" but it does a worse job of conveying the message than the user's own error-ridden text. Even from a purely stylistic standpoint, the ChatGPT output is worse. One of the hallmarks of bad English prose is using extra words to say nothing, and ChatGPT is guilty of this in virtually every sentence. It's not the perfect being the enemy of the good. The ChatGPT output is not good.
This is a very bad way to showcase ChatGPT's writing chops. When I use it, by virtue of personally having native fluency, I am far better equipped to critique its output. The original user, a Japanese person somewhat conversational in the language, is simply not in a position to be able to tell where it could have made better stylistic choices. Note that my objection above is to do with the fact that the availability of ChatGPT represents a massive improvement over the previous status-quo. If he had simply asked it to translate his original Japanese, it would have done a perfectly fine job.
LLMs are not a monolithic entity. When most people talk about ChatGPT, they're ignorant of the fact that there are options beyond the slop-tier 4o. You may or may not know the difference, but the average normie complaining doesn't. My own workflow involves using multiple different models, multiple attempts, prompts that I know qualitatively improve results and my own ability to discern between outputs.
For example, when I ask for a translation that preserves the writers' rather unique character:
Kimi K2:
Gemini 2.5 Pro:
I don't know basically any Japanese, and I'm pretty far from streaming culture -- but based on my knowledge of how these things work I'm pretty sure both are failing badly in their interpretation of 'support on live stream'; I'm pretty sure he means that he plans to, like, give her money? Probably not all of his money ("support her with everything I've got"), but certainly "cheering my lungs out" would be atypical behaviour on a livestream, no?
He probably means cheering. The Japanese message uses the loanword "live", which refers to special events like concerts, not regular live streams. You support concerts by showing up and cheering or clapping, and she's describing her birthday stream as a "horror live", so he's probably intending to watch the whole stream and spam emoji in chat whenever something noteworthy happens. This is pretty common behavior for concerts on YouTube.
More options
Context Copy link
Does he? Wouldn't surprise me, but I think we need
weebssubject matter experts to disambiguate on our behalf.More options
Context Copy link
More options
Context Copy link
I mentioned it in another comment but the typical workflow of intermediate speakers would be to write the work directly in the target language, rather than translating it. Machine translation with LLM is certainly pretty good right now but using it doesn't help anyone learn the language.
We certainly might be heading towards a world where everyone uses machine translation by default and nobody bothers learning a new language, but I'm certainly a luddite in that lane.
I don't agree. Better translation tools like deepl have been around for a while, and arxiv papers haven't shown that gpt series models seriously dominate dedicated translation models. But on the other hand ChatGPT is giving everyone a huge gun that people can shoot themselves with because it does things besides translate.
I would even argue that by virtue of using ChatGPT wrong, the user ended up with a worse result versus just using a shitty translation tool like Google translate.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
IDK if you specifically disagree, but I strongly prefer the original English, errors and all, over the ChatGPT output.
More options
Context Copy link
More options
Context Copy link