This weekly roundup thread is intended for all culture war posts. 'Culture war' is vaguely defined, but it basically means controversial issues that fall along set tribal lines. Arguments over culture war issues generate a lot of heat and little light, and few deeply entrenched people ever change their minds. This thread is for voicing opinions and analyzing the state of the discussion while trying to optimize for light over heat.
Optimistically, we think that engaging with people you disagree with is worth your time, and so is being nice! Pessimistically, there are many dynamics that can lead discussions on Culture War topics to become unproductive. There's a human tendency to divide along tribal lines, praising your ingroup and vilifying your outgroup - and if you think you find it easy to criticize your ingroup, then it may be that your outgroup is not who you think it is. Extremists with opposing positions can feed off each other, highlighting each other's worst points to justify their own angry rhetoric, which becomes in turn a new example of bad behavior for the other side to highlight.
We would like to avoid these negative dynamics. Accordingly, we ask that you do not use this thread for waging the Culture War. Examples of waging the Culture War:
-
Shaming.
-
Attempting to 'build consensus' or enforce ideological conformity.
-
Making sweeping generalizations to vilify a group you dislike.
-
Recruiting for a cause.
-
Posting links that could be summarized as 'Boo outgroup!' Basically, if your content is 'Can you believe what Those People did this week?' then you should either refrain from posting, or do some very patient work to contextualize and/or steel-man the relevant viewpoint.
In general, you should argue to understand, not to win. This thread is not territory to be claimed by one group or another; indeed, the aim is to have many different viewpoints represented here. Thus, we also ask that you follow some guidelines:
-
Speak plainly. Avoid sarcasm and mockery. When disagreeing with someone, state your objections explicitly.
-
Be as precise and charitable as you can. Don't paraphrase unflatteringly.
-
Don't imply that someone said something they did not say, even if you think it follows from what they said.
-
Write like everyone is reading and you want them to be included in the discussion.
On an ad hoc basis, the mods will try to compile a list of the best posts/comments from the previous week, posted in Quality Contribution threads and archived at /r/TheThread. You may nominate a comment for this list by clicking on 'report' at the bottom of the post and typing 'Actually a quality contribution' as the report reason.
Jump in the discussion.
No email address required.
Notes -
Dan Ariely Is In The News Again
[previous discussion here, hat-tip to TraceWoodgrains for bringing it to my twitter feed]
Data Colada Reports:
For those who missed the first one, at least one of the studies in this data is fake. There's an orgy of suspicious problems with it: incredibly implausible reported numbers, ridiculously strong effect size, negative correlations for things that should have been strongly tied together, sample 'twins', inconsistent rounding where only three out of 360 survey values were not rounded to the nearest 5%, miscalculated statistics, means changing from a preprint without derived statistics changing with them. About the only thing missing is the text format mistake showing exactly what was copied and pasted in the Honesty trials; we should be glad there's not a spurious western blot that someone made its way in. DataColada doesn't even cover all of it: there's numbers in the underlying spreadsheet that don't match in the basic math sense.
There's also a bit of a buried lead, though. DataColada needed the raw data to do this analysis. That doesn't mean anyone needed that data to make a serious criticism. The errors here are not subtle. You can tell there's something fucky wucky with this paper just by looking at its charts, and there's only one for this study.
The experimental protocol described people receiving three papers with a hundred errors each, and were told that they would receive 10 cents for every correction, but would be penalized a dollar for every day they were late.
(errors and delay calculated from the underlying source data, but nothing within a round-to-10% value works, either, so this fails the MK1 Eyeball Test. These study's test involve machine-generated papers intended to be meaningless, so a best-case scenario of 70% errors found, while suspiciously even, is at least plausible even for MIT's best and brightest.)
The chart isn't, to be clear, close to a possible combination. The chart doesn't even have a negative value reported, period. And if you start thinking about what a negative value for earnings means -- any further work would have no value, and soon no plausible expected value -- and quickly many the numbers don't really make sense. At best, the trial used a different protocol than its own minimal results show, in a form clearly visible by just reading the paper. A flat rate for participation isn't impossible, but if it was included in the measurement, it should have been in the methods section. It wasn't.
But now we actually have e-mail conversations showing that wasn't the case, either: "There was no show up fee and we did not say anything about the fact that the payment rule could mean that subjects will lose money (and as far as I remember we never had to deal with this problem)". That's for a trial where, by the published protocol and data, a full 19 out of 60 participants had negative earnings, and even assuming a flat $10 participation fee needed to make the chart work, 8 participants would still have had negative earnings.
A fellow academic, Kyle Hyndman, received that e-mail in 2014.
Hyndman, to his credit, did forward the data and conversations to DataColada in 2023. And he was working on a replication attempt most of the time, albeit probably as a low-priority for a decade. He did, after his replication effort, publish in a footnote that : "In October 2024, at the request of the editors, we shared with Dan Ariely an analysis of the contents from the file purportedly for their Study 2 and asked for permission to include a summary of it in the paper. Dan Ariely denied our request, arguing, among other things, that the files we received may not be the actual data."
That does seem to be the defense, for another way this study rhymes with past scandals. Ariely can't remember the actual study protocol, can't be sure this data was what actually got published, and doesn't want a summary of the data that does exist being republished. It is quite possible no records of the experimental protocol exist to prove or disprove anything. I've got a search tool running through old MIT classifieds, and that's more a hope than a process.
There's a plausible, if disturbing, option that had the data fabrication been revealed before the replication failure and the separate scandals in other papers, it would have blown over. There's a plausible, if even more disturbing, option that even with those other data points, it will just blow over again. Maybe a retraction -- and to be fair, there's at least been a request for one -- probably Ariely doesn't get another tv show, probably not a slapped wrist, almost certainly no serious investigation by his school.
But that still leaves a lot of questions. Ariely has over a hundred other papers he's authored or coauthored listed on his own CV. It was plausible for the Honesty trial that no one else should have looked at Ariely's data, or looked at his conclusions with a skeptical eye. It remains plausible that, for two papers, his coauthors and peer reviewers didn't look at the paper with a skeptical eye. Two isn't feeling like a very real number here.
Trace mentioned this story with the opening line "I feel for the coauthor [Wertenbroch] here – he seems to have acted generally honorably and been caught off guard by having a liar for a collaborator." Before I read the paper, my gutcheck was "On one hand, it's nice to see Wertenbroch moving on it. After Gino, I'd seriously worried about the possibility Ariely had just been able to thrive so well because no one he was working with wanted honesty, either."
And then I saw that chart, counted on my fingers, and blinked.
Wertenbroch is, notably, CC'd on that 2014 e-mail where Ariely said that his study did not have any students with a negative balance, and had no participation fee to explain that 10 dollar offset. He could, plausibly, have not reviewed the data or the paper in a decade, and not realized the discrepancy even then. Leaves a bit of a question about what he does do, though. He could, plausibly, have not touched or looked at a single part of the study methodology or data in this entire paper.
In the Honesty trials from the previous scandal, there was an absolute mess where Ariely was responsible for Study 3, and there were serious questions about what, if any, exposure to the fraud the other authors might face. Gino seems to have had minimal responsibility of exposure to Ariely's original data... and separately produced some falsified data in a separate study in the same paper, and multiple other studies in other papers.
DataColada ends today's post with "In our next post, we will share analyses of the Study 1 data file [...]. That experiment is quite different. Our analyses are quite different. But our conclusions are quite similar." I'm working on writing up a post on the aftermath -- or lack thereof -- on the Hindawi scandal.
Eat at Arby's.
EDIT: while DataColada does not spell out the chart discrepancy, it is in their underlying R code. So the description here is more a dumbass noticing the same thing, not me finding something they missed.
I'm starting to think that Futurama was rather prescient in calling the turn of the century from 20th to 21st as the Stupid Ages. More and more of the facade of credibility of organizations, institutions, people, ideas, etc. that people used to believe were not stupid - justifying using them to guide their decisions to be less stupid - have revealed themselves to be deeply stupid. I hope that LLMs will become so cheap and reliable that all the previously hidden stupidity will be aired out for all to see soon enough. I just dread what unexpected stupidity will be found - sociology, [x] studies, psychology are the obvious no-brainer candidates which have already largely gone through the process, but what if it turns out even very "hard" or concrete fields like electrical engineering or chemistry are built on a bunch of stupidity that has just enough facade to be convincing to the in-field expert and certainly more than enough to the layman?
Longtime readers will know what my take on this is.
The institutions in question have not be held accountable for outcomes in decades, don't hold their internal staff accountable, and oftentimes don't measure outcomes at all. The Institutions generally in charge of holding people accountable and punishing the misdeeds are, themselves, usually compromised as well. I suspect everyone prefers an arrangement where the institutions shamble along based on their prior reputation, everyone getting paid to maintain a facade, than risk collapsing it by exposing the extent of the rot. To the extent there are punishments and consequences they're rarely sufficient and proportional enough to truly dissuade the behavior in question.
And finally the institutions that manage to maintain integrity eventually get outnumbered and even if they are able to hold some people to account, can't possibly keep up with the volume of the problem. If that becomes widely known, even average people will be inclined to cheat/defect since not doing so makes you a bit of a sucker and there's rarely true consequences for doing so.
Re-insert skin in the game.
The good news is that in such hard fields, nature itself tends to have a correction mechanism for this, where the feedback loop for doing something wrong is tight and the punishments can be harsh.
Synthesize the wrong chemical, the reaction won't work, and you possibly kill yourself. Design the plane wrong it crashes, eventually. Reducing your safety margins too much will eventually come back to bite you, and possibly remove you from the system, and provide a warning to the next guy.
I'm sure there's stupidity to be found there, but when the real world REQUIRES you get things right in order to actually succeed, the accountability is to some extent baked into the cake, even if various institutions will try to put a few layers of obsfuscation between themselves and the outcomes.
That's exactly why if all that turned out to be a facade, it would be absolutely wild and crazy and turn my world upside down. I'd guess that the odds are low, but also, I'm quite sure that the true believers in those already-discredited fields would also judge the odds as being low for whatever field they're a true believer to.
Yeah solid point.
If it turned out that foundational papers in Engineering and studies in chemistry were false and everyone else had just been winging it and lucking out (or maintaining a massive coverup) then I would have to re-examine a lot of my core assumptions.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link