Dan Ariely has a long history of controversies and studies with falsified data. He wrote 3 popular books and gained a lot of fame before others started uncovering the problems in his studies
It’s amazing that problems with his studies are still being uncovered. Equally amazing that Duke still maintains a relationship with him after so many problems have been identified with his research.
The fraud that Data Colada presents here is unambiguous. It was also no accident in my opinion that they are specifically mentioning that the 3 Excel files were “Last saved by” “Dan Ariely”
I'm sure the statute of limitations has expired for this, but I really wish the Feds would prosecute cases like this as wire fraud instead of leaving insular academic institutions to self-police. This type of fraud is done for personal gain and self-enrichment and causes real, identifiable harms.
I'd be pretty wary of stretching wire fraud law to cover this. Prosecutors deciding whether scientific results are fraudulent sounds like it could create its own set of ugly incentives
Fraud law doesn't need to be stretched, academic fraud is covered and criminally prosecutable in the USA for decades. It has to be because they're using these results to apply for grants paid from tax monies.
Occasionally someone is even prosecuted! Unfortunately, even when the government wins, the institutions just treat the resulting fines as a cost of business. See the Dana Farber Cancer Center: prosecuted by a third party on behalf of the government, found guilty of faking their science for a decade+, paid the fine - and nothing anywhere on the DFCC website mentions anything about it.
> and nothing anywhere on the DFCC website mentions anything about it
That doesn't really surprise or bother me - I wouldn't exactly say it was a coverup just because they aren't heralding some major Fed case against them on their own website. But I do find it shitty that the Wikipedia page of Dr. Laurie Glimcher, who was president of the institute at the time, had some papers retracted and apparently resigned over the issue, makes it sound like she just retired because she completed her mission and has no mention of the scandal.
The bigger issue I see is that someone wasn't prosecuted, just the institution. I didn't spend a ton of time looking, but AFAICT no individual faced any serious consequences beyond "reputational damage". This is why the "corporations are people" shit is so aggravating. If corporations are people, then some people who committed these crimes should have faced legal consequences, not just have their malfeasance paid out by the organization.
Prosecutors (and juries) already are expected to adjudicate fraud across cases that IMO are much more ambiguous than this.
I think it's fair to debate whether it's a good thing that, as I heard it put once, "everything is wire fraud" (i.e. lots of crimes require lying on documents that are sent electronically and thus become federal crimes), but I see zero reason to think that committing academic fraud (which, again, is done for personal gain and causes real, measurable harm to others) is any less of a crime than lots of other things that are prosecuted as wire fraud.
Perhaps we need a new/updated law to cover this kind of thing. This sort of academic fraud should absolutely be criminalized, but I'm not sure I'm the biggest fan of the "everything is wire fraud" approach to achieving that.
>Peer reputation and self policing works in a system where no money is at stake.
It works in systems where money is at stake too. You need a high trust, conscientous society. Doesn't mean that having one solves the issue entirely. The ocassional bad apple will fraud there too, but it wont be rampant enough to be a systemic problem.
>When your career and income are at stake, why would you take the risk of opposing Dan Ariely? Only someone who was already rich could do that.
You don't need independently wealthy scientists to expose fraud.
You just need an environment where if regular salary dependent scientists expose fraud, they're not punished for it.
If you don't have that, sure, then you need independently wealthy scientists to dare.
>Do you know a trick to pay my bills in self-respect? Most of my creditors demand I pay in money.
Nobody said that shit. You can pay your builds with the monetary salary from doing your job honestly. That was my first point.
If many people discover that such a salary is not enough to get by, or if many people aren't content with "enough to get by" and decide to perform academic fraud for extras, then that's a society that already lost the plot. That was my second point.
> leaving insular academic institutions to self-police
I'm not convinced that academic institutions are actually self-policing, it may not even be correct to assume they each act as one coherent entity. There are a lot of incentives in play which may not be obvious from the outside.
<apologies for yet another anecdote from way back when>
Many decades ago during my PhD I was in working in a chemistry lab when a colleague had an accident which caused a minor explosion, big enough to blow both of the glass side panes completely out of his fume hood.
With hindsight he was extremely lucky to have only minor injuries, he was peppered with tiny cuts from the flying glass and temporarily deaf from the bang.
I called for an ambulance, which seemed like a reasonable response.
The first prof on the scene (not our prof though!) came in and immediately asked "who phoned for the emergency services?", so I said it was me. He then gave me a bit of a telling-off, ending with him saying "and now the press will find out!" before he stormed off.
I remember thinking "wtf?! my colleague has just blown up our lab, and your top concern is that it will be in the newspapers"...
> The fallout from the saga is likely to continue. Simonsohn and several of Gino’s co-authors last year launched a project to assess the reliability of her past papers. So far, this has led to a further retraction in February and a correction in July. Science reported in April that Gino also appeared to have plagiarized passages of text in some of her publications.
It’s also hilarious how the judge let her continue the suit against Harvard. Like “hey I plagiarized fine, but you focused on my plagiarism more than others’ that not fair”. Even as more retractions due to plagiarism kept coming…
I think we've seen recently that it is a dark kind of funny. When your standards for catching fraud are as low as top academia, and expectations might be beyond the probable without it, it creates a conundrum when someone with a prejudice starts to pick targets to actually investigate.
IANAL, but it seems the suit against Harvard is still in progress, as far as I can tell: Harvard is seeking a summary judgement, and on August 5 the judge granted Gino's request to file "a renewed Statement of Additional Material Facts that complies with the Federal Rules of Civil Procedure and the Court's Standing Order, including the requirements governing format and organization" [1] with 14 days to reply and 14 days thereafter for Harvard to respond in turn. It looks like Gino's team filed the former on August 26 [2].
From the Introduction, the gist of this statement seems (to me) to be that Harvard's finding of grave misconduct cannot simply be taken as being correct, and so the issue should be put to a jury.
Harvard has counter-sued, apparently alleging that Gino had altered and backdated an Excel file presented as evidence that Harvard's investigation had used the wrong data - a claim which which she, of course, denies.
A year ago, the judge denied Data Colada's request for Gino to reimburse their legal fees after the dismissal of the case against them, which strikes me as unfortunate.
We'd find problems with many if not most published studies if we scutinized them the same way. People like him get discovered because they step out of their academic bubble into the light of the world stage.
And mind you this is psychology, the entire discipline is only borderline scientific anyway. But even studies in the hard sciences are very often flawed if not outright fabricated. Adjusting data sets to fit the desired result (who'd ever know as long as you do it properly), changing the experiment and measure until they get what they they want, repeated measuring until the "right" data for the paper appeared randomly - those are all things I personally observed in the lab.
Not to even get started on the house of cards that is basing ones flawed assumptions on other flawed studies without even the slightest attempt at replicating their results first. It's standard practice to simply presume what's been published (especially by friendly authors) can be regarded as reliable.
It is still seared into my mind when a PhD openly stated in a meeting once that she likes to draw the graphs she needs for her papers and then goes looking for the data she needs to support it. I’m sure I can imaging what kind of looking she did based on the fact that she was little more than a slick, fast talking, skin revealing, fast moving, usual suspect con man.
> draw the graphs she needs for her papers and then goes looking for the data she needs to support it
This is a vague statement and it's super important to be precise when blaming someone. I think what you meant is - sketch out the graphs with patterns that support the alternative hypothesis/interesting result and then look for data that make the 'real' patterns look like those in the sketch?
The problem in my view is that everything can be measured like hard science in the first place (not to mention the issue with fraud you point out in hard science).
I am think the idea that a thing is only a science if and only if it's empirical is actually harmful, science used to refer to a more generalized idea of study 'scientia' this could also be mathematics or theology. Then natural science begins to eat the term and becomes more limited. I think that the current place we are is one that is actually has a very dim view of anyone thinking about a topic and is naively optimistic about experimentation in general.
For if one looks at Aristotle theories ideas one may laugh given what we know now, despite using empirical reasoning for lots of his observations, Never the less I think where useful at the time as it's great for a theory to be good but failing that if a theory can usefully explain phenomena and provide one with mastery of phenomena in some limited way it's still useful.
Occam's often abused razor is a way of building theories like this and this whole process is similar to compression, given observable phenomena how do we explain phenomena in the most simple way, this process uses experimental and empirical data but requires thinking about the results of all experiments deeply.
From an outsiders view Natural science's largely seem to have pretty good major theories that lots of results of experiments can be used to correct constants of some kind in the theory and a well established field of study of theoretical physics that largely seems to be operating in the way I outlined.
But the social sciences are rather stupidly obsessed by these little experiment and no one has the guts to build bigger theories these days, for any theory of the human mind has to deal with many variables and if one wants to have descriptive theories of human mind then one needs to do a lot of thinking and theorizing and sometimes separate form empirical data and then arguing the theories using the data and reject the theories using the data, but also other kinds of analysis.
I have the most dim view of modern political science and sociology even more so, they tend to adopt this way of operating the most and to actually operate in that realm they need to theorize the most and experimental data is hard to do properly (and do not neglect history one bit, this is are experimental data and is as good as we can get).
Until you have an objective way of measuring pain in a subject it’s not truly scientific. Hence the endless stream of opioid users lying at the hospital to get their fix.
At this point the surprising part isn't that another Ariely result has problems, it's that some of these datasets apparently went decades without this kind of basic scrutiny
> In 2006, when he was a professor at the MIT Media Lab, Ariely conducted experiments including administering electric shocks with a research assistant who had no human–subject training. […] Ariely confirmed this and said that he wasn't aware that the research assistant did not have the needed one-hour online human–subject training.
A one-hour online training is all that’s needed to be considered ready to conduct experiments involving things like electric shocks on human subjects?
Forget about this guy, the entire field needs to be burned to the ground and rebuilt from scratch.
I would not make such a harsh judgment just because electric shocks are involved.
During my master’s I took a class at the Media Lab called Human 2.0 on prostheses, orthoses, exoskeletons, and human augmentation. Unsurprisingly, all student final projects involved some form of human experimentation.
To run our final project’s experiments, we had to complete a few online modules on bioethics, first aid, and safe practices, as well as submit a proposal to the IRB. From what I saw across projects, the board applied appropriate scrutiny and made helpful contributions.
It classified our experiment as low risk, and a lot of care went into participant selection and adherence to the approved protocol, but they did not require any health professional to be present during the sessions.
Other projects, however, did require such support and even had some MIT Medical involvement.
Incidentally, my project also involved administering electrical shocks to subjects, albeit very mild ones (tACS to induce directed visual artifacts).
I don't know the details here but I am a psychologist.
The "electric shock" normally used in psych studies is tiny, it's like the equivalent of one of those little toy prank buzzers. You couldn't possibly hurt someone with it, this is really a non-issue.
Related side note: Electroconvulsive therapy uses much larger electric shocks but, administered safely by a clinic, is extremely safe, has relatively minor side effects compared to e.g. SSRIs, and is very effective for severe depression. It's a huge shame that it's undeserved reputation prevents it from being more popular. Blame "One Flew Over the Cuckoo's Nest" for its inaccurate portrayal.
ECT can be very, very unsafe. 70% of patients experience some form of memory loss and at least 30% of patients experience self-reported permanent memory loss. The rate of major adverse cardiac events found in a 2019 meta-analysis was 1 in 39 patients. For the people that it helps, I'm happy they've found relief, but it's a tool with significant costs and I know a fair amount of people who underwent it as children or otherwise were pressured to undergo it and I don't know any of them who have a positive impression of it.
This isn't responsive to the parent's question, in a way that ironically underscores the point: you can tell me that "1 in 50 patients experience X", but that's meaningless, unless I know what X is in the selected population at baseline.
I don't mean to pick on you. This problem is widespread in academic research.
> Electroconvulsive therapy uses much larger electric shocks but, administered safely by a clinic, is extremely safe, has relatively minor side effects Blame "One Flew Over the Cuckoo's Nest" for its inaccurate portrayal.
YMMV. I’ve encountered people who have had ECT forced on them whilst inadequately sedated. They have been older/elderly but it would be a mistake assuming it’s always without causing harm.
There are multiple accounts that describe severe pain and the New Zealand government has paid out compensation for the torture that it was. What was being done here was most definitely intended to be painful torture, so maybe isn’t a fair comparison with clinical ECT for actual treatment?
‘Bryon Nichol spent six months in the Lake Alice unit at the age of 12. He said he received electroconvulsive therapy (ECT) many times, which was always unmodified, and without a muscle relaxing anaesthetic.
"The nurses would come down and drag me upstairs. I was often so petrified that I soiled myself. They would put a mouthguard in my mouth and held me down while Dr Leeks gave me ECT. The ECT would give me vicious pains in my head and make me feel dizzy and as it continued the pain got worse. My arms and legs flailed about. It was absolute agony."’
And another:
‘Rangi Wickcliffe, 59, is another survivor of the Lake Alice unit.
He was 10 when he was first given electric shock treatment. Dr Selwyn Leeks turned the machine on, the pain that you feel from the electricity surging through your head is indescribable. The scream that comes out of your mouth is also indescribable.
"The terror was so intense that you lost all bowel movement."
He said it felt like his bones would break.’
For what it’s worth, I wasn’t even thinking about the physical danger when I was… shocked… that you only need a one hour online course. It’s that they’re interacting with people in a somewhat clinical way as an implicit authority. The shock would obviously cause discomfort and therefore stress (or why even do it).
Do you think a potential bad actor would change their behavior if they took, say, a 12 hour course instead?
I am of course biased, but I have often been exasperated by the various "trainings", IRB processes, ethics classes, etc, that are required to run even basic psychology studies such as surveys.
Meanwhile, down the hall, the medical department is opening the skulls of mice, implanting electrodes, giving them supercancer, and then euthanizing them en masse. Often with less approval burden.
The hard truth is that it's tricky to decide what's ethical in science; we mostly allow or disallow things because they've been done that way for years, and all the added "process" is nothing but paper pushing and ass covering to quiet concerned outsiders.
> Do you think a potential bad actor would change their behavior
What is the appropriate way to help a righteous but underpaid lab assistant find the guts to call out a process as unethical and insist that the boss’s claim that the IRB has approved it is bogus?
The most important thing is they know how to report issues and have assurance that reports will not hurt them. If you can show other reports were acted on and the reporter not harmed that helps.
It also helps to always have more than one person who could call so they compete with each other to be first because they don't want to be the one seen as an accompanist to fraud.
It’s not about bad actors. It’s about putting untrained people in a position where subjects might be vulnerable and come to depend on them being the trusted expert in the room.
> Electroconvulsive therapy uses much larger electric shocks but, administered safely by a clinic, is extremely safe, has relatively minor side effects compared to e.g. SSRIs, and is very effective for severe depression.
The evidence base for these claims appears to be missing.
It's liked by some practitioners, not by others, but actual studies in this area are low quality, didn't follow up with longer term checks, and are generally inadequate.
"The failure to find any meaningful long-term benefits compared to placebo groups are particularly distressing."
“Dr. Venkman, we believe that the purpose of science is to serve mankind. You, however, seem to regard science as some kind of dodge or hustle. Your theories are the worst kind of tripe, your methods are sloppy, and your conclusions are highly questionable; you are a poor scientist, Dr. Venkman. And you have no place in this department or in this university.”
> It revealed that Ariely had a longstanding relationship with Epstein, including having visited him at his New York home on several occasions.[45] One of the documents is an image of an email from Ariely to Epstein, in which he requests the name and email of a "redhead" that Epstein had previously introduced to him
I mean, I've always consigned 100% of NYT self help slop to the bin of books for people who aren't smart but desperately want to seem like they but this is just a little much now, Dan
Ah yes also the author of “The Honest Truth about Dishonesty”. The most amazing thing is he is still a professor at Duke. Wonder if he focused much on academic dishonesty in his book…
Institutions like Duke benefit enormously from having people like this around. They can claim they have lots of highly cited and well published researchers, a bit of controversy is free publicity, etc. They make money off him and so they defend him. It helps support the myth that they are good educational institutions. US private universities are grift mills.
This happens at any scale. Manager picks a bad candidate to join the team? Well now they are responsible for the decision. Their worth is tied to their bad decision and they will defend the bad choice to the detriment others on the team. So it’s an element of Duke also defending their honor since they invited and supposedly vetted him.
I mean being a known fraudster, requesting money from Epstein, and asking about some girl he met through Epstein is context too which is not a good look.
He may be innocent, but he certainly has a large stain and stink about him.
This. Significant portions of the Epstein files are random noise. Though even for a popularish author it'd be strange to be in enough 1990-2018 news snippets to reach that many EF hits. In any case "is in the Epstein files" should in general be replaced with "is in the Epstein files, including this email to/from them I found"
The most interesting section from the large paragraph on Wikipedia: “One of the documents is an image of an email from Ariely to Epstein, in which he requests the name and email of a "redhead" that Epstein had previously introduced to him”
That still doesn't mean anything. Sure we know Epstein did evil, but not everything he did was evil. One should assume at least a few people he interacted with had no reason to suspect his evil actions.
I guess that's why the same passage says Ariely maintained the relationship long after the Epstein's pedophile convictions. Not definitive either, but pretty suggestive.
Go read it yourself. There isn't enough information to draw concussions. Everything is consistent with contact with a major funder who you can't cut off just because of evil in a part of life you knew nothing about.
You can read the article with a bias and ignore his counter claims. However that is drawing a conclusion not warranted.
Now perhaps someone has read the letters and seen other evidence and can draw conclusions. However at this point I can't say who knew what.
Both people are of course evil, but they may or may not have an evil in common.
Asking THE sex trafficker pimp extraordinaire about the contact data of a “redhead”… I guess you CAN read this whole interaction as benevolent if you squint hard enough to the point that the nature of this guy’s transactions with people becomes invisible
I have a policy of "innocent until proven guilty". (even proven guilty doesn't mean guilty, but by that time so much evidence has been examined that odds are high enough to finally say guilty). All too often somebody pushes only evidence that supports their bias without full care for the full truth.
I agree it looks pretty bad, but I have not seen all the evidence and thus I must hold out the possibility he is innocent until someone does take the time to examine it all. That doesn't mean I think he is innocent, just that I'm carefully withholding judgement because I know I do not have all the facts. It also doesn't mean I'm against investigation of the facts - I just want the investigation done properly, and all discussions like this do not contribute to a proper investigation.
That’s fair. I just fear the justice system or whatever it should be to decide what the “truth” is might not be up to the task or not that interested in truth anymore. Like for example there’s this one person that everybody in the world knows who bragged about doing despicable things and there’s dozens of women who for years have been accusing him of doing similarly despicable or worse things to them, but it doesn’t matter because he’s innocent until proven guilty… So at some point I just allow myself to draw my own conclusions
> The messages revealed that his friendship with Epstein lasted for nine years, including well after Epstein was initially charged with soliciting sex from a minor.[52] Ariely also suggested that Epstein use learnings from his research to help rehabilitate his public image
what's clownish is defending anybody who would associate with a billionaire pedophile sex trafficker
anybody in the files who helped him rehab his image should rightly be consigned to the dustbin of history
>what's clownish is defending anybody who would associate with a billionaire pedophile sex trafficker
>anybody in the files who helped him rehab his image should rightly be consigned to the dustbin of history
It's possible to denounce someone, but still push back on false claims. Otherwise you're just condoning misinformation just because it aligns with your beliefs . eg.
"Did you know Donald Trump literally eats babies?"
"Wtf are you talking about? There's no evidence of that."
"Wow, I can't believe you're defending a convicted felon, sex offender, and authoritarian! Anyone who's trying to rehab his image should be consigned to the dustbin of history."
sure, that's a great principle but is it applicable here? Ariely is noted having sent an email asking Epstein about 'that redhead' (along with hundreds of others and his work rehabbing the image of someone with a known pedophilia sex trafficking indictment)
if you assess the original claim against the facts, is it as ludicrously out-of-step as someone claiming that Trump found A Modest Proposal persuasive?
To be clear: the implied answer is "pedophiles". Pedophiles benefit the most from "being mentioned" == pedophile
Because obviously, there are many people mentioned in different contexts and it is unclear how many of them are pedophiles.
There are people who may disagree, but I'm from the generation who considers molesting children a serious crime--much much more serious than being mentioned in some email.
Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematics. I was excited to read it, but it didn't take long for me to find out that it's complete BS-derivations that don't follow, made-up equation terms, and conflation of basic terms in the field. I was really surprised that this passed the editor and at least two peer reviewers. I dug deeper and found that the same author has published 11 research articles in top journals in 2026 alone — mostly physics and chemistry, and adjacent fields— and I'm talking about really top journals like Scientific Reports and Physics Letters B. Most of these have serious mathematical flaws that are unfixable. I wrote to a couple of editors, but only one took it seriously enough to initiate an investigation. I made a promise to myself that if no real action is taken, I will seriously consider changing careers.
Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability.
The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it:
1. What exactly is the scientific method? It's not like there's a constitutional amendment that lays it out. Try and pin people down and you'll discover nobody agrees on what science actually requires.
2. Even if some subset of people did write down a definition, nobody is checking anyone is following it.
So people conflate science the Platonic methodological ideal with Science™, the institutions that claim most forcefully to be its avatar. But those institutions have neither a definition nor any enforcement arm, so fraud and abuse runs rampant. There's no reason why anything should self correct in this environment outside of corporate science, where the self-correction mechanism is managerial intervention or, in the absence of that, bankruptcy.
> The core problem is people conflating science with Science™.
You fell into the same trap by criticizing scientific institutions and the human spirit in an attempt to criticize the scientific method.
> What exactly is the scientific method?
What a ridiculous argument. What exactly is a hamburger? Just ask around and you will find that nobody agrees if a veggie burger should be considered a hamburger, or if you can replace the bun with lettuce and still call it a hamburger. And even if you get people to agree what a hamburger is, nobody is checking that the recipe is followed to the letter. Therefore, real hamburgers do not exist, and nobody can claim in good faith that they are making a hamburger. In fact, why don't we sue McDonalds for their fraud hamburgers?
> Science isn't self correcting.
The very fact that fraud is identified is clear evidence that is is not true. You are going to need to sharpen your rethorical devices if you want to argue against hundreds of years of scientific progress, including for example everything that enabled us to argue about this in front of our devices while barely 150 years ago people had to use candles to see each other after dinner. Mind you, not all of this was developed by industrial research labs, in fact the origins of very many modern technologies lie in academic research labs pursuing ideas that are too risky for a commercial entity to bet on. And of course there are plenty of examples of industrial fraud happening despite "managerial intervention" (lol).
Reality is more complex. Not many things, if at all, exist in this world under their platonic ideal form. That does not make them any less real. Yes, we all know that humans are not perfect, sometimes are selfish and chase money and power, not only in academia but also in business, politics, and so on. Wherever humans are involved this sometimes happens. But saying that nothing good ever happens because of this goes against everything one can see when looking out of the window. Scientific progress is real, and needs both industrial and academic labs.
We ought to be able to trust the public with complex realities and contradictions. The scientific method and dogged researchers have brought us many wonderful things, but at the same time fraud happens and the layman might reasonably question to what extent the academy attracts fraudsters and to what extent it creates them.
If we can get over the absolutes then we can discuss policy changes.
Graduate student collective bargaining is one experiment; how can anyone blow a whistle on malfeasance with such an extraordinary and widening power imbalance?
Totally agree with this framing: trust in public institutions (scientific, governmental, etc.) is a common good that needs to be promoted and safeguarded.
Identifying and punishing fraud is a part of it, whistleblower protection another, modernizing the system so that graduate students are not so dependent on individual advisors yet another idea. More and more personal contact between the public and scientists could be useful too, for example via science fairs, meet & greets and so on.
Are you living in the same reality as the rest of us?
Now, if he'd laid out the financial incentives of all of it and said "everyone is only in it for the money" I'd say that's a a little too cynical. But he stopped short of doing that.
> The very fact that fraud is identified is clear evidence that is is not true
That's not merely enough, you need a clear trend that the rate of fraud discovery is greater than the rate of fraud production, not just that "some cases are found out". Good luck with that, since it's by definition not measurable.
Science is very simply a commitment towards objectivity. That's it. You see data, you conclude x,y, or z. and thats it. No external subjective influences changes the conclusions. If the data trends towards "vaccines saves millions of lives", thats it.
You're right that we don't need a universal definition of hamburger. That's because the government stays out of it, so private individuals can use things like Google Maps reviews to police a common sense definition of hamburger along with quality and other nuance.
If governments were forcing everyone to buy anything advertised as a hamburger no questions asked, then the definition of hamburger would matter a lot.
The devices on my desk are 99.9% the product of the private sector and owe nearly nothing to academia.
If people can't define it, then that seems to me like a valid criticism of the scientific method. You can't have institutions that claim to be doing science if they don't use the same definition of science as other people do.
They said "nearly nothing". As a fraction of total effort I'd say that the contribution of academia is relatively low. If you're going to say something dumb like overweighting foundationality, I can counter by reminding you that all of modern statistics and thus academic science is literally downstream of industrial beermaking (and gambling) and not academia
Yup, that is also how it works with scientific research: Once an article is published, anybody is free to read it, criticize it, reproduce it. That's because the government stays out of it: private individuals can use things like Google Scholar and Google Search to find other people's reviews and follow-up studies to better understand the quality and nuances of the original research. Consensus is an important part of science just as it is for reviews on Google Maps. Is consensus sometimes wrong? Yes, no doubt. Is it useless? Nope.
> If people can't define it, then that seems to me like a valid criticism of the scientific method.
As we all know, oeer review is not a very reliable signal about the veridicity of the claims, it is merely an approximate quality gate indicating that a study was done following appropriate standards. There needs to be more awareness of this. Nonetheless, in some fields peer review have been public for a long time, and more and more journals are making peer reviews publicly visible. This is a good trend that we should all encourage.
I also think that it sounds like a good idea to hold scientists accountable for the quality of their research, but moving away from fraud (which should unquestionably be punished) towards merely "low quality" is a very slippery slope. Who determines when the quality is good enough and under which criteria? Citations and follow-up research already are an approximate signal, why is that not enough? Proving a study false does not automatically make it low quality. How much money and resources would these additional measures cost? Would the savings from fewer "bad quality studies" offset these costs? How many fewer "good quality studies" would be prevented by redirecting these resources.
My take on the self correction is, that later research usually cannot be build on wrong results. So over the decades or centuries the correct results prevail.
Later correct research can't build on previous wrong results. Later incorrect research most assuredly can build on previous wrong results essentially indefinitely.
I'm not constructing a full differential equation model for an HN comment, but I'm pretty sure any realistic model of likelihood of detection of the falseness of past research and other similar parameters would produce a surprisingly sharp cutoff when looked at over time, above which the science does tend to self-correct and below which it disintegrates into endlessly recursive garbage drowning out any real science being produced. And I'm pretty sure a number of fields are below that threshold right now. That doesn't mean the problem is permanent... the parameters can be changed over time, too, but the problem does get worse and harder to fix if allowed to accrue too much.
>My take on the self correction is, that later research usually cannot be build on wrong results.
AFIK, it does not seem to happen much with mainstream medicinal research. Many layers of fraud/incompetence/assumptions for centuries with no correction in sight. Of course there are certain sub-areas that may have advanced. In fact it's the modern day religion, having replaced the god centric religion of yesteryears.
I'll very happily (as a significant benefactor) take this century's medicine over the last. And the one or two centuries before that aren't even worth starting to think about. So something must be working wioth mainstream medicinal research.
You'd think so. I used to believe this too. Surely you try and build on a result which creates an implicit replication study, then your experiments don't work and the problem is discovered pretty quick. Hence, self correction.
It often doesn't happen. The reasons seem to vary by field. We do see it in a few very hard fields like materials science.
In the softer end of social sciences like anthropology, education or the humanities it's obvious why not: they don't make empirically testable claims about reality anyway so there's nothing to replicate. This is the Arday problem, where he was publishing what they call auto-ethnography: effectively just blog posts about the author's own feelings. He even interviewed himself in the third person.
In the harder social sciences it's because nothing actually builds on anything else (see "A problem in theory" [1]). Psychology and related fields like sociology, criminology, etc are basically just a pile of random ideas pooped out in brainstorming sessions. There are hardly any overarching theories generating testable predictions, so studies test whatever random claims the researcher thinks is interesting enough to get in the news. Everything is just rocks lying on the ground rather than towers of understanding, so when one paper is overturned it has no impact on anything else. Citation counts obscure this fact, but do some reverse-citation checks and you'll find a lot of citations are worthless or outright invalid.
In modeling based fields like economics, epidemiology or climatology, it's because they can't put the thing they study in a lab and do high speed experimentation. "Replication" comes to mean repeating the same analysis on the same data set. Often they literally just run the same program on the same input file, get the same result and call it replicated - or sometimes they call it replicated even if the outputs don't match! See [2]. But the program is just a set of assumptions, not real discoveries, so building more papers on top of that program's output doesn't detect invalidity. It will all replicate in some very narrow technical sense without that proving anything useful.
Sometimes outsiders notice what's happening but over time academia has built up a complex web of lore that protects them psychologically. For example, they routinely reject or ignore feedback from outside their field on the basis that it doesn't come from "experts", meaning themselves, or it comes from "right wing" people, meaning anyone who isn't an academic or close political ally.
Whilst I weakly agree with this characterisation, it's not at all fair or accurate to put epidemiology and climatology in the same category as your other examples. Yes they have some weaknesses in replication practices (especially for purely computational work), yes the feedback loop for self-correction can be slow, but it would be a mistake to conclude or imply that the major results of those fields are therefore unreliable.
That's because those fields do still make objective empirical predictions about the world, which might take years to be testable but are still ultimately testable. For example, it's been convincingly shown that climate models from decades ago predicted our current climate pretty well. [0]
These fields are more comparable to subfields of physics which only get to do big experiments a handful of times, e.g. because they can only observe so many planets/supernovae/universes, or because they have to spend decades building a new billion-dollar machine to test new hypotheses.
We should also not confuse these arguments with uselessness - soft social science work can still be societally valuable without being easily empirically testable.
Disagree, those two fields are extremely unreliable. It takes a lot of things happening simultaneously to be scientific. Yes, those two do make testable predictions, sort of. But that isn't enough.
Modern (computational) epidemiology is rife with unscientific practices. They ignore data that shows a model was invalidated so the fact they make testable predictions isn't really useful. They also engage in a lot of circular reasoning, buggy coding and logical fallacies. During COVID I wrote a whole report on this topic for some politicians [1]. But the biggest issue is "A problem in theory" again - epidemiologists conflate fitting a curve in R with developing a hypothesis, so the field is overrun with overfit models that aren't based on any refinable theory of disease, just misuses of statistics. Even if you prove a paper's predictions were wrong it changes nothing because nothing built on it anyway.
The problem in climatology is that when the models don't fit the data they just change the data and claim victory, e.g.
> The problem in climatology is that when the models don't fit the data they just change the data and claim victory
That is... not remotely true, nor supported by either of the links you shared. (Nor by any cursory look at the state of the world today, with well-anticipated Himalayan glacial floods as front-page news.) You therefore appear to be leaping over the line from healthy skepticism to irrational conspiratorial thinking, so I'm going to stop engaging, sorry.
The two links I give show the problem in action. In 2013 the IPCC was concerned because temperatures had gone sideways for a decade. Der Spiegel's first sentence is "Data shows global temperatures aren't rising the way climate scientists have predicted" and the next says they were trying to "hush it up". In 2015 climatologists announced a new temperature record that replaced the entire 21st century with different data, as reported by Nature.
That's an unambiguous sequence of events. You can't make statements about replicability in an environment where they routinely decide their already published data is wrong post-hoc, without retracting the papers built on that data.
The situation of [2] is crazy. Because both the author of [2] and the people using Ferguson’s model are wrong. And the author's advice in one of the last paragraphs of we just need better programmers is also wrong.
The issue is that pandemics are extremely fat-tailed in terms of fatalities [3]. You need to collect a shit tons of data to estimate any reliable parameters and no one has time for that. The only thing you need to know is: Is this a super contagious deadly disease? If yes, lock everything down! This is pretty much what they did in Asia, and countries like Vietnam had zero cases for months and was open while the rest of the world was closed.
How to decide a disease is deadly and super contagious? You need disease experts (who most likely also academics) who understand the nature of the disease, and not better modellers and programmers as suggested in [2]. The issue is that western countries will never let any expert to make that call, so a lot of convincing needs to be done through complicated modelling and real life suffering.
Western countries won’t let an expert make the call, because there hasn’t been recent enough pain and suffering to put it in recent memory the necessity for it.
After all, why suffer if it isn’t actually necessary?
Asia had recent experience with the first SARS, which had very real and extremely high fatality rates, so for them it was a relatively easy (not an easy) call.
For the west, when was the last time we had a real pandemic that actually killed a lot of people? Almost a century ago.
Somewhat ironically because the west listened to experts for a very long time.
Notably, the last time it had a major issue also coincided with a world war.
Economics is a particularly slippery subject, since double blind experiments are practically impossible. No doubt some of the things Marx wrote correctly describe reality, even if some of the social experiments built on his work were massive failures.
I mean more the Marxist school of thought, not just the economics. Things like critical theory and post modernism.
But also, he made predictions of the near future in his time. They were incorrect. No updates were made. Those who did try to refine and update his ideas are far less influential. His original theory survives like gospel. I wish I could say only by non-academics.
Edit - back to economics. It's funny. I would say in economics the closest thing we have to double blind wrt economic systems is socialism/communism vs. capitalism. It's clearly not a true double blind. But also clearly some people just don't care to feed that data back in to their models. Because they aren't scientific.
Data Colada is a handful of volunteers who do this kind of thing as extra-curricular activities, and rely primarily on tips. That is a tiny drop in an ocean of fraud, for which the institutions get no credit at all. Academic institutions don't systematically check for fraud, but they do systematically ignore fraud reports that come from outside the academy (of which there are many).
That's covered by trying and seeing what sticks. That's really all there is to it. I already got downvoted for saying this in other words - feedback against observed phenomena is what matters. Everywhere you see shitty "science" you see either missing or ignored feedback.
I am not sure what your proposal is? It seems to be the solution is only more science? This meme that science can't solve anything or is ineffective is also a meme that needs to die. Science is about the only thing that we have to make sense of the world. What else is there?
> 1. What exactly is the scientific method? It's not like there's a constitutional amendment that lays it out. Try and pin people down and you'll discover nobody agrees on what science actually requires.
I am pretty sure the only necessary requirement is repeatability. As far as science is concerned, it does not matter if Jesus did in fact born from a virgin mother, it is not repeatable, and therefore not scientific. We could be entirely wrong about gravity, maybe Earth is indeed only 5 years old and God just created everything to appear the way it is.
Replicability isn't enough to be scientific. Many studies have invalid methodologies, so replicating them by following the same method just propagates wrongness and makes the problems worse. And if you don't follow the same method the original authors will claim it's not a replication study.
> Replicability isn't enough to be scientific. Many studies have invalid methodologies, so replicating them by following the same method just propagates wrongness and makes the problems worse. And if you don't follow the same method the original authors will claim it's not a replication study.
Replicability test also happens in the real world. If gravity is not actually 9.81 m/s^2 then we will eventually have to come to term to that. For all we know, we could be completely wrong about quantum mechanics and the reality of subatomic particles, but it doesn't matter because we can use the predicted behavior and it actually works.
The question is to ask: what is one thing that I would never discard from a scientific principle. If you remove replicability, it does not matter if you have accurately captured a measurement of the universe once, it is completely pointless if you are not able to replicate it again.
>My proposal is: let science be done by those who need its results to be correct.
A tangential proposal - individuals (not orgs) should offer to upfront fund ( in lump sum) promising researchers for life. That way a researcher is not worried about losing his livelihood. I suspect this may already be happening in a few circles without much fanfare.
Why are individuals better that individuals acting together as an organisation better?
If that’s your criteria, then shouldn’t said funding go to an individual scientist who has no lab, no staff, no PhD candidates, and no access to the internet or existing research.
Because organisations are bad and only individuals are capable of making Correct Decisions.
>Why are individuals better that individuals acting together as an organisation better?
Most organizations are made of of very mediocre (or political) individuals, and when it comes to wrong (or political) decisions individuals cannot be held responsible, that how orgs deflect responsibility.
On the hand when an individual (often wealthy) or a bunch of them decide to fund something (often from his own pocket), they know what they are doing and are personally willing to bear the risk.
>Because organisations are bad and only individuals are capable of making Correct Decisions.
I should have been more nuanced.
Org = generally public orgs where the people in the org do not have a stake beyond their jobs in the org. Example: govt. As an exception to this rule there can be (private) orgs of very competent people who deploy they own money to get things done, including research.
(p.s - I realize that I cannot delve into all all nuances with a few sentences I put here, but hopefully you get the drift)
Science that is science is self-correcting via prediction and experiment. The key piece is the feedback. The "scientific method" as taught in grade school hardly matters and is basically cargo cult. The "is this science" lint is "are you working toward more accurate and especially predictive models of observed phenomena? Are you actively trying to destroy your theory?" If the feedback is missing or ignored, it's just opinion pieces.
Edit- I don't know what's controversial about this? Feedback is the self-correcting piece.
As long as people mislead or make mistake unknowingly, it's fine to rely to self-correcting mechanism of science. The institutional rot is much deeper. It is not good to leave all this fraud to self-correcting mechanisms of science. We also have "trust" to worry about.
Maybe instead of throwing around ideas of "institutional rot" we can realize that there are hundreds of thousands of people doing "science" in the right way, are not out for material gain, do not engage in fraud, and are interested in truth and scientific advancement.
The fact that there are a vanishingly small # of cases where bad things happen in the process are the exceptions that prove the rule, IMHO. Certain segments of society like to hype these up (or even lie about their frequency) in order to discredit the scientific method.
"Institutional rot" is a meme that needs to go away. Identify the problems at the edges, fix them, and move on to letting the bulk of the work continue as it always has - quietly building upon previous valid results to the betterment of society.
It's a bit like how people believe markets are self-correcting. Yes, only over an infinite time horizon, in theory.
Edit: and like chemical equilibrium, you could say it depends on the "rate of wrongness production" compared to the "rate of correction".
Edit edit: self-correction of course must fight against the higher virality of work selected for its impact rather than correctness, and incorrect work can then spawn further incorrect work, like an autocatalytic process or epidemic.
They wouldn't dread being included there, not at all. The institutions don't care about fraud and reflexively defend it, ignore it or shoot the messenger. Look at the Arday nonsense. People are still discovering new lies he was telling but Cambridge went to the mat for him.
Dan Ariely in particular has faced convincing fraud accusations for years:
Journalism is an interesting case study. Their revenues went into sharp decline around the start of the century, dropping by more than 50% in some cases. Concurrently polls showed trust in journalism was plummeting, that people were tuning out because they felt that what they read wasn't reliable.
Journalism didn't solve this problem by rebuilding trust and their audience. Instead they started reforming as non-profit institutions and taking huge payments from philanthropists to push specific agendas, without alerting readers to that fact.
These grants aren't tied to readership but rather to the content of what's published, so they can produce lots of articles even if nobody reads them. Those articles are then used as "reliable sources" to control Wikipedia, and from there what search engines and LLMs say.
If students start checking out at scale (already happening), or if governments cut funding (already happening in the USA), their most likely move is to just replace the revenue with money from the Gates Foundation et al, and keep going just as they are. Their primary role will switch from education or research to the propagation of approved narratives.
News reporting and journalism has always mostly been a for-profit enterprise. And no matter what business model one might choose (ads, subs, charity, etc) there will be perverse incentives working in conflict with pure truth seeking and newsworthiness. Audience capture, pleasing advertisers, the politics/interests of the owners, etc.
I'm sure non-profit sponsorship has it's own issues, but I don't think those issues are particularly new or unique or transformative.
The demand side is arguably the more intractable issue with journalism, since loads of people don't select for truth or importance in their news - or at least not very carefully. The institutions that seem to hold up the best are the ones with the broadest audiences.
On the issue of public trust in journalism, I don't think it's actually a function of the quality or trustworthiness of journalism. I think it has more to do with the rise of radical right-wing propaganda infotainment, a core pillar of which was (and is) delegitimizing mainstream media (and academic) institutions. The mistrust was created and cultivated. Consequently, there's not much they can do to regain trust, except to work harder at discrediting the radical right-wing propaganda that has been flooding all the zones for the past several decades.
I think ads were better because everyone was aware of the conflict of interest issues and took steps to manage it. They admitted it could be an issue and would have internal firewalls where ad sales weren't allowed to talk to the journalists, things like that. Advertisers accepted it because their goal wasn't to manipulate news reporting.
The problem with the "philanthropic" funding is not only is there no firewall, but it's explicitly pay-to-play agenda pushing. So that culture of keeping revenue and news separated has vanished without the public being made aware of it.
The right wing are saying things like, journalism is fake because they're just pushing a left wing agenda for money. That's the claim that delegitimizes journalism. But it turns out that the rightists are correct and that's exactly what journalists are doing, so it's hardly their fault that mistrust was created and cultivated. You should mistrust people who mislead you about their agenda, and if you go to the AP website today and look at some random article paid for by these grants there's no mention of where the funding comes from. Actually you get a popup begging for donations, saying that because the AP is a non-profit it's extra trustworthy.
If they want to regain trust, a good start would be by refusing to accept money for stories, relabelling all the stories where they did that, and then pay for journalism purely out of subscription revenues or ads brokered through exchanges (i.e. where the advertiser doesn't have any contact with the publishers).
There certainly are problems, but the biggest is that reality puts two other factors ahead of publishing the truth:
1) People don't like having their beliefs challenged. This will slant all news organizations towards the biases of their readers. We see this at all levels. The shorter and more personalized the feedback loop the faster it happens. This makes the news with things like ChatGPT, the feedback is very quick and personalized so it's very easy for people to tumble down the rabbit hole. Same thing, slower, with social media recommendation engines. And online hate communities--they lead people to extremes. But it also happens with traditional media, just much more diffuse. Everybody moves away from the center.
2) News organizations care very much about being able to get images and video of events. This means they are readily manipulated by restricting access to those who report the locally-approved version of the "facts". We used to understand that "news" from Russia was the state-sponsored version, we don't pay attention to the fact that "news" from areas with Islamist control are just about as bad.
And news is always slanted towards the side of controversy. No controversy, people aren't going to care.
However, this does not remotely mean that the right wing news is better. The reality is that it's even worse. All the same forces are at work and then there's an ideological bias on top of that. With news having become a la carte I doubt there's any way to remain in business as an honest news source anymore.
Point (1) is why news institutions that cover a wide range of interests and have a broad audience have tended to fare better. Depending too much on a particular niche audience leads to audience capture.
WSJ and Fox News have the same owners but WSJ still has one of the best and most credible newsrooms in the business (their opinion section is another matter).
Meanwhile Fox News is a lie factory that only tells it's large (but niche) audience what it wants to hear. And now they have rival networks eating away at the pie by being more sensational and even more shamelessly obsequies. They're stuck in a radicalization doom loop.
If the theory is that advertising makes conflicts of interest more transparent, I don't think I agree. Advertisers can be stakeholders across diverse ranges of concerns - conflicts of interest are often indirect. They can influence what doesn't get written, as much as what does. The real bad actors can even just disguise ads as content. We still need to have some basic trust that institutions are respecting firewalls that - by and large - they have to self police
> The problem with the "philanthropic" funding is not only is there no firewall, but it's explicitly pay-to-play agenda pushing
> if you go to the AP website today and look at some random article paid for by these grants
Why do you say there is no firewall? Is that really true? The AP has their guidelines here and I don't see it as substantially different than what a publication might do for advertisers and other stakeholders who may be tempted to exert editorial control:
If you have specific instances where these arrangements have clearly compromised their journalism, that would be good to see.
In the meantime, I think we can do two things at the same time here: resist overdosing on maximal cynicism, but also be clear eyed about the ways money can tempt institutions to throw integrity out the window.
> The right wing are saying things like, journalism is fake because they're just pushing a left wing agenda for money.
> But it turns out that the rightists are correct and that's exactly what journalists are doing, so it's hardly their fault that mistrust was created and cultivated.
Generally speaking, no they are not correct, and most of them are just outright cynically lying, while engaging in the very behaviors of which they accuse others.
> The Associated Press said Tuesday that it is assigning more than two dozen journalists across the world to cover climate issues, in the news organization’s largest single expansion paid for through philanthropic grants.
> The grant is for more than $8 million over three years, and about 20 of the climate journalists will be new hires. The AP has appointed Peter Prengaman as its climate and environment news director to lead the team.
> For many years, Journalists and philanthropists were more wary of each other. News organizations were concerned about maintaining independence and, until the past two decades, financially secure enough not to need help. Philanthropists didn’t see the need, or how journalists could help them achieve their goals.
> Nonprofit news organizations like ProPublica and Texas Tribune led the way in changing minds. The Salt Lake Tribune, which in 2019 became a nonprofit to attract more donors, and The Seattle Times are other pioneers.
> “As far as we’re concerned, it has been really seamless,” said Monica Drake, assistant managing editor at the Times. “It has allowed us to focus on things we want to cover and to do it independently.”
> Both sides had things to learn. For Carovillano, it was getting used to the idea that funders weren’t just being generous; they had their own goals to achieve. “This is a mutually beneficial arrangement,” he said.
> “I feel like our language has gone from people talking about local news dying to there being a renaissance of local news,” she said. “If you can provide good, high-quality journalism, people are willing to pay for that, people are willing to support it. I think there’s a lot of room for optimism. The trick is, can we move quickly enough?”
The article leads with climate grants but is really about the phenomenon in general, which covers many topics.
The quoted journalists are duplicitous, trying to reassure themselves. The text says that what they're doing used to be taboo because rich guys had "goals to achieve", but now it's considered "pioneering" and "mutually beneficial". Their justification is they didn't have any money and felt sad, but now they've got lots of money and are happy. They say the transition has been "seamless", i.e. readers aren't aware of it. And hilariously, at the end Carivillano claims there's room for optimism because people are willing to pay for high quality news - except "people" aren't paying, that's the reason they need to take money from a handful of donors.
It's obvious this compromises their journalism, especially in the stories they don't publish. These news orgs have basically been bought by the donors, large chunks of their staff only exist because of these grants. Thinking it doesn't compromise their news would be like saying Facebook simply buying all the news teams that cover the internet wouldn't have any effect.
This is a puff piece. They are always a bit icky, but they're common and I don't think you'll find any outlet of renown that doesn't succumb to these from time to time.
In this case, it reads like their way of stroking their benefactors egos a bit, but without promising them anything real in return.
FTA: "AP often needs to educate funders upon first approach, explaining the company’s worldwide reach and mission to report independently. AP accepts money to cover certain areas but without strings attached; the funders have no influence on the stories that are done, Carovillano said."
You left out a pretty important force: political forces lying about the trustworthiness of journalism in a blatant effort to gain political power. Traditional journalism didn't have any ready defenses against that.
Even as imperfect as it is, it's infinitely better than the alternatives like belief in religion/religious texts/pastors or anybody else who is just making shit up based on whatever they think sounds good. None of that is falsifiable or makes any attempt to iteratively arrive at anything true or real.
There is literally nothing better that we have other than the scientific method. Yes, science is self-correcting, even if it is slow or whatever else. Nothing else that we have can approach even that bar.
>I made a promise to myself that if no real action is taken, I will seriously consider changing careers.
Personally I think you should name and shame the author as well as the publications and the names of the peer "reviewers" who clearly did sweet-fuck-all.
It does seem like its way too easy to do this kind of fraud.
The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now. Is there some way to change incentives to get people to try and replicate? Assign undergrads to replicate random research papers?
Replication is unpaid QA on someone else's product. Nobody gets tenure for finding the bug, hence the field has been meaning to get around to it for 20 years.
And the incentives are arguably worse than QA: you get little credit if the result replicates, and potentially a lot of grief if it doesn't. Novel results build careers. Checking them mostly doesn't
On the other hand, this seems like a super interesting path for an (unglamorous) PhD -- replicate the N most important findings in a field, subject to some restrictions (maybe focus on the ones that are "easy", or haven't been replicated yet, or the ones that could be patched together to make an acceptable PhD thesis). This sort of stuff really appeals to me, I think I might end up doing something like that in 5-10 years.
I share your enthusiasm for this. Creating a map of the most influential research across all the fields to identify the highest priority work, any work replicating that work or validating that work by building from it would be a good start. Ideally funded by governments, but institutions, companies, and individuals could put up bounties for replicating particular research.
I think you're right that this will be the most common objection, but wrong that it will be exceedingly difficult to find an advisor that would support this. I assume that if you _can_ overturn some foundational result, you won't face criticism over novelty (potentially you can argue that overturning important accepted results is inherently novel). Then the rest is just math:
Assume that n% of foundational studies are demonstrably incorrect if replication were attempted. Then if you replicate X foundational studies, there's a 1 - (1-n/100)^X chance that you'll show one foundational study to be incorrect. Then I think you just need to find an advisor who believes the incorrectness rate n, and the replication attempts X is high enough for you to revert a foundational study over the course of your PhD.
Universities, institutions and publications who care about their reputation should pay people to replicate studies performed by people they employ in their name. Otherwise they will loose their trustworthiness.
But perhaps it is not as big a shame for them as some think it should be...
My default position for a long time has been that all results in the social sciences are garbage. If they go against intuition or my biases, then they are most likely garbage. If they confirm my intuition or my biases, then they are definitely garbage.
Sure but I mean the community as a whole. Why has there been allowed to be a reproducibility problem at all? Why has the consensus not always been that without provably reproducible results it’s not to be taken seriously?
> Meditation and mindfulness practices may have a variety of health benefits and may help people improve the quality of their lives. Recent studies have investigated if meditation or mindfulness helps people manage anxiety, stress, depression, pain, or symptoms related to withdrawal from nicotine, alcohol, or opioids.
> Other studies have looked at the effects of meditation or mindfulness on weight control or sleep quality.
> However, much of the research on these topics has been preliminary or not scientifically rigorous. Because the studies examined many different types of meditation and mindfulness practices, and the effects of those practices are hard to measure, results from the studies have been difficult to analyze and may have been interpreted too optimistically.
This is just a polite way of saying "there's no evidence of them actually doing anything*".
I mean, that's exactly what you'd tell someone who believed in the healing powers of homeopathy or chiropracty(sp?).
That’s different as I don’t think anyone needs science to feel the benefits of yoga or breathwork but lots of people want to enact policies based social studies that are not able to be reproduced.
You don't. It is too costly and thus risky. Instead you do a charade and convince authorities (who just want to cover their asses citing some vetted studies) and people that your stuff work.
It is "ass covering" all the way. Not "does it work" all the way...
My proposal, once you complete your study, if you find a significant positive result you should be required (prior to publication) to describe your methodology in full, and have another lab (who you never have any contact with) blindly take your methodology, and repeat it (on video), and then do their own independent analysis of the data.
It's okay if it's $200k or so to do each of these, the university should pay. The point is if this methodology were applied people would have incentive to cheat anymore. It would also serve as an immediate replication.
Schools won't do this because they have a perverse incentive for big successes and quiet retractions. It's corruption all the way up.
Would be a great way for up and coming researchers to learn and apply skills before attempting to work in labs pursuing novel research.
Maybe some joint effort across a large number of research schools which function as “proportional to number of students in your program, X number of validation studies must be performed” with maybe some monetary or access incentives aligned.
Since we’ve had student loans all but guaranteed, and therefore cost of schooling skyrocketing, the flywheel of pursing higher profits and more bureaucratic cruft (looking at ratios of university staff to students), its no wonder that incentive has extended to research.
N.B. nobody needed to replicate this study. It only took basic sanity checks to spot that it was very unlikely to be true (and thus, very unlikely to replicate). And they went much further than needed. They could have stopped at "the claimed effect is much larger than the height difference between men and women". This obviousness is a common pattern:
> Prediction markets work well because predicting replication is easy. There's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.
So you could also say, how do we change the incentives to get people to peer review properly? That would be a lot cheaper.
The obvious solution is for people to get paid to audit work professionally, and be fired if they sign off on bad results. But that exists already, it's called a corporate lab. Managers are incentivized by carrots and sticks to properly understand what their researchers are doing and if anyone is doing fraud they will be fired, maybe prosecuted. Universities don't do this because so much of what they do exists purely to spend money, with no higher purpose. There's no specific goal or outcome they're trying to achieve, and so no reason to do unpleasant things like enforcing incentives to get there.
So in the end the solution is just to defund university research. The social structures are fundamentally unfixable. You'll never get people to care if the only purpose of their job's existence is to convince voters governments are funding science.
Fund it. Unfortunately there's hardly any money for regular studies, let alone replication studies. If you want better science there has to be tax dollars behind it. Otherwise corporations choose what gets funded, and they only fund narratives and experiments they think will be profitable for them.
> Otherwise corporations choose what gets funded, and they only fund narratives and experiments they think will be profitable for them.
Well, that's the incentive mechanism that keeps the accountability for the results whether we like it or not. The government doesn't immediately go out of business if the research they fund is full of fraud or goes nowhere useful. Moral outrage at all the fraud in research associated with these Universities and government funded studies isn't a powerful enough incentive mechanism to correct that fraud, apparently.
"easy to replicate (no fancy equipment needed, just ask some people to proofread)"
? This was a human subject study, that would cost $$ and there would be very little upside to confirmation and only a little notoriety with a failure to replicate (which by the way is not dispositive, people might need multiple failed replications to know who to believe, depending on the size of the replication). Maybe you didn't read the post: this was data fraud discovered by statistical irregularities. That is indeed becoming easy to spot with more widespread statistical literacy, at least for unsophisticated jobs like this from decades ago.
If the culture shifted such that people were skeptical of original results but excited about replication, replication would be the big news and people would want to do it. How to get there, I don't know.
I seem to remember a lot of hype about room-temperature superconductors, that croaked upon [lack of] replication. That wasn't too long ago.
Before that, cold fusion.
We still have the ion drive that doesn't require fuel. I don't think it has ever been successfully replicated, but I think that it also hasn't been successfully repudiated, either.
This kind of thing has been going on for a long time. I suspect that not all of it is deliberate fraud, but rather, people wanting their preconceived bias to be satisfied.
No, that would never happen. People would just be skeptical of most findings until they see replications (where we are headed in the social sciences, anyway). If people see replications the original discoverer will get all the accolades, not the (several) replications.
No paper should ever be referenced until it has been replicated by an independent group. The replication crisis in science is real, and it won't take long for no one to believe anything anymore.
Does some sort of batched style of research already exist? By batched, I mean a study where n number of teams all agree on a single methodology, and then conduct the same experiment simultaneously. Each team could then publish their results independently and the teams could publish one aggregate.
I know meta studies attempt to do this in a way, but they can sometimes have their own problems too.
I cannot recall coming across this style of research in much of what I have read as a hobby, but maybe it is more common in some domains than others? I am not smart, and there is nothing new under the Sun, so surely something like this exists, no?
Not quite what you're asking, but pre-registration does alleviate e.g. p-hacking and such without the need for multiple teams. It doesn't do much for outright fraud, but still...
In the physical sciences, any sufficiently important paper is independently verified. Sounds like cost/effort of these multi year studies is not faesible to replicate, but then again, I know nothing about social science so w/e
Dan Ariely is well past the point that he needs to not be a faculty member at Duke University. Data Coda do a fantastic job ferreting out academic fraud. With these fraudulent behavioral science experiments like this relying on printed papers/markups, and coauthors not present and who did not even see the raw data, I wonder if the experiment was ever actually conducted. I would love to see that question answered where fraud is suspected, something I'd hope IRBs would step up to do.
Actually the post is ambiguous: did the authors want the original article retracted, or the replication by Hyndman and Bisin? I hope the former!
"Finally, it should be noted that when we shared these posts with Ariely and Wertenbroch a few weeks ago, they reached out to Psychological Science to request that the article be retracted. As of this writing, that process is ongoing."
One thing I didn't know before reading this: an unusually large effect size can itself be a red flag. You don't have to understand all the statistics to at least notice when a result looks almost too good to be true.
It is interesting how all these famous non-replicating psych paper results feel reasonably intuitive to me. Like I can buy that having a more structured and detailed deadline helps you procrastinate less. I can even buy that doing a goofy little "power pose" exercise kind of hypes yourself up to give some self confidence and do better on an interview or whatever.
You would almost expect the opposite and that all the famous studies would be the ones that run counter to intuition. Maybe those get more replication attempts, or people feel less confident faking them.
The direction is intuitive, but the effect sizes are, as you say, unusually large. I'm no statistician, but if you have an effect size that's several multiples of the standard deviation, that would mean that it dominates the natural variation in results right? So you're kind of making an implicit claim that other factors in play (for eg in this proofreading task, how busy someone is or their reading level) are peanuts in comparison.
Further, if there are these relatively simple factors like how a deadline is expressed that make you several times better, you'd expect anybody who stumbles into several of them and stacks them together to be absolutely off the charts. Somebody who listens to Mozart, gets the right priming, doesn't deplete their ego, and receives short deadlines is suddenly like 50X more productive, which doesn't seem to match reality.
Your exact intuition about the implausibility of many large effect sizes is called the “Piranha problem” by my favourite statistician Andrew Gelman first here in a blog post:
https://statmodeling.stat.columbia.edu/2017/12/15/eats/
This is one of the things that did in Jan Hendrik Schön.
I can't remember the exact measurement but he was showing a set of curves that were simply too neat for a relatively small sample size - roughly 100 or so plastic transistors. In reality you cannot get that neat an effect without a huge sample population
Having skimmed this article and the replication study itself (which references some studies that complicate our common understanding of the positive effects of deadlines) I’m left wondering what the constructive alternative is to deadlines and/or how deadlines should be maintained or determined.
There is no alternative to deadlines because time is money, and those downstream of you can ill afford to wait on your deliverables. Part of management's job is to determine how much time it should take so a reasonable timetable that stakeholders can agree to can be set, and then hold you accountable for delivering within that time frame.
I wish we could revive Tim Bryce's mind with AI, because I'd love to hear what he'd have to say in a podcast about this topic:
"Fraud in a procrastination study? You don't say! I don't understand why all this taxpayer money is going to study procrastination in the first place. Procrastination is a well-known phenomenon with a well-understood cause: laziness. Procrastinators didn't have a boot placed to their behinds growing up, so they reached adulthood slacking and coasting on their trust-fund money or couch-surfing at their friends' places, mooching off others. You see this a lot in modern software organizations, especially among the programmers. And now those programmers are all crying because they're about to get replaced by Claude Code. Boo hoo! The military knew how to deal with procrastinators, too: send them to scrub latrines with a toothbrush, and drum them out if they didn't shape up after that! We don't need millions of dollars in government grants going to academic research to study this phenomenon. It's as old as humanity itself—we called it the deadly sin of sloth. Friends, keep the faith!"
Well, no, he was not a pleasant person. Back in the day he was a management consultant, best known for an essay which rustled many programmer jimmies at the time: "Theory P: The Philosophy of Managing Programmers" (2005):
Theory P advances the thesis that because programmers are lazy pseudointellectuals who chafe at discipline and use their arcane technical knowledge to gain power and leverage in the workplace, they need to be tightly managed, their small roles within the company clearly delineated, and discipline and organization strictly enforced.
A more recent (2018) essay of his is much more blunt and to the point, shows that he hadn't changed his mind at all, and concludes with "Is it any wonder why I despise programmers?":
a) for good or ill, he said what your boss—or at least top management at your company—was thinking
b) he was right about many things. He was an exponent of PRIDE, the first widely available commercial software development methodology created by his father, Milt Bryce. Unlike Scrum or even SAFe, which are more like loose frameworks, PRIDE is a complete soup-to-nuts instruction manual on how to develop, implement, and integrate business information systems (including, but not limited to, those which operate via computer). As Bryce himself said: "PRIDE answers the 5 W's of information systems: WHO is to perform WHAT task, WHERE, WHEN, and WHY." PRIDE is pretty much Spec-Driven Development writ large; in PRIDE, the majority of the work and the source of nearly all the value is in systems analysis, not programming: determining how the business operates and what information systems need to be in place to support the business. This is stepwise-refined into a series of more and more detailed specifications, which are only then handed off to programmers to code. Doing it this way saves time and money and leads to more correct software on the first go. It's also pretty close to the state of the art in AI agentic workflows.
Tim Bryce would be giddy with glee had he lived to see today, because finally all those lazy programmers would be put out of work! Since under PRIDE, programming is mere translation of requirements into code, that can now be automated and asymptotically approaches requiring no human effort at all! The fact that you and I dislike his personality doesn't change the fact that what he advocated for is exactly what's happening now, and that business has for centuries and will continue to operate as if what he believed were true—including the bits about procrastination being simple laziness, correctable with stricter discipline, rather than some sort of complex psychological phenomenon.
> and that business has for centuries and will continue to operate as if what he believed were true
Then maybe we should do away with "business". Small operations do not tend to operate that way, and broadly everyone prefers their output (if not price) in the domains in which they operate – with a few notable exceptions. In theory, software should allow small operations to each serve millions of people, such that we (e.g.) only need a few thousand search engine providers to meet the needs of the entire world's population. Not everything has to be enterprise-scale, and indeed many things should not be.
If "business" means treating people badly, then we don't need it.
I am not lazy (for work. I am in my personal life) but when I procrastinate it's due to lack of clarity or due to issues acquiring focus. Both I have ways to address, so I completely agree with you
I hope that with LLMs we'll see a lot faster identification of these anomalies. Imagine that this guy taught at MIT and has his own lab at Duke, basically more than 20 years of taking up some of the most prestigious positions in his field, all based on fraud.
The butterfly effect this has on science as a practice is huge. Some marginal researcher who actually deserved it was bumped out of those places, and somewhere down the line someone was bumped all the way out of academia because of this guy's lies. And this is hardly the only person in this situation.
What's scary is that the next fraudster will read this take down and see what they have to do to avoid detection. It wouldn't be that hard to form data that looks a lot more plausible. Replication may be the only solution, unfortunately it's so expensive and time consuming, without any of the actual benefits of doing the original study. Maybe it should be government funded, the US replication lab, its only mission to replicate the top X papers from some named fields each year.
It's shocking to me how badly the data was falsified. You'd think a researcher who is A. well versed in statistics and B. has seen a lot of actual research data would be able to produce more believable data.
Does this mean we're only seeing the tip of the iceberg? how many papers are based on hard to detect falsifications?
Why bother? Low effort data falsification is enough to land a high-ranking paper. Sure, really simulating data with some noise would be much more convincing but its just not necessary. Raw data is very rarely inspected during peer review, much less so after publication, and most reviewers do not address statistics — if they do, the usually refer to their pet model and/or use the same incorrect tools as most psychologists.
There is such a smell around this whole literature.
I wonder what it's like to be a grad student in social psychology right now. When the norm is surprising results, any responsible, competent researcher is going to realize that they can't do it honestly. So this leaves behind people who are willing to bend the rules, or people who don't see the problem with, for example, the garden of forking paths.
In good universities replication crisis and the underlying structures are taught. (Un?)surprisingly, not much has changed though. Except for some unis, some journals, and a bubble of rigorous research continuing to work on meta-science and replications. Have to admire their stamina.
People for a long time have had far too much faith in peer review as and had a dim view of anything other then experimental results and work, despite how much fraud and just bad experiments their are. Not to mention how many variables their might actually be to account for in something like psychology or any of the social science's.
Deriving things from axioms and postulates alone is problematic but some one who can do both begins to actually do useful work in my opinion and some one who can accept the difficult their actually is in knowing anything about human psychology simply from studying it and underneath all experiments is assumptions and the people in the experiment who have so many variables. This is not like medicine human psychology is very flexible and for me though not a social scientist I feel of all theories of human psychology they are only applicable to people now and not even 300 years ago, though of course one must concede that the micro psychological principles might be applicable to humans more generally.
Short of blindly outsourcing the work to an LLM and checking to see if its hallucinated problems are actually problems[0], how can more laymen (without the ability to run replication studies, and without degrees in statistics) learn to more easily identify questionable/potentially fraudulent research like this?
This is a question that has been really bothering me for a while. I have a short list of things I’ve learnt to check, but it’s all stuff I’ve picked up in a very ad hoc way (reading and listening to these kinds of critical analyses, mostly). For example, until now, I wouldn’t think about implausibly large Cohen’s d, despite this seeming like an easily generalisable rule that anyone could eyeball. But without being told, and without the reference points given in the article, I wouldn’t have a clue. And I would like to have more of a clue. And I would like other people to have more of a clue. How?
The reason why people(or ai agents) have not been able to systematically destroy bad research in fields where fraud is so rampant is because they data are not published or easily accessible without emailing the author and asking them.
I sincerely hope that any self respectful journal that publishes papers should have a minimum bar: submit the data on a public link with no paywall or blockers and feature the link prominently on your paper.
In this case even emailing the author didn't work. The data was obtained via some roundabout method.
But it's not the real reason IMHO. You could use LLMs to write takedowns like this one. I gave ChatGPT the paper PDF and some neutral prompts. Although it didn't go straight to 'this is clearly fraud' - it's too nice for that - it did compute and point out the absurdly huge effect sizes, and give a relatively low confidence the paper's central claim would replicate.
The real reason there aren't bots systematically destroying papers is because universities defend fraud, they don't punish it. You'd just be burning tokens to reveal lots of fraud with no outcome.
Dan Ariely remaining at Duke, after it has become clear that much if not most of his work is fraudulent, is at least as big a black mark for Duke as Jason Arday was for Cambridge.
It's arguably worse. Jason Arday was doing qualitative work in education. Nobody pays any attention to anything UK educationalists write. Dan Ariely was a genuine superstar and writer of popular bestsellers (with academic content).
I agree Ariely was more influential and his fraud more flagrant (obviously Arday's lies were more absurd, but they were less academically relevant), but it's not true that Arday had no attention given to him. He did provide advice to government groups on policy [1,2]. Furthermore, the high publicity around his story about being non-verbal until he was 11 probably gave a lot of false hope to people with non-verbal children.
Dan Ariely was one of the biggest disappointments and betrayals of my intellectual life. I found the "research" he did to be mind-blowing, groundbreaking stuff at the time, and I told many people his work, sharing his papers and podcasts as a true enthusiast.
I am astonished that he holds an academic-adjacent position anywhere, no matter what his regrets might be today. Private redemption shouldn't be out of reach, but it shouldn't be possible for him to hold a position at any institution concerned with uncovering Truth. He should have resigned out of respect, if his institution was too gutless to do so. It's a tragedy.
The last point in the OA about rounding the estimate of the time spent on the assigned tasks. Just wondering if the response would depend on the wording of the instructions to participants and the nature of the incentivisation.
Me being a bit of a boy scout / Belbin 'company worker', I would tend to use a timer (I'm a pomodoro method person for things like marking or checking references &c).
The rest, oh yes. I stopped paying much attention to studies based on dozens of undergraduates doing weird concocted tasks years ago.
So this paper appears to be fabricated, but was the conclusion actually wrong? I do appear to perform better on tasks with an external deadline than on self-set ones.
The conclusion is faulty given that it was derived from faulty data. The 2002 paper had two pilot studies and 2 bigger studies. The replication of study 2 failed[1], and the original data has substantial concerning features the rest of the datacolada article lays out. This has two unfortunate implications.
First, if study 2 was not necessary to support the conclusions, it seems likely it would not have been performed or included in this paper. So given it seems necessary, the conclusion is invalid. In past examples (the Reinhart-Rogoff paper comes to mind) when this happens, the authors claim it wasn't necessary and the conclusion is still valid and the professional embarrassment of a retraction is not called for. But in this case the replication failure might stand as a strike _against_ the theory.
But second, this is not the first questionable data coming from Ariely's lab, and it seems unlikely this was a data entry mistake. If study 2 is not trustworthy, we should update our priors about the trustworthiness of study 1. Note it's not guaranteed to be doctored in some way, just worthy of additional scrutiny. And if that one also fails to replicate, the paper and its conclusion seems unsalvageable.
Presumably his coauthor is now panicking about not keeping data from 25 years ago to exhonerate and distance himself.
At the risk of bringing up a controversial topic, in the midst of the Jason Arday furore last month when everyone was like "this wouldn't happen to a white professor", the person who kept coming to mind for me was Ariely. I became a data scientist in no small part due to his early work, and seeing academia totally incapable of punishing him is deeply disappointing.
And yet it will live forever, just like Dunning-Kruger despite its statistical issues. I fully expect to see it cited during some workshop or other pointless training exercise.
Dan Ariely has a long history of controversies and studies with falsified data. He wrote 3 popular books and gained a lot of fame before others started uncovering the problems in his studies
https://en.wikipedia.org/wiki/Dan_Ariely
It’s amazing that problems with his studies are still being uncovered. Equally amazing that Duke still maintains a relationship with him after so many problems have been identified with his research.
The fraud that Data Colada presents here is unambiguous. It was also no accident in my opinion that they are specifically mentioning that the 3 Excel files were “Last saved by” “Dan Ariely”
I'm sure the statute of limitations has expired for this, but I really wish the Feds would prosecute cases like this as wire fraud instead of leaving insular academic institutions to self-police. This type of fraud is done for personal gain and self-enrichment and causes real, identifiable harms.
I'd be pretty wary of stretching wire fraud law to cover this. Prosecutors deciding whether scientific results are fraudulent sounds like it could create its own set of ugly incentives
Fraud law doesn't need to be stretched, academic fraud is covered and criminally prosecutable in the USA for decades. It has to be because they're using these results to apply for grants paid from tax monies.
Occasionally someone is even prosecuted! Unfortunately, even when the government wins, the institutions just treat the resulting fines as a cost of business. See the Dana Farber Cancer Center: prosecuted by a third party on behalf of the government, found guilty of faking their science for a decade+, paid the fine - and nothing anywhere on the DFCC website mentions anything about it.
Thanks for this comment, and I totally agree.
Only important things I'd add
> and nothing anywhere on the DFCC website mentions anything about it
That doesn't really surprise or bother me - I wouldn't exactly say it was a coverup just because they aren't heralding some major Fed case against them on their own website. But I do find it shitty that the Wikipedia page of Dr. Laurie Glimcher, who was president of the institute at the time, had some papers retracted and apparently resigned over the issue, makes it sound like she just retired because she completed her mission and has no mention of the scandal.
The bigger issue I see is that someone wasn't prosecuted, just the institution. I didn't spend a ton of time looking, but AFAICT no individual faced any serious consequences beyond "reputational damage". This is why the "corporations are people" shit is so aggravating. If corporations are people, then some people who committed these crimes should have faced legal consequences, not just have their malfeasance paid out by the organization.
Prosecutors (and juries) already are expected to adjudicate fraud across cases that IMO are much more ambiguous than this.
I think it's fair to debate whether it's a good thing that, as I heard it put once, "everything is wire fraud" (i.e. lots of crimes require lying on documents that are sent electronically and thus become federal crimes), but I see zero reason to think that committing academic fraud (which, again, is done for personal gain and causes real, measurable harm to others) is any less of a crime than lots of other things that are prosecuted as wire fraud.
Perhaps we need a new/updated law to cover this kind of thing. This sort of academic fraud should absolutely be criminalized, but I'm not sure I'm the biggest fan of the "everything is wire fraud" approach to achieving that.
If it gets so bad that a society/academic system needs to criminalize such fraud, they have already lost that race.
The only thing that really works is a culture where this rarely if ever happens, because of the peer respect and self-respect of academics.
Peer reputation and self policing works in a system where no money is at stake.
In the old days scientific research was only done by self-funded rich people with tons of money and free time.
Nowadays science is mostly done by people whose income and food depends on getting grants.
When your career and income are at stake, why would you take the risk of opposing Dan Ariely? Only someone who was already rich could do that.
>Peer reputation and self policing works in a system where no money is at stake.
It works in systems where money is at stake too. You need a high trust, conscientous society. Doesn't mean that having one solves the issue entirely. The ocassional bad apple will fraud there too, but it wont be rampant enough to be a systemic problem.
>When your career and income are at stake, why would you take the risk of opposing Dan Ariely? Only someone who was already rich could do that.
You don't need independently wealthy scientists to expose fraud.
You just need an environment where if regular salary dependent scientists expose fraud, they're not punished for it.
If you don't have that, sure, then you need independently wealthy scientists to dare.
Sure, if culture were different, or human nature we different, we wouldn't have the problems we currently do.
However, given the culture we have, and the human tendencies we have, I think laws can make things better. Like criminalizing various sorts of fraud.
[flagged]
> I wasn't expecting an account named after a racial slur to last so long.
That's a good point. We've banned the account.
>Do you know a trick to pay my bills in self-respect? Most of my creditors demand I pay in money.
Nobody said that shit. You can pay your builds with the monetary salary from doing your job honestly. That was my first point.
If many people discover that such a salary is not enough to get by, or if many people aren't content with "enough to get by" and decide to perform academic fraud for extras, then that's a society that already lost the plot. That was my second point.
> leaving insular academic institutions to self-police
I'm not convinced that academic institutions are actually self-policing, it may not even be correct to assume they each act as one coherent entity. There are a lot of incentives in play which may not be obvious from the outside.
<apologies for yet another anecdote from way back when>
Many decades ago during my PhD I was in working in a chemistry lab when a colleague had an accident which caused a minor explosion, big enough to blow both of the glass side panes completely out of his fume hood.
With hindsight he was extremely lucky to have only minor injuries, he was peppered with tiny cuts from the flying glass and temporarily deaf from the bang.
I called for an ambulance, which seemed like a reasonable response.
The first prof on the scene (not our prof though!) came in and immediately asked "who phoned for the emergency services?", so I said it was me. He then gave me a bit of a telling-off, ending with him saying "and now the press will find out!" before he stormed off.
I remember thinking "wtf?! my colleague has just blown up our lab, and your top concern is that it will be in the newspapers"...
Reminds me of Francesca Gino for data fabrication. She was also a best seller of sorts.
https://en.wikipedia.org/wiki/Francesca_Gino
It should - one of the papers that Data Colada reported on about Gino was co-authored with Dan Ariely: https://datacolada.org/98
She had sued Data Colada, I believe about that post. Seems like it got dismissed: https://www.science.org/content/article/honesty-researcher-s...
Oh wow. From your link
> The fallout from the saga is likely to continue. Simonsohn and several of Gino’s co-authors last year launched a project to assess the reliability of her past papers. So far, this has led to a further retraction in February and a correction in July. Science reported in April that Gino also appeared to have plagiarized passages of text in some of her publications.
It’s also hilarious how the judge let her continue the suit against Harvard. Like “hey I plagiarized fine, but you focused on my plagiarism more than others’ that not fair”. Even as more retractions due to plagiarism kept coming…
I think we've seen recently that it is a dark kind of funny. When your standards for catching fraud are as low as top academia, and expectations might be beyond the probable without it, it creates a conundrum when someone with a prejudice starts to pick targets to actually investigate.
Kudos to Data Colada for sticking to their guns and not being cowed by that law suit. More power to them.
For your amusement, from the post:
> a week after Francesca Gino sued us for $25 million […]
That wasn’t successful but she also sued Harvard and won
IANAL, but it seems the suit against Harvard is still in progress, as far as I can tell: Harvard is seeking a summary judgement, and on August 5 the judge granted Gino's request to file "a renewed Statement of Additional Material Facts that complies with the Federal Rules of Civil Procedure and the Court's Standing Order, including the requirements governing format and organization" [1] with 14 days to reply and 14 days thereafter for Harvard to respond in turn. It looks like Gino's team filed the former on August 26 [2].
From the Introduction, the gist of this statement seems (to me) to be that Harvard's finding of grave misconduct cannot simply be taken as being correct, and so the issue should be put to a jury.
Harvard has counter-sued, apparently alleging that Gino had altered and backdated an Excel file presented as evidence that Harvard's investigation had used the wrong data - a claim which which she, of course, denies.
A year ago, the judge denied Data Colada's request for Gino to reimburse their legal fees after the dismissal of the case against them, which strikes me as unfortunate.
[1] https://www.courtlistener.com/docket/67659904/gino-v-harvard...
[2] https://www.courtlistener.com/docket/67659904/258/gino-v-pre...
Harvard, nice.
We'd find problems with many if not most published studies if we scutinized them the same way. People like him get discovered because they step out of their academic bubble into the light of the world stage.
And mind you this is psychology, the entire discipline is only borderline scientific anyway. But even studies in the hard sciences are very often flawed if not outright fabricated. Adjusting data sets to fit the desired result (who'd ever know as long as you do it properly), changing the experiment and measure until they get what they they want, repeated measuring until the "right" data for the paper appeared randomly - those are all things I personally observed in the lab.
Not to even get started on the house of cards that is basing ones flawed assumptions on other flawed studies without even the slightest attempt at replicating their results first. It's standard practice to simply presume what's been published (especially by friendly authors) can be regarded as reliable.
It is still seared into my mind when a PhD openly stated in a meeting once that she likes to draw the graphs she needs for her papers and then goes looking for the data she needs to support it. I’m sure I can imaging what kind of looking she did based on the fact that she was little more than a slick, fast talking, skin revealing, fast moving, usual suspect con man.
> draw the graphs she needs for her papers and then goes looking for the data she needs to support it
This is a vague statement and it's super important to be precise when blaming someone. I think what you meant is - sketch out the graphs with patterns that support the alternative hypothesis/interesting result and then look for data that make the 'real' patterns look like those in the sketch?
That's just coming up with a hypothesis.
The problem in my view is that everything can be measured like hard science in the first place (not to mention the issue with fraud you point out in hard science).
I am think the idea that a thing is only a science if and only if it's empirical is actually harmful, science used to refer to a more generalized idea of study 'scientia' this could also be mathematics or theology. Then natural science begins to eat the term and becomes more limited. I think that the current place we are is one that is actually has a very dim view of anyone thinking about a topic and is naively optimistic about experimentation in general.
For if one looks at Aristotle theories ideas one may laugh given what we know now, despite using empirical reasoning for lots of his observations, Never the less I think where useful at the time as it's great for a theory to be good but failing that if a theory can usefully explain phenomena and provide one with mastery of phenomena in some limited way it's still useful.
Occam's often abused razor is a way of building theories like this and this whole process is similar to compression, given observable phenomena how do we explain phenomena in the most simple way, this process uses experimental and empirical data but requires thinking about the results of all experiments deeply.
From an outsiders view Natural science's largely seem to have pretty good major theories that lots of results of experiments can be used to correct constants of some kind in the theory and a well established field of study of theoretical physics that largely seems to be operating in the way I outlined.
But the social sciences are rather stupidly obsessed by these little experiment and no one has the guts to build bigger theories these days, for any theory of the human mind has to deal with many variables and if one wants to have descriptive theories of human mind then one needs to do a lot of thinking and theorizing and sometimes separate form empirical data and then arguing the theories using the data and reject the theories using the data, but also other kinds of analysis.
I have the most dim view of modern political science and sociology even more so, they tend to adopt this way of operating the most and to actually operate in that realm they need to theorize the most and experimental data is hard to do properly (and do not neglect history one bit, this is are experimental data and is as good as we can get).
There is nothing scientific about psychology. There can be no science where everything is subjective.
Pain is entirely subjective, but we are pretty sure it exists, and there is a lot of science around pain management.
Until you have an objective way of measuring pain in a subject it’s not truly scientific. Hence the endless stream of opioid users lying at the hospital to get their fix.
Dan Ariely's fraud is pretty widespread.
He sold his research to an insurance company, who should have known better: https://www.lemonade.com/blog/oh-behave/
At this point the surprising part isn't that another Ariely result has problems, it's that some of these datasets apparently went decades without this kind of basic scrutiny
> In 2006, when he was a professor at the MIT Media Lab, Ariely conducted experiments including administering electric shocks with a research assistant who had no human–subject training. […] Ariely confirmed this and said that he wasn't aware that the research assistant did not have the needed one-hour online human–subject training.
A one-hour online training is all that’s needed to be considered ready to conduct experiments involving things like electric shocks on human subjects?
Forget about this guy, the entire field needs to be burned to the ground and rebuilt from scratch.
I would not make such a harsh judgment just because electric shocks are involved.
During my master’s I took a class at the Media Lab called Human 2.0 on prostheses, orthoses, exoskeletons, and human augmentation. Unsurprisingly, all student final projects involved some form of human experimentation.
To run our final project’s experiments, we had to complete a few online modules on bioethics, first aid, and safe practices, as well as submit a proposal to the IRB. From what I saw across projects, the board applied appropriate scrutiny and made helpful contributions.
It classified our experiment as low risk, and a lot of care went into participant selection and adherence to the approved protocol, but they did not require any health professional to be present during the sessions.
Other projects, however, did require such support and even had some MIT Medical involvement.
Incidentally, my project also involved administering electrical shocks to subjects, albeit very mild ones (tACS to induce directed visual artifacts).
I don't know the details here but I am a psychologist.
The "electric shock" normally used in psych studies is tiny, it's like the equivalent of one of those little toy prank buzzers. You couldn't possibly hurt someone with it, this is really a non-issue.
Related side note: Electroconvulsive therapy uses much larger electric shocks but, administered safely by a clinic, is extremely safe, has relatively minor side effects compared to e.g. SSRIs, and is very effective for severe depression. It's a huge shame that it's undeserved reputation prevents it from being more popular. Blame "One Flew Over the Cuckoo's Nest" for its inaccurate portrayal.
ECT can be very, very unsafe. 70% of patients experience some form of memory loss and at least 30% of patients experience self-reported permanent memory loss. The rate of major adverse cardiac events found in a 2019 meta-analysis was 1 in 39 patients. For the people that it helps, I'm happy they've found relief, but it's a tool with significant costs and I know a fair amount of people who underwent it as children or otherwise were pressured to undergo it and I don't know any of them who have a positive impression of it.
ECT is a very specific subset of electrical shocks. Just because ECT can have adverse effects doesn't mean all electric shocks will.
Yes, memory loss surrounding the hours/days of the treatment is the major side effect.
Care to share the meta-analysis? How does 1/39 compare to the base rate of those with severe depression who did not receive ECT?
https://pubmed.ncbi.nlm.nih.gov/30557212/
About 1 in 50 patients may experience a major cardiac event (or death) during the course of ECT
This isn't responsive to the parent's question, in a way that ironically underscores the point: you can tell me that "1 in 50 patients experience X", but that's meaningless, unless I know what X is in the selected population at baseline.
I don't mean to pick on you. This problem is widespread in academic research.
The stats are from the Wikipedia page for ECT--I'm not sure about the base rate/risk ratio.
> Electroconvulsive therapy uses much larger electric shocks but, administered safely by a clinic, is extremely safe, has relatively minor side effects Blame "One Flew Over the Cuckoo's Nest" for its inaccurate portrayal.
YMMV. I’ve encountered people who have had ECT forced on them whilst inadequately sedated. They have been older/elderly but it would be a mistake assuming it’s always without causing harm.
That does sound rough. The initial shock, without sedation, could cause mild to moderate pain and would certainly be a scary experience.
Even then, though, it's a far cry from being the antiquated mental lobotomy that many people wrongly assume it to be.
There are multiple accounts that describe severe pain and the New Zealand government has paid out compensation for the torture that it was. What was being done here was most definitely intended to be painful torture, so maybe isn’t a fair comparison with clinical ECT for actual treatment?
‘Bryon Nichol spent six months in the Lake Alice unit at the age of 12. He said he received electroconvulsive therapy (ECT) many times, which was always unmodified, and without a muscle relaxing anaesthetic. "The nurses would come down and drag me upstairs. I was often so petrified that I soiled myself. They would put a mouthguard in my mouth and held me down while Dr Leeks gave me ECT. The ECT would give me vicious pains in my head and make me feel dizzy and as it continued the pain got worse. My arms and legs flailed about. It was absolute agony."’
And another:
‘Rangi Wickcliffe, 59, is another survivor of the Lake Alice unit. He was 10 when he was first given electric shock treatment. Dr Selwyn Leeks turned the machine on, the pain that you feel from the electricity surging through your head is indescribable. The scream that comes out of your mouth is also indescribable. "The terror was so intense that you lost all bowel movement." He said it felt like his bones would break.’
https://www.rnz.co.nz/news/national/444800/abuse-in-care-inq...
https://www.rnz.co.nz/news/national/444800/abuse-in-care-inq...
For what it’s worth, I wasn’t even thinking about the physical danger when I was… shocked… that you only need a one hour online course. It’s that they’re interacting with people in a somewhat clinical way as an implicit authority. The shock would obviously cause discomfort and therefore stress (or why even do it).
Do you think a potential bad actor would change their behavior if they took, say, a 12 hour course instead?
I am of course biased, but I have often been exasperated by the various "trainings", IRB processes, ethics classes, etc, that are required to run even basic psychology studies such as surveys.
Meanwhile, down the hall, the medical department is opening the skulls of mice, implanting electrodes, giving them supercancer, and then euthanizing them en masse. Often with less approval burden.
The hard truth is that it's tricky to decide what's ethical in science; we mostly allow or disallow things because they've been done that way for years, and all the added "process" is nothing but paper pushing and ass covering to quiet concerned outsiders.
> Do you think a potential bad actor would change their behavior
What is the appropriate way to help a righteous but underpaid lab assistant find the guts to call out a process as unethical and insist that the boss’s claim that the IRB has approved it is bogus?
The most important thing is they know how to report issues and have assurance that reports will not hurt them. If you can show other reports were acted on and the reporter not harmed that helps.
It also helps to always have more than one person who could call so they compete with each other to be first because they don't want to be the one seen as an accompanist to fraud.
It’s not about bad actors. It’s about putting untrained people in a position where subjects might be vulnerable and come to depend on them being the trusted expert in the room.
> Electroconvulsive therapy uses much larger electric shocks but, administered safely by a clinic, is extremely safe, has relatively minor side effects compared to e.g. SSRIs, and is very effective for severe depression.
The evidence base for these claims appears to be missing.
https://theconversation.com/no-evidence-that-ect-works-for-d...
It's liked by some practitioners, not by others, but actual studies in this area are low quality, didn't follow up with longer term checks, and are generally inadequate.
"The failure to find any meaningful long-term benefits compared to placebo groups are particularly distressing."
Look up violet wands, you can shock humans with 0 training when it's a consensual hobby.
You can buy them without training, but that doesn’t mean you should be allowed to use them in a research setting on strangers without training.
This scientific experiment escalated quickly...
> Ariely conducted experiments including administering electric shocks
Obligatory https://www.youtube.com/watch?v=A6bJHalRnX4
“Dr. Venkman, we believe that the purpose of science is to serve mankind. You, however, seem to regard science as some kind of dodge or hustle. Your theories are the worst kind of tripe, your methods are sloppy, and your conclusions are highly questionable; you are a poor scientist, Dr. Venkman. And you have no place in this department or in this university.”
> It revealed that Ariely had a longstanding relationship with Epstein, including having visited him at his New York home on several occasions.[45] One of the documents is an image of an email from Ariely to Epstein, in which he requests the name and email of a "redhead" that Epstein had previously introduced to him
I mean, I've always consigned 100% of NYT self help slop to the bin of books for people who aren't smart but desperately want to seem like they but this is just a little much now, Dan
Ah yes also the author of “The Honest Truth about Dishonesty”. The most amazing thing is he is still a professor at Duke. Wonder if he focused much on academic dishonesty in his book…
Institutions like Duke benefit enormously from having people like this around. They can claim they have lots of highly cited and well published researchers, a bit of controversy is free publicity, etc. They make money off him and so they defend him. It helps support the myth that they are good educational institutions. US private universities are grift mills.
This happens at any scale. Manager picks a bad candidate to join the team? Well now they are responsible for the decision. Their worth is tied to their bad decision and they will defend the bad choice to the detriment others on the team. So it’s an element of Duke also defending their honor since they invited and supposedly vetted him.
Beautifully ironic that "Dishonesty and ethics" is listed as one of his research fields.
In person immersion is a great way to research!
He IS a specialist.
"Ariely is named 636 times in the Epstein files".
Yikes.
Context matters. Otherwise we’re just being clowns.
I mean being a known fraudster, requesting money from Epstein, and asking about some girl he met through Epstein is context too which is not a good look.
He may be innocent, but he certainly has a large stain and stink about him.
This. Significant portions of the Epstein files are random noise. Though even for a popularish author it'd be strange to be in enough 1990-2018 news snippets to reach that many EF hits. In any case "is in the Epstein files" should in general be replaced with "is in the Epstein files, including this email to/from them I found"
The most interesting section from the large paragraph on Wikipedia: “One of the documents is an image of an email from Ariely to Epstein, in which he requests the name and email of a "redhead" that Epstein had previously introduced to him”
That still doesn't mean anything. Sure we know Epstein did evil, but not everything he did was evil. One should assume at least a few people he interacted with had no reason to suspect his evil actions.
I guess that's why the same passage says Ariely maintained the relationship long after the Epstein's pedophile convictions. Not definitive either, but pretty suggestive.
Go read the article instead of nitpicking people doing the citations for you.
If you ask a known pedophile for funding and introductions while accepting favors, you can rightly be accused of bad judgment - at best.
Go read it yourself. There isn't enough information to draw concussions. Everything is consistent with contact with a major funder who you can't cut off just because of evil in a part of life you knew nothing about.
You can read the article with a bias and ignore his counter claims. However that is drawing a conclusion not warranted.
Now perhaps someone has read the letters and seen other evidence and can draw conclusions. However at this point I can't say who knew what.
Both people are of course evil, but they may or may not have an evil in common.
Asking THE sex trafficker pimp extraordinaire about the contact data of a “redhead”… I guess you CAN read this whole interaction as benevolent if you squint hard enough to the point that the nature of this guy’s transactions with people becomes invisible
I have a policy of "innocent until proven guilty". (even proven guilty doesn't mean guilty, but by that time so much evidence has been examined that odds are high enough to finally say guilty). All too often somebody pushes only evidence that supports their bias without full care for the full truth.
I agree it looks pretty bad, but I have not seen all the evidence and thus I must hold out the possibility he is innocent until someone does take the time to examine it all. That doesn't mean I think he is innocent, just that I'm carefully withholding judgement because I know I do not have all the facts. It also doesn't mean I'm against investigation of the facts - I just want the investigation done properly, and all discussions like this do not contribute to a proper investigation.
That’s fair. I just fear the justice system or whatever it should be to decide what the “truth” is might not be up to the task or not that interested in truth anymore. Like for example there’s this one person that everybody in the world knows who bragged about doing despicable things and there’s dozens of women who for years have been accusing him of doing similarly despicable or worse things to them, but it doesn’t matter because he’s innocent until proven guilty… So at some point I just allow myself to draw my own conclusions
Context matters and the more context you have the worse it looks.
> The messages revealed that his friendship with Epstein lasted for nine years, including well after Epstein was initially charged with soliciting sex from a minor.[52] Ariely also suggested that Epstein use learnings from his research to help rehabilitate his public image
what's clownish is defending anybody who would associate with a billionaire pedophile sex trafficker
anybody in the files who helped him rehab his image should rightly be consigned to the dustbin of history
>what's clownish is defending anybody who would associate with a billionaire pedophile sex trafficker
>anybody in the files who helped him rehab his image should rightly be consigned to the dustbin of history
It's possible to denounce someone, but still push back on false claims. Otherwise you're just condoning misinformation just because it aligns with your beliefs . eg.
"Did you know Donald Trump literally eats babies?"
"Wtf are you talking about? There's no evidence of that."
"Wow, I can't believe you're defending a convicted felon, sex offender, and authoritarian! Anyone who's trying to rehab his image should be consigned to the dustbin of history."
sure, that's a great principle but is it applicable here? Ariely is noted having sent an email asking Epstein about 'that redhead' (along with hundreds of others and his work rehabbing the image of someone with a known pedophilia sex trafficking indictment)
if you assess the original claim against the facts, is it as ludicrously out-of-step as someone claiming that Trump found A Modest Proposal persuasive?
iirc Epstein funded the media lab and had an office there
Ask yourself who benefits from equating "being mentioned in a file" and "having evidence this person is a pedophile"
The answer appears to be "people who spend more time on offense than defense".
Not to say that tilting things in their direction is a good idea, but that's a pretty wide class of beneficiaries.
To be clear: the implied answer is "pedophiles". Pedophiles benefit the most from "being mentioned" == pedophile
Because obviously, there are many people mentioned in different contexts and it is unclear how many of them are pedophiles.
There are people who may disagree, but I'm from the generation who considers molesting children a serious crime--much much more serious than being mentioned in some email.
Ask yourself why nobody is being prosecuted.
In the same way that the primary beneficiaries of witch hunts are... witches?
[flagged]
Science is self-correcting, but sometimes, the correction mechanism it is very slow and inconsequential enough that people are incentivized to just publish anything. Peer review is merely a quality check—and not a very good one. Just the other day, I ran into a paper in my field that claims to have solved a central problem that scientists have been working on for 50 years, single-authored, using some fancy mathematics. I was excited to read it, but it didn't take long for me to find out that it's complete BS-derivations that don't follow, made-up equation terms, and conflation of basic terms in the field. I was really surprised that this passed the editor and at least two peer reviewers. I dug deeper and found that the same author has published 11 research articles in top journals in 2026 alone — mostly physics and chemistry, and adjacent fields— and I'm talking about really top journals like Scientific Reports and Physics Letters B. Most of these have serious mathematical flaws that are unfixable. I wrote to a couple of editors, but only one took it seriously enough to initiate an investigation. I made a promise to myself that if no real action is taken, I will seriously consider changing careers.
Science isn't self correcting. This is a harmful meme that we need to collectively put in the trash can, as it's constantly used by academics to evade accountability.
The core problem is people conflating science with Science™. The scientific method is a set of techniques that help people discover and report true things. But there are two huge problems with it:
1. What exactly is the scientific method? It's not like there's a constitutional amendment that lays it out. Try and pin people down and you'll discover nobody agrees on what science actually requires.
2. Even if some subset of people did write down a definition, nobody is checking anyone is following it.
So people conflate science the Platonic methodological ideal with Science™, the institutions that claim most forcefully to be its avatar. But those institutions have neither a definition nor any enforcement arm, so fraud and abuse runs rampant. There's no reason why anything should self correct in this environment outside of corporate science, where the self-correction mechanism is managerial intervention or, in the absence of that, bankruptcy.
This is unnecessarily cynical.
> The core problem is people conflating science with Science™.
You fell into the same trap by criticizing scientific institutions and the human spirit in an attempt to criticize the scientific method.
> What exactly is the scientific method?
What a ridiculous argument. What exactly is a hamburger? Just ask around and you will find that nobody agrees if a veggie burger should be considered a hamburger, or if you can replace the bun with lettuce and still call it a hamburger. And even if you get people to agree what a hamburger is, nobody is checking that the recipe is followed to the letter. Therefore, real hamburgers do not exist, and nobody can claim in good faith that they are making a hamburger. In fact, why don't we sue McDonalds for their fraud hamburgers?
> Science isn't self correcting.
The very fact that fraud is identified is clear evidence that is is not true. You are going to need to sharpen your rethorical devices if you want to argue against hundreds of years of scientific progress, including for example everything that enabled us to argue about this in front of our devices while barely 150 years ago people had to use candles to see each other after dinner. Mind you, not all of this was developed by industrial research labs, in fact the origins of very many modern technologies lie in academic research labs pursuing ideas that are too risky for a commercial entity to bet on. And of course there are plenty of examples of industrial fraud happening despite "managerial intervention" (lol).
Reality is more complex. Not many things, if at all, exist in this world under their platonic ideal form. That does not make them any less real. Yes, we all know that humans are not perfect, sometimes are selfish and chase money and power, not only in academia but also in business, politics, and so on. Wherever humans are involved this sometimes happens. But saying that nothing good ever happens because of this goes against everything one can see when looking out of the window. Scientific progress is real, and needs both industrial and academic labs.
Science as a monolithic institution is a trap.
We ought to be able to trust the public with complex realities and contradictions. The scientific method and dogged researchers have brought us many wonderful things, but at the same time fraud happens and the layman might reasonably question to what extent the academy attracts fraudsters and to what extent it creates them.
If we can get over the absolutes then we can discuss policy changes.
Graduate student collective bargaining is one experiment; how can anyone blow a whistle on malfeasance with such an extraordinary and widening power imbalance?
Totally agree with this framing: trust in public institutions (scientific, governmental, etc.) is a common good that needs to be promoted and safeguarded.
Identifying and punishing fraud is a part of it, whistleblower protection another, modernizing the system so that graduate students are not so dependent on individual advisors yet another idea. More and more personal contact between the public and scientists could be useful too, for example via science fairs, meet & greets and so on.
>This is unnecessarily cynical.
Are you living in the same reality as the rest of us?
Now, if he'd laid out the financial incentives of all of it and said "everyone is only in it for the money" I'd say that's a a little too cynical. But he stopped short of doing that.
> The very fact that fraud is identified is clear evidence that is is not true
That's not merely enough, you need a clear trend that the rate of fraud discovery is greater than the rate of fraud production, not just that "some cases are found out". Good luck with that, since it's by definition not measurable.
Science is very simply a commitment towards objectivity. That's it. You see data, you conclude x,y, or z. and thats it. No external subjective influences changes the conclusions. If the data trends towards "vaccines saves millions of lives", thats it.
You're right that we don't need a universal definition of hamburger. That's because the government stays out of it, so private individuals can use things like Google Maps reviews to police a common sense definition of hamburger along with quality and other nuance.
If governments were forcing everyone to buy anything advertised as a hamburger no questions asked, then the definition of hamburger would matter a lot.
The devices on my desk are 99.9% the product of the private sector and owe nearly nothing to academia.
If people can't define it, then that seems to me like a valid criticism of the scientific method. You can't have institutions that claim to be doing science if they don't use the same definition of science as other people do.
Sorry your device owes nothing to academia? Are you referring to a computer? If that’s the case are you mad?
They said "nearly nothing". As a fraction of total effort I'd say that the contribution of academia is relatively low. If you're going to say something dumb like overweighting foundationality, I can counter by reminding you that all of modern statistics and thus academic science is literally downstream of industrial beermaking (and gambling) and not academia
Yup, that is also how it works with scientific research: Once an article is published, anybody is free to read it, criticize it, reproduce it. That's because the government stays out of it: private individuals can use things like Google Scholar and Google Search to find other people's reviews and follow-up studies to better understand the quality and nuances of the original research. Consensus is an important part of science just as it is for reviews on Google Maps. Is consensus sometimes wrong? Yes, no doubt. Is it useless? Nope.
> If people can't define it, then that seems to me like a valid criticism of the scientific method.
Please read the hamburger example once more.
Peer reviews don't appear in Google Scholar.
The involvement of governments is when they force the public to pay for the studies via tax funded grants, without caring about the quality.
As we all know, oeer review is not a very reliable signal about the veridicity of the claims, it is merely an approximate quality gate indicating that a study was done following appropriate standards. There needs to be more awareness of this. Nonetheless, in some fields peer review have been public for a long time, and more and more journals are making peer reviews publicly visible. This is a good trend that we should all encourage.
I also think that it sounds like a good idea to hold scientists accountable for the quality of their research, but moving away from fraud (which should unquestionably be punished) towards merely "low quality" is a very slippery slope. Who determines when the quality is good enough and under which criteria? Citations and follow-up research already are an approximate signal, why is that not enough? Proving a study false does not automatically make it low quality. How much money and resources would these additional measures cost? Would the savings from fewer "bad quality studies" offset these costs? How many fewer "good quality studies" would be prevented by redirecting these resources.
[dead]
My take on the self correction is, that later research usually cannot be build on wrong results. So over the decades or centuries the correct results prevail.
Later correct research can't build on previous wrong results. Later incorrect research most assuredly can build on previous wrong results essentially indefinitely.
I'm not constructing a full differential equation model for an HN comment, but I'm pretty sure any realistic model of likelihood of detection of the falseness of past research and other similar parameters would produce a surprisingly sharp cutoff when looked at over time, above which the science does tend to self-correct and below which it disintegrates into endlessly recursive garbage drowning out any real science being produced. And I'm pretty sure a number of fields are below that threshold right now. That doesn't mean the problem is permanent... the parameters can be changed over time, too, but the problem does get worse and harder to fix if allowed to accrue too much.
>My take on the self correction is, that later research usually cannot be build on wrong results.
AFIK, it does not seem to happen much with mainstream medicinal research. Many layers of fraud/incompetence/assumptions for centuries with no correction in sight. Of course there are certain sub-areas that may have advanced. In fact it's the modern day religion, having replaced the god centric religion of yesteryears.
I'll very happily (as a significant benefactor) take this century's medicine over the last. And the one or two centuries before that aren't even worth starting to think about. So something must be working wioth mainstream medicinal research.
Only if people fight for that to happen, or there is some external factor forcing it to be so (like a war or other existential threat).
Otherwise people will happily go along with BS.
You'd think so. I used to believe this too. Surely you try and build on a result which creates an implicit replication study, then your experiments don't work and the problem is discovered pretty quick. Hence, self correction.
It often doesn't happen. The reasons seem to vary by field. We do see it in a few very hard fields like materials science.
In the softer end of social sciences like anthropology, education or the humanities it's obvious why not: they don't make empirically testable claims about reality anyway so there's nothing to replicate. This is the Arday problem, where he was publishing what they call auto-ethnography: effectively just blog posts about the author's own feelings. He even interviewed himself in the third person.
In the harder social sciences it's because nothing actually builds on anything else (see "A problem in theory" [1]). Psychology and related fields like sociology, criminology, etc are basically just a pile of random ideas pooped out in brainstorming sessions. There are hardly any overarching theories generating testable predictions, so studies test whatever random claims the researcher thinks is interesting enough to get in the news. Everything is just rocks lying on the ground rather than towers of understanding, so when one paper is overturned it has no impact on anything else. Citation counts obscure this fact, but do some reverse-citation checks and you'll find a lot of citations are worthless or outright invalid.
In modeling based fields like economics, epidemiology or climatology, it's because they can't put the thing they study in a lab and do high speed experimentation. "Replication" comes to mean repeating the same analysis on the same data set. Often they literally just run the same program on the same input file, get the same result and call it replicated - or sometimes they call it replicated even if the outputs don't match! See [2]. But the program is just a set of assumptions, not real discoveries, so building more papers on top of that program's output doesn't detect invalidity. It will all replicate in some very narrow technical sense without that proving anything useful.
Sometimes outsiders notice what's happening but over time academia has built up a complex web of lore that protects them psychologically. For example, they routinely reject or ignore feedback from outside their field on the basis that it doesn't come from "experts", meaning themselves, or it comes from "right wing" people, meaning anyone who isn't an academic or close political ally.
[1] https://www.nature.com/articles/s41562-018-0522-1
[2] https://dailysceptic.org/2020/05/06/code-review-of-fergusons...
Whilst I weakly agree with this characterisation, it's not at all fair or accurate to put epidemiology and climatology in the same category as your other examples. Yes they have some weaknesses in replication practices (especially for purely computational work), yes the feedback loop for self-correction can be slow, but it would be a mistake to conclude or imply that the major results of those fields are therefore unreliable.
That's because those fields do still make objective empirical predictions about the world, which might take years to be testable but are still ultimately testable. For example, it's been convincingly shown that climate models from decades ago predicted our current climate pretty well. [0]
These fields are more comparable to subfields of physics which only get to do big experiments a handful of times, e.g. because they can only observe so many planets/supernovae/universes, or because they have to spend decades building a new billion-dollar machine to test new hypotheses.
We should also not confuse these arguments with uselessness - soft social science work can still be societally valuable without being easily empirically testable.
[0] https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/201...
Disagree, those two fields are extremely unreliable. It takes a lot of things happening simultaneously to be scientific. Yes, those two do make testable predictions, sort of. But that isn't enough.
Modern (computational) epidemiology is rife with unscientific practices. They ignore data that shows a model was invalidated so the fact they make testable predictions isn't really useful. They also engage in a lot of circular reasoning, buggy coding and logical fallacies. During COVID I wrote a whole report on this topic for some politicians [1]. But the biggest issue is "A problem in theory" again - epidemiologists conflate fitting a curve in R with developing a hypothesis, so the field is overrun with overfit models that aren't based on any refinable theory of disease, just misuses of statistics. Even if you prove a paper's predictions were wrong it changes nothing because nothing built on it anyway.
The problem in climatology is that when the models don't fit the data they just change the data and claim victory, e.g.
September 2013: https://www.spiegel.de/international/world/climate-scientist...
June 2015: https://www.nature.com/articles/nature.2015.17700
[1] https://plan99.net/~mike/epidemiology.pdf
> The problem in climatology is that when the models don't fit the data they just change the data and claim victory
That is... not remotely true, nor supported by either of the links you shared. (Nor by any cursory look at the state of the world today, with well-anticipated Himalayan glacial floods as front-page news.) You therefore appear to be leaping over the line from healthy skepticism to irrational conspiratorial thinking, so I'm going to stop engaging, sorry.
Disengaging isn’t part of the scientific method.
Disengaging is sticking your fingers in your ears and blurting out “la la la la I can’t hear you!”
And calling that a win.
The problem isn’t science, the problem is that there are adults with the maturity of a four year old acting as sciences hype man.
Feel free to not engage, but for anyone reading the Nepal flash flood was caused by bedrock giving way, which isn't climate related.
https://www.theweek.in/news/world/2026/08/28/new-satellite-i...
The two links I give show the problem in action. In 2013 the IPCC was concerned because temperatures had gone sideways for a decade. Der Spiegel's first sentence is "Data shows global temperatures aren't rising the way climate scientists have predicted" and the next says they were trying to "hush it up". In 2015 climatologists announced a new temperature record that replaced the entire 21st century with different data, as reported by Nature.
That's an unambiguous sequence of events. You can't make statements about replicability in an environment where they routinely decide their already published data is wrong post-hoc, without retracting the papers built on that data.
Shoosh. And if you won’t shoosh yourself, a handful of The True Adherents Of The Science will down vote you.
Because Down Votes Are Truth.
And that, ladies n gennelman, is How Science Works.
That’s how we end up with Jason Arday: Up Votes.
[dead]
> Sometimes outsiders notice what's happening but over time academia has built up a complex web of lore that protects them psychologically
The word you're looking for is "triangulation" - https://en.wikipedia.org/wiki/Triangulation_(social_science)
As long as you can find [1] a few [2] citations [3] that kind of all point in the same kind of direction then you're good!
The situation of [2] is crazy. Because both the author of [2] and the people using Ferguson’s model are wrong. And the author's advice in one of the last paragraphs of we just need better programmers is also wrong.
The issue is that pandemics are extremely fat-tailed in terms of fatalities [3]. You need to collect a shit tons of data to estimate any reliable parameters and no one has time for that. The only thing you need to know is: Is this a super contagious deadly disease? If yes, lock everything down! This is pretty much what they did in Asia, and countries like Vietnam had zero cases for months and was open while the rest of the world was closed.
How to decide a disease is deadly and super contagious? You need disease experts (who most likely also academics) who understand the nature of the disease, and not better modellers and programmers as suggested in [2]. The issue is that western countries will never let any expert to make that call, so a lot of convincing needs to be done through complicated modelling and real life suffering.
[3] https://arxiv.org/pdf/2004.08658
Western countries won’t let an expert make the call, because there hasn’t been recent enough pain and suffering to put it in recent memory the necessity for it.
After all, why suffer if it isn’t actually necessary?
Asia had recent experience with the first SARS, which had very real and extremely high fatality rates, so for them it was a relatively easy (not an easy) call.
For the west, when was the last time we had a real pandemic that actually killed a lot of people? Almost a century ago.
Somewhat ironically because the west listened to experts for a very long time.
Notably, the last time it had a major issue also coincided with a world war.
Considering the prosecution of Fauci, which does not happen anywhere else, I don't think America is ready for the next pandemic.
Do you think America was ready for the last one?
[flagged]
Economics is a particularly slippery subject, since double blind experiments are practically impossible. No doubt some of the things Marx wrote correctly describe reality, even if some of the social experiments built on his work were massive failures.
I mean more the Marxist school of thought, not just the economics. Things like critical theory and post modernism.
But also, he made predictions of the near future in his time. They were incorrect. No updates were made. Those who did try to refine and update his ideas are far less influential. His original theory survives like gospel. I wish I could say only by non-academics.
Edit - back to economics. It's funny. I would say in economics the closest thing we have to double blind wrt economic systems is socialism/communism vs. capitalism. It's clearly not a true double blind. But also clearly some people just don't care to feed that data back in to their models. Because they aren't scientific.
marxism (a materialist ideology) is incompatible with post-modernism, a post-truth ideology
How do you square your position with all the ... self corrections ... that have happened in science in the last, oh, 500 years or so?
Most of the problems are with academic fields that were barely funded before WW2.
Your comment claims that academics do not correct other academics.
Your comment appears on a post in which, literally, academics are correcting other academics.
Data Colada is a handful of volunteers who do this kind of thing as extra-curricular activities, and rely primarily on tips. That is a tiny drop in an ocean of fraud, for which the institutions get no credit at all. Academic institutions don't systematically check for fraud, but they do systematically ignore fraud reports that come from outside the academy (of which there are many).
Yup
Motte-and-bailey fallacy abound
Nature (not the magazine) doesn't care if you published a peer-reviewed paper in a high-impact journal.
It doesn't care if reviewer 2 thinks you need to quote his friend
It also doesn't care that you think a dewormer works against a virus because it doesn't
> What exactly is the scientific method?
Trying and seeing what sticks. That's literally it.
Of course publishing (and publicity) forces like to jiggle that one way or another
> Trying and seeing what sticks. That's literally it.
I'd like to add a readiness to abandon that which doesn't stick. Otherwise you end up with people going 'round in circles around a campfire.
That's covered by trying and seeing what sticks. That's really all there is to it. I already got downvoted for saying this in other words - feedback against observed phenomena is what matters. Everywhere you see shitty "science" you see either missing or ignored feedback.
Do you mind if nature also considers it a lice and mite control agent?
Or are you planning on continuing to call it a “dewormer”.
Completely factual vs factually complete.
Why wouldn't I agree
You might be confusing me with your friends that failed HS biology
> Science isn't self correcting....
I am not sure what your proposal is? It seems to be the solution is only more science? This meme that science can't solve anything or is ineffective is also a meme that needs to die. Science is about the only thing that we have to make sense of the world. What else is there?
> 1. What exactly is the scientific method? It's not like there's a constitutional amendment that lays it out. Try and pin people down and you'll discover nobody agrees on what science actually requires.
I am pretty sure the only necessary requirement is repeatability. As far as science is concerned, it does not matter if Jesus did in fact born from a virgin mother, it is not repeatable, and therefore not scientific. We could be entirely wrong about gravity, maybe Earth is indeed only 5 years old and God just created everything to appear the way it is.
Replicability isn't enough to be scientific. Many studies have invalid methodologies, so replicating them by following the same method just propagates wrongness and makes the problems worse. And if you don't follow the same method the original authors will claim it's not a replication study.
https://blog.plan99.net/replication-studies-cant-fix-science...
My proposal is: let science be done by those who need its results to be correct.
> Replicability isn't enough to be scientific. Many studies have invalid methodologies, so replicating them by following the same method just propagates wrongness and makes the problems worse. And if you don't follow the same method the original authors will claim it's not a replication study.
Replicability test also happens in the real world. If gravity is not actually 9.81 m/s^2 then we will eventually have to come to term to that. For all we know, we could be completely wrong about quantum mechanics and the reality of subatomic particles, but it doesn't matter because we can use the predicted behavior and it actually works.
The question is to ask: what is one thing that I would never discard from a scientific principle. If you remove replicability, it does not matter if you have accurately captured a measurement of the universe once, it is completely pointless if you are not able to replicate it again.
>My proposal is: let science be done by those who need its results to be correct.
A tangential proposal - individuals (not orgs) should offer to upfront fund ( in lump sum) promising researchers for life. That way a researcher is not worried about losing his livelihood. I suspect this may already be happening in a few circles without much fanfare.
Why are individuals better that individuals acting together as an organisation better?
If that’s your criteria, then shouldn’t said funding go to an individual scientist who has no lab, no staff, no PhD candidates, and no access to the internet or existing research.
Because organisations are bad and only individuals are capable of making Correct Decisions.
>Why are individuals better that individuals acting together as an organisation better?
Most organizations are made of of very mediocre (or political) individuals, and when it comes to wrong (or political) decisions individuals cannot be held responsible, that how orgs deflect responsibility.
On the hand when an individual (often wealthy) or a bunch of them decide to fund something (often from his own pocket), they know what they are doing and are personally willing to bear the risk.
>Because organisations are bad and only individuals are capable of making Correct Decisions.
I should have been more nuanced.
Org = generally public orgs where the people in the org do not have a stake beyond their jobs in the org. Example: govt. As an exception to this rule there can be (private) orgs of very competent people who deploy they own money to get things done, including research.
(p.s - I realize that I cannot delve into all all nuances with a few sentences I put here, but hopefully you get the drift)
> it does not matter if Jesus did in fact born from a virgin mother, it is not repeatable, and therefore not scientific
https://www.newscientist.com/article/1837276-the-boy-whose-b...
Even better, from the BMJ christmas edition https://www.bmj.com/content/347/bmj.f7102
> Around 0.5% of women consistently affirmed their status as virgins and did not use assisted reproductive technology, yet reported virgin births.
Not necessarily.
The article I linked reported that the white blood cell DNA of the child did not contain Y chromosome.
That's different than conducting a survey.
In the true spirit of science this needs to be pursued however outlandish it may seem.
Fully concur with you here. Ultimately it boils down to integrity and objectivity of the individuals involved, which IMO is in short supply.
Also my way of phrasing what you said: science = verb, Science™ = noun. George Orwell had a point about the play of words.
Indeed. “Science” is modern religion.
3. Publish or perish, scientific publications (count x impact factor) as careeer credits. What a terrible yet hard to kill idea.
Science that is science is self-correcting via prediction and experiment. The key piece is the feedback. The "scientific method" as taught in grade school hardly matters and is basically cargo cult. The "is this science" lint is "are you working toward more accurate and especially predictive models of observed phenomena? Are you actively trying to destroy your theory?" If the feedback is missing or ignored, it's just opinion pieces.
Edit- I don't know what's controversial about this? Feedback is the self-correcting piece.
> Science is self-correcting, but sometimes
As long as people mislead or make mistake unknowingly, it's fine to rely to self-correcting mechanism of science. The institutional rot is much deeper. It is not good to leave all this fraud to self-correcting mechanisms of science. We also have "trust" to worry about.
Maybe instead of throwing around ideas of "institutional rot" we can realize that there are hundreds of thousands of people doing "science" in the right way, are not out for material gain, do not engage in fraud, and are interested in truth and scientific advancement.
The fact that there are a vanishingly small # of cases where bad things happen in the process are the exceptions that prove the rule, IMHO. Certain segments of society like to hype these up (or even lie about their frequency) in order to discredit the scientific method.
"Institutional rot" is a meme that needs to go away. Identify the problems at the edges, fix them, and move on to letting the bulk of the work continue as it always has - quietly building upon previous valid results to the betterment of society.
It's a bit like how people believe markets are self-correcting. Yes, only over an infinite time horizon, in theory. Edit: and like chemical equilibrium, you could say it depends on the "rate of wrongness production" compared to the "rate of correction". Edit edit: self-correction of course must fight against the higher virality of work selected for its impact rather than correctness, and incorrect work can then spawn further incorrect work, like an autocatalytic process or epidemic.
>really top journals like Scientific Reports
I wouldn't consider Scientific Reports to be a top journal, it's basically pay-to-publish
I have been having the opposite problem with my papers lately.
Instead of glossing over mistakes, the reviewer report missing information which is clearly stated in the text.
This is quite frustrating and hard to stay polite after the fifth "please just read the text."
I really wish there was some serious rework of the publishing system. Nobody is happy about it but nothing seems to be moving.
Would be nice if you kepy ypur career and created some magazine / journal that focuses on checks and pointing out flaws.
That cheaters would dread being included there (would probably be a thick journal too)
They wouldn't dread being included there, not at all. The institutions don't care about fraud and reflexively defend it, ignore it or shoot the messenger. Look at the Arday nonsense. People are still discovering new lies he was telling but Cambridge went to the mat for him.
Dan Ariely in particular has faced convincing fraud accusations for years:
https://en.wikipedia.org/wiki/Dan_Ariely#Controversies_and_c...
Nothing ever happens. Universities do not care.
The real power move would be to start targeting prospective students.
Universities will start caring of enrolment goes down.
They might not care even then.
Journalism is an interesting case study. Their revenues went into sharp decline around the start of the century, dropping by more than 50% in some cases. Concurrently polls showed trust in journalism was plummeting, that people were tuning out because they felt that what they read wasn't reliable.
Journalism didn't solve this problem by rebuilding trust and their audience. Instead they started reforming as non-profit institutions and taking huge payments from philanthropists to push specific agendas, without alerting readers to that fact.
https://apnews.com/article/science-business-arts-and-enterta...
These grants aren't tied to readership but rather to the content of what's published, so they can produce lots of articles even if nobody reads them. Those articles are then used as "reliable sources" to control Wikipedia, and from there what search engines and LLMs say.
If students start checking out at scale (already happening), or if governments cut funding (already happening in the USA), their most likely move is to just replace the revenue with money from the Gates Foundation et al, and keep going just as they are. Their primary role will switch from education or research to the propagation of approved narratives.
News reporting and journalism has always mostly been a for-profit enterprise. And no matter what business model one might choose (ads, subs, charity, etc) there will be perverse incentives working in conflict with pure truth seeking and newsworthiness. Audience capture, pleasing advertisers, the politics/interests of the owners, etc.
I'm sure non-profit sponsorship has it's own issues, but I don't think those issues are particularly new or unique or transformative.
The demand side is arguably the more intractable issue with journalism, since loads of people don't select for truth or importance in their news - or at least not very carefully. The institutions that seem to hold up the best are the ones with the broadest audiences.
On the issue of public trust in journalism, I don't think it's actually a function of the quality or trustworthiness of journalism. I think it has more to do with the rise of radical right-wing propaganda infotainment, a core pillar of which was (and is) delegitimizing mainstream media (and academic) institutions. The mistrust was created and cultivated. Consequently, there's not much they can do to regain trust, except to work harder at discrediting the radical right-wing propaganda that has been flooding all the zones for the past several decades.
I think ads were better because everyone was aware of the conflict of interest issues and took steps to manage it. They admitted it could be an issue and would have internal firewalls where ad sales weren't allowed to talk to the journalists, things like that. Advertisers accepted it because their goal wasn't to manipulate news reporting.
The problem with the "philanthropic" funding is not only is there no firewall, but it's explicitly pay-to-play agenda pushing. So that culture of keeping revenue and news separated has vanished without the public being made aware of it.
The right wing are saying things like, journalism is fake because they're just pushing a left wing agenda for money. That's the claim that delegitimizes journalism. But it turns out that the rightists are correct and that's exactly what journalists are doing, so it's hardly their fault that mistrust was created and cultivated. You should mistrust people who mislead you about their agenda, and if you go to the AP website today and look at some random article paid for by these grants there's no mention of where the funding comes from. Actually you get a popup begging for donations, saying that because the AP is a non-profit it's extra trustworthy.
If they want to regain trust, a good start would be by refusing to accept money for stories, relabelling all the stories where they did that, and then pay for journalism purely out of subscription revenues or ads brokered through exchanges (i.e. where the advertiser doesn't have any contact with the publishers).
There certainly are problems, but the biggest is that reality puts two other factors ahead of publishing the truth:
1) People don't like having their beliefs challenged. This will slant all news organizations towards the biases of their readers. We see this at all levels. The shorter and more personalized the feedback loop the faster it happens. This makes the news with things like ChatGPT, the feedback is very quick and personalized so it's very easy for people to tumble down the rabbit hole. Same thing, slower, with social media recommendation engines. And online hate communities--they lead people to extremes. But it also happens with traditional media, just much more diffuse. Everybody moves away from the center.
2) News organizations care very much about being able to get images and video of events. This means they are readily manipulated by restricting access to those who report the locally-approved version of the "facts". We used to understand that "news" from Russia was the state-sponsored version, we don't pay attention to the fact that "news" from areas with Islamist control are just about as bad.
And news is always slanted towards the side of controversy. No controversy, people aren't going to care.
However, this does not remotely mean that the right wing news is better. The reality is that it's even worse. All the same forces are at work and then there's an ideological bias on top of that. With news having become a la carte I doubt there's any way to remain in business as an honest news source anymore.
Point (1) is why news institutions that cover a wide range of interests and have a broad audience have tended to fare better. Depending too much on a particular niche audience leads to audience capture.
WSJ and Fox News have the same owners but WSJ still has one of the best and most credible newsrooms in the business (their opinion section is another matter).
Meanwhile Fox News is a lie factory that only tells it's large (but niche) audience what it wants to hear. And now they have rival networks eating away at the pie by being more sensational and even more shamelessly obsequies. They're stuck in a radicalization doom loop.
If the theory is that advertising makes conflicts of interest more transparent, I don't think I agree. Advertisers can be stakeholders across diverse ranges of concerns - conflicts of interest are often indirect. They can influence what doesn't get written, as much as what does. The real bad actors can even just disguise ads as content. We still need to have some basic trust that institutions are respecting firewalls that - by and large - they have to self police
> The problem with the "philanthropic" funding is not only is there no firewall, but it's explicitly pay-to-play agenda pushing
> if you go to the AP website today and look at some random article paid for by these grants
Why do you say there is no firewall? Is that really true? The AP has their guidelines here and I don't see it as substantially different than what a publication might do for advertisers and other stakeholders who may be tempted to exert editorial control:
- https://www.ap.org/the-definitive-source/behind-the-news/how...
If you have specific instances where these arrangements have clearly compromised their journalism, that would be good to see.
In the meantime, I think we can do two things at the same time here: resist overdosing on maximal cynicism, but also be clear eyed about the ways money can tempt institutions to throw integrity out the window.
> The right wing are saying things like, journalism is fake because they're just pushing a left wing agenda for money.
> But it turns out that the rightists are correct and that's exactly what journalists are doing, so it's hardly their fault that mistrust was created and cultivated.
Generally speaking, no they are not correct, and most of them are just outright cynically lying, while engaging in the very behaviors of which they accuse others.
Yes, it's true. No more firewalls. Go read the article I originally linked, here it is again:
https://apnews.com/article/science-business-arts-and-enterta...
Some key paragraphs are:
> The Associated Press said Tuesday that it is assigning more than two dozen journalists across the world to cover climate issues, in the news organization’s largest single expansion paid for through philanthropic grants.
> The grant is for more than $8 million over three years, and about 20 of the climate journalists will be new hires. The AP has appointed Peter Prengaman as its climate and environment news director to lead the team.
> For many years, Journalists and philanthropists were more wary of each other. News organizations were concerned about maintaining independence and, until the past two decades, financially secure enough not to need help. Philanthropists didn’t see the need, or how journalists could help them achieve their goals.
> Nonprofit news organizations like ProPublica and Texas Tribune led the way in changing minds. The Salt Lake Tribune, which in 2019 became a nonprofit to attract more donors, and The Seattle Times are other pioneers.
> “As far as we’re concerned, it has been really seamless,” said Monica Drake, assistant managing editor at the Times. “It has allowed us to focus on things we want to cover and to do it independently.”
> Both sides had things to learn. For Carovillano, it was getting used to the idea that funders weren’t just being generous; they had their own goals to achieve. “This is a mutually beneficial arrangement,” he said.
> “I feel like our language has gone from people talking about local news dying to there being a renaissance of local news,” she said. “If you can provide good, high-quality journalism, people are willing to pay for that, people are willing to support it. I think there’s a lot of room for optimism. The trick is, can we move quickly enough?”
The article leads with climate grants but is really about the phenomenon in general, which covers many topics.
The quoted journalists are duplicitous, trying to reassure themselves. The text says that what they're doing used to be taboo because rich guys had "goals to achieve", but now it's considered "pioneering" and "mutually beneficial". Their justification is they didn't have any money and felt sad, but now they've got lots of money and are happy. They say the transition has been "seamless", i.e. readers aren't aware of it. And hilariously, at the end Carivillano claims there's room for optimism because people are willing to pay for high quality news - except "people" aren't paying, that's the reason they need to take money from a handful of donors.
It's obvious this compromises their journalism, especially in the stories they don't publish. These news orgs have basically been bought by the donors, large chunks of their staff only exist because of these grants. Thinking it doesn't compromise their news would be like saying Facebook simply buying all the news teams that cover the internet wouldn't have any effect.
This is a puff piece. They are always a bit icky, but they're common and I don't think you'll find any outlet of renown that doesn't succumb to these from time to time.
In this case, it reads like their way of stroking their benefactors egos a bit, but without promising them anything real in return.
FTA: "AP often needs to educate funders upon first approach, explaining the company’s worldwide reach and mission to report independently. AP accepts money to cover certain areas but without strings attached; the funders have no influence on the stories that are done, Carovillano said."
You’re not helping whatever it is you think you’re trying to achieve by calling everything you don’t like radical right-wing propaganda.
Radical right-wing infotainment propaganda is a specific set of things (like Rush Limbaugh, Fox News), not "everything I don't like"
You left out a pretty important force: political forces lying about the trustworthiness of journalism in a blatant effort to gain political power. Traditional journalism didn't have any ready defenses against that.
Even as imperfect as it is, it's infinitely better than the alternatives like belief in religion/religious texts/pastors or anybody else who is just making shit up based on whatever they think sounds good. None of that is falsifiable or makes any attempt to iteratively arrive at anything true or real.
There is literally nothing better that we have other than the scientific method. Yes, science is self-correcting, even if it is slow or whatever else. Nothing else that we have can approach even that bar.
And you protect them by not sharing their name.
>I made a promise to myself that if no real action is taken, I will seriously consider changing careers.
Personally I think you should name and shame the author as well as the publications and the names of the peer "reviewers" who clearly did sweet-fuck-all.
Not to forget the editors responsible at time. Why not leadership and board members if they can be found.
Making responsible people responsible publicly is most effective way. Fully deserved shame can do wonders to world.
It does seem like its way too easy to do this kind of fraud.
The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now. Is there some way to change incentives to get people to try and replicate? Assign undergrads to replicate random research papers?
Replication is unpaid QA on someone else's product. Nobody gets tenure for finding the bug, hence the field has been meaning to get around to it for 20 years.
And the incentives are arguably worse than QA: you get little credit if the result replicates, and potentially a lot of grief if it doesn't. Novel results build careers. Checking them mostly doesn't
On the other hand, this seems like a super interesting path for an (unglamorous) PhD -- replicate the N most important findings in a field, subject to some restrictions (maybe focus on the ones that are "easy", or haven't been replicated yet, or the ones that could be patched together to make an acceptable PhD thesis). This sort of stuff really appeals to me, I think I might end up doing something like that in 5-10 years.
I share your enthusiasm for this. Creating a map of the most influential research across all the fields to identify the highest priority work, any work replicating that work or validating that work by building from it would be a good start. Ideally funded by governments, but institutions, companies, and individuals could put up bounties for replicating particular research.
Good luck finding an advisor/a department that awards PhDs for that.
“It’s not novel”.
I think you're right that this will be the most common objection, but wrong that it will be exceedingly difficult to find an advisor that would support this. I assume that if you _can_ overturn some foundational result, you won't face criticism over novelty (potentially you can argue that overturning important accepted results is inherently novel). Then the rest is just math:
Assume that n% of foundational studies are demonstrably incorrect if replication were attempted. Then if you replicate X foundational studies, there's a 1 - (1-n/100)^X chance that you'll show one foundational study to be incorrect. Then I think you just need to find an advisor who believes the incorrectness rate n, and the replication attempts X is high enough for you to revert a foundational study over the course of your PhD.
Universities, institutions and publications who care about their reputation should pay people to replicate studies performed by people they employ in their name. Otherwise they will loose their trustworthiness.
But perhaps it is not as big a shame for them as some think it should be...
> The study seems extremely easy to replicate (no fancy equipment needed, just ask some people to proofread), and yet nobody did until now
They were just putting it off.
Why do we even accept anything as true without replication by default?
Speak for yourself!
My default position for a long time has been that all results in the social sciences are garbage. If they go against intuition or my biases, then they are most likely garbage. If they confirm my intuition or my biases, then they are definitely garbage.
Sure but I mean the community as a whole. Why has there been allowed to be a reproducibility problem at all? Why has the consensus not always been that without provably reproducible results it’s not to be taken seriously?
Why do people on this site love nutrition and self-medication fads?
People are surprisingly thoughtless.
Or as one might say, predictably irrational.
Because “science” doesn’t know everything, and people don’t want to wait until their options are scientifically-validated.
Yoga, breathing exercises, meditation and the like were once considered fads and derided by your ‘science’.
> Yoga, breathing exercises, meditation and the like were once considered fads and derided by your ‘science’.
What do you mean "once"?
In case you don't follow the field: https://www.nccih.nih.gov/health/meditation-and-mindfulness-...
> Meditation and mindfulness practices may have a variety of health benefits and may help people improve the quality of their lives. Recent studies have investigated if meditation or mindfulness helps people manage anxiety, stress, depression, pain, or symptoms related to withdrawal from nicotine, alcohol, or opioids.
> Other studies have looked at the effects of meditation or mindfulness on weight control or sleep quality.
> However, much of the research on these topics has been preliminary or not scientifically rigorous. Because the studies examined many different types of meditation and mindfulness practices, and the effects of those practices are hard to measure, results from the studies have been difficult to analyze and may have been interpreted too optimistically.
This is just a polite way of saying "there's no evidence of them actually doing anything*".
I mean, that's exactly what you'd tell someone who believed in the healing powers of homeopathy or chiropracty(sp?).
I recommend to skip the intro and go to specific articles.
That’s different as I don’t think anyone needs science to feel the benefits of yoga or breathwork but lots of people want to enact policies based social studies that are not able to be reproduced.
Scientists know that they don’t know everything and act accordingly, they don’t make claims for the efficacy of the things they don’t know.
People here routinely do.
How do you decide whether something works or not?
You don't. It is too costly and thus risky. Instead you do a charade and convince authorities (who just want to cover their asses citing some vetted studies) and people that your stuff work.
It is "ass covering" all the way. Not "does it work" all the way...
Are you a vendor of orgone accumulators or a staunch believer and customer?
[dead]
My proposal, once you complete your study, if you find a significant positive result you should be required (prior to publication) to describe your methodology in full, and have another lab (who you never have any contact with) blindly take your methodology, and repeat it (on video), and then do their own independent analysis of the data.
It's okay if it's $200k or so to do each of these, the university should pay. The point is if this methodology were applied people would have incentive to cheat anymore. It would also serve as an immediate replication.
Schools won't do this because they have a perverse incentive for big successes and quiet retractions. It's corruption all the way up.
Would be a great way for up and coming researchers to learn and apply skills before attempting to work in labs pursuing novel research.
Maybe some joint effort across a large number of research schools which function as “proportional to number of students in your program, X number of validation studies must be performed” with maybe some monetary or access incentives aligned.
Since we’ve had student loans all but guaranteed, and therefore cost of schooling skyrocketing, the flywheel of pursing higher profits and more bureaucratic cruft (looking at ratios of university staff to students), its no wonder that incentive has extended to research.
Replication is overrated. There's no point in doing more of it. You'd just replicate a ton of bad science:
https://blog.plan99.net/replication-studies-cant-fix-science...
N.B. nobody needed to replicate this study. It only took basic sanity checks to spot that it was very unlikely to be true (and thus, very unlikely to replicate). And they went much further than needed. They could have stopped at "the claimed effect is much larger than the height difference between men and women". This obviousness is a common pattern:
https://www.fantasticanachronism.com/p/whats-wrong-with-soci...
> Prediction markets work well because predicting replication is easy. There's no need for a deep dive into the statistical methodology or a rigorous examination of the data, no need to scrutinize esoteric theories for subtle errors—these papers have obvious, surface-level problems.
So you could also say, how do we change the incentives to get people to peer review properly? That would be a lot cheaper.
The obvious solution is for people to get paid to audit work professionally, and be fired if they sign off on bad results. But that exists already, it's called a corporate lab. Managers are incentivized by carrots and sticks to properly understand what their researchers are doing and if anyone is doing fraud they will be fired, maybe prosecuted. Universities don't do this because so much of what they do exists purely to spend money, with no higher purpose. There's no specific goal or outcome they're trying to achieve, and so no reason to do unpleasant things like enforcing incentives to get there.
So in the end the solution is just to defund university research. The social structures are fundamentally unfixable. You'll never get people to care if the only purpose of their job's existence is to convince voters governments are funding science.
Fund it. Unfortunately there's hardly any money for regular studies, let alone replication studies. If you want better science there has to be tax dollars behind it. Otherwise corporations choose what gets funded, and they only fund narratives and experiments they think will be profitable for them.
> Otherwise corporations choose what gets funded, and they only fund narratives and experiments they think will be profitable for them.
Well, that's the incentive mechanism that keeps the accountability for the results whether we like it or not. The government doesn't immediately go out of business if the research they fund is full of fraud or goes nowhere useful. Moral outrage at all the fraud in research associated with these Universities and government funded studies isn't a powerful enough incentive mechanism to correct that fraud, apparently.
"easy to replicate (no fancy equipment needed, just ask some people to proofread)"
? This was a human subject study, that would cost $$ and there would be very little upside to confirmation and only a little notoriety with a failure to replicate (which by the way is not dispositive, people might need multiple failed replications to know who to believe, depending on the size of the replication). Maybe you didn't read the post: this was data fraud discovered by statistical irregularities. That is indeed becoming easy to spot with more widespread statistical literacy, at least for unsophisticated jobs like this from decades ago.
> Is there some way to change incentives to get people to try and replicate?
Sure. Pay them to do it.
If the culture shifted such that people were skeptical of original results but excited about replication, replication would be the big news and people would want to do it. How to get there, I don't know.
I seem to remember a lot of hype about room-temperature superconductors, that croaked upon [lack of] replication. That wasn't too long ago.
Before that, cold fusion.
We still have the ion drive that doesn't require fuel. I don't think it has ever been successfully replicated, but I think that it also hasn't been successfully repudiated, either.
This kind of thing has been going on for a long time. I suspect that not all of it is deliberate fraud, but rather, people wanting their preconceived bias to be satisfied.
Replicating superconductor experiment gives you the conductor and the skill.
Both of them are useful as is.
Replicating a social experiment, when successful, don't give you anything new.
No, that would never happen. People would just be skeptical of most findings until they see replications (where we are headed in the social sciences, anyway). If people see replications the original discoverer will get all the accolades, not the (several) replications.
Or maybe we shouldnt call anything "science" until it assumes replication of foundational results.
Physics solved this quite some time ago.
Then it "replication" becomes a target and thus would be gamed instead.
This is a constant tug of war..
[dead]
No paper should ever be referenced until it has been replicated by an independent group. The replication crisis in science is real, and it won't take long for no one to believe anything anymore.
Not all science has this issue. But to reference Thiel, a lot of the sciences with "science" in the name, seem to.
That rule would make it difficult to report on a failure to replicate a published paper.
Does some sort of batched style of research already exist? By batched, I mean a study where n number of teams all agree on a single methodology, and then conduct the same experiment simultaneously. Each team could then publish their results independently and the teams could publish one aggregate.
I know meta studies attempt to do this in a way, but they can sometimes have their own problems too.
I cannot recall coming across this style of research in much of what I have read as a hobby, but maybe it is more common in some domains than others? I am not smart, and there is nothing new under the Sun, so surely something like this exists, no?
It doesn’t seem very popular but that exact thing does seem to exist, here called a ‘Registered Replication Report’:
https://www.psychologicalscience.org/publications/replicatio...
There are some completed which is cool. I agree that this seems like a great initiative.
Not quite what you're asking, but pre-registration does alleviate e.g. p-hacking and such without the need for multiple teams. It doesn't do much for outright fraud, but still...
Only if the study matches its pre-registration and someone is checking that. Which it turns out, they often don't.
https://www.compare-trials.org/
That's a fair point. Ultimately, incentives are all screwy at the moment.
Gotta tie some actual prestige/shame to not following up.
If the failure is replicated by multiple groups, that would be reportable.
Otoh, not sure that anyone wants to read a report that some finding that nobody reported on couldn't be replicated.
Every paper needs to include an independent replication of the study.
So build a second particle collider to validate what comes out of CERN?
If only we had that level of science funding.
>The replication crisis in science is real
In psychology, you mean. Many contest if it even qualifies as a science anymore after the complete annihilation of trust it has undergone.
In the physical sciences, any sufficiently important paper is independently verified. Sounds like cost/effort of these multi year studies is not faesible to replicate, but then again, I know nothing about social science so w/e
> The replication crisis in science is real, and it won't take long for no one to believe anything anymore.
If you think replication in the sciences is in crises, wait till you see the field of psychology.
I came here for a procrastination joke comment and didn't find it, but I suppose someone will get around to it eventually.
Here's one. I always leave my procrastination for later.
> I came here for a procrastination joke comment and didn't find it, but I suppose someone will get around to it eventually.
Too soon; they'll all do it later.
I’ll do it later
Dan Ariely is well past the point that he needs to not be a faculty member at Duke University. Data Coda do a fantastic job ferreting out academic fraud. With these fraudulent behavioral science experiments like this relying on printed papers/markups, and coauthors not present and who did not even see the raw data, I wonder if the experiment was ever actually conducted. I would love to see that question answered where fraud is suspected, something I'd hope IRBs would step up to do.
I'm glad the authors seem to have acknowledged the problem and not doing much about it :-)
Per the post, the original authors have requested that the original paper be retracted. That’s well above the bar of “not doing much”.
Actually the post is ambiguous: did the authors want the original article retracted, or the replication by Hyndman and Bisin? I hope the former!
"Finally, it should be noted that when we shared these posts with Ariely and Wertenbroch a few weeks ago, they reached out to Psychological Science to request that the article be retracted. As of this writing, that process is ongoing."
The post is unambiguous if you make it to the very end, where there are particular statements from each of the people involved.
Look for:
> Klaus Wertenbroch […] indicated that he has asked the editor to retract the paper.
That's pathetic.
How did he agree to be a co-author when "he never had access to the data for any of the studies"?
One thing I didn't know before reading this: an unusually large effect size can itself be a red flag. You don't have to understand all the statistics to at least notice when a result looks almost too good to be true.
It is interesting how all these famous non-replicating psych paper results feel reasonably intuitive to me. Like I can buy that having a more structured and detailed deadline helps you procrastinate less. I can even buy that doing a goofy little "power pose" exercise kind of hypes yourself up to give some self confidence and do better on an interview or whatever.
You would almost expect the opposite and that all the famous studies would be the ones that run counter to intuition. Maybe those get more replication attempts, or people feel less confident faking them.
The direction is intuitive, but the effect sizes are, as you say, unusually large. I'm no statistician, but if you have an effect size that's several multiples of the standard deviation, that would mean that it dominates the natural variation in results right? So you're kind of making an implicit claim that other factors in play (for eg in this proofreading task, how busy someone is or their reading level) are peanuts in comparison.
Further, if there are these relatively simple factors like how a deadline is expressed that make you several times better, you'd expect anybody who stumbles into several of them and stacks them together to be absolutely off the charts. Somebody who listens to Mozart, gets the right priming, doesn't deplete their ego, and receives short deadlines is suddenly like 50X more productive, which doesn't seem to match reality.
Your exact intuition about the implausibility of many large effect sizes is called the “Piranha problem” by my favourite statistician Andrew Gelman first here in a blog post: https://statmodeling.stat.columbia.edu/2017/12/15/eats/
And then as a published paper: https://sites.stat.columbia.edu/gelman/research/published/pi...
> how many explanatory variables can have large effects on an outcome?
This is one of the things that did in Jan Hendrik Schön.
I can't remember the exact measurement but he was showing a set of curves that were simply too neat for a relatively small sample size - roughly 100 or so plastic transistors. In reality you cannot get that neat an effect without a huge sample population
Having skimmed this article and the replication study itself (which references some studies that complicate our common understanding of the positive effects of deadlines) I’m left wondering what the constructive alternative is to deadlines and/or how deadlines should be maintained or determined.
[dead]
There is no alternative to deadlines because time is money, and those downstream of you can ill afford to wait on your deliverables. Part of management's job is to determine how much time it should take so a reasonable timetable that stakeholders can agree to can be set, and then hold you accountable for delivering within that time frame.
I wish we could revive Tim Bryce's mind with AI, because I'd love to hear what he'd have to say in a podcast about this topic:
"Fraud in a procrastination study? You don't say! I don't understand why all this taxpayer money is going to study procrastination in the first place. Procrastination is a well-known phenomenon with a well-understood cause: laziness. Procrastinators didn't have a boot placed to their behinds growing up, so they reached adulthood slacking and coasting on their trust-fund money or couch-surfing at their friends' places, mooching off others. You see this a lot in modern software organizations, especially among the programmers. And now those programmers are all crying because they're about to get replaced by Claude Code. Boo hoo! The military knew how to deal with procrastinators, too: send them to scrub latrines with a toothbrush, and drum them out if they didn't shape up after that! We don't need millions of dollars in government grants going to academic research to study this phenomenon. It's as old as humanity itself—we called it the deadly sin of sloth. Friends, keep the faith!"
> I wish we could revive Tim Bryce's mind with AI, because I'd love to hear what he'd have to say in a podcast about this topic:
If he's anything like how the rest of your post read, he sounds like a pretty awful person.
Well, no, he was not a pleasant person. Back in the day he was a management consultant, best known for an essay which rustled many programmer jimmies at the time: "Theory P: The Philosophy of Managing Programmers" (2005):
https://web.archive.org/web/20190416200125/https://it.toolbo...
Theory P advances the thesis that because programmers are lazy pseudointellectuals who chafe at discipline and use their arcane technical knowledge to gain power and leverage in the workplace, they need to be tightly managed, their small roles within the company clearly delineated, and discipline and organization strictly enforced.
A more recent (2018) essay of his is much more blunt and to the point, shows that he hadn't changed his mind at all, and concludes with "Is it any wonder why I despise programmers?":
https://patch.com/florida/palmharbor/amp/28237018/the-proble...
The, ahem, problem with Tim Bryce is twofold:
a) for good or ill, he said what your boss—or at least top management at your company—was thinking
b) he was right about many things. He was an exponent of PRIDE, the first widely available commercial software development methodology created by his father, Milt Bryce. Unlike Scrum or even SAFe, which are more like loose frameworks, PRIDE is a complete soup-to-nuts instruction manual on how to develop, implement, and integrate business information systems (including, but not limited to, those which operate via computer). As Bryce himself said: "PRIDE answers the 5 W's of information systems: WHO is to perform WHAT task, WHERE, WHEN, and WHY." PRIDE is pretty much Spec-Driven Development writ large; in PRIDE, the majority of the work and the source of nearly all the value is in systems analysis, not programming: determining how the business operates and what information systems need to be in place to support the business. This is stepwise-refined into a series of more and more detailed specifications, which are only then handed off to programmers to code. Doing it this way saves time and money and leads to more correct software on the first go. It's also pretty close to the state of the art in AI agentic workflows.
Tim Bryce would be giddy with glee had he lived to see today, because finally all those lazy programmers would be put out of work! Since under PRIDE, programming is mere translation of requirements into code, that can now be automated and asymptotically approaches requiring no human effort at all! The fact that you and I dislike his personality doesn't change the fact that what he advocated for is exactly what's happening now, and that business has for centuries and will continue to operate as if what he believed were true—including the bits about procrastination being simple laziness, correctable with stricter discipline, rather than some sort of complex psychological phenomenon.
> and that business has for centuries and will continue to operate as if what he believed were true
Then maybe we should do away with "business". Small operations do not tend to operate that way, and broadly everyone prefers their output (if not price) in the domains in which they operate – with a few notable exceptions. In theory, software should allow small operations to each serve millions of people, such that we (e.g.) only need a few thousand search engine providers to meet the needs of the entire world's population. Not everything has to be enterprise-scale, and indeed many things should not be.
If "business" means treating people badly, then we don't need it.
upvoted for an interesting and informative response. This is why I come to HN. Thanks :)
You're basically replacing the label "procrastination" with the label "laziness" without understanding anything.
Are all people who postpone a task lazy?
What constitutes laziness? Can we define laziness?
Why are people lazy? Is laziness negative? Are there positive aspects to laziness? Why did we evolve laziness?
Are people consistently lazy, or are they only lazy in some areas of their lives?
Does scrubbing a toilet with a toothbrush actually reduce laziness? Are there other methods of reducing it?
What are the effects of reducing laziness? Are there emotional or physical penalties from reducing laziness?
Answering some of these would be useful. Just saying "everyone who procrastinates is lazy" is not.
> Are there positive aspects to laziness?
Hard work often pays off in the future, but laziness always pays off now.
I am not lazy (for work. I am in my personal life) but when I procrastinate it's due to lack of clarity or due to issues acquiring focus. Both I have ways to address, so I completely agree with you
I don't even think procrastination is laziness as you end up having to do the work later.
Not if you're procrastinating on the right things.
Defered tasks are sometimes completed by others, or otherwise become obsolete while you wait.
I hope that with LLMs we'll see a lot faster identification of these anomalies. Imagine that this guy taught at MIT and has his own lab at Duke, basically more than 20 years of taking up some of the most prestigious positions in his field, all based on fraud.
The butterfly effect this has on science as a practice is huge. Some marginal researcher who actually deserved it was bumped out of those places, and somewhere down the line someone was bumped all the way out of academia because of this guy's lies. And this is hardly the only person in this situation.
What's scary is that the next fraudster will read this take down and see what they have to do to avoid detection. It wouldn't be that hard to form data that looks a lot more plausible. Replication may be the only solution, unfortunately it's so expensive and time consuming, without any of the actual benefits of doing the original study. Maybe it should be government funded, the US replication lab, its only mission to replicate the top X papers from some named fields each year.
They will be generated faster than they can be detected.
It's shocking to me how badly the data was falsified. You'd think a researcher who is A. well versed in statistics and B. has seen a lot of actual research data would be able to produce more believable data. Does this mean we're only seeing the tip of the iceberg? how many papers are based on hard to detect falsifications?
Why bother? Low effort data falsification is enough to land a high-ranking paper. Sure, really simulating data with some noise would be much more convincing but its just not necessary. Raw data is very rarely inspected during peer review, much less so after publication, and most reviewers do not address statistics — if they do, the usually refer to their pet model and/or use the same incorrect tools as most psychologists.
There is such a smell around this whole literature.
I wonder what it's like to be a grad student in social psychology right now. When the norm is surprising results, any responsible, competent researcher is going to realize that they can't do it honestly. So this leaves behind people who are willing to bend the rules, or people who don't see the problem with, for example, the garden of forking paths.
In good universities replication crisis and the underlying structures are taught. (Un?)surprisingly, not much has changed though. Except for some unis, some journals, and a bubble of rigorous research continuing to work on meta-science and replications. Have to admire their stamina.
Source: did my PhD on exactly this topic
People for a long time have had far too much faith in peer review as and had a dim view of anything other then experimental results and work, despite how much fraud and just bad experiments their are. Not to mention how many variables their might actually be to account for in something like psychology or any of the social science's.
Deriving things from axioms and postulates alone is problematic but some one who can do both begins to actually do useful work in my opinion and some one who can accept the difficult their actually is in knowing anything about human psychology simply from studying it and underneath all experiments is assumptions and the people in the experiment who have so many variables. This is not like medicine human psychology is very flexible and for me though not a social scientist I feel of all theories of human psychology they are only applicable to people now and not even 300 years ago, though of course one must concede that the micro psychological principles might be applicable to humans more generally.
Short of blindly outsourcing the work to an LLM and checking to see if its hallucinated problems are actually problems[0], how can more laymen (without the ability to run replication studies, and without degrees in statistics) learn to more easily identify questionable/potentially fraudulent research like this?
This is a question that has been really bothering me for a while. I have a short list of things I’ve learnt to check, but it’s all stuff I’ve picked up in a very ad hoc way (reading and listening to these kinds of critical analyses, mostly). For example, until now, I wouldn’t think about implausibly large Cohen’s d, despite this seeming like an easily generalisable rule that anyone could eyeball. But without being told, and without the reference points given in the article, I wouldn’t have a clue. And I would like to have more of a clue. And I would like other people to have more of a clue. How?
[0] https://news.ycombinator.com/item?id=49484925
Sure. The guy put it off until the last minute, and then had to fake his way through it. It writes itself.
The reason why people(or ai agents) have not been able to systematically destroy bad research in fields where fraud is so rampant is because they data are not published or easily accessible without emailing the author and asking them.
I sincerely hope that any self respectful journal that publishes papers should have a minimum bar: submit the data on a public link with no paywall or blockers and feature the link prominently on your paper.
In this case even emailing the author didn't work. The data was obtained via some roundabout method.
But it's not the real reason IMHO. You could use LLMs to write takedowns like this one. I gave ChatGPT the paper PDF and some neutral prompts. Although it didn't go straight to 'this is clearly fraud' - it's too nice for that - it did compute and point out the absurdly huge effect sizes, and give a relatively low confidence the paper's central claim would replicate.
The real reason there aren't bots systematically destroying papers is because universities defend fraud, they don't punish it. You'd just be burning tokens to reveal lots of fraud with no outcome.
There is a lot of bad ML research for the same reason.
Dan Ariely remaining at Duke, after it has become clear that much if not most of his work is fraudulent, is at least as big a black mark for Duke as Jason Arday was for Cambridge.
Not to mention the Epstein ties.
It's arguably worse. Jason Arday was doing qualitative work in education. Nobody pays any attention to anything UK educationalists write. Dan Ariely was a genuine superstar and writer of popular bestsellers (with academic content).
I agree Ariely was more influential and his fraud more flagrant (obviously Arday's lies were more absurd, but they were less academically relevant), but it's not true that Arday had no attention given to him. He did provide advice to government groups on policy [1,2]. Furthermore, the high publicity around his story about being non-verbal until he was 11 probably gave a lot of false hope to people with non-verbal children.
[1] https://committees.parliament.uk/oralevidence/2482/html/
[2] https://faculty.educ.cam.ac.uk/230223-jason-arday
Dan Ariely was one of the biggest disappointments and betrayals of my intellectual life. I found the "research" he did to be mind-blowing, groundbreaking stuff at the time, and I told many people his work, sharing his papers and podcasts as a true enthusiast.
I am astonished that he holds an academic-adjacent position anywhere, no matter what his regrets might be today. Private redemption shouldn't be out of reach, but it shouldn't be possible for him to hold a position at any institution concerned with uncovering Truth. He should have resigned out of respect, if his institution was too gutless to do so. It's a tragedy.
The last point in the OA about rounding the estimate of the time spent on the assigned tasks. Just wondering if the response would depend on the wording of the instructions to participants and the nature of the incentivisation.
Me being a bit of a boy scout / Belbin 'company worker', I would tend to use a timer (I'm a pomodoro method person for things like marking or checking references &c).
The rest, oh yes. I stopped paying much attention to studies based on dozens of undergraduates doing weird concocted tasks years ago.
They probably procrastinated all their deadlines and then fibbed things.
So this paper appears to be fabricated, but was the conclusion actually wrong? I do appear to perform better on tasks with an external deadline than on self-set ones.
The conclusion is faulty given that it was derived from faulty data. The 2002 paper had two pilot studies and 2 bigger studies. The replication of study 2 failed[1], and the original data has substantial concerning features the rest of the datacolada article lays out. This has two unfortunate implications.
First, if study 2 was not necessary to support the conclusions, it seems likely it would not have been performed or included in this paper. So given it seems necessary, the conclusion is invalid. In past examples (the Reinhart-Rogoff paper comes to mind) when this happens, the authors claim it wasn't necessary and the conclusion is still valid and the professional embarrassment of a retraction is not called for. But in this case the replication failure might stand as a strike _against_ the theory.
But second, this is not the first questionable data coming from Ariely's lab, and it seems unlikely this was a data entry mistake. If study 2 is not trustworthy, we should update our priors about the trustworthiness of study 1. Note it's not guaranteed to be doctored in some way, just worthy of additional scrutiny. And if that one also fails to replicate, the paper and its conclusion seems unsalvageable.
Presumably his coauthor is now panicking about not keeping data from 25 years ago to exhonerate and distance himself.
[1]: https://journals.sagepub.com/doi/full/10.1177/09567976261460...
That’s not how science works
Figures, even the studies about procrastination are fraught with issues. Guess I'll put off reading the details until tomorrow.
this image https://datacolada.org/wp-content/uploads/scatter_like_vs_in... itself visually shows fake data. Too perfectly clustered.
What should I do now with the 4+ books authored by Ariely that I have sitting on my shelf?
Books on procrastination? Now you don't have to get around to reading them! (I jest)
At the risk of bringing up a controversial topic, in the midst of the Jason Arday furore last month when everyone was like "this wouldn't happen to a white professor", the person who kept coming to mind for me was Ariely. I became a data scientist in no small part due to his early work, and seeing academia totally incapable of punishing him is deeply disappointing.
There's a bit of irony that probably 50% of us are procrastinating while reading this
I'll get around to reading this at some point.
Procrastination: They couldn't be bothered.-
"Trust the science" is one of the most anti-science phrases ever invented. The whole point of science is to challenge everything.
The name of the place should be enough of a giveaway for any thinking person Center for Advanced Hindsight.
SMH.
And yet it will live forever, just like Dunning-Kruger despite its statistical issues. I fully expect to see it cited during some workshop or other pointless training exercise.
[dead]
[dead]
[dead]
[dead]
[dead]
[flagged]
Oh the irony...