In the lyceum where I studied, there was one lab on Physics, where the book that accompanied the lab was deliberately wrong. We were told to perform an experiment that "should" support a certain conclusion, but actually neither the "correct" conclusion nor its opposite could be done because of the flawed setup which measured something slightly different. A lot of students (in some groups, all students) fell into this trap and submitted paperwork with the "correct" conclusion according to the book.

I've come to think things works as well as it does largely because a whole lot of what people do has no effect either way. I see so much stupidity where the only saving grace is that it is directed into pointless efforts that won't be allowed to do any real damage.

A bio professor of mine said something that stuck with me: “life doesn’t work perfectly, it just works.”

It has to work well enough to… work… and reproduce. That’s it. It’s not “survival of the fittest.” It’s “survival of a randomized subset of the fit.”

There’s even a set of thermodynamic arguments to the effect that systems are unlikely to exceed such minimum requirements for a given threshold. For example, if we are visited by interstellar travelers they are likely to be the absolute dumbest and most dysfunctional possible examples of beings capable of interstellar travel since anything more is a less likely thermodynamic state.

So much for Star Trek toga wearing utopian aliens.

The problem is the incentives. To do well, you must publish. To publish, you must have a good story, and ‘we tried this and it didn’t work’ is not one.

So after a certain time spent, you are left with a choice of ‘massaging’ the data to get some results, or not and getting left behind those that do or were luckier in their research.

Distracting from the main point, "let's just try something and hope nothing bad happens" (trial and error) is precisely the reason civilization made it this far :)

> It's really a miracle the civilisation works as well as it does.

I think this all the time.

Evo psych is questionable to me for more basic reasons. It seems full of untestable just so stories to explain apparent biases that are themselves hard to pin down or prove are a result of nature not nurture.

It’s probably not all bullshit but I would bet a double digit percentage of it is.

> “how many of these zillion studies cited that are actually replicable?” This doubt remains despite knowing that the scientist is one of the leading experts in the field, and very down-to-earth.

I think the problem is much bigger than simply a binary is it replicable or not. It’s extremely easy to find papers by “leading experts” that have valid data with replicable results where the conclusions have been generalized beyond the experiments. The media does this more or less by default when reporting on scientific results, but researchers do it themselves to a huge degree, use very specific conditions and results to jump to a wider conclusion that is not actually supported by the results.

A high profile example of this is the “Dunning Kruger” effect; the data in paper did not show what the flowery narrative in the paper claimed to show, but there’s no reason to think they falsified the results. Some researchers have reproduced the results, as long as the conditions were very similar. Other researchers have tried to reproduce the results under different conditions that should have worked according to the paper’s narrative and conclusions, but found that they could not reproduce, because there were specific factors in the original experiment that were not discussed in the original paper’s conclusions -- in other words, Dunning and Kruger overstated what they measured such that the conclusion was not true. They both enjoyed successful academic careers and some degree of academic fame as a result of this paper that is technically reproducible but not generally true.

To make matters worse, the public has generally misinterpreted and misunderstood even the incorrect conclusions the authors stated, and turned it into something else. Almost never in discussions where the DK effect is invoked do people talk about the context or methodology of the experiments, or the people who participated in them.

This human tendency to tell a story and lose the context and details and specificity of the original evidence, the tendency to declare that one piece of evidence means there is a general truth, that is scarier to me than whether papers are replicable or not, because it casts doubt on all the replicable papers too.

> The book is extremely competently written, and is awash with statistics on almost every page, "70% of men this; 30% of women that ... on and on". And much to my solace, the scientist was super-careful to distinguish studies that were replicable, and those that were not.

Out of curiosity, what's the title of the book?

One approach that can be adopted on a personal level is simply changing the way one thinks. For example, switch from a binary (true/false) method of epistemology to trinary (true/false/unknown), defaulting to unknown, and consciously insist on a high level of certainty to reclassify an idea.

There's obviously more complexity than this, but I believe that if even a relatively small percentage of the population started thinking like this (particularly, influential people) it could make a very big difference.

Unfortunately, this seems to be extremely counter to human nature and desires - people seem seem compelled to form conclusions, even when it is not necessary ("Do people have ideas, or do ideas have people?").

Yes. A famous recent example was stress-induced stem cells:

https://www.nature.com/news/failed-replications-put-stap-ste...

Yes, but you have to convince your readers that you did a more careful and meticulous job than 'Top Institution's Best Paper Award' did. After all, a failure to replicate only means that one of you is wrong, but it doesn't give any hint as to who.

Just don't forget that the guy who wrote the Best Paper will probably review your articles in future.

Maybe all papers should have a replication score ?

The problem with experimental results is that they are difficult to replicate. In software you can "git clone x.git & cd x & make" and replicate the correct or incorrect results. In hardware, it's more difficult.

The main problem is that even if you reproduce their experiment, they can claim that you did some step wrong, perhaps you are mixing it too fast or too slow, or the temperature is not correctly controlled, or that one of your reactive have a contamination that destroy the effect, or magically realize that their reactive that is important.

It's very difficult to publish papers with negative results. So there is a high chance it will not count in your total number of publications. Also, expect a low number of citation, so it's not useful for other metrics like citation count or h.

For the same reason, you will not see publications of exact replications. A good paper X will be followed by almost-replications by another teams, like "we changed this and got X with a 10% improvement" or "we mixed the methods of X and Y and unsurprisingly^W got X+Y". This is somewhat good because it shows that the initial result is robust enough to survive small modifications.

It's harder to publish negative results.

The problem isn't just the scientists, it goes all the way up. Let's say we implement your system. Who decides how many 'smart professors' the Type Theory group gets to hire? What if the Type Theory and Machine Learning departments both want to hire a new 'smart professor' but the Computer Science department only has money to hire one more person?

One reasonable approach might be to look at which group has produced the 'best' research over the past few years. But how do you judge that in a way that seems fair? Once you have a criteria to judge that, then people will start to game that criteria.

Or taking a step up, The university needs to save money. How do you judge if the Chemistry department or the Computer Science department should have its funding cut.

No matter how you slice it at some point you're going to need a way for someone to judge which of two departments is producing the 'best' research and thus deserves more money, and that will incentivize people to game that metric.

Smart isn't the biggest criteria for success as a professor. The PhD degree is a good filter because it trains and tests research aptitude, work ethic, ability to collaborate, ability to focus on a single problem for a long period of time, and others.

One problem is PhD degrees are too costly to those who don't get academic or industrial success from them. But as long as talented people are willing to try to become a professor I don't see the system changing.

Who is going to decide whether professor is smart?

Not my turf but I'll chime in.

In the past people who did science could do so with less personally on the line. In the early days you had men of letters like Cavendish who didn't really need to care if you liked what he wrote, he'd be fine without any grants. That obviously doesn't work for everyone, but then the tenure system developed for a similar reason: you have to be able to follow an unproductive path sometimes without starving. And that can mean unproductive in that you don't find anything or in that your peers don't rate your work. There'd be a gap between being a young researcher and tenured, sure.

Nowadays there's an army of precariously employed phds and postdocs. Publish or perish is a trope. People get really quite old while still being juniors in some sense, and during that time everyone is thinking "I have to not jeopardise my career".

When you have a system where all the agents are under huge pressure, they adapt in certain ways: take safer bets, write more papers from each experiment, cooperate with others for mutual gain, congregate around previous winners, generally more risk reducing behaviour.

Perhaps the thing to do is make a hard barrier: everyone who wants to be a researcher needs to get tenure after undergrad, or not at all. (Or after masters or whatever, I wouldn't know.) Those people then get a grant for life. It will be hard to get one of these, but it will be clear if you have to give up. Lab assistants and other untenured staff know what they are negotiating for. Tenured young people can start a family and not have the rug pulled out when they write something interesting.

Solution is to publish data, not „papers“ first and assign it a replication score - how many times it was verified by independent research. The paper can follow with the explanation, but citations will no longer be important - what will matter is the contribution to the replication score (will also work as an incentive to confirm other‘s results).

Im not sure prosecuting academics is particularly obvious: you'd need to prove malicious intent (rather than ignorance) which is always difficult.

For me a better solution would be to properly incentivise replication work and solid scientific principles. If repeating an experiment and getting a contradictory result carried the same kudos as running the original experiment then I think we'd be in a healthier place. Similarly if doing the 'scientific grind work' of working out mistakes in experimental practice that can affect results and, ultimately, our understanding of the universe around us.

I think an analogy with software development works pretty well: often the incentives point towards adding new features above all else. Rarely is sitting down and grinding through the litany of small bugs prioritised, but as any dev will tell you doing that grind work is as important otherwise you'll run in to a wall of technical debt and the whole thing will come tumbling down.

I really don’t know.

One perspective is that, “knowledge generation wise,” the current system really does work from a long term perspective. Evolutionary pressure keeps the good work alive while bad work dies. Like that [Top Institution] paper: if nobody else could reproduce it, then the ideas within it die because nobody can extend the work.

But that comes at the heavy short term cost of good researchers getting duped into wasting time and bad researchers seeing incentives in lying. Which will make academia less attractive to the kind of people that ought to be there, dragging down the whole community.

This is a recent HN thread and Post you might find interesting.

https://nintil.com/newton-hypothesis

https://news.ycombinator.com/item?id=25787745

Due to career and other reasons, there is a publish or perish crisis today.

Maybe we can do better by accepting not everyone can publish ground breaking results, and it's okay.

There are lots of incompetent people in academia, who later go to upper positions and decide your promotions by citation counts and how much papers you published. I have no realistic ideas how to counter this.

> Honest question: how do we fix this?

We need to create new a social institution of Anti-Science, which would work on other stimuli correlated with the amount of refuted articles. No tenures, no long-term contracts. If anti-scientist wished to have income it would need to refute science articles.

Create a platform allowing to hold a scientific debate between scientists and anti-scientists, for a scientist had an ability to defend his/her research.

No need to do anything special to prosecute, because Science is a very competitive, and availability of refutations would be used inevitable to stop career progressions of authors of refuted articles.

Data and code archives, along with better methods training.

Data manipulation generally doesn't happen by changing values in a data frame. It's done by running and rerunning similar models with slightly different specifications to get a P value under .05, or by applying various "manipulations" to variables or the models themselves for the same effect. It's much easier to identify this when you have the code that was used to recreate whatever was eventually published.

I don't think prosecution is the right tool but if we were going down that road material misrepresentations only would fit with anti-fraud standard for companies. Just drawing dumb, unpopular, or 'biased' conclusions shouldn't be a crime but data tampering would fall into the scope. Not a great idea as it would add a chilling effect, lawyer-friction and expenses and still be hard to enforce for little direct gain.

I personally favor requirements which call for bundling raw datasets with the "papers". The data storage and transmission is very cheap now so there isn't a need to restrict ourselves to just texts. We should still be able to check all of the thrown out "outliers" from the datasets. An aim should be to make the tricks for massaging data nonviable. Even if you found your first data set was full of embarassing screw ups due to doing it hungover and mixing up step order it could be helpful to get a collection of "known errors" to analyze. Optimistically it could also uncover phenomenon scientests thought was them screwing up like say cosmic background radiation being taken as just noise and not really there.

Paper reviewing is already a problem but adding some transparency should help.

We could use public funding to do the work OP tried to do.

Something like a well funded ten year campaign to do peer review, retrying experiments and publishing papers on why results are wrong.

I have a co-worker who had a job than involved publishing research papers. Based on his horror stories it seems like the most effective course of action is to attack the credibility of those who fudges results.

The single biggest impediment to "fixing this" is that you haven't identified what "this" is or in what manner it is broken.

There will always be cases of fraud if someone deeps deeply enough into large institutions. That doesn't actually indicate that there is a problem.

Launching in to change complex systems like the research community based on a couple of anecdotes and just-so stories is a great way not actually achieving anything meaningful. There needs to be a very thorough, emotionally and technically correct enumeration of what the actual problem(s) are.

By waiting until scientists address this? Note that the 'replication crisis' is something that originated inside science itself, so, despite there being problems science has not lost its self-correcting abilities. The scientists themselves can do something by insisting on reliable and correct methods and pointing it out wherever such methods are not in use. It is also not like there are no gains in doing this. Brian Nosek became rather famous.

Scientists with a proven track record should have life-long funding of their laboratory without any questions asked. So they can act as they want without fear of social repercussions. Of course some money will be wasted and the question of determining whether a track record is proven is still open, but I think that's the only way for things to work (except when the scientist himself have enough money to fund his own work).

at least in some parts of computer science the solution is easy: do not ever publish results without the public source code of all the experiments.

My personal opinion is this problem fixes itself over time.

When I was in graduate school papers from one lab at Harvard were know to be “best case scenario”. Other labs had a rock solid reputation - if they said you could do X with their procedure, you could bet on it.

So basically we treated every claim as potential BS unless it came from a reputable lab or we or others had replicated it.

An approach to how go about it is to include a replication package with the paper, including the dataset... This should be regarded as standard practice today, as sharing something was never easier. However, adding a replication package is still done by the minority of researchers...

You can't, except trying to fix human nature..

Instead of adding a punishment, maybe we should remove the reward. How, that I don't know.

More transparency in some form, requiring researchers to publish code and data openly for instance.

While I would have agree with that when I was younger. I learned there is a lot of possibilities why PhD students (the guys who do studies) fail to replicate anything (and I am talking about fundamental solid engineering).

Their username is publicly linked to their real-life identity. Revealing the name and institution has a reasonable chance of provoking a potentially messy dispute in real life. Maybe eob has justice on their side, but picking fights has a lot of downsides, especially if your evidence is secondhand.

I worked at a prestigious physics lab working for the top researcher in a field. It absolutely happens there and probably everywhere.

The only way to fix replication issues is to give financial and career incentives for doing replication work. Right now there are few carrots and many sticks.

I suppose it depends on how you define nice? If you cheat at some point people will catch on, even if you don't face any real consequences. So if you want prestige within your community then cheating isn't the way to go.

If you want to look impressive to non-experts and get lots of grant money/opportunities, I'd go for lots of straightforward publications in top-tier venues. Star findings will come under greater scrutiny.

For grants and tenure, 100 tiny increments over 10 years are much better for your research career then 1 major paper in 5 years that is better than all of them put together.

If you want to write a pop book and on TV and sell classes, you need one interesting bit of pseudoscience and a dozen followup papers using the same bad methodology.

You don't make a nice career in a vacuum. With very few exceptions, you don't get star findings in a social desert. You get star findings by being liked by influential supervisors who are liked by even more influential supervisors.

There's a book called science fictions that pretty much goes over the standard bullshit packages in modern science.

> I want to answer the question "if I were a researcher and were willing to cheat to get ahead, what should be the objective of my cheating?"

"Academic politics is the most vicious and bitter form of politics, because the stakes are so low."

https://en.wikipedia.org/wiki/Sayre%27s_law

If your main interest is reading papers and not being political about it, just use sci-hub to read the papers.

In many journals it's abuse of process to exclude reviewers you don't like. Much of the times this is supposed to be used to declare conflicts of interest based on relationships you have in the field.

I need money to do research, available grants require achieving specific measurable results during the grant (mostly publications fitting specific criteria e.g. "journal that's rated above 50% of average citation rating in your subfield" or "peer reviewed publication that's indexed in SCOPUS or WebOfScience", definitely not a preprint or blog), and getting one is also conditional on earlier publications like that.

In essence, the evaluators (non-scientific organizations who fund scientific organizations) need some metric to compare and distinguish decent research from weak, one that's (a) comparable across fields of science; (b) verifiable by people outside that field (so you can compare across subfields); (c) not trivially changeable by the funded institutions themselves; (d) describable in an objective manner so that you can write up the exact criteria/metrics in a legal act or contract. There are NO reasonable metrics that fit these criteria; international peer-reviewed-publications fitting certain criteria are bad but perhaps least bad from the (even worse) alternatives like direct evaluation by government committees.

When you are looking for a job, are up for promotion/tenure, or applying for grants, a long publication record in prestigious journals is helpful.

Metrics. You can’t manage what you can’t measure!

At some point, there's not going to be enough budget for both the football coach and the Latin philology professor. We should hire another three layers of housing diversity deans just to be safe.

If you add 'self funded' to this hypothetical person, then it would not matter if they play any games. Getting published is really not that hard if your work is good. And if it is good it will get noticed (hopefully during the hypothetical person's lifetime). Conferences have less of these games in my experience and would help.

Also, I know of no researchers personally who are enthralled by the existing system.

This has nothing to do with gatekeeping. I agree that the current publication and incentive system is broken, but it's completely unrelated to the question if outsiders are being published. The reason why you see very little work from outsiders is because research is difficult. It typically requires years of full time dedicated work, you can't just do it on the side. Moreover, you need to first study and understand the field to identify the gaps. If you try to identify gaps on your own, you are highly likely to go off into a direction which is completely irrelevant.

BTW I can tell you the the vast majority of researchers are not "enthralled" by the system, but highly critical. They simply don't have a choice but to work with it.

I think this is a bit naive. One thing that stops a smart person doing research without a PhD is that it takes a long time to learn enough to be at the scientific frontier where new research can be done. About a PhD’s length of time, in fact. So, many people without a PhD who try to do scientific research are cranks. I don’t say all.

Some quality journals and conferences have double blind reviews now. So the work is reviewed without knowing who the work belongs to. It's not so much the politics of the system as the skills required to write a research paper being hard to learn outside of a PhD. You need to understand how to identify a line of work in a very narrow field so that you can cite prior work and demonstrate a proper understanding of how your work compares and contrasts to other closely related work. That's an important part of demonstrating your work is novel and it's hard to do (especially for the first time) without expert guidance. Most students trying this for the first time cite far too broadly (getting work that's somewhat related but not related to the core of their ideas) and miss important closely related work.

There's lots of good science done in the commercial sector.

(There's lots of crowding out happening, of course, from the government subsidized science. But that can't be helped at the moment.)

I don't tell lies about bad papers, only give some peremptory praise so that reviewers don't have ammunition to kill my submission. E.g. if a paper makes a false contribution X and a true contribution Y, I only mention Y. If I were to say "So-and-so claimed X but actually that's false" I would have to prove it, and unless it's a big enough issue to warrant its own paper, I don't want to prove it. Any ways, without having the raw data, source code etc for the experiments, there is no way for me to prove that X is false (I'm not a mathematician). Then the reviews will ask why I believe X is not true when peer-review accepted it. Suddenly all of my contributions are out the window, and all anybody cares about is X.

The situation is even worse when the paper claiming X underwent artifact review, where reviewers actually DID look at the raw data and source code but simply lacked the attention or expertise to recognize errors.

I'm not taking bribes, I'm paying a toll.

I don’t really buy the comparison entirely. Presumably the sports boss is doing something patently illegal, and obviously there are alternative career paths. OP is working in academia, which is socially acceptable, and feels that this is what is normal in their academic field, necessary for their goals, and isn’t actively harmful.

I wouldn’t necessarily condone the behavior, but what would you do in the situation? To always whistleblow whenever something doesn’t feel right and risk the politics? To quit working in the field if your concerns aren’t heard? To never cite papers that have absolutely any errors? I think it’s a tough situation and not productive to say OP isn’t behaving morally.

Your example of looking at a problem from more than one angle made me think of the problem of finding the Hubble constant that describes the rate of expansion of the universe. There are two recent methods which have different estimates for this rate of expansion.

PBS Space time has an excellent video on the topic: https://www.youtube.com/watch?v=72cM_E6bsOs

I think you might be mistaken. The study of Nature/Science papers was "Evaluating replicability of social science experiments published in Nature and Science between 2010 and 2015"

I assume you're thinking of N-Rays and Blondlot, but there is another phenomenon that more or less fits your description.

https://en.wikipedia.org/wiki/Cosmic_ray_visual_phenomena

It's interesting that according to the Wikipedia article it's not entirely certain whether the radiation is producing actual light or just the sensation of light.

Yea, evolution and climate change are pretty solid. But the projected models do no favor to climate science. Those are bogus. They've mostly been wrong in the past and will be mostly wrong again in the future. There are way too many variables and unknowns, and those accumulate the further out the model reaches.

And your evidence for that is what exactly? Since this replication problem is already known to appear in multiple disciplines, it's quite likely that the same misconduct is happening in other areas too. I think you're being a little too quick to hope that it doesn't affect those areas where you have a vested interest.

In the main people don't literally mean it that way. they're expressing belief int he validity of the scientific method, but the more they explaina nd justify the scientific method the more time consuming it is. When dealing with stupid people or obdurate trolls, the sincere adherent of the scientific method can be tricked into wasting a great deal of time by being peppered with questions that are foolish or posed in bad faith.

You're free to treat it as unquestionable, but the fact remains there is ample evidence of our scientific process being deeply broken with a really bad incentive structure if nothing else.

If you think there is no incorrect science at all in regards to evolution and climate change you're no better than the zealots of any religion.

While I agree that "most" researchers start out with good intentions, I'm afraid I've directly and indirectly witnessed so many examples of fraud, data manipulation, wilful misrepresentation and outright incompetence, that I think we need some proper checks and balances put in place.

When people deliberately fake lab data to further their career, and that fake data is used to perform clinical trials on actual people, that's not just fraudulent, it's morally destitute. Yet this has happened.

People deliberately use improper statistics all the time to make their data "significant". It's outright fraud.

I've seen people doing sloppy work in the lab, and when questioning them, was told "no one cares so long as it's publishable". Coming from industry, where quality, accuracy and precision are paramount, I found the attitude shocking and repugnant. People should take pride and care in their work. If they can't do that, they shouldn't be working in the field.

PIs don't care so long as things are publishable. They live in wilful ignorance. Unless they are forced to investigate, it's easiest not to ask any questions and get unpleasant answers back. Many of them would be shocked if they saw the quality of work done by their underlings, but they live in an office and rarely get directly involved.

I've since gone back to industry. Academia is fundamentally broken.

When you say "double-checking" won't solve anything, I'd like to propose a different way of thinking about this:

* lab notebooks are supposed to be kept as a permanent record, checked and signed off. This rarely happens. It should be the responsibility of a manager to check and sign off every page, and question any changes or discrepancies.

* lab work needs independent validation, and lab workers should be able to prove their competence to perform tasks accurately and reproducibly; in industry labs do things like sending samples to reference labs, and receiving unknown samples to test, and these are used to calculate any deviation from the real value both between the reference lab and others in the same industry. They get ranked based upon their real-world performance.

* random external audits to check everything, record keeping, facilities, materials, data, working practices, with penalties for noncompliance.

Now, academic research is not the same as industry, but the point I'm making here is that what's largely missing here is oversight. By and large, there isn't any. But putting it in place would fix most of the problems, because most of the problems only exist because they are permitted to flourish in the absence of oversight. That's a failure of management in academia, globally. PIs aren't good managers. PIs see management in terms of academic prestige, and expanding their research group empires, but they are incompetent at it. They have zero training, little desire to do it, and it could be made a separate position in a department. Stop PIs managing, let them focus on science, and have a professional do it. And have compliance with oversight and work quality part of staff performance metrics, above publication quantity.

The trouble is, industrial researchers usually don't publish negative results or failures to reproduce. So it takes a long time to correct the published scientific record even if privately some people know it's wrong.

This is like the xkcd test for weird science: Is some big boring company making billions with it? If so (quantum physics) then it’s legit. If not (healing crystals, orgone energy, essential oils...) it probably doesn’t work.

I don't really see why "being shunned" or "being a visionary" has anything to do with this, to be honest. If you set up a simple rule: "the results have to be reproducable", then surely it shouldn't matter whether or not the theory is considered "crackpot" or "brilliant"?

The "Jury Duty" could easily be implemented in the existing grant structure - condition some new research grant on also doing an audit of some previous grant in your field (and fund it as part of the grant).