There should be Nutrition Facts but for scientific trials. Independent agency just publishing quality assessments of the trial.
This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.
There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).
Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).
There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.
There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.
And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.
Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.
Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.
There's the Cochrane Collaboration. They don't tick off every item on your list but it's fairly close to what you're asking for. It's mentioned in the article as they do a lot of meta-studies. Unfortunately they only started trying to spot fraudulent RCTs in 2021. Also in recent times some people don't like them, because they did a big review of mask studies and found there was no reliable evidence that masks worked against COVID.
Cochrane (formerly known as the Cochrane Collaboration) is a British international charitable organisation formed to organise medical research findings to facilitate evidence-based choices about health interventions involving health professionals, patients and policy makers.[4][5] It includes 53 review groups that are based at research institutions worldwide. Cochrane has approximately 30,000 volunteer experts from around the world.[6]
The group conducts systematic reviews of health-care interventions and diagnostic tests and publishes them in the Cochrane Library.[7][4]
> Also in recent times some people don't like them, because they did a big review of mask studies and found there was no reliable evidence that masks worked against COVID.
Oh.
I quickly found this:
"The new scientific review on masks and Covid isn’t what you think" Kelsey Piper
"Many commentators have claimed that a recently-updated Cochrane Review shows that 'masks don't work', which is an inaccurate and misleading interpretation."
The success of the Nutrition Facts labeling does not get enough publicity.
Rather than outlawing certain ingredients, or creating some kind of health score which a product must be above, Nutrition Facts is a way for suppliers to attest to information about a product in way that is legally binding. If it isn't accurate the penalties are steep.
Consumers then have the information they need to vote with their wallets. Markets cannot function properly without symmetric information, and Nutrition Facts essentially creates a functioning market where one did not exist previously.
Any effort to regulate what's in food would probably be better spent expanding what must be in the Nutrition Facts label. I guess it's nice that we are finally getting around to banning artificial trans fats, but anyone who can read has been able to keep those out of their diet for years. The same can be said about the next bad ingredient, and the one after that.
Interestingly, there is some gaming of the main number everyone looks at on the nutrition facts chart: calories per serving.
All snacks aim to fall at or below a certain number the FDA (or some other agency) put out as being considered a snack. Planet Money did an episode on different M&M varieties having different total weights to account for their different calorie counts. So you get fewer by weight peanut butter M&Ms because they’re more calorie rich
> Ultimately, a lingering question is — as with paper mills — why so many suspect RCTs are being produced in the first place. Mol, from his experiences investigating the Egyptian studies, blames lack of oversight and superficial assessments that promote academics on the basis of their number of publications, as well as the lack of stringent checks from institutions and journals on bad practices.
A substantial part of what's happening here is that first-world countries with generally good cultures of research integrity are basing medical policy on studies done in countries where the system encourages researchers to cheat. British and US authorities can't put Egyptian or Chinese researchers in prison, can they?
Imagine same approach in politics. If some politics put lies in their speeches to manipulate with people, we should call them liers and put them in prison. But somehow it doesn't happen. Looks like society prefer conformity over responsibility.
Why would the statisticians be anonymous? I'm aware of at least a couple cases in which an independent set of statisticians were provided the data from a clinical trial specifically for a re-analysis. In one case, they showed some pretty concerning inconsistencies and the other confirmed no effect on the primary analysis, but suggested some sub-populations that might have shown an effect if a future study was properly powered. That follow-up clinical trial was just published showing pretty remarkable effect in the sub-population. I don't think there's reason to believe either independent analysis was anything other than independent.
There is already a mechanism for companies to submit questions to the FDA prior to clinical trial initiation. I'm not in these conversations, but I know the type of questions can be things like: would you accept this endpoint as a proxy for this indication, would you be satisfied with the effect size we expect, and are there other safety concerns you would expect us to evaluate other than those in our current plan. I assume EMA and other regulatory bodies have a similar process, but I'm not positive.
Disclaimer: I work in pharma, but pre-clinically. I am not involved in these clinical or regulatory issues.
Yes, I have said this as "The UX of [medical] study papers is terrible". Some people do not agree, they think that it should not be made easier to understand, that non-experts cannot really understand medical studies, so they should not be more approachable. I think that's dead wrong.
Im struggling to see the difference between this and the current FDA process, and think it is 90% the same.
Drugs have "prescribing information", referred to in industry as "labeling", which follows a consistent format containing safety, trial results, side effects, and mechanism of action.[1] I recommend people read them for drugs they take.
This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.
Third party analysis is the main difference here. In the current state, firms run the analysis for FDA review using standard practices, and must explain and get approval for any unconventional methods
>There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).
Labeling contains many relevant numbers. Trial sizes, how many per arm, what was measured, and and final results. Maybe there could be some squishy qualitative summary, but that seems more risky. I would rather know that 1 out of 20 patients died than it got a "2" on the safety scale.
>Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).
Labeling contains drug specific information.
There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.
Drug labeling requires pre-application and and a standard 12 month review period by the FDA prior approval
There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.
The FDA provides advice on structuring trials and acceptable design, size, power, endpoints. Firms do this by scheduling calls with FDA staticians and experts. [2]
And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.
Maybe there is a difference here. I'm not sure what you mean by punishment? In the current system, The FDA can use the label as "punishment". The FDA may require addition "black box warnings" for drugs that are found to have serious side effects (e.g. high chance of death). They can also pull the label entirely, meaning the drug can not be sold.
Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.
This is basically how it works for medical device labeling in the EU. There are several "notified bodies" [3] which are private agencies to review the safety and efficacy. The firm then takes their mark of approval to the government agency.
They say "medicine". I would say science in general, perhaps we could generalise even further to "any human activity is full of unethical people trying to exploit it". But 25%!? That suggest there is a big problem with how we "do science". Unfortunately I have no solution to the problem. Publishing everything (including raw data) for every research would probably help somewhat, but only teams repeating experiments/trials would ensure it.
I wonder if we suddenly took 10% of all money spent on science (let's say in medicine) and instead of novel research we used it to redo randomly chosen previous research. Would we loose or gain in terms of new cures? And if we gained, what if we spent 15%, or 25%? That's a great idea for a scientific study to find a point of diminishing returns on "research verification".
Would someone please write a research grant request for this?
Of course there are big problems with how we do science. Much of it is garbage. Imagine most software was written by junior engineers, without any code review or input from seniors. That is today's science.
Most scientific legwork is done by absolute beginners, i.e., graduate students. They often lack a support structure to focus on what they have learned so far. Most of the world is not Oxbridge, MIT, Stanford.
Where are the beginners' supervisors, you might ask? Chasing the latest trend to secure funding. Pondering how their line of research can be formulated as buzzword-du-jour markov chain. Ass kissing the dean to get department funding.
Having worked in research for 15 years, I am certain about two things: (1) The scientific method yields better results than doing things freehand. (2) Randomly axing 50% of academia would improve the situation.
That 10% investment would have a huge payoff too because it would shut down avenues that were only opened due to p-hacking long before they'd had a chance to seek further investment.
And as a bonus, read books by controversial figures who talk about these challenges through their own published studies and decided it isn't worth fighting.
What would be cool is to see "trust indicators" as part of a study's metadata as it is reviewed through time and continually shared by others. Could be a "study health score" or a checklist showing the study isn't biased heavily by sponsors, methods, or misleading language.
Especially given that medicine is becoming more personal with the advancements of AI and accessibility of tests, someone should be able to understand the health of a study at-a-glance reviewed by peer reviews rather than disclosed only by the authors.
A primary trust indicator of any research result is the reputation of the journal publishing the paper. Better journals demand more, not just in terms of the impact/insight of the results, but also the rigor of the experimental methodology used and how well the data was curated and the confounding variables identified and isolated.
And of course, the criteria for observational-based research differ a lot from mechanism-based, especially since the former can't control for nearly as many variables. The same goes for simulations or interpretive modeling, where experimentation is minimal.
Research studies differ quite a lot in how precisely the mechanism of action is identified, isolated, tested, and results interpreted. IMHO, blackening the trustworthiness of all of science is unhelpful, especially when some models and methods are surely more trustworthy and replicable than others. Better to identify and catalog specific sources of error (or imprecision) in order to remedy them than to just toss the baby.
If you read the article, the headline is wildly editorialized.
Whatever. Par for the course in 2023, right?
But this is Nature, a paragon of scientific literature, fueling the distrust of medicine.
There are great reasons to be skeptical of all trials and strengthen peer review and transparency, but this kind of headline is editorial malpractice, in my opinion.
No, not really. The headline says "plagued with", the subhead says "in some fields, at least one-quarter of clinical trials might be problematic or even entirely made up", and the article substantiates that. One quarter of all trials is more than adequate to plague the whole enterprise, and there isn't a problem with this headline.
Agreed that the headline makes it sound like a strong majority, when the article isn't nearly as strong on that.
Still, "Carlisle rejected every zombie trial, but by now, almost three years later, most have been published in other journals — sometimes with different data to those submitted with the manuscript he had seen. He is writing to journal editors to alert them, but expects that little will be done." is concerning. I'm almost afraid to know what the list of rejected papers covers.
We should be distrustful of medicine, of all science in general. Ignoring flawed methodologies or inconclusive results just means it's no longer science, it's ideology. I'm a physicist though so perhaps I could be overly jaded about science and peer review compared to most scientists
You simply can't apply the rules of publication in physics to medical biology research. Even highly quantitative biology is noticeably different in terms of standards of proof and quality of models.
It has been since the introduction of peer review that science has taken a nose-dive. Peer-review is just such a terrible idea. All it does is bring politics and back-stabbing into science.
it's 2023, we have the means to cheaply record and store audio and video evidence for basically any medical experiment. we can record every patient reaction and opinion without relying on the reasearchers' hearsay. we also have the means to store and distribute all the binary/textual raw data gathered throughout the experiments.
maybe as an intermediate step we could make available all the recordings to the peer reviewers and only offer the raw experimental data bundled in the paper publicly? maybe in the future we can have 1TB studies without breaking a sweat? maybe all the money we give to publishers can be spent on servers to archive all the primary data so at least we aren't simply filling the pockets of MBAs?
> maybe as an intermediate step we could make available all the recordings to the peer reviewers
The issue is clearly not the amount of data available to peer reviewers considering it's already easy to detect major flaws in a quarter of published peer reviewed research. The issue is that peer reviewers do a shoddy job which should surprise no one having ever published peer reviewed research.
And to be fair why should they do better? It's generally unpaid, it's poorly paid when it is paid and it's not particularly well considered.
If we discover that we can´t trust researchers then what else are we left with? Doctor-patient privacy works if the doctor is truthful in their reporting
It's generally permitted to share de-identified patient data. As long as you're not sharing patients names, medical record numbers, birthdays, and a couple of other fields, you should be fine.
Maybe we could do a double blind (including scientists) study where everyone waived their rights & are recorded then in another "typical" conditions study do none of that and compare the two and see which one seems to have the best, most accurate results.
>it's 2023, we have the means to cheaply record and store audio and video evidence for basically any medical experiment. we can record every patient reaction and opinion without relying on the reasearchers' hearsay. we also have the means to store and distribute all the binary/textual raw data gathered throughout the experiments
This is a great approach IMO. Additionally skeptics (for example anti-vaxxers), should be physically present at the trials.
This is a problem everywhere where the raw data isn't released (suitably anonymized).
In cognitive science, psychology, even computer science / ai / ml, business.
And the problem with rejected papers getting in somewhere else while being total garbage is pervasive. I've rejected a lot of papers because they were mathematically or statistically bogus only to see them get published elsewhere where reviewers were not so careful (a few times in Nature and Science).
We need an open science movement where you must release everything with your paper. The full pipeline to reconstruct every single result from the raw data. No hiding data. No hiding fmri scans. No "our code only runs on our machine". Etc.
I'm empathetic to what you're suggesting — I've published on open science and meta-science specifically, and think open data should be the default norm. The problem with clinical research, though, is that it starts running into conflicting considerations about patient and participant privacy. Even when people aren't patients per se, the focus often involves sensitive information.
You can just say "anonymize it" but that turns out to be more difficult than it seems initially, especially with many questions of interest.
Also, there's often too many opportunities to do science that is of real public benefit that comes with privacy expectations attached for all kinds of reasons. Cases where there is legitimate consented access but an expectation of privacy without data sharing.
People have tried to solve this problem in different ways (for example, methods where someone can analyze data without having access to it directly) and maybe those solutions will lead to a good resolution. But they often have problems of their own (overhead costs associated with providing anonymized remote data analysis), and don't solve all problems (guarantees of absolutely restricted access to personal data).
I think value-based care is the only real incentive on this. Otherwise, there is simply no reason for anyone to care enough. Even the insurers, they found a way to make money by making sure their premiums factor in these things. In the expense of the patient. As long as the drug doesn't kill people, who cares if it works if I make money off it as a pharma? Unfortunately, value-based care can only be pushed top-down. Patients are not in a position of power against pharma companies on this matter.
I now know of a few more people who have completely lost both of their hearing after taking just one Wellburtin-class pill.
At the time, no mention was made in the pill’s warning pamphlet.
It is still difficult to secure a class-action suit in America.
Meanwhile, such quality of life would plummet into a silent world, even if one knew American Sign Language fluently beforehand, that tidbit can go against the victim in court.
There unfortunately are too many perverse incentives that encourage fraudulent studies. Everyone should take findings "established in the literature" with a grain of salt. We should also incorporate more intuition and first-principles reasoning, i.e. obesity is a harmful state for humans even if 100 RCTs proved otherwise.
Readit News
This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.
There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).
Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).
There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.
There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.
And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.
Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.
Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.
Cochrane (formerly known as the Cochrane Collaboration) is a British international charitable organisation formed to organise medical research findings to facilitate evidence-based choices about health interventions involving health professionals, patients and policy makers.[4][5] It includes 53 review groups that are based at research institutions worldwide. Cochrane has approximately 30,000 volunteer experts from around the world.[6]
The group conducts systematic reviews of health-care interventions and diagnostic tests and publishes them in the Cochrane Library.[7][4]
https://en.wikipedia.org/wiki/Cochrane_(organisation)
Oh.
I quickly found this:
"The new scientific review on masks and Covid isn’t what you think" Kelsey Piper
https://www.vox.com/future-perfect/2023/2/22/23609499/masks-...
Based on the criticisms, I expect Cochrane will revisit this topic.
Progress isn't a straight line.
No, that was a misinterpretation of the review in the Covid-skeptic sphere. Cochrane have had to issue a statement to clarify : https://www.cochrane.org/news/statement-physical-interventio...
tl;dr : half of the people given masks in these studies didn't wear them consistently or at all, dragging efficacy results down.
https://www.cochrane.org/news/statement-physical-interventio...
Rather than outlawing certain ingredients, or creating some kind of health score which a product must be above, Nutrition Facts is a way for suppliers to attest to information about a product in way that is legally binding. If it isn't accurate the penalties are steep.
Consumers then have the information they need to vote with their wallets. Markets cannot function properly without symmetric information, and Nutrition Facts essentially creates a functioning market where one did not exist previously.
Any effort to regulate what's in food would probably be better spent expanding what must be in the Nutrition Facts label. I guess it's nice that we are finally getting around to banning artificial trans fats, but anyone who can read has been able to keep those out of their diet for years. The same can be said about the next bad ingredient, and the one after that.
All snacks aim to fall at or below a certain number the FDA (or some other agency) put out as being considered a snack. Planet Money did an episode on different M&M varieties having different total weights to account for their different calorie counts. So you get fewer by weight peanut butter M&Ms because they’re more calorie rich
No, there should be prison time for scientists who conduct unethical trials or publish fake results.
The public (and policy makers) place such immense trust in those people and what they publish that nothing less is even remotely adequate.
When someone puts arsenic in food, they go to prison – labeling the food with "contains arsenic" doesn't cut it.
Do this and watch science magically fix itself.
> Ultimately, a lingering question is — as with paper mills — why so many suspect RCTs are being produced in the first place. Mol, from his experiences investigating the Egyptian studies, blames lack of oversight and superficial assessments that promote academics on the basis of their number of publications, as well as the lack of stringent checks from institutions and journals on bad practices.
A substantial part of what's happening here is that first-world countries with generally good cultures of research integrity are basing medical policy on studies done in countries where the system encourages researchers to cheat. British and US authorities can't put Egyptian or Chinese researchers in prison, can they?
Imagine same approach in politics. If some politics put lies in their speeches to manipulate with people, we should call them liers and put them in prison. But somehow it doesn't happen. Looks like society prefer conformity over responsibility.
There is already a mechanism for companies to submit questions to the FDA prior to clinical trial initiation. I'm not in these conversations, but I know the type of questions can be things like: would you accept this endpoint as a proxy for this indication, would you be satisfied with the effect size we expect, and are there other safety concerns you would expect us to evaluate other than those in our current plan. I assume EMA and other regulatory bodies have a similar process, but I'm not positive.
Disclaimer: I work in pharma, but pre-clinically. I am not involved in these clinical or regulatory issues.
Deleted Comment
Drugs have "prescribing information", referred to in industry as "labeling", which follows a consistent format containing safety, trial results, side effects, and mechanism of action.[1] I recommend people read them for drugs they take.
This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.
Third party analysis is the main difference here. In the current state, firms run the analysis for FDA review using standard practices, and must explain and get approval for any unconventional methods
>There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).
Labeling contains many relevant numbers. Trial sizes, how many per arm, what was measured, and and final results. Maybe there could be some squishy qualitative summary, but that seems more risky. I would rather know that 1 out of 20 patients died than it got a "2" on the safety scale.
>Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).
Labeling contains drug specific information.
There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.
Drug labeling requires pre-application and and a standard 12 month review period by the FDA prior approval
There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.
The FDA provides advice on structuring trials and acceptable design, size, power, endpoints. Firms do this by scheduling calls with FDA staticians and experts. [2]
And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.
Maybe there is a difference here. I'm not sure what you mean by punishment? In the current system, The FDA can use the label as "punishment". The FDA may require addition "black box warnings" for drugs that are found to have serious side effects (e.g. high chance of death). They can also pull the label entirely, meaning the drug can not be sold.
Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.
This is basically how it works for medical device labeling in the EU. There are several "notified bodies" [3] which are private agencies to review the safety and efficacy. The firm then takes their mark of approval to the government agency.
https://www.accessdata.fda.gov/drugsatfda_docs/label/2014/20...https://www.fda.gov/media/72253/downloadhttps://climedo.de/en/blog/list-of-mdr-certified-notified-bo...
Generally, reading a paper is much less risky to a person's health than taking a drug so the differences in review process add up meaningfully.
I wonder if we suddenly took 10% of all money spent on science (let's say in medicine) and instead of novel research we used it to redo randomly chosen previous research. Would we loose or gain in terms of new cures? And if we gained, what if we spent 15%, or 25%? That's a great idea for a scientific study to find a point of diminishing returns on "research verification".
Would someone please write a research grant request for this?
Most scientific legwork is done by absolute beginners, i.e., graduate students. They often lack a support structure to focus on what they have learned so far. Most of the world is not Oxbridge, MIT, Stanford.
Where are the beginners' supervisors, you might ask? Chasing the latest trend to secure funding. Pondering how their line of research can be formulated as buzzword-du-jour markov chain. Ass kissing the dean to get department funding.
Having worked in research for 15 years, I am certain about two things: (1) The scientific method yields better results than doing things freehand. (2) Randomly axing 50% of academia would improve the situation.
https://peterattiamd.com/ns001/
https://biolayne.com/reps/how-to-read-research-a-biolayne-gu...
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7392212/
And as a bonus, read books by controversial figures who talk about these challenges through their own published studies and decided it isn't worth fighting.
What would be cool is to see "trust indicators" as part of a study's metadata as it is reviewed through time and continually shared by others. Could be a "study health score" or a checklist showing the study isn't biased heavily by sponsors, methods, or misleading language.
Especially given that medicine is becoming more personal with the advancements of AI and accessibility of tests, someone should be able to understand the health of a study at-a-glance reviewed by peer reviews rather than disclosed only by the authors.
And of course, the criteria for observational-based research differ a lot from mechanism-based, especially since the former can't control for nearly as many variables. The same goes for simulations or interpretive modeling, where experimentation is minimal.
Research studies differ quite a lot in how precisely the mechanism of action is identified, isolated, tested, and results interpreted. IMHO, blackening the trustworthiness of all of science is unhelpful, especially when some models and methods are surely more trustworthy and replicable than others. Better to identify and catalog specific sources of error (or imprecision) in order to remedy them than to just toss the baby.
Whatever. Par for the course in 2023, right?
But this is Nature, a paragon of scientific literature, fueling the distrust of medicine.
There are great reasons to be skeptical of all trials and strengthen peer review and transparency, but this kind of headline is editorial malpractice, in my opinion.
Still, "Carlisle rejected every zombie trial, but by now, almost three years later, most have been published in other journals — sometimes with different data to those submitted with the manuscript he had seen. He is writing to journal editors to alert them, but expects that little will be done." is concerning. I'm almost afraid to know what the list of rejected papers covers.
Dead Comment
maybe as an intermediate step we could make available all the recordings to the peer reviewers and only offer the raw experimental data bundled in the paper publicly? maybe in the future we can have 1TB studies without breaking a sweat? maybe all the money we give to publishers can be spent on servers to archive all the primary data so at least we aren't simply filling the pockets of MBAs?
The issue is clearly not the amount of data available to peer reviewers considering it's already easy to detect major flaws in a quarter of published peer reviewed research. The issue is that peer reviewers do a shoddy job which should surprise no one having ever published peer reviewed research.
And to be fair why should they do better? It's generally unpaid, it's poorly paid when it is paid and it's not particularly well considered.
Or do subjects need to wave all their doctor-patient privacy rights before joining any trial?
TL;DR: IRBs are a mess, hyper-individualized, and the problem ain't getting any better any time soon.
This is a great approach IMO. Additionally skeptics (for example anti-vaxxers), should be physically present at the trials.
for example, if you have 50 partecipants but only provide the multimedia evidence for 20 of them your study should be thrown out the window
In cognitive science, psychology, even computer science / ai / ml, business.
And the problem with rejected papers getting in somewhere else while being total garbage is pervasive. I've rejected a lot of papers because they were mathematically or statistically bogus only to see them get published elsewhere where reviewers were not so careful (a few times in Nature and Science).
We need an open science movement where you must release everything with your paper. The full pipeline to reconstruct every single result from the raw data. No hiding data. No hiding fmri scans. No "our code only runs on our machine". Etc.
You can just say "anonymize it" but that turns out to be more difficult than it seems initially, especially with many questions of interest.
Also, there's often too many opportunities to do science that is of real public benefit that comes with privacy expectations attached for all kinds of reasons. Cases where there is legitimate consented access but an expectation of privacy without data sharing.
People have tried to solve this problem in different ways (for example, methods where someone can analyze data without having access to it directly) and maybe those solutions will lead to a good resolution. But they often have problems of their own (overhead costs associated with providing anonymized remote data analysis), and don't solve all problems (guarantees of absolutely restricted access to personal data).
Whatever the costs and challenges are, they are not nearly as high as maintaining the status quo.
At the time, no mention was made in the pill’s warning pamphlet.
It is still difficult to secure a class-action suit in America.
Meanwhile, such quality of life would plummet into a silent world, even if one knew American Sign Language fluently beforehand, that tidbit can go against the victim in court.