The researchers in this paper use an astonishingly biased "fake paper detector", requiring only two conditions to be met for any paper to be considered "fake":
1. Use a non-institutional email address, or have a hospital affiliation,
2. Have no international co-authors.
And they acknowledge 86% sensitivity and 44% specificity. It's a coin-toss which biases massively against research from outside the US and Western Europe.
No. They use 400 known fakes and 400 matched (presumed) non-fakes to estimate the sensitivity and specificity of their indicator, then apply that indicator to the full universe, then employ the estimated sensitivity and specificity to the obtained measurement to estimate the approximate actual rate of false papers.
If you know the true prevalence of a disease in a population, and the sensitivity and specificity of your test, you can predict how many positive measurements you obtain. Vice versa, from the (flawed raw) measurement, given sensitivity and specificity, you can estimate the true prevalence.
Furthermore, they’re explicitly saying that “red flagging” by their simple indicator doesn’t mean that the paper is fake, but that it merits higher scrutiny.
ETA: I mean, it could still all be bullshit (by virtue of some bias or so), but you’ll need to argue a bit harder to establish that.
ETA2: Actually, not sure that’s what they’ve done. They might have just reported the raw (very bad) measurement (that they call “potential red flagged fake paper”), without doing the obvious next step outlined above, and without applying any confidence intervals. So, it might actually be a pretty crap paper (though possibly technically correct) coupled with some mediocre reporting layered on top. Isn’t basic statistics taught anymore?
I've worked on research estimating prevalence from imperfect tests, and something that concerns me about this study is that they aren't showing the error bars for their estimates. Typically, you would report a confidence interval for prevalence rather than just a point estimate, and the confidence intervals can often be fairly wide. There's two sources of uncertainty here, the assumed probabilistic nature of the diagnostic test, and uncertainty in our estimates of the sensitivity and specificity.
I think this paper by Peter J Diggle [0], gives a solid methodology. Instead of treating sensitivity and specificity as fixed values using sample estimates, you can model them as each having a beta distribution. In this case these beta distributions can be found using a Bayesian treatment of Bernoulli trials.
> Furthermore, they’re explicitly saying that “red flagging” by their simple indicator doesn’t mean that the paper is fake, but that it merits higher scrutiny.
Then they and science should change their sensationalist headline. It's ironic that a paper about fakeness of something uses a borderline misleading title.
You can’t directly calculate both sensitivity and specificity using equal numbers of positives and negatives groups unless the actual population has that ratio.
A completely random test given equal populations results in 50% accuracy and 50% specificity. Things don’t look nearly as good if only 1% of the actual population has the condition.
So, in other words, the signal they get from it is around 70% of the noise, but it's ok because you can indeed do that with good enough statistics?
They better have a flawless methodology, because any tiny problem is enough to ruin their analysis. And well, just flagging almost any paper not from the EU or US as fraud doesn't usually come together with a flawless methodology.
Paper mills are a $3-4 billion dollar industry that is growing rapidly. That money isn't coming from nowhere. There are a lot of fake papers, and the fake paper industry is growing steadily.
So then the question becomes "where are those fake papers being published, and by whom."
You can converge on answers to those questions in a lot of ways. The fake paper detection method is suggested as one tool to aid journals tackle fraud.
If you don't think the conditions are valid, well, ok. But why not? How would you improve on the validation methodology? Obviously having more known fakes would be nice.
Saying the article is "bigoted nonsense" doesn't make a lot of sense without more information (to be fair, I might be lacking crucial context). Are the authors known bigots with history of pushing bigotry? What I read seemed to be a sincere attempt to improve scientific publication practices by identifying the scope and scale of the fraud problem, while also developing means to address it. That doesn't strike me as bigoted nonsense.
That said, the headline of the article is pretty click-baity, and shame on science's editors for that.
> The researchers in this paper use an astonishingly biased "fake paper detector"
I havent looked at the details here, but if you make a prediction model and if that prediction model is robust enough to explain with great accuracy something with 2 or 3 variables, it's not going to be "biased", it's just going to be robust and right more often than not using only these few variables (as long as the training data was broad enough).
The metrics used in this paper are... deeply flawed, to the point that the authors admit that they label nearly half of known good papers in a curated sample as "fake" -- and particularly likely to generate false positives for researchers whose institutions don't, say, run their own email systems (as is common in large chunks of the world). Here's a rundown of the flaws from an epidemiologist with a sideline in scientific communication:
You know what's funny? Even if the numbers are hot garbage, they proved the point about how easy it is to publish fake science papers, since it got published.
Kinda similar to those researchers years back who proved how easy it was to go into certain social science journals as long as you copied their ideology.
Well, there is a difference between "fake science" and "tried to do correct science but ending being wrong". If the second is "fake science", then basically all that Newton has ever produced is "fake science".
Ironically, this study has generated a lot of "fake news" on the field of social science. The conclusions of this study were widely spread mainly by people for ideological reason. When we look at the study in question, it's clear the conclusions are quite different than what the rumors say. For example, the same researchers tried such hoax before the ones they mention in their study, except that these hoaxes failed to be published, and they "forgot" to mention it. They did not have any control group, neither as "correct article" or "article defending the opposite ideology" (so, how can we conclude that the reason these bad articles were published were because of ideology if you don't know how many articles are published without being critically reviewed). They also count as valid a lot of journals that are pay-to-publish and not seriously used in the field. One of the author, ironically, ended up supporting platforms publishing conspiracy theories (and he was even banned from Twitter) (not that the study should be judged based on that, but it's a funny anecdote: the author who, according to some, had the courage to defend real science against bad woke ideology, who ends up demonstrating that he never cared about real science and is driven by ideology not science)
Unfortunately, it's an open secret that fake or low-effort almost useless papers are very common in every area of scientific research.
Typically, it doesn't affect people working in that specific area - they develop/have a sixth sense to detect bullshit papers - it comes with experience but depends on several factors including the authors reputation, their institution (for the first screening), what journal/conference the paper was published in, authors other work, and sometimes things as simple as how much effort was put into the figures, polishing the text, etc. Some of these things are LLM proof, some of them are not - e.g. a senior professor I was talking to, who's been getting like 50-100 emails a week from non-english speaking countries (primarily India, China, Pakistan, and Bangladesh) mentioned that the quality of the text in the emails went up significantly almost overnight after ChatGPT was made open to public. It'll be interesting to see how things change in the next few months/years.
Right- academic papers are written for academicians who don't have any issues separating good papers and journals from bad. The fact that many journals have set themselves up as or allowed themselves to become part of the tenure and promotion metrics game, is more of an issue with tenure and promotion. If the requirement for simple metrics dissapeared, the fake papers would go away on their own. In any event, it's not really a problem for researchers.
Yup, that’s sums up the incentive for publishing so many papers and get citations.
Some professor put it in a nice way - the current system motivates us to think of research in terms of LPUs - least publishing units. No matter how established your lab is, you’d try to publish as soon as possible, leading to a lot of papers with not a lot of contribution. If tenure committees and all other systems that gauge academicians require people to say present their only top 3 or 5 seminal papers, then people would try to put their best work out there without the constant pressure of always publishing - win win for everyone. Unfortunately, the ones with the power to make these changes are the ones gaining the most in the current system so it’s unlikely to happen.
I mean it is a problem for researchers though. The blatantly fake paper mill ones (which seem to be the topic of this article anyway) aren't, but scientific fraud or even just minor misconduct from people that know how to mask it can waste a great deal of grant funding and scientist time to figure out.
Like look how many times that 2006 Nature paper on amyloid beta in Alzheimer's was cited, turns out some of the images were completely fabricated.
My favourite example is the grievance studies (https://en.wikipedia.org/wiki/Grievance_studies_affair). The authors published, among others, portions of directly copied Mein Kampf. In one "study" submitted, they claimed to have observed thousands of hours of dogs having sex in parks, to observe the "patriarchal" linked to "rape culture". The entire thing is a horrible indictment on the level of scrutiny undertaken in the various activist and social "science" oriented journals.
I have no sympathy for social science journals, but when you look at the details of this study, it's way less obvious that the rumor says it is. On the tens of articles they have proposed, the majority was rejected. The article "copied from Mein Kampf" was taking sentence from the book but changing words to create sentences that were scientifically correct (for example: "this social class is bad and we should avoid it" into "stress is bad and we should avoid it"), which means that the article content in itself had no reason to be refused.
It's very ironical that this study that was all about "bad science" since then created a totally whimsical rumor on the real situation.
Right. If one can develop a sixth sense about bullshit papers then so too can an LLM. If you have a bullshit paper you will need to pipe it through an LLM to debullshittify it so that reviewers cannot tell. The reviewers themselves may need LLM to fight the rising tide of passable bullshit papers. None of that seems productive to me, just throwing gasoline on the dumpster fire of phacking, credential inflation, publish or perish, etc..
Love that! I never understood why so many of us would use their affiliation's email address in print if they know that they would only be there for another 2--3 years.
Reading the paper it seems like a pretty accurate description. The paper just calls it a "private email" instead of a "non-institutional email". For example (@@@ emphasis is mine):
> To identify indicators able to red-flagged fake publications (RFPs), we sent
questionnaires to authors. Based on author responses, three indicators were identified: @@@“author's private email”@@@, “international co-author” and “hospital affiliation”.
> For Studies 1 to 6 we identified two easy-to-detect indicators, where a publication was labelled as RFP: @@@if an author used a private email@@@ and had no international partner.
> Then we combined the two best indicators (@@@“author's private email”@@@ and “hospital affiliation”) to form a classification (tallying) rule: “If both indicators are present, classify as a potential fake, otherwise not” (the “AND” rule) (Katsikopoulos et al., 2020).
Fun bonus there with the 2020 book citation for the concept of an AND gate in a classifier.
The rule I omitted from the quote was "hospital affiliation". In the paper, they try a variety of combinations of rules, including some where failing any one rule classifies the paper as fake.
The meat of my complaint remains even when they're intersecting with other rules. We should not be incentivizing people to use emails that predictably go dead in O(years). It is quite a common annoyance to read a paper, want to contact the author, and not be able to because the email they listed is dead, requiring searching for where they currently work and trying to find their email at that new place, with mixed results.
Yes, a private email is predictive of a paper being fake, in the literal sense that P(fake|privateemail) > P(fake|institutionemail). I get weird looks at work for using my permanent email address because of it. And probably if we select on that as a way to discard papers, it will initially appear to work and then start to look like it's working even better because anyone trying to give permanent contact info will be forced to switch to be published/cited/taken-seriously. But that's a bad outcome. Also, if you systematize this rule, paper mills will just start using emails that appear institutional, because this is a simple rule to defeat.
> Such manuscripts threaten to corrupt the scientific literature, misleading readers and potentially distorting systematic reviews.
Is treating "the scientific literature" as a single thing perhaps a habit worth giving up?
As convenient as it would be to be able to just blindly trust something because of where it is published, that model hasn't shown itself to be especially robust in other cases (e.g. the news media).
Elsewhere, this is a red flag:
> I trust it because of which aggregator aggregated it
Should we really make an exception for science? I think that academia is a bit biased towards optimism about publisher-based root-of-trust models because scientific publishing is a relatively unweaponized space. Sure, shenanigans happen, but not at the same scale as elsewhere. The fakers are just trying to get another published paper, they're for the most part not trying to mislead. It's only fake news with a lowercase-f.
Sure, let's try to create a medium we can trust, but let's not get our hopes too high about it. That's energy better spent augmenting the ability of a reader or researcher to decide whether to trust a paper based on it's content or based on it having been endorsed or authored by somebody that they explicitly (or transitively) trust.
I disagreed with you until the last paragraph. Lots of things authentically just rely on a high degree of trust and I suspect trying to engineer human systems to be zero trust will make them deeply pathological.
But tempering our expectations while working to meaningfully improve on conditions? Aces, all for it.
I agree that zero trust is in most cases a problematic goal. It's really root-of-trust vs web-of-trust that I'm on about here.
If peer review is the product then the trust should be peer to peer. It feels like we're treating the publishers themselves as an authority, which I dislike.
One option is to provide a (perhaps less prestigious) avenue to publish non-novel or unsurprising findings. I suspect many people “fake” their results so all their effort isn’t in vain.
> STM hasn’t yet generated figures on accuracy or false-positive rates because the project is too new. But catching as many fakes as possible typically produces more false positives. Sabel’s tool correctly flagged nearly 90% of fraudulent or retracted papers in a test sample. However, it marked up to 44% of genuine papers as fake
> so results still need to be confirmed by skilled reviewers
So there is some human review involved. Which is presumably how they got to the headline figures of 34% of neuroscience papers and 24% of medicine papers are fake.
Still, flagging 44% of genuine papers as fake doesn't sound very useful. The process only about halves your workload compared to just checking all the papers. In any large-scale rollout they would have to set a way higher threshold, and hope they still catch a useful number of fraudulent papers when using a threshold that detects 10% or 1% of genuine papers as fake.
If they thing that people can do research in medicine without having legal access to the patients (AKA some kind of Hospital affiliation), they are clueless about medical research. They don't seem to understand how much hospitals, medical companies and academy are interwined now. Lots of relevant physicians are also teachers, direct a research team or are testers from new products.
They are also tagging all independent non affiliated researchers as fake. Do they know how many young people are doing science in the universities as temporary collabo-slaves without right to a nice personal mail?. Their detector would tag Einstein and Erdos as fake scientists by Pete's sake!. They just have a narrow vision tunnel about how the real research works
I am amazed at how well Alexandra Elbakyan has created and promoted sci-hub to fight these journal cash cows, and appalled at the way these journals have tried to block her. They now digitally watermark every journal downloaded at colleges etc, so they can ID the provenance of the journals = she must obfuscate this as best she can. The journals try to punish universities that leak papers to sci-hub
Give her a wave....
https://sci-hub.se/alexandra
The cash cows, AKA Elsevier et al, need to do more to stem the flow of BS. The problem is the proliferation of well crafted, but fake, papers has grown enormously over the past 25 years as the cows rely on free paper editors - who are swamped by this duty = time for paid scientists to winnow the chaff. Sadly the cows are a greedy lot.
Only way out is fully open. Back in the day when Nobel was born, the journals and authors circulated as near free resources, with authors mailing free copies on request, and now emailing them (often this is interdicted by the cows) and journal fees being modest - covering production costs.
Nobel would be (IMHO) royally pissed at the present state.
So I suggest the Nobel Committee introduce a policy that only openly published papers would be read and considered by the committee - This would put a tiger among the pigeons(Cows) and change things - say, after Jan 1 2024?
Readit News
1. Use a non-institutional email address, or have a hospital affiliation, 2. Have no international co-authors.
And they acknowledge 86% sensitivity and 44% specificity. It's a coin-toss which biases massively against research from outside the US and Western Europe.
This "paper" is bigoted nonsense.
https://fediscience.org/@ct_bergstrom/110357278154604907
If you know the true prevalence of a disease in a population, and the sensitivity and specificity of your test, you can predict how many positive measurements you obtain. Vice versa, from the (flawed raw) measurement, given sensitivity and specificity, you can estimate the true prevalence.
Furthermore, they’re explicitly saying that “red flagging” by their simple indicator doesn’t mean that the paper is fake, but that it merits higher scrutiny.
ETA: I mean, it could still all be bullshit (by virtue of some bias or so), but you’ll need to argue a bit harder to establish that.
ETA2: Actually, not sure that’s what they’ve done. They might have just reported the raw (very bad) measurement (that they call “potential red flagged fake paper”), without doing the obvious next step outlined above, and without applying any confidence intervals. So, it might actually be a pretty crap paper (though possibly technically correct) coupled with some mediocre reporting layered on top. Isn’t basic statistics taught anymore?
I think this paper by Peter J Diggle [0], gives a solid methodology. Instead of treating sensitivity and specificity as fixed values using sample estimates, you can model them as each having a beta distribution. In this case these beta distributions can be found using a Bayesian treatment of Bernoulli trials.
[0] https://www.hindawi.com/journals/eri/2011/608719/
Then they and science should change their sensationalist headline. It's ironic that a paper about fakeness of something uses a borderline misleading title.
A completely random test given equal populations results in 50% accuracy and 50% specificity. Things don’t look nearly as good if only 1% of the actual population has the condition.
They better have a flawless methodology, because any tiny problem is enough to ruin their analysis. And well, just flagging almost any paper not from the EU or US as fraud doesn't usually come together with a flawless methodology.
Paper mills are a $3-4 billion dollar industry that is growing rapidly. That money isn't coming from nowhere. There are a lot of fake papers, and the fake paper industry is growing steadily.
So then the question becomes "where are those fake papers being published, and by whom."
You can converge on answers to those questions in a lot of ways. The fake paper detection method is suggested as one tool to aid journals tackle fraud.
If you don't think the conditions are valid, well, ok. But why not? How would you improve on the validation methodology? Obviously having more known fakes would be nice.
Saying the article is "bigoted nonsense" doesn't make a lot of sense without more information (to be fair, I might be lacking crucial context). Are the authors known bigots with history of pushing bigotry? What I read seemed to be a sincere attempt to improve scientific publication practices by identifying the scope and scale of the fraud problem, while also developing means to address it. That doesn't strike me as bigoted nonsense.
That said, the headline of the article is pretty click-baity, and shame on science's editors for that.
I havent looked at the details here, but if you make a prediction model and if that prediction model is robust enough to explain with great accuracy something with 2 or 3 variables, it's not going to be "biased", it's just going to be robust and right more often than not using only these few variables (as long as the training data was broad enough).
https://fediscience.org/@ct_bergstrom/110357259338364341
Kinda similar to those researchers years back who proved how easy it was to go into certain social science journals as long as you copied their ideology.
For the social science journals bit, are you thinking of the "grievance studies affair": https://en.wikipedia.org/wiki/Grievance_studies_affair ?
Ironically, this study has generated a lot of "fake news" on the field of social science. The conclusions of this study were widely spread mainly by people for ideological reason. When we look at the study in question, it's clear the conclusions are quite different than what the rumors say. For example, the same researchers tried such hoax before the ones they mention in their study, except that these hoaxes failed to be published, and they "forgot" to mention it. They did not have any control group, neither as "correct article" or "article defending the opposite ideology" (so, how can we conclude that the reason these bad articles were published were because of ideology if you don't know how many articles are published without being critically reviewed). They also count as valid a lot of journals that are pay-to-publish and not seriously used in the field. One of the author, ironically, ended up supporting platforms publishing conspiracy theories (and he was even banned from Twitter) (not that the study should be judged based on that, but it's a funny anecdote: the author who, according to some, had the courage to defend real science against bad woke ideology, who ends up demonstrating that he never cared about real science and is driven by ideology not science)
Not by the definition of "fake" used in the article, as the data wouldn't be plagiarized or fabricated. It'd just be shitty data.
Typically, it doesn't affect people working in that specific area - they develop/have a sixth sense to detect bullshit papers - it comes with experience but depends on several factors including the authors reputation, their institution (for the first screening), what journal/conference the paper was published in, authors other work, and sometimes things as simple as how much effort was put into the figures, polishing the text, etc. Some of these things are LLM proof, some of them are not - e.g. a senior professor I was talking to, who's been getting like 50-100 emails a week from non-english speaking countries (primarily India, China, Pakistan, and Bangladesh) mentioned that the quality of the text in the emails went up significantly almost overnight after ChatGPT was made open to public. It'll be interesting to see how things change in the next few months/years.
Some professor put it in a nice way - the current system motivates us to think of research in terms of LPUs - least publishing units. No matter how established your lab is, you’d try to publish as soon as possible, leading to a lot of papers with not a lot of contribution. If tenure committees and all other systems that gauge academicians require people to say present their only top 3 or 5 seminal papers, then people would try to put their best work out there without the constant pressure of always publishing - win win for everyone. Unfortunately, the ones with the power to make these changes are the ones gaining the most in the current system so it’s unlikely to happen.
Like look how many times that 2006 Nature paper on amyloid beta in Alzheimer's was cited, turns out some of the images were completely fabricated.
It's very ironical that this study that was all about "bad science" since then created a totally whimsical rumor on the real situation.
Deleted Comment
Uh huh.
I didn't realize until today that all my papers are fake because I give contact information that won't go stale in 3 years, instead of my work email.
> To identify indicators able to red-flagged fake publications (RFPs), we sent questionnaires to authors. Based on author responses, three indicators were identified: @@@“author's private email”@@@, “international co-author” and “hospital affiliation”.
> For Studies 1 to 6 we identified two easy-to-detect indicators, where a publication was labelled as RFP: @@@if an author used a private email@@@ and had no international partner.
> Then we combined the two best indicators (@@@“author's private email”@@@ and “hospital affiliation”) to form a classification (tallying) rule: “If both indicators are present, classify as a potential fake, otherwise not” (the “AND” rule) (Katsikopoulos et al., 2020).
Fun bonus there with the 2020 book citation for the concept of an AND gate in a classifier.
I would allow just one valid paper with that inability.
The meat of my complaint remains even when they're intersecting with other rules. We should not be incentivizing people to use emails that predictably go dead in O(years). It is quite a common annoyance to read a paper, want to contact the author, and not be able to because the email they listed is dead, requiring searching for where they currently work and trying to find their email at that new place, with mixed results.
Yes, a private email is predictive of a paper being fake, in the literal sense that P(fake|privateemail) > P(fake|institutionemail). I get weird looks at work for using my permanent email address because of it. And probably if we select on that as a way to discard papers, it will initially appear to work and then start to look like it's working even better because anyone trying to give permanent contact info will be forced to switch to be published/cited/taken-seriously. But that's a bad outcome. Also, if you systematize this rule, paper mills will just start using emails that appear institutional, because this is a simple rule to defeat.
Is treating "the scientific literature" as a single thing perhaps a habit worth giving up?
As convenient as it would be to be able to just blindly trust something because of where it is published, that model hasn't shown itself to be especially robust in other cases (e.g. the news media).
Elsewhere, this is a red flag:
> I trust it because of which aggregator aggregated it
Should we really make an exception for science? I think that academia is a bit biased towards optimism about publisher-based root-of-trust models because scientific publishing is a relatively unweaponized space. Sure, shenanigans happen, but not at the same scale as elsewhere. The fakers are just trying to get another published paper, they're for the most part not trying to mislead. It's only fake news with a lowercase-f.
Sure, let's try to create a medium we can trust, but let's not get our hopes too high about it. That's energy better spent augmenting the ability of a reader or researcher to decide whether to trust a paper based on it's content or based on it having been endorsed or authored by somebody that they explicitly (or transitively) trust.
But tempering our expectations while working to meaningfully improve on conditions? Aces, all for it.
If peer review is the product then the trust should be peer to peer. It feels like we're treating the publishers themselves as an authority, which I dislike.
So there is some human review involved. Which is presumably how they got to the headline figures of 34% of neuroscience papers and 24% of medicine papers are fake.
Still, flagging 44% of genuine papers as fake doesn't sound very useful. The process only about halves your workload compared to just checking all the papers. In any large-scale rollout they would have to set a way higher threshold, and hope they still catch a useful number of fraudulent papers when using a threshold that detects 10% or 1% of genuine papers as fake.
They are also tagging all independent non affiliated researchers as fake. Do they know how many young people are doing science in the universities as temporary collabo-slaves without right to a nice personal mail?. Their detector would tag Einstein and Erdos as fake scientists by Pete's sake!. They just have a narrow vision tunnel about how the real research works
Note: I paid for the reprints and the postage, often expensive foreign rates.