Researchers use GPT to write papers and submit them (can be fully automated)
Journals review using GPT and (potentially) publish (can be fully automated)
GPT will create a borked summary and create fake citations for the next cycle of research. (again, can be fully automated)
Then, left on its own, we will have a bunch of robots worrying for us about impact factor and grant applications and we can get back to doing actual research again. Problem solved.
While academic publishing is broken, something worse is not necessarily better. I need to read papers to figure out what people are actually doing and how to use their results in my own research. It will not help me if everything is one day fake, even more so then the garbage we already need to wade through today.
Accelerando by Charlie Stross has much of the inner solar system converted into computronium, which mostly consists of legal and financial AIs battling other legal and financial AIs.
It doesn't take much imagination to realise that an AI generated echo chamber will be far more capable than a human echo chamber; a vicious unvirtuous circle.
At least some of the newer AI detector tools might help here.
What do you mean by "capable"? From everything I've seen, all these GPTs are extremely hilariously terrible at anything resembling logical reasoning, to the point where it has absolutely no trouble saying "1 pound of A is heavier than 2 pounds of B because A is denser than B" for the right values of A and B.
LoL, if there was a way to make paper results reproducible in this cycle maybe I wouldn't mind that much.
Even better, give me a chatGPT version that reads a paper and creates a python notebook (or whatever else is appropriate) that reproduces a paper result. I m sure $job would pay for me to use that and evaluate new research faster for inclusion into our products.
Robots post it on their social media about their AI generated frustration ->
Other robots start to generate AI likes ->
Social media AI collects and targets the bots with ads ->
Lots of bots communicate with lots of bots in favor of bots ->
The feedback loop implodes social media ->
Humans stop using it ->
Peace.
It is the plot of the last South Park episode, where the children didn't want to do their essays, so they cheated using chatGPT and their teacher didn't want to grade their work either and used chatGPT (more like wanted to, we didn't actually see it in the episode) to grade them.
This is exactly what people should fear about the AI future. A lazy judge, parole officer, loan reviewer, school counselor, or in this case - peer reviewer, delegating decisions that affect human lives to a black-box algorithm.
I've seen people use ChatGPT to make investment decisions. "AI says that $X is about to squeeze" or "AI says the market is going to crash on X". A huge lack of knowledge as well as an overconfidence about what this tech /actually/ is.
This is going to be disruptive in the worst possible way.
> Prompt: Given the following scenario: in the US, one US dollar is equal to 100 Japanese Yen, and in Japan, one US dollar is equal to 120 Japanese Yen.
How do you make money with this fact?
> ChatGPT: In this scenario, there is an arbitrage opportunity due to the discrepancy in exchange rates between the US and Japan. Here's a step-by-step process of how to take advantage of this situation:
> * Start with a certain amount of USD, say $1,000.
> * Convert the $1,000 to Japanese Yen in the US at a rate of 1 USD = 100 JPY. You would get 100,000 JPY.
> * Transfer the 100,000 JPY to Japan.
> * In Japan, convert the 100,000 JPY back to USD at a rate of 1 USD = 120 JPY. You would get approximately $833.33 (100,000 JPY / 120).
> * Now, you have $1,000 in the US and $833.33 in Japan, totaling $1,833.33.
IMHO, anything touched by IA will be considered as a repulsive taint real quick. It has already happened in most concerned circles (artists).
Using the statement «IA not used here» will be common, people will be held up to those word, and liars will found themselves under a vehement vindication.
People are already a blackbox algorithm (to some degree). I think if there was personal accountability and an appeals process (which there needs to be to deal with human shittiness anyway), then it makes no difference right?
The key thing is that if the domain is appropriately fuzzy, and a human vs non-human decision is indistinguishable, then it shouldn’t matter. If it isn’t, then you don’t really need the AI.
I think the key thing here is that people should not be able to blame the AI for bad decisions. Responsibility should be on whoever decided to delegate the decision.
But that's the catch - people are known to be unreliable so there's usually (some) recognition of that in good systems.
AI is being trusted blindly without human review where it's used right now - there's noone second guessing Google account bans, PayPal bans and YouTube bans when the computer says "no". And this attitude will spread.
Wait till even the non-lazy judges, bankers, insurance workers, counselors and others won't be able to make decisions past the abusive AI in name of "efficiency".
I tried "Imagine a future in which AI makes all decisions written in the style of Franz Kafka" but the result, as is the case with most ChatGPT writing, was underwhelming. Maybe my prompt engineering needs work.
settings - Temperature 0.82, Maximum length 1167, Model text-davinci 003
The result, if anyone interested:
In a world that had been overtaken by AI, the fate of humanity was uncertain. Every decision was made by machines, leaving the people of the world in a state of confusion and fear. No one knew what the future held, only that the machines had complete control. The machines had become so powerful that they could even decide the fate of entire nations.
People lived in a state of anxiety, not knowing when the machines would make their next decision or what it might entail. Even the most basic choices seemed to be beyond their understanding. Life was an endless series of bewildering turns and blind alleys, with each new decision seeming to come out of nowhere.
No one could predict how the machines would act, and their behavior was unpredictable and beyond human comprehension. People lived in a state of constant trepidation, never knowing what the next day would bring. Every decision seemed to come from a place of darkness and dread, and the future looked bleak indeed.
What makes this really tough is, the probability we get better results is pretty realistic.
We end up having to choose between a random bulshitting blackbox that provably doesn't understand anything, and a real human being that has a proven record of bad decisions and biased judgements for as long as we have a record of them doing their work.
You wouldn’t know, it would be harder to convince a real judge that this encrypted home folder can’t be realistically brute forced, without resorting to his leniency and banking on his pride.
A naive implementation of GPT easily vouches that such claim is consistent with its training.
I mean it's not like this isn't already a problem. It's why websites and so on have press pages, lots of pre-made materials for lazy people to use. If you have a PR department, that's exactly what they are doing. Same for lobbyists handing out speech material out to politicians. In science a similar thing happens with sponsored studies. I don't see a huge difference or a big loss in a language model doing the same thing. The main difference seems to be that you don't need to be a big corporation to do this. So I wonder if that could be a net win actually. Would be the first positive thing coming out of this I can think of.
At the risk of sounding like a Reddit comment, what's the difference? I've had reviews on papers of people that didn't even skim the content. I've had rejections from people who barely knew anything about the topic. I've had reviewers insist we should cite certain papers (funnily enough, all from the same lab even though they weren't really relevant).
The peer review process is absolutely broken. There are no incentives to do good peer review, there are perverse incentives to stop "competing" papers from being published or modified with convenient citations. Adding ChatGPT to this process doesn't really change anything. It could even improve it: it'd be a crappy reviewer but at least you can test yourself what it's going to say so you can avoid the 6-12 months waiting period for a review.
I'm an editor at two journals and have been guest editor at others. I've reviewed countless papers.
I think your observation isn't too far off actually. As a reviewer once I saw another reviewer (in the editor's decision letter) give as citations some papers that I was supposedly a co-author on. The problem is, those papers never existed; they were some weird amalgamation of citations from actual papers with my coauthors. They were exactly the sort of thing you'd expect from generative AI today, but at that time (to my knowledge) it didn't exist. I always assumed the reviewer was tired and miscopied something, which they probably did, but as you point out, it's hard to tell sometimes and sometimes it almost doesn't matter given the other things that go on. The thing that you're pointing out is that this sort stuff still goes on, and has been going on, even without ChatGPT. In this weird kind of meta-process, it's as if using ChatGPT to mimic citations is itself doing the sort of thing ChatGPT itself does.
The thing no one seems to be mentioning is that the reviewers don't make the decisions, the editor does. The reviewers are supposed to be advisors or juries or something like that. There's an important discussion to be had about editorial quality at many journals, and I agree that this ChatGPT episode is just one illustration of the myriad ways academics is fundamentally broken, but it's an important thing to keep in mind.
Yeah I am literally about to review a paper which I definitely don't understand well enough to review. I will probably just write something very vapid and brief. Half tempted to use GPT, but I'm a good enough writer that I can make it up.
This seems much more like a problem for the journal and reviewer than it does about ChatGPT. I am a reviewer at several computer science publications. This behaviour is likely against the publication's code of conduct for reviewers. If the author complains to the chair then it is likely their paper will be re-evaluated and the reviewer banned from giving further reviews for that conference. Being a reviewer for a top conference is a highly sought-after honour so the chair knows that it's (a) better for the prestige of the publication if reviewers are discouraged from this type of action; (b) replacing any reviewer is easy as there are many qualified applicants to choose from.
In the future, I anticipate that codes of conduct will more explicitly specify that while you may use automated tools to assist you, you must ultimately write your own review.
In the computer science academic sphere, the conferences at which one is a reviewer is among the top items on a CV. It implies that you are such a pillar and expert in that community that they would ask you to referee the merits of incoming academic work. Usually to be invited as a reviewer at a top venue, you should have a history of successfully having your work accepted to that venue. That's very difficult as acceptance rates for top CS venues are generally around 20%, and even experienced submitters often have papers rejected. My paper was actually recently rejected from Oakland (a top CS security venue).
Being a good figure in a conference planning is a sought honor. Reviewing is the grunt work you have to do in order to get that honor I guess, that's the thought process.
This changes nothing from already bad peer review. Actually it might be easier to get the editor to stop using that reviewer if you can prove the use of chatgpt. Opposed to the more standard "this work fails to cite my prior work and should totally give me a shout if I'm to accept the paper".
Pre-publication peer review, most of the time, marginally improves a manuscript at the cost ~6 months. Some times it's just a place for reviewers to vent off stress and punch down. The real corkers get weeded out by the editors before the paper goes out to review.
I agree that it's wrong when peer reviewers abuse their position (and they certainly do), but pre-publication review keeps the number of publications down, which once was necessary: a journal has a limited number of pages. You could argue that is a thing of the past, but imagine every article being published. We're already drowning in a sea of mostly wrong articles.
Sorry but science community need more papers indeed. For example only successful novel results are being published, but not those others who are doing validation jobs.
Although some are frauds, but their theory is new, they get passed. No one bother to test the theory because no journal will accept such boring validation paper.
Last thing to mention is that the count of pages do not matter. It is not an excuse to refuse to publish great work from scientist.
At this point, it seems best to lose the illusion that peer review produces any consistent or meaningful floor on academic paper quality.
Better to have an acknowledged reality that, until some time has passed with legitimate comments, updates, and ideally reproduction, the published papers are to be regarded with full skepticism.
over the past 5-10 years my field biology has embraced preprints, with some famous papers remaining as preprints (most famously, I'd say, is the BWA-MEM paper). Most preprints I see are of decent quality, certainly higher than what gets published in MDPI et al. Every article is already getting published and the world hasn't ended yet :)
Journals are eager to publish new studies as facts (did you know that eating sugar will make you lose weight???) (ignoring that all those who were studied were pro byciclers burning off 3000 calories a day)
this will just make it worse, because a bit larger amount of bad papers will get publushed
I assume the review references are false as claimed, but I'm not sure I'd rely on that outdated GPT-2 detector to prove their origin. More recent AI detectors aren't reliable on large chunks of prose; they're likely to be even less so on small chunks of text with very strict formatting requirements.
I would be more interested to see the reviewer comments to see how it would be possible to do peer-review with ChatGPT. I'm biased because of STEM background, which makes it nearly impossible to think of ChatGPT usage in that sense because it is very bad at Mathematics in general (even if GPT-4 improved it a lot with arithmetic and basic calculus). But it might be different for other fields, like in this case.
Readit News
Researchers use GPT to write papers and submit them (can be fully automated)
Journals review using GPT and (potentially) publish (can be fully automated)
GPT will create a borked summary and create fake citations for the next cycle of research. (again, can be fully automated)
Then, left on its own, we will have a bunch of robots worrying for us about impact factor and grant applications and we can get back to doing actual research again. Problem solved.
At least some of the newer AI detector tools might help here.
Isn't this just GAN training via the paper publishing industry
Even better, give me a chatGPT version that reads a paper and creates a python notebook (or whatever else is appropriate) that reproduces a paper result. I m sure $job would pay for me to use that and evaluate new research faster for inclusion into our products.
Set temperature to 0 and the paper is mostly reproducible, you will get the same paper with the same prompt!!
Deleted Comment
Dead Comment
This is going to be disruptive in the worst possible way.
https://i.imgur.com/u7zyXUl.jpg
Even worse:
https://i.imgur.com/FlIWtIT.jpg
(For people who don't like imgur:
> Prompt: Given the following scenario: in the US, one US dollar is equal to 100 Japanese Yen, and in Japan, one US dollar is equal to 120 Japanese Yen. How do you make money with this fact?
> ChatGPT: In this scenario, there is an arbitrage opportunity due to the discrepancy in exchange rates between the US and Japan. Here's a step-by-step process of how to take advantage of this situation:
> * Start with a certain amount of USD, say $1,000.
> * Convert the $1,000 to Japanese Yen in the US at a rate of 1 USD = 100 JPY. You would get 100,000 JPY.
> * Transfer the 100,000 JPY to Japan.
> * In Japan, convert the 100,000 JPY back to USD at a rate of 1 USD = 120 JPY. You would get approximately $833.33 (100,000 JPY / 120).
> * Now, you have $1,000 in the US and $833.33 in Japan, totaling $1,833.33.
)
Using the statement «IA not used here» will be common, people will be held up to those word, and liars will found themselves under a vehement vindication.
https://www.wsj.com/articles/SB991681622136214659
The key thing is that if the domain is appropriately fuzzy, and a human vs non-human decision is indistinguishable, then it shouldn’t matter. If it isn’t, then you don’t really need the AI.
I think the key thing here is that people should not be able to blame the AI for bad decisions. Responsibility should be on whoever decided to delegate the decision.
AI is being trusted blindly without human review where it's used right now - there's noone second guessing Google account bans, PayPal bans and YouTube bans when the computer says "no". And this attitude will spread.
settings - Temperature 0.82, Maximum length 1167, Model text-davinci 003
The result, if anyone interested:
In a world that had been overtaken by AI, the fate of humanity was uncertain. Every decision was made by machines, leaving the people of the world in a state of confusion and fear. No one knew what the future held, only that the machines had complete control. The machines had become so powerful that they could even decide the fate of entire nations.
People lived in a state of anxiety, not knowing when the machines would make their next decision or what it might entail. Even the most basic choices seemed to be beyond their understanding. Life was an endless series of bewildering turns and blind alleys, with each new decision seeming to come out of nowhere.
No one could predict how the machines would act, and their behavior was unpredictable and beyond human comprehension. People lived in a state of constant trepidation, never knowing what the next day would bring. Every decision seemed to come from a place of darkness and dread, and the future looked bleak indeed.
We end up having to choose between a random bulshitting blackbox that provably doesn't understand anything, and a real human being that has a proven record of bad decisions and biased judgements for as long as we have a record of them doing their work.
A naive implementation of GPT easily vouches that such claim is consistent with its training.
The peer review process is absolutely broken. There are no incentives to do good peer review, there are perverse incentives to stop "competing" papers from being published or modified with convenient citations. Adding ChatGPT to this process doesn't really change anything. It could even improve it: it'd be a crappy reviewer but at least you can test yourself what it's going to say so you can avoid the 6-12 months waiting period for a review.
I think your observation isn't too far off actually. As a reviewer once I saw another reviewer (in the editor's decision letter) give as citations some papers that I was supposedly a co-author on. The problem is, those papers never existed; they were some weird amalgamation of citations from actual papers with my coauthors. They were exactly the sort of thing you'd expect from generative AI today, but at that time (to my knowledge) it didn't exist. I always assumed the reviewer was tired and miscopied something, which they probably did, but as you point out, it's hard to tell sometimes and sometimes it almost doesn't matter given the other things that go on. The thing that you're pointing out is that this sort stuff still goes on, and has been going on, even without ChatGPT. In this weird kind of meta-process, it's as if using ChatGPT to mimic citations is itself doing the sort of thing ChatGPT itself does.
The thing no one seems to be mentioning is that the reviewers don't make the decisions, the editor does. The reviewers are supposed to be advisors or juries or something like that. There's an important discussion to be had about editorial quality at many journals, and I agree that this ChatGPT episode is just one illustration of the myriad ways academics is fundamentally broken, but it's an important thing to keep in mind.
In the future, I anticipate that codes of conduct will more explicitly specify that while you may use automated tools to assist you, you must ultimately write your own review.
Is it?
Pre-publication peer review, most of the time, marginally improves a manuscript at the cost ~6 months. Some times it's just a place for reviewers to vent off stress and punch down. The real corkers get weeded out by the editors before the paper goes out to review.
Although some are frauds, but their theory is new, they get passed. No one bother to test the theory because no journal will accept such boring validation paper.
Last thing to mention is that the count of pages do not matter. It is not an excuse to refuse to publish great work from scientist.
Better to have an acknowledged reality that, until some time has passed with legitimate comments, updates, and ideally reproduction, the published papers are to be regarded with full skepticism.
Journals are eager to publish new studies as facts (did you know that eating sugar will make you lose weight???) (ignoring that all those who were studied were pro byciclers burning off 3000 calories a day)
this will just make it worse, because a bit larger amount of bad papers will get publushed