Note how the author talks about "main conferences", "major conferences", and "top conferences". That's the root issue. Whenever there is prestige available, people will compete for it. And if you have a competition, you should formalize the rules to make it fair.
When I was doing PhD ~15 years ago, I noticed that I rarely cited work that appeared in the top conferences of the subfield. Those conferences covered so wide range of topics that often only 1 or 2 papers were in the same subsubfield as me. And even those were often not directly relevant to my work.
But then there were small specialized conferences that had plenty of interesting papers every year. I left CS for another field a decade ago, but I still regularly attend some of those conferences and review for them. The papers published in them are still interesting and relevant to my work.
I 100% agree with this comment. In fact, it is very common to receive reviews at top conference which say "you should submit to a smaller/more specialized venue". Those "more specialized" conference are indeed both more interesting AND less valued by universities. Sometimes universities will not even agree to pay the travel cost to supposedly "smaller" conferences!
Another issue I have observed is that, exactly like the article highlights, conference reviewing has become about rejecting papers and not accepting them: there are too many decent papers! This is very perverse because it may only take a negative review from an unfamiliar reviewer to get a paper rejected. And even if your paper is considered decent, it will often be necessary for at least one reviewer to "champion" your article. This means that if your paper happens to be reviewed by one of the "top researcher" in your field, this person not championing your paper is a death sentence. And if for some reason this researcher does not like your work, you just have to pray that they will not review your paper. But of course, reviewers get to bid on the papers that they want to review so this person is very likely to review your paper every time.
All of this means that in our modern conference system, the role of few loud gatekeepers is highly exaggerated and they get to shape a lot the output of "top conferences". This is really more of human problem, but the lack of hierarchy in academia means that those people can get their way without major opposition for a long time...
Note of course that this is *highly* dependent on the field that your work in. There are still many fields of CS with very reasonable community.
I could say algorithmic bioinformatics, but that's not really correct. The center of mass may be there, but the actual work varies from theoretical computer science to genomics.
A big part of the problem here is that Universities have increasingly begun attaching prestige to specific “top” conference publications for both ranking and faculty promotions. A good example of the phenomenon can be seen in [1] (sorry for the noun-citation!) which only gives credit for approximately three conferences in each field. Combine this with a flood of new researchers entering CS, you have a recipe for “top” conferences being essentially destroyed and filled with uninspired work.
(And contrary to the joke in the article, even your own work becomes uninspired when you ship it to those conferences. You can’t afford to be quirky or interesting.)
Fortunately every field has a fourth or fifth-tier conference that isn’t on this list (or a specialized topic conference that the rankings folks don’t care about), and those still serve the purposes that conferences were made for. You just might not be able to convince a ranking-obsessed administrator that your work has any value if you publish there.
(I imagine you agree, so this is just to expand) a secondary, insidious issue is that administrators diffuse their rules through the bureaucracy. In the case of CS, you start seeing references to csrankings in recommendation letters for grad applications, faculty applications, or even tenure letters. At that point, it can be hard to fight against it.
csrankings considered harmful. Three conferences per area (networking only gets two as apparently it is not an important fiedl) seems crazy - note they had to split AI into a number of subfields.
IIRC Usenix ATC (and maybe Eurosys?) were not originally included on csrankings.
Coincident with Meyer's "fossilization" complaint, csrankings seems to ignore many conferences focused on emerging work, new ideas, or industry developments.
There also seem to be various field omissions such as theory, cloud computing, multimedia, optical networking and computing, storage and file systems, quantum, etc.
Although a good venue can increase visibility, and impact to some extent, ultimately the quality of the work is what matters most.
This article is mostly whining that evidence-free speculation about how to write good software is no longer publishable in top conferences. And the major evidence cited is that there's a specific citation style required, a standard feature of every kind of publishing since forever. I promise (having reviewed many times for the specific conference under discussion) that no one's paper is rejected (or even denigrated) for failing to use appropriate citation style, people comment on it the same way they would comment on any other style issue.
I think that's a pretty uncharitable take; I thought there were several interesting questions raised by the author:
1. Should conference "service" be something we expect of postdocs (and even PhD candidates) rather than established experts?
> Often, as a result, the PC is staffed by junior, ambitious academics intent on filling their résumés. Note that it does not matter for these résumés whether the person did a good or bad job as a referee! [...] I very much doubt that the submissions of Einstein, Curie, Planck, and such to the Solvay conferences were assessed by postdocs. Top conferences should be the responsibility of the established leaders in the field.
2. Should programme chairs strive to maintain exclusivity of their conference track, or look for important ideas that deserve to be communicated?
> As a simple example, consider a paper that introduces a new concept, but does not completely work out its implications and has a number of imperfections. In the careerist view, it is normal to reject it as not ready for full endorsement. In the scientific view, the question for the program committee (PC) becomes: is the idea important enough to warrant publication even if it still has rough edges? The answer may well be yes. [...] Since top conferences boast of their high rejection rates, typically 80% to 90%, referees must look for reasons to reject the papers in their pile rather than arguments for accepting them.
3. Is computer science suffering from a focus on orthopraxy rather than scientific method?
> What threatens to make conferences irrelevant is a specific case of the general phenomenon of bureaucratization of science. Some of the bureaucratization process is inevitable: research no longer involves a few thousand elite members in a dozen countries (as it did before the mid-1900s), but is a global academic and industry business drawing in enormous amounts of money and millions of players for whom a publication is not just an opportunity to share their latest results, but a career step.
I think the example of how to "correctly" cite a paper actually makes this issue seem smaller than it is. In reality, these conferences have very complicated (and unstated) "rules" for how a paper is supposed to look. If an "outsider" wanders in and submits a paper with new ideas, it will be very obvious that they are not a "member of the community" and their paper will usually be treated much more harshly as a result. This adds a huge amount of friction to research.
And what's particularly frustrating is that many organizers will try to combat this by writing papers saying they "particularly encourage" papers that are interdisciplinary, or focused on less fashionable topics, etc. It's good that they are trying to change things, but I think the main effect in practice is to encourage people to spend their time writing papers that have little chance of being accepted.
This issue isn't at all unique to computer science, though. Try publishing a paper in a top economics journal as an outsider!
I am fairly certain this rule was there against an obnoxious citing style of "The lambda calculus [1] was intended as a foundation for mathematics". It is especially obnoxious in the case of CS because when you cite e.g. "as Johns comments in his article about future developments of the programming languages [1963a]" it is quite important to know that this paper is actually from 1963 and can be mostly disregarded except as a historic curiosity; yet I've seen people vehemently defending this "[1]" style.
Is citation style really an issue? Even if they don't state which style they expect, surely you can tell their expected style from their existing publications? With proper tooling (e.g. LaTeX+BibTeX) it's pretty painless to switch styles.
The only time I like numbers is writing proposals and I only like it because it saves space. Other than that I much prefer (name, year) if I am to have a preference at all.
Adding to the frustration, (the lack of) these shibboleths partially undermine double-blind reviewing, which is on the rise in prestigious conferences. A reviewer from the in-group may immediately spot that a submission comes from the out-group.
The critique of "An Axiomatic Basis for Computer Programming" is pretty much on spot though. The humble "procedure call" was not given a satisfactory Hoare's rule until the late 80s, for instance.
Pretty funny OOPSLA/SPLASH is the first example given because that's exactly the conference that came to mind when I clicked to read the article.
Honestly though, I find much better luck in the workshops. They don't really have the same reach as the main conference, and the specificity of the topics means that the reviewers are usually much more focused on content rather than checking boxes. They want to make a good workshop for the attendees, so it's far more important for workshops to focus on actual content rather than the resume building activities. The LIVE workshop for instance doesn't even really require a paper: https://2024.splashcon.org/home/live-2024#Call-for-Submissio...
Other workshops require just an extended abstract. Or maybe a short paper that doesn't have to be archived. I find these venues easier to get into, easier to present at, and easier to have a good discussion with the attendees.
The main issue I see is that papers are actually becoming so focussed on form that they are now unreadable. People prefer reading my blog for my papers than reading the papers themselves. In fact I hear people telling me they understood the blog _better_. The whole academic writing shtick has become so obtuse that not only writing is cumbersome, but so is reading.
The other side of all this academic brownie points via papers (and doing reviews, which has become "brownie points for gatekeeping") is that most academic software is not only unmaintained, but actually unusable. They rarely even compile, and if they do, there is no --help, no good defaults, no README, and no way to maintain them. They are single-use software and their singular use is to write the paper. Any other use-case is almost frowned upon.
One of the worst parts of Academic software is that if you re-write it in a ways that's actually usable and extensible, you can't publish that -- it's not new ("research") work. And you will not only have to cite the person who wrote the first useless version forever, but they will claim they have done it if your tool actually takes off.
BTW, there are academics who don't follow this trend. I am glad that in my field (SAT), some of the best, e.g. Armin Biere and Randal Bryant are not like this at all. Their software is insanely nice and they fix bugs many-many years after release. Notice that they are also incredibly good engineers.
> In fact I hear people telling me they understood the blog _better_. The whole academic writing shtick has become so obtuse that not only writing is cumbersome, but so is reading.
Legible writing has little incentive. You can write in a simple down-to-earth manner and risk having someone objecting "Your work is isomorphic to X, done years ago." Your counter-objection, "Well, X is so incomprehensible that no one had any chance of understanding it", will fall flat. Better not risk it, write in an obtuse form and withhold the source code.
I've studied political science in Germany. Unfortunately, most German political science people confused opaque writing with wisdom - basically hiding invalid theory behind an obtuse language. Nobody wanted to say out loud that they don't understand the language, so it helped to fortify their shit theories.
Yes. Exactly. If you make it readable and understandable, somehow the reviewers think it's easy. NO! It's easy because I made it easy for you to understand, and that's a good thing! If you invented BDDs or Petri Nets today, and didn't explain it over 5 pages of unreadable Greek alphabet, it'd be rejected. It's hilarious and really sad. Both are extremely useful, and intuitively "easy" tools. But they are so incredibly good and useful.
Exactly. The format is wrong. The expectation of publication in a journal in the way that they are formatted today is not good. At one point, an aspiring PhD student used a novel technique from a library I wrote and published on GitHub. The student cited my library as an example of the technique out in the wild, but claimed that he invented it. This is easily falsifiable by the GitHub history, but because there's no scientific paper, apparently this is fine by the journal. Not that I really care because for me, it's NBD, but I think it shows how myopic, isolated, and simply silly the whole scientific publishing industry is, especially in CS.
Haha yes. Some people don't publish work in the form of a paper, because it's just so much hassle and you'll get a bunch of nitpicking nonsense reviews while the tool is used by hundreds if not thousands of people/researchers/etc. It'd be hilarious if it wasn't so sad.
there is only 1 implementation available for an algorithm on which i did part of my bs internship, and it uses a lot of harcoding. It of course makes benchmarking and extending it much, much harder if you want to change some things
Unfortunately academic resource doesn't seem to be focused on the hard problems of today. A fast algorithm is nice, but we need to be able to maintain code long term, understand code, and fix bugs. I have 15 million lines of code I'm supposed to maintain and I know many others reading this work on larger projects. There is no way one human can write that much software. There is no way we can afford to throw it away even if there is a better way (well we could, but based on the last time we did that the cost would be over 1 billion dollars and 8 years - not a useful investment since there is no reason to think anything will make those prices go down). We need to keep it working.
This is a shock to many of our leaders - who were writing 8 bit assembly to do similar things. They commonly did throw away all the work of the last version since it only took them a few months to rewrite it for the exact features they needed. (having experience because they wrote it just a year ago means the rewrite as much faster, and the limitations of 8 bit means it was worth rewriting since they had to remove one feature to add a new one).
That's not a part of computer science that's organizational and process management at that point not computer science so I'm not shocked CS academics aren't interested in it. You're looking at the wrong group to answer your question, we have a great amount of discussion about how to organize large software to make it manageable already too but most of it comes down to breaking it into more manageable chunks with defined interfaces and testing that interface thoroughly.
Not to mention, academia likes to think it is isolated from the real world, which is advertising and marketing all the way down. We should probably mint more dual CS and marketing majors.
The reason behind the citation style is to serve the automated parsing of citations by research information systems, which can then roll up all of your contributions to the field into a single score which determines one’s entire worth in academia.
The author really should have recognized this, as it serves his point about careerism and brownie points.
The idea that being forced into a citation style stifles innovation is hilarious, especially coming from a computer scientist - formal systems are all we do. It’s not so hard, is it? Use a citation manager and have them generated for you!
Readit News
When I was doing PhD ~15 years ago, I noticed that I rarely cited work that appeared in the top conferences of the subfield. Those conferences covered so wide range of topics that often only 1 or 2 papers were in the same subsubfield as me. And even those were often not directly relevant to my work.
But then there were small specialized conferences that had plenty of interesting papers every year. I left CS for another field a decade ago, but I still regularly attend some of those conferences and review for them. The papers published in them are still interesting and relevant to my work.
Another issue I have observed is that, exactly like the article highlights, conference reviewing has become about rejecting papers and not accepting them: there are too many decent papers! This is very perverse because it may only take a negative review from an unfamiliar reviewer to get a paper rejected. And even if your paper is considered decent, it will often be necessary for at least one reviewer to "champion" your article. This means that if your paper happens to be reviewed by one of the "top researcher" in your field, this person not championing your paper is a death sentence. And if for some reason this researcher does not like your work, you just have to pray that they will not review your paper. But of course, reviewers get to bid on the papers that they want to review so this person is very likely to review your paper every time.
All of this means that in our modern conference system, the role of few loud gatekeepers is highly exaggerated and they get to shape a lot the output of "top conferences". This is really more of human problem, but the lack of hierarchy in academia means that those people can get their way without major opposition for a long time...
Note of course that this is *highly* dependent on the field that your work in. There are still many fields of CS with very reasonable community.
Fortunately Arxiv seems to be changing the publication landscape.
(And contrary to the joke in the article, even your own work becomes uninspired when you ship it to those conferences. You can’t afford to be quirky or interesting.)
Fortunately every field has a fourth or fifth-tier conference that isn’t on this list (or a specialized topic conference that the rankings folks don’t care about), and those still serve the purposes that conferences were made for. You just might not be able to convince a ranking-obsessed administrator that your work has any value if you publish there.
[1] https://csrankings.org/
IIRC Usenix ATC (and maybe Eurosys?) were not originally included on csrankings.
Coincident with Meyer's "fossilization" complaint, csrankings seems to ignore many conferences focused on emerging work, new ideas, or industry developments.
There also seem to be various field omissions such as theory, cloud computing, multimedia, optical networking and computing, storage and file systems, quantum, etc.
Although a good venue can increase visibility, and impact to some extent, ultimately the quality of the work is what matters most.
1. Should conference "service" be something we expect of postdocs (and even PhD candidates) rather than established experts?
> Often, as a result, the PC is staffed by junior, ambitious academics intent on filling their résumés. Note that it does not matter for these résumés whether the person did a good or bad job as a referee! [...] I very much doubt that the submissions of Einstein, Curie, Planck, and such to the Solvay conferences were assessed by postdocs. Top conferences should be the responsibility of the established leaders in the field.
2. Should programme chairs strive to maintain exclusivity of their conference track, or look for important ideas that deserve to be communicated?
> As a simple example, consider a paper that introduces a new concept, but does not completely work out its implications and has a number of imperfections. In the careerist view, it is normal to reject it as not ready for full endorsement. In the scientific view, the question for the program committee (PC) becomes: is the idea important enough to warrant publication even if it still has rough edges? The answer may well be yes. [...] Since top conferences boast of their high rejection rates, typically 80% to 90%, referees must look for reasons to reject the papers in their pile rather than arguments for accepting them.
3. Is computer science suffering from a focus on orthopraxy rather than scientific method?
> What threatens to make conferences irrelevant is a specific case of the general phenomenon of bureaucratization of science. Some of the bureaucratization process is inevitable: research no longer involves a few thousand elite members in a dozen countries (as it did before the mid-1900s), but is a global academic and industry business drawing in enormous amounts of money and millions of players for whom a publication is not just an opportunity to share their latest results, but a career step.
What do you think about these?
And what's particularly frustrating is that many organizers will try to combat this by writing papers saying they "particularly encourage" papers that are interdisciplinary, or focused on less fashionable topics, etc. It's good that they are trying to change things, but I think the main effect in practice is to encourage people to spend their time writing papers that have little chance of being accepted.
This issue isn't at all unique to computer science, though. Try publishing a paper in a top economics journal as an outsider!
Honestly though, I find much better luck in the workshops. They don't really have the same reach as the main conference, and the specificity of the topics means that the reviewers are usually much more focused on content rather than checking boxes. They want to make a good workshop for the attendees, so it's far more important for workshops to focus on actual content rather than the resume building activities. The LIVE workshop for instance doesn't even really require a paper: https://2024.splashcon.org/home/live-2024#Call-for-Submissio...
Other workshops require just an extended abstract. Or maybe a short paper that doesn't have to be archived. I find these venues easier to get into, easier to present at, and easier to have a good discussion with the attendees.
The other side of all this academic brownie points via papers (and doing reviews, which has become "brownie points for gatekeeping") is that most academic software is not only unmaintained, but actually unusable. They rarely even compile, and if they do, there is no --help, no good defaults, no README, and no way to maintain them. They are single-use software and their singular use is to write the paper. Any other use-case is almost frowned upon.
One of the worst parts of Academic software is that if you re-write it in a ways that's actually usable and extensible, you can't publish that -- it's not new ("research") work. And you will not only have to cite the person who wrote the first useless version forever, but they will claim they have done it if your tool actually takes off.
BTW, there are academics who don't follow this trend. I am glad that in my field (SAT), some of the best, e.g. Armin Biere and Randal Bryant are not like this at all. Their software is insanely nice and they fix bugs many-many years after release. Notice that they are also incredibly good engineers.
Legible writing has little incentive. You can write in a simple down-to-earth manner and risk having someone objecting "Your work is isomorphic to X, done years ago." Your counter-objection, "Well, X is so incomprehensible that no one had any chance of understanding it", will fall flat. Better not risk it, write in an obtuse form and withhold the source code.
Even small deviations from academic style get negative responses from reviewers.
examples here https://github.com/search?q=repo%3AIBM%2FFedMA%20hard&type=c...
these hard coded parts are not easily adjustable
This is a shock to many of our leaders - who were writing 8 bit assembly to do similar things. They commonly did throw away all the work of the last version since it only took them a few months to rewrite it for the exact features they needed. (having experience because they wrote it just a year ago means the rewrite as much faster, and the limitations of 8 bit means it was worth rewriting since they had to remove one feature to add a new one).
The author really should have recognized this, as it serves his point about careerism and brownie points.
The idea that being forced into a citation style stifles innovation is hilarious, especially coming from a computer scientist - formal systems are all we do. It’s not so hard, is it? Use a citation manager and have them generated for you!