A lot of this seems to be a problem with academic publishing, peer review, and the pedantry contained therein.
I tried (twice!) to publish a paper that was scientifically sound, but written such that it could be understood by a lay audience. It was rejected; quoting a reviewer: "This paper lacks math." That sentiment made me lose a lot of faith in academia. Now, I simply refuse to contribute any more time to writing papers or performing peer review (esp. in non-open publications). I'm sure I'm not alone; I know of at least one "seminal" robotics paper that was rejected from top venues multiple times for "simplicity" (lack of mathematics content) that went on to become a foundational paper in the field after appearing in a lower-tier venue years later.
The irony: it takes researchers a lot of time to make a paper dense & concise. If they "showed all steps", it would probably improve researcher productivity & make the material more approachable to newcomers. Instead publishers enforce length restrictions... for which authors dedicate upto 25% to related work (some of which is useful; much of which is pandering to likely peer reviewers in small, niche fields). Length restrictions seem equally foolish in the age of digital publishing. And again, inadvertent pedantry is the only explanation I can imagine... but happy to be wrong.
This may be intentional - for most academic publishing venues, their target audience is other researchers of that domain. It's counterproductive to optimize an article for a lay audience if everybody in the actual audience has (or is working on) a PhD in that domain and is generally expected to be (or become) up to speed with the terminology and earlier results on that topic. The journal or conference is generally not meant to communicate their results to wider audience, it's generally a tool that some particular research community made for themselves to help their research by an exchange of ideas and results. An article in academic publishing is meant for someone who will use that to further their own research in a related field - and the needs of a person like that are very far from a lay audience, they will want to see entirely different things in that paper - you want the paper to focus on the novelty, on the "delta" between this result and what was bleeding edge a few months ago; not spend two thirds of the paper describing what was already known before.
There are publishing venues intended for a lay audience, but most academic publishing is not, they have incompatible goals.
Unfortunately, the 'lay' audience is often not all that 'lay.' For example, you might be working on a phd in an adjacent field to a given paper and still have a hard time coping with the sub-field-specific jargon. This happens all the time in math research, and contributes heavily to silo'ing of subfields. This silo'ing is actually super counter productive, and can lead to the same things being independently rediscovered, and important, useful discoveries simply failing to be communicated across silo boundaries.
I agree with your basic premise: Researchers are not supposed to be writing for a lay audience. No arguments here.
But too often, the language (both english & mathematical) of research papers is not intended to enhance academic communication. Rather, it's a faux optimization to game peer review or make results look more "technical" in a way that is actually detrimental to communication.
In effect, the language is counterproductive. And the time spent on faux conciseness is wasted; much like that old quote, "I would've written a shorter letter, if only I'd had more time."
The reason why researchers write for a niche audience is because they are maximizing publication count. We're essentially targeting the papers for their reviewers. If researchers did not care about publication count, they would target the widest possible audience so that their results would have the most impact.
And perversely, this makes it so that people feel compelled to add mathy things that add no additional explanatory power to the results.
I've frequently heard complaints that people in some CS subfields in particular will just add complicated math because they have to rather than because it helps. It sucks because the authors don't have that great a grasp on the math anyway, the readers don't care about the math -- they just care about the intuition and process, and communication suffers overall.
My best papers had at most like two equations, and they were just there to clarify models for fits and the like in case the reader wasn't familiar with terms like "Arrhenius relation". So, you know, to actually make reading it less burdensome rather than just making it look more sophisticated.
Exactly! In the end, we ended up just publishing on arXiv.org and writing up a blog post [1]. It's not earth shattering... just an entirely new idea being demonstrated for the first time. The mathy background certainly exists, and was explained in other cited papers, but wasn't necessary to explain the new ideas.
I've definitely had reviewers tell me in a paper that I needed to make my math more formal and really prove things, when all I really wanted to say was, "We take the limit of foo as bar goes to zero, and get baz. Look, baz is interesting!"
Here are my thoughts based on my experience with peer reviewed publication.
There are two high level criteria for publication: novelty and difficulty (this is in my field of Programming Languages and Systems so keep that in mind).
The novelty requirement is important and I trust that you satisfied it but (as you pointed out in a child comment) you may not have met the difficulty requirement and the reviewer did their "best" to articulate that in a way that isn't the all together ridiculous "not hard enough".
Naturally we might wonder why "difficulty" is a requirement at all. Shouldn't the importance or impact of the work be the thing that matters regardless of how difficult it was to achieve? The problem is that it's _extremely_ hard to know what work will be impactful and so reviewers, who have to reject something like 90% of submissions, use the heuristic of "difficulty" to estimate.
This is a problem to be sure but I think it would be a problem in other settings as well.
I wanted to echo this experience. I agree that reviewers are generally looking for novelty and difficulty. Unfortunately, difficulty can translate into a system where reviewers create small fiefdoms: This isn't difficult enough for my journal, so I'm going to reject it. I say fiefdoms because I don't often see criteria for what difficult means, so it's left to individual reviewers and the editor to decide. Anyway, this creates some perverse incentives where it's now necessary to make a piece of work appear more difficult than it really is.
That said, I don't really think this is all that unique to academic publishing. At some point, I think we've all met someone professionally who makes problems sound more difficult than they really are and seen this person rewarded by management because they're the person who can work on really difficult problems.
I'm more on the theory end of PL, but in my experience program committees generally regard difficulty as a negative for a paper. The main thing PCs want is for papers which make progress on important problems. Now, you can do this either by (1) attacking existing open problems, or (2) by finding new important problems.
It's easy to tell that a long-standing open problem is important, but generally these problems are open because they are hard, and consequently the technical sophistication for new attacks on them is generally very high.[] New problems can often be solved with relatively simple techniques, but it's also difficult to tell if a new problem is genuinely important or just some random thing made up for submission.
With a low acceptance rate, PCs tend to favor type (1) papers over type (2) papers, because few people feel comfortable arguing for a paper that
might* be doing something important over one that is doing something important. This is widely seen as a failure mode of low acceptance-rate venues.
So over the last several years many of the top PL conferences (such as POPL, LICS and PLDI) have moved to a multiple track format in order to increase the number of papers they accept. This is precisely to give program committees the freedom to take chances on "risky" papers that might become important, or might become footnotes.
[*] Occasionally, someone will solve a difficult open problem with simple techniques, and these papers are immediately accepted with much rejoicing. It's easy to have your cake and eat it, when you have two cakes...
I don't know about the details of the OP's paper, but I'm not sure "difficulty" is the right word... Mathematical rigor is needed in many domains. In my field, "not enough math" could well signify that the description of the experimental results is sloppy, or that you can't check whether the results presented are sound.
> A lot of this seems to be a problem with academic publishing, peer review, and the pedantry contained therein.
Ayup. It's too annoying to translate the academic speak, but you can shortcut it.
For example, anything CS nowadays (I include crypto in this) must be 1) publicly available on the web, 2) come with running code that runs some basic tests, and 3) be limited to a single compilable file. Failure in any of the criteria means I move on. If the source code involves anything like "configure" it's an immediate fail. If the code passes all three of those criteria, it's probably more useful to read the code rather than the paper.
I used to love ISSCC for the reason that they used to demand both A) a die photograph of a chip and B) actual oscilloscope traces. You can't hide when you have to make test equipment produce data. Sadly, they got rid of that requirement in the late 90's, and the information content of the conference suffered correspondingly.
If what you write is difficult to understand people will assume you're smart. If you've ever learned a foreign language you'll recognize this: Blithering idiots talking in that language sound smart because you're having a hard time understanding them.
In my experience, papers often take more than just two submissions to be published. It's often a question of finding a good community fit. One paper I coauthored took 8 years and probably 10 submissions before it was finally accepted.
Length restrictions can be a bummer, but if you have a publishable result that you simply can't squeeze into 10-20 pages (depending on the venue), typically you split it in to two or more publications. This has the added advantage of ensuring that each published unit is a smaller, tighter piece of work. I don't think it's just pedantry.
Many people would say that if you submitted 10 times, you are playing the system. You've had 10 sets of reviewers spend time to review your work, even though 9 of them told you that it's not worth publishing.
I guess that's the advantage that tenured researchers have over everyone else--you can't run out of career before you published all of your papers.
Carl Sagan was famously denied tenure at Harvard and membership in the National Academy of Sciences for his science advocacy. Different takes on it range from "jealousy" of his peers over his broader popularity to anger from them for his making science accessible and explaining concepts in a way that allowed readers to understand them:
I used to see this with my peers in computer programming in the 1990s. There was a lot of anger and jealousy when everyday normal people started putting up websites. Several of my CS friends were of the opinion that this was almost polluting the WWW with bad code.
(Also: The mathematical underpinnings are related to radar systems, whereas the application is in robotics. The amount of background information to bring a roboticist up to speed would've consumed entire papers or books -- some of which were previously published by myself & other coauthors.)
"I know of at least one "seminal" robotics paper that was rejected from top venues multiple times for "simplicity" (lack of mathematics content) that went on to become a foundational paper in the field after appearing in a lower-tier venue years later."
Correct me if I'm wrong, but if you don't value what a publisher is offering, you could publish your paper on your own, make it accessible to an audience of your choosing with a delivery of your choice (for example, the wonderful work of Aphyr, here: http://jepsen.io/).
I don't get why people would try to publish in high-tier venue. To me it seems much more about polishing one's ego instead of improving the research quality.
As a young researcher, you need the publications when you apply for academic positions. High-tier venue publications are easily recognized as such. To recognize high-quality work published in a low-tier venue, the only metric is citations. Accumulating citations takes a long time (you might not have) and is helped by active marketing (you might not do).
This is more about quick-polishing your image than your ego. ;)
Researcher has to have some amount of authority in his field to receive grants. Publishing in high-tier venue is a good way to get that authority because it is how people-with-money judge researchers :-)
Well I really enjoy reading articles from top-tier venues (PL theory). It means that committee does a good work filtering and improving articles, definitely not without mistakes.
"What is the role of human scientists in an age when the frontiers of scientific inquiry have moved beyond the comprehension of humans?"
The above quote is from Ted Chiang's short story "The Evolution of Human Science," originally published in Nature as "Catching crumbs from the table" [0]. It's a brilliant depiction of this very problem: when new developments contribute to an increasing gap between those who can make new developments, and those attempting to understand the state of the art, the entire process of scientific inquiry becomes less efficient. In fact, the scenario depicted is one where the majority of researchers become "distillers," to use the language of the original post.
While Chiang posits a science-fiction reason for the divide, "normal" research/technical debt is insidious as well. Without incentives to reduce debt, the knowledge gap widens until only a handful of experts can make significant contributions. It's a problem that needs to be tackled head-on in both research and engineering. I'd love to see more initiatives like Distill.
There's another subtle aspect to this: the same or very similar ideas, methods, and tools show up or are reinvented again and again, under different guises, in different disciplines and subfields that have their own jargon, unnecessarily making human comprehension even harder.
If you look ahead to a future age, and consider the state of literature after the printing press, which never rests, has filled huge buildings with books, you will find again a twofold division of labor. Some will not do very much reading, but will instead devote themselves to investigations which will be new, or which they will believe to be new (for if we are even now ignorant of a part of what is contained in so many volumes published in all sorts of languages, they will know still less of what is contained in those same books, augmented as they will be by a hundred—a thousand—times as many more). The others, day laborers incapable of producing anything of their own, will be busy night and day leafing through these books, taking out of them fragments they consider worthy of being collected and preserved.
I can't count how many times I've invested meaningful time and effort to grok the key ideas and intuition of a new AI/DL/ML paper, only to feel that those ideas and intuitions could have been explained much better, less formally, with a couple of napkin diagrams.
Alas, authors normally have no incentive (or time, for that matter!) to publish nice concise explanations of their intuitions with easy-to-follow diagrams and clear notation... so the mountain of research debt continues to grow to the detriment of everyone.
I LOVE what Olah, Carter et al are trying to do here.
I really love this effort. Research papers are low bandwidth way to get information into our brains. They take a lot of effort to read, even if the ideas are not particularly complicated. Often when reading complicated material, I have to come up with metaphors in my mind to make sense of it. This is somewhat of a wasted effort, as the author who wrote the material surely had metaphors for their own mind when writing, but too often they don't share these metaphors, and stick to purely technical writing. I think this is one reason why ideas like general relativity are so popular, even though the material is actually quite complicated. The average educated person can give a reasonable explanation of general relativity because the metaphors used to explain it are so powerful, even though its very unlikely they understand any of the math involved.
I've often thought it would be great for the Arxiv to make it easy to link to a video of the 'talk' that generally goes with a paper. The talk is very often the distillation of the ideas in the paper, as conducted by one of the involved researchers. Indeed, one of the main points of conferences is to allow us to trade these distilled versions of our research with one another and place them in the context of everything else going on...
> I've often thought it would be great for the Arxiv to make it easy to link to a video of the 'talk' that generally goes with a paper.
This has been in the idea pipeline for a while, but the arXiv is slow-moving. (They probably wouldn't host the actual video files for cost reasons, but that's not a big deal.)
> The talk is very often the distillation of the ideas
This seemed a major omission of the OP.
The distillation quality hierarchy goes something like 'conversation among best-in-field researchers', 'conversation with principals' and 'research talk by principals, plus subsequent questions/conversations', 'review paper', 'journal commentary' and 'papers' but... in many fields, papers really need some of the above to provide critical context.
So a distillation system exists - it's just very very narrowly scoped. It's sad to see graduate students and young faculty who haven't "plugged-in", and so are not spending their time well.
And it's depressing to see a 'conversation among best-in-field researchers' excoriating a perspective on a topic, knowing that broken perspective is how the topic is pervasively taught, graduate down to high-school, and it's not going to change any year soon. There being few incentives or mechanism to pipeline such insight into educational content.
So yes, starting to build an expectation that talks become available as video, and are easily found, could be a non-trial step forward.
> Noise – Being a researcher is like standing in the middle of a construction site. Countless papers scream for your attention and there’s no easy way to filter or summarize them.2 We think noise is the main way experts experience research debt.
This is a big part of how I don't understand why some type of annotation standard hasn't taken off for research papers. Everyone does the same duplicative, time-consuming work of turning a paper into knowledge in their own head, so many wheels are reinvented. Where is the GitHub for research ideas?
I like the ideas put forth in this article. I wonder, though, if "distillation" is a re-casting of "scholarship" as considered by the humanities.
People studying topics ranging from Biblical Studies to History to Literature often do not create new source material, unlike in STEM. Yet there is a large degree of effort taken to "distill" existing facts through new lenses, producing novel concepts and interpretations. These efforts can transform our understanding of many areas of human endeavour.
One thing I believe to be of great value which is not made explicit in this article (which I think is an awesome article), is that research debt, as they describe it, is basically _education_.
In other words, improve the educational resources for complex subjects.
In our age where we're blessed with cheap printing of books and the possibility of creating complex interactive media, I think the question of designing user-friendly, powerful, and beautiful educational resources is a huge opportunity and pressing question.
Not just for people seeking to achieve a research-level understanding of a complex subject, but for all subjects and all people.
Consider the social value of beautiful, well-designed and nontrivial educational material for mathematics or basic science being widely available for all classes of people at all ages.
I'd argue that when news organizations use infographics or interactive journalism at its best, they are also performing this educational function.
Sorry for the long post, but to summarize, I think it's useful to recognize research debt as a specific case of the art and practice of creating media for education.
There are 'distillers' of large bodies of scientific research. Traditionally, they are science communicators, and more specifically science journalists.
The goal of a practicing scientist is very much at odds with someone whose job is to translate science into larger audiences. I've had very well-intentioned rational research scientists tell me with a straight face that "my job is to produce science results, not to communicate it. that's someone else's job", usually with the attitude that it's less respected or somehow self-aggrandizing. "The best science will be self-evident" attitude that all researchers secretly aspire for, not realizing that 99% of impactful science has had effort spent to promote, frame, or distribute it.
This weird stereotype is somehow beaten into scientists from the very beginning, and I haven't been able to figure out where this comes from. Obviously, yes, it's a lack of tools and accessibility into letting scientists also become distillers themselves. But the motivations and incentives at the center of the whole system is what's making this whole imbalance. I think there are parts of our research system that actually say "No, you cannot and should not distill your science".
Ultimately, for me, it gets back to funding. If review articles and outreach weighed just as much as citation count in tenure and grant committees, then maybe this could start to change. Yet, these committees still don't value open access, and look how tough that battle has been.
Also - this solution is really great and commendable, but I don't see how this works outside of ML/CS where research outputs are more like software development - gists, snippets, prototypes that are immediately shared, pushed, forked. More science fields like ecology, synthetic biology, anthropology, will look like this, but it will take a few human generations.
I think you miss the point. The article proposes a path for distillers whose target audience is researchers, not the lay public.
Science journalists are fairly poor distillers of knowledge, actually.
What's needed is more like a way for senior researchers to write more and better review papers that lay out and summarize all of the issues around a particular sub-field, for active researchers in that field.
Science communicators/journalists can turn "Methodological observation of the sociometrical behavior tendencies of prematurated isolates indicates that a casual relationship exists between groundward tropism and lachrimatory, or 'crying,' behavior forms." into "Scientists find that falling down makes babies cry" but they're less good at expressing concepts like "math equation forms the basis for all current modeling, papers A, B, and C each use a different special case of the equation to reach their conclusions."
Isaac Asimov anticipated the idea of a research distiller in "The Dead Past" with the character of Nimmo, a professional science writer:
"Nimmo received his first assignment at the age of twenty-five, after he had completed his apprenticeship and been out in the field for less than three months. It came in the shape of a clotted manuscript whose language would impart no glimmering of understanding to any reader, however qualified, without careful study and some inspired guesswork. Nimmo took it apart and put it together again (after five long and exasperating interviews with the authors, who were biophysicists), making the language taut and meaningful and smoothing the style to a pleasant gloss."
In the story Nimmo has less prestige than a "real researcher" but the role pays well and he is in high demand.
Readit News
I tried (twice!) to publish a paper that was scientifically sound, but written such that it could be understood by a lay audience. It was rejected; quoting a reviewer: "This paper lacks math." That sentiment made me lose a lot of faith in academia. Now, I simply refuse to contribute any more time to writing papers or performing peer review (esp. in non-open publications). I'm sure I'm not alone; I know of at least one "seminal" robotics paper that was rejected from top venues multiple times for "simplicity" (lack of mathematics content) that went on to become a foundational paper in the field after appearing in a lower-tier venue years later.
The irony: it takes researchers a lot of time to make a paper dense & concise. If they "showed all steps", it would probably improve researcher productivity & make the material more approachable to newcomers. Instead publishers enforce length restrictions... for which authors dedicate upto 25% to related work (some of which is useful; much of which is pandering to likely peer reviewers in small, niche fields). Length restrictions seem equally foolish in the age of digital publishing. And again, inadvertent pedantry is the only explanation I can imagine... but happy to be wrong.
There are publishing venues intended for a lay audience, but most academic publishing is not, they have incompatible goals.
But too often, the language (both english & mathematical) of research papers is not intended to enhance academic communication. Rather, it's a faux optimization to game peer review or make results look more "technical" in a way that is actually detrimental to communication.
In effect, the language is counterproductive. And the time spent on faux conciseness is wasted; much like that old quote, "I would've written a shorter letter, if only I'd had more time."
I've frequently heard complaints that people in some CS subfields in particular will just add complicated math because they have to rather than because it helps. It sucks because the authors don't have that great a grasp on the math anyway, the readers don't care about the math -- they just care about the intuition and process, and communication suffers overall.
My best papers had at most like two equations, and they were just there to clarify models for fits and the like in case the reader wasn't familiar with terms like "Arrhenius relation". So, you know, to actually make reading it less burdensome rather than just making it look more sophisticated.
[1] http://www.hizook.com/blog/2015/08/10/mobile-robots-and-long...
There are two high level criteria for publication: novelty and difficulty (this is in my field of Programming Languages and Systems so keep that in mind).
The novelty requirement is important and I trust that you satisfied it but (as you pointed out in a child comment) you may not have met the difficulty requirement and the reviewer did their "best" to articulate that in a way that isn't the all together ridiculous "not hard enough".
Naturally we might wonder why "difficulty" is a requirement at all. Shouldn't the importance or impact of the work be the thing that matters regardless of how difficult it was to achieve? The problem is that it's _extremely_ hard to know what work will be impactful and so reviewers, who have to reject something like 90% of submissions, use the heuristic of "difficulty" to estimate.
This is a problem to be sure but I think it would be a problem in other settings as well.
That said, I don't really think this is all that unique to academic publishing. At some point, I think we've all met someone professionally who makes problems sound more difficult than they really are and seen this person rewarded by management because they're the person who can work on really difficult problems.
It's easy to tell that a long-standing open problem is important, but generally these problems are open because they are hard, and consequently the technical sophistication for new attacks on them is generally very high.[] New problems can often be solved with relatively simple techniques, but it's also difficult to tell if a new problem is genuinely important or just some random thing made up for submission.
With a low acceptance rate, PCs tend to favor type (1) papers over type (2) papers, because few people feel comfortable arguing for a paper that
might* be doing something important over one that is doing something important. This is widely seen as a failure mode of low acceptance-rate venues.So over the last several years many of the top PL conferences (such as POPL, LICS and PLDI) have moved to a multiple track format in order to increase the number of papers they accept. This is precisely to give program committees the freedom to take chances on "risky" papers that might become important, or might become footnotes.
[*] Occasionally, someone will solve a difficult open problem with simple techniques, and these papers are immediately accepted with much rejoicing. It's easy to have your cake and eat it, when you have two cakes...
Ayup. It's too annoying to translate the academic speak, but you can shortcut it.
For example, anything CS nowadays (I include crypto in this) must be 1) publicly available on the web, 2) come with running code that runs some basic tests, and 3) be limited to a single compilable file. Failure in any of the criteria means I move on. If the source code involves anything like "configure" it's an immediate fail. If the code passes all three of those criteria, it's probably more useful to read the code rather than the paper.
I used to love ISSCC for the reason that they used to demand both A) a die photograph of a chip and B) actual oscilloscope traces. You can't hide when you have to make test equipment produce data. Sadly, they got rid of that requirement in the late 90's, and the information content of the conference suffered correspondingly.
What do you mean by this? Surely some things require some configuration. Don't they?
Academese exists because it works.
Length restrictions can be a bummer, but if you have a publishable result that you simply can't squeeze into 10-20 pages (depending on the venue), typically you split it in to two or more publications. This has the added advantage of ensuring that each published unit is a smaller, tighter piece of work. I don't think it's just pedantry.
I guess that's the advantage that tenured researchers have over everyone else--you can't run out of career before you published all of your papers.
https://en.wikipedia.org/wiki/Carl_Sagan
I used to see this with my peers in computer programming in the 1990s. There was a lot of anger and jealousy when everyday normal people started putting up websites. Several of my CS friends were of the opinion that this was almost polluting the WWW with bad code.
Why didn't you place the additional detail in an appendix, so as to not detract from the main points?
(Also: The mathematical underpinnings are related to radar systems, whereas the application is in robotics. The amount of background information to bring a roboticist up to speed would've consumed entire papers or books -- some of which were previously published by myself & other coauthors.)
Curious. Which paper is this?
Direct pdf link: https://arxiv.org/pdf/1507.02373.pdf
I don't get why people would try to publish in high-tier venue. To me it seems much more about polishing one's ego instead of improving the research quality.
This is more about quick-polishing your image than your ego. ;)
Well I really enjoy reading articles from top-tier venues (PL theory). It means that committee does a good work filtering and improving articles, definitely not without mistakes.
Deleted Comment
The above quote is from Ted Chiang's short story "The Evolution of Human Science," originally published in Nature as "Catching crumbs from the table" [0]. It's a brilliant depiction of this very problem: when new developments contribute to an increasing gap between those who can make new developments, and those attempting to understand the state of the art, the entire process of scientific inquiry becomes less efficient. In fact, the scenario depicted is one where the majority of researchers become "distillers," to use the language of the original post.
While Chiang posits a science-fiction reason for the divide, "normal" research/technical debt is insidious as well. Without incentives to reduce debt, the knowledge gap widens until only a handful of experts can make significant contributions. It's a problem that needs to be tackled head-on in both research and engineering. I'd love to see more initiatives like Distill.
[0] http://www.nature.com/nature/journal/v405/n6786/full/405517a... - a highly recommended companion piece to the original post.
Tibshirani's "glossary" of ML and Statistics terms is a canonical example: http://statweb.stanford.edu/~tibs/stat315a/glossary.pdf
-- Denis Diderot, 1755
https://ello.co/dredmorbius/post/ayavb8clhm61natfm5lfca
http://www.historyofinformation.com/expanded.php?id=2877
(The History of Information is an absolute treasure of a website, BTW.)
Expansion while a problem space is explored, drunken sailor style. Contraction and consolidation as best fit solutions are identified and adopted.
Technical debt due to entropy, obsolescence, communication lag (diffusion of innovation), pride, etc.
---
Oh. This reminds me. TODO: read up on facilities management, see how they deal with this. Stuff like scheduling capex and funding maintenance.
I can't count how many times I've invested meaningful time and effort to grok the key ideas and intuition of a new AI/DL/ML paper, only to feel that those ideas and intuitions could have been explained much better, less formally, with a couple of napkin diagrams.
Alas, authors normally have no incentive (or time, for that matter!) to publish nice concise explanations of their intuitions with easy-to-follow diagrams and clear notation... so the mountain of research debt continues to grow to the detriment of everyone.
I LOVE what Olah, Carter et al are trying to do here.
This has been in the idea pipeline for a while, but the arXiv is slow-moving. (They probably wouldn't host the actual video files for cost reasons, but that's not a big deal.)
http://blog.jessriedel.com/2015/09/02/additional-material-on...
This seemed a major omission of the OP.
The distillation quality hierarchy goes something like 'conversation among best-in-field researchers', 'conversation with principals' and 'research talk by principals, plus subsequent questions/conversations', 'review paper', 'journal commentary' and 'papers' but... in many fields, papers really need some of the above to provide critical context.
So a distillation system exists - it's just very very narrowly scoped. It's sad to see graduate students and young faculty who haven't "plugged-in", and so are not spending their time well.
And it's depressing to see a 'conversation among best-in-field researchers' excoriating a perspective on a topic, knowing that broken perspective is how the topic is pervasively taught, graduate down to high-school, and it's not going to change any year soon. There being few incentives or mechanism to pipeline such insight into educational content.
So yes, starting to build an expectation that talks become available as video, and are easily found, could be a non-trial step forward.
This is a big part of how I don't understand why some type of annotation standard hasn't taken off for research papers. Everyone does the same duplicative, time-consuming work of turning a paper into knowledge in their own head, so many wheels are reinvented. Where is the GitHub for research ideas?
People studying topics ranging from Biblical Studies to History to Literature often do not create new source material, unlike in STEM. Yet there is a large degree of effort taken to "distill" existing facts through new lenses, producing novel concepts and interpretations. These efforts can transform our understanding of many areas of human endeavour.
In other words, improve the educational resources for complex subjects.
In our age where we're blessed with cheap printing of books and the possibility of creating complex interactive media, I think the question of designing user-friendly, powerful, and beautiful educational resources is a huge opportunity and pressing question.
Not just for people seeking to achieve a research-level understanding of a complex subject, but for all subjects and all people.
Consider the social value of beautiful, well-designed and nontrivial educational material for mathematics or basic science being widely available for all classes of people at all ages.
I'd argue that when news organizations use infographics or interactive journalism at its best, they are also performing this educational function.
Sorry for the long post, but to summarize, I think it's useful to recognize research debt as a specific case of the art and practice of creating media for education.
The goal of a practicing scientist is very much at odds with someone whose job is to translate science into larger audiences. I've had very well-intentioned rational research scientists tell me with a straight face that "my job is to produce science results, not to communicate it. that's someone else's job", usually with the attitude that it's less respected or somehow self-aggrandizing. "The best science will be self-evident" attitude that all researchers secretly aspire for, not realizing that 99% of impactful science has had effort spent to promote, frame, or distribute it.
This weird stereotype is somehow beaten into scientists from the very beginning, and I haven't been able to figure out where this comes from. Obviously, yes, it's a lack of tools and accessibility into letting scientists also become distillers themselves. But the motivations and incentives at the center of the whole system is what's making this whole imbalance. I think there are parts of our research system that actually say "No, you cannot and should not distill your science".
Ultimately, for me, it gets back to funding. If review articles and outreach weighed just as much as citation count in tenure and grant committees, then maybe this could start to change. Yet, these committees still don't value open access, and look how tough that battle has been.
Also - this solution is really great and commendable, but I don't see how this works outside of ML/CS where research outputs are more like software development - gists, snippets, prototypes that are immediately shared, pushed, forked. More science fields like ecology, synthetic biology, anthropology, will look like this, but it will take a few human generations.
Science journalists are fairly poor distillers of knowledge, actually.
What's needed is more like a way for senior researchers to write more and better review papers that lay out and summarize all of the issues around a particular sub-field, for active researchers in that field.
"Nimmo received his first assignment at the age of twenty-five, after he had completed his apprenticeship and been out in the field for less than three months. It came in the shape of a clotted manuscript whose language would impart no glimmering of understanding to any reader, however qualified, without careful study and some inspired guesswork. Nimmo took it apart and put it together again (after five long and exasperating interviews with the authors, who were biophysicists), making the language taut and meaningful and smoothing the style to a pleasant gloss."
In the story Nimmo has less prestige than a "real researcher" but the role pays well and he is in high demand.