This article includes a throwaway link to the wikipedia page at the end of that quote. I recommend reading the relevant section (https://en.wikipedia.org/wiki/Line_length#Electronic_text), because it's pretty limited. There is really no way to tell if it (or Glyph) are accurately summarizing the research.
> What this situation does, in fact, is cause many more problems than it solves - catastrophic failures on a website are ensured total destruction with the addition of ROBOTS.TXT.
Of course an archival pedant [1] will tell you it's a bad idea (because it makes their archival process less effective)—but this is one of those "maybe you should think for yourself and not just implement what some rando says on the internet" moments.
If you're using version control, running backups, and not treating your production env like a home computer (i.e., you're aware of the ephemeral nature of a disk on a VPS), you're fine.
[1] Archivists are great (and should be supported), but when you turn it into a crusade, you get foolish, generalized takes like this wiki.