TeX and Literate Programming (and Lisp) are my fundamental, day-to-day tools.
Code it, explain it, generate a Literate PDF containing the code.
The programming cycle is simple. Write the code in a latex block. Run make.
The makefile extracts the code from the latex, compiles it, runs the test cases (also in the latex), and regenerates the PDF. Code and explanations are always up to date and in sync.
You may be interested in using Org [1], in conjunction with literate-lisp [2] (for Common Lisp) and literate-elisp [3] (for Emacs Lisp). Org provides various outlining commands (among other things), letting you view your program at different levels of granularity. literate-lisp/elisp advise the Lisp reader so the Org file can be loaded and/or compiled directly, without requiring tangling. Consequently, tools like xref will jump to the source block in your Org file, rather than the tangled source. I wrote a little hack/guide to extend this to errors raised in Emacs' compilation-mode. [4]
(Unfortunately, the package that lets each Org source block behave as though it was using the corresponding language's Emacs major mode - poly-org-mode - has a ton of bugs. It was part of why I stopped using literate programming entirely for later projects.)
I should have mentioned that emacs is also a fundamental tool for all of my work. But since everyone, everywhere uses emacs that would be stating the obvious :-)
I tried orgmode. I even attended a course at CMU that used it for the "live notes". It is excellent for teaching. But it has the same flaw as the "live notebook" idea. There is no generally accepted structure to the approach.
Everyone "understands" books. They have a preface, chapters, an index, a bibliography, pictures, credits, and a table of contents. Literate programs leverage that shared understanding of the structure.
Think of a physics textbook. If you just copy every equation out of the book then you "have the code". All the rest is explanation. They belong together so the explanation and equations (code) are intermingled.
I'm a "primitivist". I work in straight text at an emacs buffer in fundamental mode.
The point of my code is to "talk to the machine". The point of my literate program is to "explain to other programmers (mostly 'future' me)". Note that this is NOT DOCUMENTATION. It is explanation, best presented in book form.
I work with the author of your link [2], and his literate programming documents for Common Lisp are awesome. Watching him during Zoom screen-shares is fun. I am not currently a user of literate-lisp but it is on my non-urgent to-do list.
Not really a programmer (sysadmin) but I've tried the literate way with org-mode for my stuff and I use LaTeX for all "pretty docs" needs, however while for document production LaTeX excel literate programming AS A MERE CONCEPT for me does not work that much:
- simple things became uselessly lengthy and prolix
- complex things became too big to be easy read, while the equivalent "pure-code with comments" and a small new developers doc while fail to give an equally deep knowledge of the code base is FAR more quick and easy accessed.
Maybe it's my style, I do not know but while I do not feel LaTeX as painful, since most my docs are of the same kind so a template/class produced once with calm get reused issueless, I can't really digest literate programming...
You may in interested to learn then, that for the past 10 months I have (slowly) been working on a LaTeX package that does exactly this [1]. It writes all the text from between come \begin{code} blocks out to a file, as well as use minted/listings to highlight it in the generated pdf. (exactly what you described but in LaTeX package form).
I am currently a physics student and one of the main problems that I have day-to-day coding is being unable to include diagrams/equations/pictures in the source code. Moreover, because there is a fair amount of 'domain complexity' the comments are often longer than the code itself.
The package is still very much unstable and not even close to being ready for any sort production. However, I did use it for my masters thesis [2], and it was quite an interesting (in a good way) experience. (I recommend everyone to actually try to write literate code at least once).
About 12 years ago, I used LaTex to write a semantic web book, but in 2 editions: Common Lisp and Java. It was very cool to have 2/3 of the manuscript common text, and about 1/3 unique to the programming language.
For the last two evenings, I have been revisiting my old manuscript materials because I am thinking of updating the material and also creating additional editions for more programming languages: Python, JavaScript, and maybe Swift and Prolog.
I had forgot how cool TeX and LaTex are. It was very easy to start working with again, even after a 12 year gap.
I'm curious what literate style you use? I have tried getting into the style a few times, but I often find I'm not taking advantage of the full scramble of the code that is often possible.
Note that the C program is just a hand translation of the Lisp code.
The lisp code has an explanation and the necessary latex macros. The idea is to scan the latex, find each named code 'chunk', and add each one to a hash table. Then the hash table is scanned to dump the requested chunk to stdout. For example:
\begin{chunk}{part1}
code for part 1
\end{chunk}
Ordinary latex code between chunks.
\begin{chunk}{part2}
code for part 2
\end{chunk}
\begin{chunk}{part1}
this code will be appended to the prior chunk
\end{chunk}
I've been struggling with keeping track of research experiments and code at the same time. This seems pretty cool! I like how this method is language agnostic and uses "matured" tools. Question: I'd love to give this a try; do you have any public code snippets?
Even if LaTeX3 brought some nice things to the TeX world, TeX programming is every year lagging more and more behind modern programming languages and practices. SILE has a long way to go before it can compete with math-heavy publications made with TeX, but I think it is already a good solution for typesetting technical documentation or prose fiction.
Better tools and formats have existed for sometime, the problem remains adoption. Part of the issue with mentioning a flawed tool is the supporters who come out of the woodwork to insist that the sharp edges are required, and anything more user friendly would needlessly curtail the power they require.
Personally, I have given up on Latex and write documents in pandoc markdown which I can then convert to pdf through the intermediary latex.
For prose fiction, I don't see much here that is going to drive people away from Indesign. Designers and typesetters aren't really interested in doing that kind of work in a system that would require generating new output for every correction. Even relatively simple tasks like balancing columns in a spread would become a pain.
The whole point is you shouldn't have to manually balance columns in a spread. Obviously a round trip from source to output to fiddle with balancing every spread would be tedious. But if you can say "For this content I want to use spread X defined as this shape and balanced in this way" and then be able to fiddle with your content and have your spread always work, then where is the tedium? That's what SILE is supposed to do.
Many publisher's workflows involve a round of content editing bouncing a word file back and forth, then a period where a typesetter uses InDesign or similar to lay it all out, then it goes to press. You can't keep copy-editing after the designer takes over. With a workflow using source documents in Markdown and typesetting handled by SILE I am able to allow copy-edits to book manuscripts up until minutes before going to press.
Does it come with a context-free syntax? If not, that would be a grave mistake. Being unparseable is one of the biggest problems of TeX, IMO. Not the least because you cannot automatically work with TeX "documents".
TeX is a program for typesetting, and IMO if one is a programmer, it is best used as a target or output format: do all your "document" stuff somewhere else (don't use LaTeX), and use TeX just for typesetting. You can have your document in some parsable format, but not require it of the output .tex file, just as we expect high-level programming-language source code to be parsable but don't really expect to get much utility out of parsing assembly or object code.
Knuth initially added macros to TeX just as a convenience to save some typing here and there, and the tradition of (ab)using that macro ability as a language in which to program, inaugurated in a small way by Knuth himself and later taken to dizzying heights by others (such as Leslie Lamport with LaTeX, and hundreds/thousands of assorted "package" authors since, and the LaTeX team currently), is IMO a mistake.
If you use TeX just for typesetting, it is quite a nice program that does what you want, and has very good error messages (yes) and debugging output.
> TeX is a program for typesetting, and IMO if one is a programmer, it is best used as a target or output format
That's the only thing I ever found it to be good for. Even then, it's a PITA. The world would not be a poorer place if it disappeared and was replaced something that that separated style definition, markup, and content in a reasonable fashion, and used modern language techniques (just a CFG would be a start) for the first two.
I work with TeX quite a lot and I can't even really understand intuitively the principles of the language design let alone the grammar. Like 15 years later and I'm still copy and pasting spells. I'd love to see an CST of my document and figure out why I'm writing the characters I'm writing!
My experience with TeX is a complete opposite. I feel like I can do pretty much anything in plain TeX, with a very good feel of what it is doing in its 'guts'. I wrote parsers in compilers in pure TeX (the latter was admittedly a silly academic exercise). It took a very thorough reading of 'The TeXbook' but after that I am constantly amazed at how deliberate and tasteful Knuth's choices were. TeX is simply ... elegant. I am not sure what 'modern' bells and whistles TeX lacks that everyone is so eager to see. CSS is a nightmare, even with recent additions, it seems clunky, most of its choices look like an afterthought.
It is also somewhat puzzling that same people who advocate for a better grammar would also push the idea of a better markup. So which is it: a better programming experience or a pure markup language that one wants? I feel TeX strikes a near perfect balance here. Most importantly, TeX has stayed nearly unchanged at its core for almost forty years! It is its greatest strength, not weakness. I cannot see what latest fads can improve in TeX
> I work with TeX quite a lot and I can't even really understand intuitively the principles of the language design let alone the grammar.
Read the TeXbook. No, really, it is a beautifully written text. The program it describes is quite elegant. Most of the complexity comes from modern LaTeX packages, not from the core tool.
TikZ is the worst for this. I don't usually have any problem picking up esoteric programming languages, but whatever the hell is going in with TikZ/TeX will be forever beyond my comprehension.
It's damning that there no truly functional grammar or spellchecker for latex, even ignoring crazy things that can be done with macros.
If some billionaire wanted to 'move the needle' and change society for the better I think building a truly better latex would be worth it. Each year some of the most brilliant people in the world sacrifice on the altar of latex, that could instead be spent on more productive research.
> Each year some of the most brilliant people in the world sacrifice on the altar of latex, that could instead be spent on more productive research.
Perhaps. But I'd wager that more time is lost to the bureaucracy of universities and grant administration than to fighting LaTeX quirks. I'm not sure that "fixing" (La)TeX would result in a noticeable improvement in research that could be done.
I'd like to know what a "better LaTeX" means, and how that would move the needle, given that the only things you can't easily do in modern (Xe)LaTeX are things that you shouldn't be using TeX itself for in the first place, but have other tools do so you can simply embed their output. Especially given the availability of WYSIWYG-esque editors with "one-click" compile buttons that run all the nonsense that we used to have to do by hand back in the dark days of "you want a GUI? Get out of my office, you simp" paired with the modern "just install missing packages without asking me, that's the whole point: I do the writing, you make that frictionless."
This is a red herring, I assert. There is no "boiling the problem down to a simple grammar" that has promise for a total solution. There are plenty of ways to combine the different parts that work well, and even more ways to combine them into nonsense. Disallowing the nonsense is probably not going to make the ways that work well any better.
Yes it comes with its own context-free syntax that (despite some surface resemblance to TeX) is quite easy to parse. It can also ingest XML and some other formats.
TeX is one of those bits of software that's so complex to replace, it'll take another few decades (and likely a few false starts) before we can get there.
The fundamentals of TeX's typesetting are amazing, but everything else is just bolted on. You'd probably want to rebuild it with its own, custom language, perhaps inspired more by modern XML/markdown rather than the old TeX language.
The biggest problem that we're going to have moving past TeX is that it's something of a standard for math representation in text. MathML never _really_ took off. Re-training millions of people skilled in typesetting math is going to be _tough_.
TeX itself is not really complex. The implementation is fine. The hard part is the ecosystem.
Any contender is going to face a daunting task of explaining why adopters should throw away 50 years of collective work of thousands of people that went into TeX packages in CTAN. Math notation is at best 1% of the reasons why people use TeX. Packages are.
I think the argument comes down to complexity. Getting Markdown --> Pandoc featureset up to a reasonable fidelity is the next stage IMO, especially if Markdown can be made to support the most common downloaded TeX extension packages.
I never figured the purpose of MathML. It's ugly/impractical to type as a text-based format. Verbose and inefficient compared to a binary format. A middle ground some people might like, but I can't stand.
I'd prefer one of these:
- A human-writable text format that's displayed in a formatted way, eg TeX or MathJax
- A binary format that represents structs and enums in code with a clear documentation of how it's packed and unpacked. That can be formatted for reading, and perhaps has a TeX-like API (or is just constructed programmatically using an API)
MathML, in contrast, provides neither the speed and small size of a binary format, while being unreadable and unwriteable.
I think it was designed as a low-level browser standard, not an authoring language. It is supposed to provide a stable, standardized target for authoring languages to compile to. Think of it as WASM for math markup. It makes sense if languages for authoring math is an unsettled design space where we need diversity and experimentation. As a browser standard, like WASM, MathML should enable people to experiment with new ideas and hopefully evolve a better authoring language than we would have got if we had tried to invent a language and standardize it at the same time.
Ideally there would be a replacement of TeX that shares little of its non-math typesetting capabilities but retains (or only slightly modifies) the math syntax. The rest of the system is where the the problems lie, or at least where I waste a lot of time on little things.
I always had the hope that Lout[0] would take over LaTeX: very clean & reasonably small implementation, functional programming instead of macros. I guess it's just hard to move against the inertia that (La)TeX carries.
I've had a look at Lout years ago, but I think the development stalled a bit and the source wasn't that inviting for contributors. So the PDF support came (too) late and still isn't feature-complete with the PS output.
I am still fond of the Scribe[1]-like syntax, but then again, I also liked it in Texinfo or Borland Sprint.
I did typeset in masters thesis in ConTeXt. It worked okay enough, but for me, it didn't provide any features that make up for the increased time spent searching for stuff because there's less documentation and a much smaller community.
I also used it to typeset a project report which had some msword specific formatting rules. I found it more customizable than LaTeX in that regard, but I did have to read the docs a lot more.
Readit News
Code it, explain it, generate a Literate PDF containing the code.
The programming cycle is simple. Write the code in a latex block. Run make. The makefile extracts the code from the latex, compiles it, runs the test cases (also in the latex), and regenerates the PDF. Code and explanations are always up to date and in sync.
I have found no better toolset.
(Unfortunately, the package that lets each Org source block behave as though it was using the corresponding language's Emacs major mode - poly-org-mode - has a ton of bugs. It was part of why I stopped using literate programming entirely for later projects.)
[1] https://orgmode.org/ [2] https://github.com/jingtaozf/literate-lisp [3] https://github.com/jingtaozf/literate-elisp [4] https://contrapunctus.codeberg.page/blog/literate-programmin...
I tried orgmode. I even attended a course at CMU that used it for the "live notes". It is excellent for teaching. But it has the same flaw as the "live notebook" idea. There is no generally accepted structure to the approach.
Everyone "understands" books. They have a preface, chapters, an index, a bibliography, pictures, credits, and a table of contents. Literate programs leverage that shared understanding of the structure.
Think of a physics textbook. If you just copy every equation out of the book then you "have the code". All the rest is explanation. They belong together so the explanation and equations (code) are intermingled.
I'm a "primitivist". I work in straight text at an emacs buffer in fundamental mode.
The point of my code is to "talk to the machine". The point of my literate program is to "explain to other programmers (mostly 'future' me)". Note that this is NOT DOCUMENTATION. It is explanation, best presented in book form.
- simple things became uselessly lengthy and prolix
- complex things became too big to be easy read, while the equivalent "pure-code with comments" and a small new developers doc while fail to give an equally deep knowledge of the code base is FAR more quick and easy accessed.
Maybe it's my style, I do not know but while I do not feel LaTeX as painful, since most my docs are of the same kind so a template/class produced once with calm get reused issueless, I can't really digest literate programming...
I am currently a physics student and one of the main problems that I have day-to-day coding is being unable to include diagrams/equations/pictures in the source code. Moreover, because there is a fair amount of 'domain complexity' the comments are often longer than the code itself.
The package is still very much unstable and not even close to being ready for any sort production. However, I did use it for my masters thesis [2], and it was quite an interesting (in a good way) experience. (I recommend everyone to actually try to write literate code at least once).
[1] https://github.com/Skipp1/fortex [2] https://github.com/Skipp1/mono-rich
For the last two evenings, I have been revisiting my old manuscript materials because I am thinking of updating the material and also creating additional editions for more programming languages: Python, JavaScript, and maybe Swift and Prolog.
I had forgot how cool TeX and LaTex are. It was very easy to start working with again, even after a 12 year gap.
Lisp program to extract latex chunks: https://github.com/daly/axiom/blob/master/books/tangle.lisp
C program to extract latex chunks: https://github.com/daly/axiom/blob/master/books/tanglec.c
Note that the C program is just a hand translation of the Lisp code.
The lisp code has an explanation and the necessary latex macros. The idea is to scan the latex, find each named code 'chunk', and add each one to a hash table. Then the hash table is scanned to dump the requested chunk to stdout. For example:
\begin{chunk}{part1} code for part 1 \end{chunk}
Ordinary latex code between chunks.
\begin{chunk}{part2} code for part 2 \end{chunk}
\begin{chunk}{part1} this code will be appended to the prior chunk \end{chunk}
\begin{chunk}{getall} \getchunk{part1} \getchunk{part2} \end{chunk}
Assuming the above chunks are in the file foo.tex then
tanglec foo.tex getall >getall.code
will print out the named chunk (getall). For an individual chunk use
tanglec foo.tex part2 >getsome.code
This is (most of) the source code for Axiom. The code is extracted from the latex for these pdfs.
Personally, I have given up on Latex and write documents in pandoc markdown which I can then convert to pdf through the intermediary latex.
Many publisher's workflows involve a round of content editing bouncing a word file back and forth, then a period where a typesetter uses InDesign or similar to lay it all out, then it goes to press. You can't keep copy-editing after the designer takes over. With a workflow using source documents in Markdown and typesetting handled by SILE I am able to allow copy-edits to book manuscripts up until minutes before going to press.
• https://www.youtube.com/watch?v=5BIP_N9qQm4 [FOSDEM 2015, "Introducing SILE: A New Typesetting System"]
• https://www.youtube.com/watch?v=t_kk20vlamo [GRANSHAN Conference 2015, "Global typesetting with SILE"]
Knuth initially added macros to TeX just as a convenience to save some typing here and there, and the tradition of (ab)using that macro ability as a language in which to program, inaugurated in a small way by Knuth himself and later taken to dizzying heights by others (such as Leslie Lamport with LaTeX, and hundreds/thousands of assorted "package" authors since, and the LaTeX team currently), is IMO a mistake.
If you use TeX just for typesetting, it is quite a nice program that does what you want, and has very good error messages (yes) and debugging output.
That's the only thing I ever found it to be good for. Even then, it's a PITA. The world would not be a poorer place if it disappeared and was replaced something that that separated style definition, markup, and content in a reasonable fashion, and used modern language techniques (just a CFG would be a start) for the first two.
It is also somewhat puzzling that same people who advocate for a better grammar would also push the idea of a better markup. So which is it: a better programming experience or a pure markup language that one wants? I feel TeX strikes a near perfect balance here. Most importantly, TeX has stayed nearly unchanged at its core for almost forty years! It is its greatest strength, not weakness. I cannot see what latest fads can improve in TeX
Read the TeXbook. No, really, it is a beautifully written text. The program it describes is quite elegant. Most of the complexity comes from modern LaTeX packages, not from the core tool.
If some billionaire wanted to 'move the needle' and change society for the better I think building a truly better latex would be worth it. Each year some of the most brilliant people in the world sacrifice on the altar of latex, that could instead be spent on more productive research.
Perhaps. But I'd wager that more time is lost to the bureaucracy of universities and grant administration than to fighting LaTeX quirks. I'm not sure that "fixing" (La)TeX would result in a noticeable improvement in research that could be done.
The fundamentals of TeX's typesetting are amazing, but everything else is just bolted on. You'd probably want to rebuild it with its own, custom language, perhaps inspired more by modern XML/markdown rather than the old TeX language.
The biggest problem that we're going to have moving past TeX is that it's something of a standard for math representation in text. MathML never _really_ took off. Re-training millions of people skilled in typesetting math is going to be _tough_.
Any contender is going to face a daunting task of explaining why adopters should throw away 50 years of collective work of thousands of people that went into TeX packages in CTAN. Math notation is at best 1% of the reasons why people use TeX. Packages are.
I'm not sure if we're getting hung up on the word "complex", but here's 60+ pages on paragraph line breaking alone:
http://www.eprg.org/G53DOC/pdfs/knuth-plass-breaking.pdf
Digital Typography is 700 pages long (not everything is going to be TeX fundamentals, but...):
https://archive.org/details/digitaltypograph0000knut/page/n7...
I'd prefer one of these:
MathML, in contrast, provides neither the speed and small size of a binary format, while being unreadable and unwriteable.As far as I could get, Sile is exactly that.
[0]: https://en.wikipedia.org/wiki/Lout_(software)
I am still fond of the Scribe[1]-like syntax, but then again, I also liked it in Texinfo or Borland Sprint.
[1]: https://en.wikipedia.org/wiki/Scribe_(markup_language)
https://en.wikipedia.org/wiki/ConTeXt
It appears that there is no development in ConTeXt for a while now. I haven't checked it, but it seems that they are working on LuaMetaTeX.
http://www.pragma-ade.com/luametatex-1.htm