Scott Bell wrote a lot of that. I was fascinated by the self-hosting technique. The whole repo is so small that you can almost dismiss it as some kind of academic toy, if you don't realize it's a full-featured production-grade lisp.
(Not to take anything away from Fennel! The more lisp, the better.)
Thanks for posting this. I took a look at Fennel a few months ago and although they list 2 web frameworks on the front page, there's little documentation (that may have changed since then) about how to interface with other Lua projects. I kept running into things I couldn't figure out when working with the existing Lua ecosystem and projects.
I’ve been using it quite successfully in a large OpenResty project.
While you’re right that there aren’t any docs on working with any of the 3rd party projects listed in the homepage - I found that it was pretty straight forward to use as a drop in Lua replacement by examining the compiled Lua from the online REPL. I was surprised how clean the Lua output is. There is really no magic here. I believe one of the project goals is to stay as close to Lua semantics as possible. Basically just lisp flavored Lua.
In my project I just use the compiled .fnl files as I would any of other .lua file.
Is something specific you’re getting stuck with? I’d be happy to try to point you in the right direction.
It’s really nice! I encourage you to give it another try.
Lua is way more common for game scripting than Python. One of the few major games which used Python I can think of was Civ IV, and Civ V switched to Lua.
Lua is smaller, easier to embed interface / interact with (e.g. it's way less ceremony to expose a native function to Lua than to Python), and tends to be faster, especially when jumping back and forth between the engine and the scripting. Embedding / scripting (within a larger program) is the original use-case of Lua, not so Python.
I believe it's also much easier to secure / sandbox (remove bits you don't want script writers to have access to) as well, the stdlib is smaller and I think it has less interactions between modules.
Python embedding / scripting tends to be more common for software where the securing / restriction aspect is smaller but flexibility & larger embeds are necessary e.g. 3D, CAD and other "production pipeline" software tends to be scripted with Python.
> I believe [Lua is] also much easier to secure / sandbox [than Python]
Yes. C code that embeds the Lua VM has total control over which functions are exposed to the Lua code. It's possible to create a VM that, for example, has no access to the filesystem.
AFAIK it's not possible to do that if you embed Python. Dangerous functions like `open` are always available and sandboxing facilities have to try and prevent untrusted code from getting a reference to them. Unfortunately, there are numerous ways to work around these restrictions and obtain access to dangerous functions, e.g. via `__builtins__`.
Another reason running untrusted Lua code is considered fairly safe is that the VM is well-engineered and has a history of very few bugs [0]. However, the latest 5.4.0 release seems to have more bugs than older releases.
It might still possible for untrusted Lua code to use 100% CPU and hang a program, or to read data from the embedding process using a Spectre-style attack (although even that's unlikely because the Lua VM is an interpreter rather than a JIT compiler). However, it's quite possible to secure an embedded Lua VM to prevent it from doing things like accessing arbitrary files.
I've made a suite of digital musical instruments for Android phones in Lua. Now I'm using same language to build some VR prototypes.
Here are some benefits of Lua over Python. It's really powerful language (anonymous functions, closures, tail-call optimization, full lexical scoping, coroutines...) that supports many paradigms, and yet it is tiny and elegant. There are few surprises and everything is explicit.
Runtime is easily embeddable, which means it will be more readily available on exotic platforms. The interpreter is much faster than Python. C code is effortless to call into with FFI and resulting code looks just like normal Lua.
Not everything is roses. I prefer readability and less verbosity of Python. The 1-indexing needs some getting used to. Batteries are not included (to be more portable), so there are dozens of implementations of basic things like serialization, OO classes and table copying. Most online material (wiki) was written very long ago and it's full of language proposals that never succeeded. Some valuable resources can be found in documentation of hosting frameworks (Defold, Solar2D, Roblox, ComputerCraft, WoW).
I wrote a console-based email-client, using Lua for all configuration and scripting needs.
Using Lua is generally done because it is easy to embed inside a host-application. While it is possible to embed Python, Perl, or other languages, they're relatively heavyweight and not so commonly used in that case.
In my own application I always felt annoyed that mutt didn't have "real" scripting. Just an ad-hoc configuration that made lots of things possible, but neglected some basics (such as loops and similar).
Configuring a mail-client in Lua was a nice exercise, but eventually I moved on to pay for gsuite rather than self-hosting a mailserver of my own so it became a "done" project. Definitely a useful learning experience though, experimenting with user-interface, embedded scripting, and going through lots of learning relating to MIME-handling & etc.
Your project sounds cool - is it possible to process email with Lua scripts, so that I could use a mailbox as a business process queue that is also user-friendly, while having a suite of Lua apps processing the mailboxes? If so, care to share details of your project if its open/available?
Lua is a tiny, relatively simple language with a simple data model built on a small runtime that's very fast for an interpreter and easy to embed into C/C++ applications (once you dealt with the stack, mentally or otherwise). Startup time is ~nil.
Python is a large, complex language with an exceedingly complicated data model built on a runtime where the core interpreter is about ~4 MB, which is also a pretty slow interpreter (mostly due to the exceedingly complicated data model). Python is certainly not hard to embed on an API level, but somewhat annoying to package up for a build. Python is very hard to sandbox, whereas Lua is basically sandboxed by default. Startup time is on the order of ~100-200 ms.
If I use Python instead of Lua for monte carlo tree search, my machine will page out, and even before it pages out, my training pipeline will be running dozens of times slower than it was with Lua.
If I use Python instead of Lua to make a game, I will not be able to update my game state and render all my images at 60hz.
If I use Python instead of Lua to write programs that do many tasks at the same time, I will have to write "await" a thousand times instead of 0 times.
the easiest way to put it is that you embed Lua into your app (say scripts that are loaded at startup or even as a command line) while it’s more natural to embed your external functionality into Python (e.e. numpy).
It’s simpler to connect lua to your c/c++ datastructures than it is to connect Python, and it’s more complicated to write little things in Python.
Sometimes it’s easier to grab a pencil and scribble a little note on a piece of paper. Sometimes it’s a lot more useful to type a note into your phone or computer than to deal with paper.
I have embedded lua in Spring Boot webapps where I required an external rules files which could be easy to read for business folks.
Example: Specialised handling for different customer types and you don't want to manage all the rules in the RDBMS but would rather run a separate script for each customer type and have this script in a plain text format and version managed using Git.
Sounds very interesting! Any chance this is open source? If not, can you give a bit of additional information on how you did this (including project structure, some insight in the lua code, etc)?
I use it at work to process SIP messages on behalf of a cellular carrier. The actual parser is written using LPEG, a parser expression grammar for Lua. Because Lua is meant to be embedded, it only has the functionality required to do the job and no more.
This and a recent feature addition to neovim (passing lua closures to vimscript functions [1]) should make a good substitute for elisp to configure neovim.
I don't see it mentioned but there is a neovim plugin called conjure [1] that is written in fennel. This is a general purpose REPL client that works for Clojure (I use it daily for Clojure and ClojureScript) but also for fennel itself. So you can have a really nice neovim plugin dev experience using this tool. Conjure works with fennel out of the box, but if you want to go deeper you may be interested in another plugin called aniseed [2] which provides interesting facilities for the fennel language and that is used internally by conjure.
Disclaimer: I am not the author of those tools, just a random Clojure dev.
Readit News
I'd like to see him visit this thread.
https://news.ycombinator.com/user?id=technomancy
\s .... but not really. I need to figure out a way to tone down my mind wandering off.
https://technomancy.us/190
https://lobste.rs/u/technomancy
> Clojure's license makes it a complete non starter for me, sorry. Live free or die.
when at first glance fennel looks a lot like clojure e.g. `fn`, square brackets, {} for maps / table literals.
Aren't those mostly inherited from lua and present in a lot of different languages invented before and after clojure?
https://fennel-lang.org/v0.6.0/
Scott Bell wrote a lot of that. I was fascinated by the self-hosting technique. The whole repo is so small that you can almost dismiss it as some kind of academic toy, if you don't realize it's a full-featured production-grade lisp.
(Not to take anything away from Fennel! The more lisp, the better.)
edit: Looking at the git history is very educational for how to bootstrap a self hosting system like this. Fascinating.
Deleted Comment
While you’re right that there aren’t any docs on working with any of the 3rd party projects listed in the homepage - I found that it was pretty straight forward to use as a drop in Lua replacement by examining the compiled Lua from the online REPL. I was surprised how clean the Lua output is. There is really no magic here. I believe one of the project goals is to stay as close to Lua semantics as possible. Basically just lisp flavored Lua.
In my project I just use the compiled .fnl files as I would any of other .lua file.
Is something specific you’re getting stuck with? I’d be happy to try to point you in the right direction.
It’s really nice! I encourage you to give it another try.
Lua is smaller, easier to embed interface / interact with (e.g. it's way less ceremony to expose a native function to Lua than to Python), and tends to be faster, especially when jumping back and forth between the engine and the scripting. Embedding / scripting (within a larger program) is the original use-case of Lua, not so Python.
I believe it's also much easier to secure / sandbox (remove bits you don't want script writers to have access to) as well, the stdlib is smaller and I think it has less interactions between modules.
Python embedding / scripting tends to be more common for software where the securing / restriction aspect is smaller but flexibility & larger embeds are necessary e.g. 3D, CAD and other "production pipeline" software tends to be scripted with Python.
Yes. C code that embeds the Lua VM has total control over which functions are exposed to the Lua code. It's possible to create a VM that, for example, has no access to the filesystem.
AFAIK it's not possible to do that if you embed Python. Dangerous functions like `open` are always available and sandboxing facilities have to try and prevent untrusted code from getting a reference to them. Unfortunately, there are numerous ways to work around these restrictions and obtain access to dangerous functions, e.g. via `__builtins__`.
Another reason running untrusted Lua code is considered fairly safe is that the VM is well-engineered and has a history of very few bugs [0]. However, the latest 5.4.0 release seems to have more bugs than older releases.
It might still possible for untrusted Lua code to use 100% CPU and hang a program, or to read data from the embedding process using a Spectre-style attack (although even that's unlikely because the Lua VM is an interpreter rather than a JIT compiler). However, it's quite possible to secure an embedded Lua VM to prevent it from doing things like accessing arbitrary files.
[0] https://www.lua.org/bugs.html
Here are some benefits of Lua over Python. It's really powerful language (anonymous functions, closures, tail-call optimization, full lexical scoping, coroutines...) that supports many paradigms, and yet it is tiny and elegant. There are few surprises and everything is explicit.
Runtime is easily embeddable, which means it will be more readily available on exotic platforms. The interpreter is much faster than Python. C code is effortless to call into with FFI and resulting code looks just like normal Lua.
Not everything is roses. I prefer readability and less verbosity of Python. The 1-indexing needs some getting used to. Batteries are not included (to be more portable), so there are dozens of implementations of basic things like serialization, OO classes and table copying. Most online material (wiki) was written very long ago and it's full of language proposals that never succeeded. Some valuable resources can be found in documentation of hosting frameworks (Defold, Solar2D, Roblox, ComputerCraft, WoW).
Using Lua is generally done because it is easy to embed inside a host-application. While it is possible to embed Python, Perl, or other languages, they're relatively heavyweight and not so commonly used in that case.
In my own application I always felt annoyed that mutt didn't have "real" scripting. Just an ad-hoc configuration that made lots of things possible, but neglected some basics (such as loops and similar).
Configuring a mail-client in Lua was a nice exercise, but eventually I moved on to pay for gsuite rather than self-hosting a mailserver of my own so it became a "done" project. Definitely a useful learning experience though, experimenting with user-interface, embedded scripting, and going through lots of learning relating to MIME-handling & etc.
Python is a large, complex language with an exceedingly complicated data model built on a runtime where the core interpreter is about ~4 MB, which is also a pretty slow interpreter (mostly due to the exceedingly complicated data model). Python is certainly not hard to embed on an API level, but somewhat annoying to package up for a build. Python is very hard to sandbox, whereas Lua is basically sandboxed by default. Startup time is on the order of ~100-200 ms.
If I use Python instead of Lua to make a game, I will not be able to update my game state and render all my images at 60hz.
If I use Python instead of Lua to write programs that do many tasks at the same time, I will have to write "await" a thousand times instead of 0 times.
It’s simpler to connect lua to your c/c++ datastructures than it is to connect Python, and it’s more complicated to write little things in Python.
Sometimes it’s easier to grab a pencil and scribble a little note on a piece of paper. Sometimes it’s a lot more useful to type a note into your phone or computer than to deal with paper.
Example: Specialised handling for different customer types and you don't want to manage all the rules in the RDBMS but would rather run a separate script for each customer type and have this script in a plain text format and version managed using Git.
[1] https://github.com/neovim/neovim/pull/12507
Disclaimer: I am not the author of those tools, just a random Clojure dev.
[1] https://github.com/Olical/conjure
[2] https://github.com/Olical/aniseed
https://news.ycombinator.com/item?id=18016168