I was having some difficulty figuring out how Hy actually is translated to Python (and wasn't even sure if it was compiled or interpreted). Eventually I found on Wikipedia the following:
> Hy is a dialect of the Lisp programming language designed to interact with Python by translating s-expressions into Python's abstract syntax tree (AST).
I kind of wish this was made more clear on the main website. Perhaps, instead of introducing Hy as "a Lisp dialect that's embedded in Python", introduce it as "a Lisp dialect that compiles to Python's AST". The words "embedded in Python" don't make it very clear just how it's embedded into Python. The various ways you can embed a Lisp look very different and have very different tradeoffs.
For example, off the top of my head, I could "embed" a Lisp by writing an interpreter (in C if I care about performance) and letting it be called from Python, perhaps passing in a Python list instead of a string to make it more "native". Or I could "embed" a Lisp by compiling to Python bytecode. Or I could "embed" a Lisp by translating it directly to Python source code. Etc.
> Hy is a Lisp dialect that's embedded in Python. Since Hy transforms its Lisp code into Python abstract syntax tree (AST) objects, you have the whole beautiful world of Python at your fingertips, in Lisp form.
> The various ways you can embed a Lisp look very different and have very different tradeoffs.
Hy itself provides options. Typically the process is that the Hy source code becomes Python AST objects, which Python then complies and executes, but you can also translate the Python AST objects into Python source text. Or you can use Python from Hy or vice versa: https://hylang.org/hy/doc/v1.0.0/interop
The "embed" part stems from the fact that you can mix Python and Hy in a project with bi-directional calling. Works great, because it is all Python byte code in the end.
The original hy annoucement makes it clear that they embed a Lisp by compiling with Python bytecode. You can see it in the following video about the 16:25 mark
I would like to make the observation that as Hy matured over the years, instead of accumulating syntactic sugar and special cases to grow more Lispy, less Pythony, it seems to have generally gone the opposite way. That is, becoming a thinner syntactic abstraction of Python's feature set, focusing on the essentials that cannot be emulated in any other way (macros)
A few examples from recent releases:
- "match" is just native Python "match" -- it doesn't even polyfill for pre-3.10 Python versions (in the TypeScript world this would be unthinkable)
- "foo?" used to mangle to "is_foo" as a special case, but this has been removed
- "hy.eval" has been overhauled to be more like Python's "eval"
- nice-to-have but non-essential utilities ("unless") get often pushed out into the Hyrule package
For me this direction was counter-intuitive at first, but it has some very nice outcomes; for one, it simplifies the learning curve when coming over to Hy from Python, and it makes it easier to consistently interact with Python packages (arguably the main reason to use Python in the first place!)
Or maybe it's just a matter of simplyfing maintenance of the language; IIRC, "let" took like 4 attempts to get right :)
In any case, congratulations on this great milestone!
Yeah, at a certain point I realized that both the maintenance and the use of the language became much slicker if unnecessary deviations from Python were minimized. After all, when I'm writing Hy code, I'm usually spending a lot more time referring to the documentation of Python or third-party Python libraries than the documentation of Hy. I felt there were a number of ways Python could be improved upon, but e.g. the old feature that let you spell `True` as `true` in deference to Clojure was just a needless complication.
It is true that Hy really shines in those cases where it adopts an existing Python feature and adds meaningful quality-of-life improvements: anonymous functions without limitations; multiple iteration in for-loops; relaxed character set for identifiers. Things that seem completely obvious, once you have them.
It also demonstrates that elegance in a Lisp-on-Python is reached in a very different way than elegance in a stand-alone language, since it becomes an art of making the best out of what is already there.
At long last! Now I can finally clean up https://github.com/rcarmo/sushy (I've been poking at it over the years, but every time I upgraded hy portions of the syntax broke, or things would get moved in and out of the hyrule package, etc.)
By the way, Hy works really well inside https://holzschu.github.io/a-Shell_iOS on the iPad, although the syntax highlighting in vim/neovim needs to catch up to the 0.29+ releases and async.
Although I've tried using Fennel and Guile instead over the years, having access to Python libraries and ecosystem is preferable to me, and with async I can do some very nice, efficient API wrangling (doing HTTPS with fine-grained control over socket re-use and headers remains a pain in various Schemes, so I very much prefer using aiohttp)
Wow! It has come such a long way since its early, humble beginnings.
I saw the original lightning talk that introduced Hy to the world at Pycon those ages ago. Soon after I met Paul and started contributing to the early versions of Hy. I was responsible for the CL-style kwargs (you’re welcome), some minor innards, and a library or two.
Whimsy is useful, especially to keep enthusiasm up. It’s nice when hackers can be hackers and not every thing is business.
While I haven’t been involved in years it brings a smile to me face to see the project continues apace. What a great milestone!
I played around with Coconut many years ago and my impression was that the compiler was not smart enough to be useful. The generated code had a big pile of helper functions hardcoded at the top, and the program was much slower than the equivalent plain Python.
By contrast Hy generates Python code that is very close to what you might write by hand, apart from some indirection when it comes to scoping with `let` and some variations around returning values.
Maybe Coconut has improved though, it's been a long time.
1. Does it support REPL-driven development? (condition system, breakloop, etc.)
2. Is there a standalone distribution? Distributing python in itself is a hassle, ideal situation would be to simply distribute a single Hy binary that contains all dependencies within it (either statically linked or as a zip file extracted in tmp directory).
1. I don't know what a breakloop is. Hy uses Python's exception system, which is more like a traditional exception system than Common Lisp's condition system.
A breakloop is a REPL operating in the context of condition handling. When a condition is signaled, you can use the breakloop to modify state and direct how the condition should be handled (including fixing something local and letting the current function proceed by ignoring the condition).
Seems like that would only be doable by altering CPython to at least have a hook in the initial exception processing (or maybe there is some magic double-underscore thing for that already?).
1. It supports the same set of features that python supports, which is pretty good when it comes to things like traditional step through and postmortem debugging. And CPython Supports a lot of internal hooks if you want to do really advanced dark magic. But it doesn't have anything like the condition system or handlers/restarts.
Readit News
Also, looking at the code on Github suggests this compiler is written in Python (see https://github.com/hylang/hy/blob/master/hy/compiler.py).
I kind of wish this was made more clear on the main website. Perhaps, instead of introducing Hy as "a Lisp dialect that's embedded in Python", introduce it as "a Lisp dialect that compiles to Python's AST". The words "embedded in Python" don't make it very clear just how it's embedded into Python. The various ways you can embed a Lisp look very different and have very different tradeoffs.
For example, off the top of my head, I could "embed" a Lisp by writing an interpreter (in C if I care about performance) and letting it be called from Python, perhaps passing in a Python list instead of a string to make it more "native". Or I could "embed" a Lisp by compiling to Python bytecode. Or I could "embed" a Lisp by translating it directly to Python source code. Etc.
Regardless, interesting project!
> Hy is a Lisp dialect that's embedded in Python. Since Hy transforms its Lisp code into Python abstract syntax tree (AST) objects, you have the whole beautiful world of Python at your fingertips, in Lisp form.
Yes, that's right. Hy is not self-hosted.
> The various ways you can embed a Lisp look very different and have very different tradeoffs.
Hy itself provides options. Typically the process is that the Hy source code becomes Python AST objects, which Python then complies and executes, but you can also translate the Python AST objects into Python source text. Or you can use Python from Hy or vice versa: https://hylang.org/hy/doc/v1.0.0/interop
https://m.youtube.com/watch?v=1vui-LupKJI
> ...because this is a frontend like LLVM or GCC that compiles instead of bytecode, uh, to Python AST, um, so this Lisp compiles entirely to Python
@ https://youtu.be/1vui-LupKJI?t=1020
A few examples from recent releases:
- "match" is just native Python "match" -- it doesn't even polyfill for pre-3.10 Python versions (in the TypeScript world this would be unthinkable)
- "foo?" used to mangle to "is_foo" as a special case, but this has been removed
- "hy.eval" has been overhauled to be more like Python's "eval"
- nice-to-have but non-essential utilities ("unless") get often pushed out into the Hyrule package
For me this direction was counter-intuitive at first, but it has some very nice outcomes; for one, it simplifies the learning curve when coming over to Hy from Python, and it makes it easier to consistently interact with Python packages (arguably the main reason to use Python in the first place!)
Or maybe it's just a matter of simplyfing maintenance of the language; IIRC, "let" took like 4 attempts to get right :)
In any case, congratulations on this great milestone!
It also demonstrates that elegance in a Lisp-on-Python is reached in a very different way than elegance in a stand-alone language, since it becomes an art of making the best out of what is already there.
[0] https://github.com/hylang/hy/discussions/2609
Hehe, clever.
By the way, Hy works really well inside https://holzschu.github.io/a-Shell_iOS on the iPad, although the syntax highlighting in vim/neovim needs to catch up to the 0.29+ releases and async.
Although I've tried using Fennel and Guile instead over the years, having access to Python libraries and ecosystem is preferable to me, and with async I can do some very nice, efficient API wrangling (doing HTTPS with fine-grained control over socket re-use and headers remains a pain in various Schemes, so I very much prefer using aiohttp)
I saw the original lightning talk that introduced Hy to the world at Pycon those ages ago. Soon after I met Paul and started contributing to the early versions of Hy. I was responsible for the CL-style kwargs (you’re welcome), some minor innards, and a library or two.
Whimsy is useful, especially to keep enthusiasm up. It’s nice when hackers can be hackers and not every thing is business.
While I haven’t been involved in years it brings a smile to me face to see the project continues apace. What a great milestone!
A Clojure-compatible(-ish) Lisp dialect targeting Python 3.8+ https://github.com/basilisp-lang/basilisp
I'd be seriously interested in hearing from people that have actually used any of these two and what their experience was.
By contrast Hy generates Python code that is very close to what you might write by hand, apart from some indirection when it comes to scoping with `let` and some variations around returning values.
Maybe Coconut has improved though, it's been a long time.
1. Does it support REPL-driven development? (condition system, breakloop, etc.)
2. Is there a standalone distribution? Distributing python in itself is a hassle, ideal situation would be to simply distribute a single Hy binary that contains all dependencies within it (either statically linked or as a zip file extracted in tmp directory).
https://docs.astral.sh/uv/guides/tools/#running-tools
(context: uv can install and manage python versions)
2. No, sorry.
Seems like that would only be doable by altering CPython to at least have a hook in the initial exception processing (or maybe there is some magic double-underscore thing for that already?).
>A convenient way to use this class to interactively debug code is to insert the following in the code you want to debug:
>Or in Python: >Note that as with `code.interact()`, changes to local variables inside the REPL are not propagated back to the original scope.