I've been making a language called TXR for ... oh, ten years this coming August now. It combines a Lisp dialect ("TXR Lisp") with a whole-document pattern-based extraction language ("TXR Pattern Language"). It takes the form of a simple executable with a few satellite files in your /usr/share.
Lots of "batteries" are included in TXR Lisp: object system with static and instance slots, single inheritance, GC finalization and a form of RAII, exception handling, delimited continuations, byte code compiler for virtual machine, file compilation, macros (of course), regexes, built-in lazy lists, a fantastic set of macros for partial application, a comprehensive declarative FFI, good amount of POSIX wrapage built-in, a decent REPL with history, completion, multi-line editing, visual copy-paste, ...
TXR Lisp has a concept similar to CL's "generalized places" implemented differently. It supports Lisp-1-style evaluation (higher order functions used without a funcall operator), in the middle of a Lisp-2 substrate. Traditional list operations are generalized so they work on other sequences; you can mapcar over a string and such.
First of all, thanks for doing this and making it free.
I do a lot of shell based work (Linux commands, Awk, Python, Perl, and Powershell) and Lisp has always interested me, so this may scratch an itch.
How easy is this to get setup? Just a few binaries?
Also, one barrier to me has always been learning lisp as it is very different from most traditional scripting languages. I took a look at your examples and it looks neat, but a little complex. Is there a tutorial document that goes over the main functionality and commands. Like with Linux, knowing grep,cut, sort, uniq, cat, and a few others can take you a long way. I should add that although I haven't coded enough lisp to get competent, I've read two books on Common Lisp and have played with Chez Scheme, Racket, and Clojure...so I get the concepts with CONS/CAR/CDR...macros...etc.
> How easy is this to get setup? Just a few binaries?
From sources, just unpack, ./configure (with --prefix=/your/dest/dir if you don't want it in /usr/local/bin), make, make tests, make install if you're going from source. The dependencies are very low. But one of them is libffi; ./configure doesn't complain about it missing, but you get a build with reduced functionality (which doesn't pass certain tests).
For every release I make tarball packages for for older Debian 5.x (64 bit), Ubuntu 12 (32 bit), Mac OS 10.7.4 (64) and Solaris 10 (x86, 32 bit) as well as Cygwin (32 and 64).
For native Windows, I make a installer. It registers TXR into your PATH environment variable and associates .txr and .tl file associations.
TXR installations are relocatable: TXR discovers its own path and then finds the satellite files relative to its own location; if you keep the relative locations of the pieces as they are, you can move it anywhere in your filesystem. That's also useful for deployment. In the TXR executable there is a 128 byte space where you can embed command-line arguments to the executable. There is a helper function for this called save-exe. https://www.nongnu.org/txr/txr-manpage.html#N-02850687
If you've read books on Common Lisp, you're in pretty good stead. My own Lisp background is heavily Common Lisp, and the CL influences in TXR Lisp are clear; yet it's not CL and will probably mess with your head if you know a bit of CL in ways that it wouldn't if you didn't know CL. The reference manual has some dialect notes in relation to CL, far and few between though.
I don't have a newbie tutorial, unfortunately; I'm hoping someone else will write that. Keeping the documentation complete and at a good level of quality has been a big effort; it's pushing close to 700 pages now if PDF-ed.
I'm quite satisfied with it. I put time into it that I was able, and no more than that; and for that time put into it, it is good.
I have a private TODO list, in point form, that is over 1300 lines long, so there is no shortage of stuff to do.
There was never supposed to be Lisp dialect, object system, FFI, or any of those things; it really was just a specialized text extraction tool.
A decade and 5200 commits later, it's interesting that I'm the only contributor (of any logic). There have only been doc fixes.
I also didn't think that a decade later, there would be more than 5200 commits, almost all of them from only me. There have been only some half dozen contributions from outsiders: most of them documentation issues, and something in the configure script. Third party code I've brought in, of which there isn't much, I took over as if they were my own. I've kind of gotten used to it being that way; if that ever changes, it will be big adjustment from "everything here is mine".
Racket is great if you are looking for a high quality general purpose language, but if you want to use important python libraries for pythons's popular use cases: numerical/scientific computing, data science/ai, etc., then Hylang is a drop-in replacement with the added benefit of Metaprogramming facilities (Lisp-2 macros) and Clojure-like semantics.
Python and Hy have 100% interoperability. Python can call Hy out of the box, and Hy can call Python out of the box.
Any Python programmer can pick up Hy basics in minutes, and will be productive in a matter of hours. If you are an intermediate level Python programmer and have experience with Lisp macros, you will be writing macros in no time as well. Writing macro wrappers for numerical/scientific applications can increase your code's signal to noise ratio by quite a lot.
Actually Hy is mentioned in the article, though the author found it didn't fit his usecase.
> Alas, Hy didn't really reach my dream. That macro expansion made debugging a nightmare as Hy would lose track of where the line numbers are; it wasn't until that when I really realized that without line numbers, you're just lost in terms of debugging in Python-land. That and Python didn't really have the right primitives; immutable datastructures for whatever reason never became first class, meaning that functional programming was hard, "cons" didn't really exist (actually this doesn't matter as much as people might think), recursive programming isn't really as possible without tail call elimination, etc
Debugging macro expanded code is definitely the number one pain with Hy, although a somewhat workable solution is to view the output of hy2py (comes with hy).
As far as the lack of functional programming semantics go (no tail call elimination is very sad indeed), there is no denying that either, but considering the applications I am recommending (c/c++ inter-op with python wrappers), you are hardly working in a functional domain in the first place.
A big use case for python is glue for high performance c/c++ code. How is racket at this? Also, is there a cython alternative for racket? I would also argue that, though racket may have more batteries included, there are definitely not as many externally developed libraries. Especially important are numpy/scipy/pandas/pysam, the list could go on...
That's not to say that I'm against racket, I really like the language and its level of design and documentation.
I've used the Racket FFI for some important tasks, and it's very easy to use. However, it's not as fully baked as the Python one (I've found bugs due to GC interactions) and it's weirdly slow (a few too many levels of wrapping). That said, I still use Racket extensively and enjoy it.
This is the write up I was looking for as this is the use case I've had in mind for Racket of late after reading Felleisen's LOP paper.
Would love to know more about your experiences with doing Racket FFI. How hard were the bugs around GC to discover and workaround? What were you building in Racket?
While it's nowhere near as mature as Racket, those interested in this use case might also be interested in Clasp, a Common Lisp implementation built on LLVM specifically to enable the combination of Lisp-style dynamic development approaches with C++ libraries.
FFI into C++ is a lot of work in any language. A lot of projects just define a plain C wrapper for the C++ code they want to call, and call that C code from whatever their glue language is.
However, Python does have CLIF[1], which is the nicest solution for calling into C++ that I'm aware of.
I hope this year it's videoed well. In past years except the last it's been fine, but last year I'm guessing whoever normally does it wasn't available for 2018.
It's one person doing the recordings and has been the same the last few years iiuc. They were just under-resourced I believe, and last year especially. They're doing a damn good job given the constraints they've been putting things together under I think.
While I'm impressed with racket on the whole, I can't quite agree to be impressed by the GUI bits. They're extremely limited, and once I got away from "how do I put a button on the screen" I didn't find the underlying implementation lent itself to being extended. In particular the drag and drop system only goes as far as file drops, and I quite quickly hit a brick wall trying to add anything more interesting.
In principle you can get pretty far with racket/gui despite it's limited amenities because you can roll your own widgets with canvas. However I too ran into wanting to create a drag source (not merely target) for files and couldn't find a reasonable way.
To be honest though, I think the biggest missed opportunity in racket is the single dispatch object system, used particularly by racket/gui. For some reason the core racket folks weren't smitten with CLOS and didn't go the Guile route of implementing something similar to it (Guile has GOOPS, which is related to Tiny-CLOS.)
The Racket class system has seemed fine to me for the GUI tookit (except for me not liking typing `send`). But if you find a limitation for that purpose, please post about it on the `racket-users` email list. The core developers are there. https://groups.google.com/forum/#!forum/racket-users/
FWIW, Racket (when it was still PLT Scheme) got a tiny-CLOS-alike around the time that Guile got GOOPS: Eli Barzilay made Swindle: https://docs.racket-lang.org/swindle/
CLOS is neat, and I'd be interested to see someone make a CLOS (maybe including a MOP?) for Racket, atop the modern Racket `struct` and other features. (If you tune it, try tuning for the tentatively forthcoming Chez backend to Racket, rather than for the current JIT. IIRC, Swindle was written before the JIT.)
Tk from python is pretty damn convenient, being as it's always there. Not the most modern toolkit for sure, but always being there is a pretty compelling feature for some GUI projects.
I am always confused when people talk about the language itself.
In my experience, python is used for Tensorflow, or Pandas, or Django, or Flask, or pytorch or something else that runs on top of it. Sometimes it is even more specialized and I need a wrapper for an API to let me talk to some web data. Maybe I need a crawler/scraper and a parser. There is a specialized language on top of the language.
So when someone says, oh this language is better with objects, or has some syntax thing or the other, or I can reason about it I am left confused.
Its like if I were talking to a professional shoe designer and I ask for hiking boots and they tell me that they're really into having at least two tones to offset the lace and the heels or something.
What am I missing? I want to reason about the language too, but doesn't that pale in comparison to being able to run a specialized library?
You're missing that people write mountains of code in Python and similar languages; they are not always just for a small amount of glue to gain access to some specialized libraries.
Well, I assume people write mountains of code in the library. If you're making a machine learning product, that is still a lot of work.
However, writing your own Tensorflow interface would take several human lifetimes to get it right, and Google already has provided it. So it seems that is not the part you would re-write no matter how good the language is.
> In my experience, python is used for Tensorflow, or Pandas, or Django, or Flask, or pytorch or something else that runs on top of it.
Is this your experience of using python or reading about it? I don't know your background so I apologize for making some assumptions about the source of your confusion. It sounds like you don't have a ton of experience programming, so let me start with a broadly: Python is a language in sense that English is a language, but these "something else that runs on top of it", are more like specialized vocabulary or jargon than "languages on top of the language".
English gives you the grammar/structure/spelling to communicate; it's a foundation. But it also gives you general vocabulary; adjectives and adverbs blend and interact with any new vocabulary that might come into play. It doesn't matter if its a poem, or novel, or a technical documentation, or a text book, there is still a lot of English-ness to it.
In the same way, Python as a language is still the substrate that each of those tools (Tensorflow, or Pandas, or Django, or Flask) are interacted with. I agree with what you're getting at, that maybe the tools are more important than the language. When people talk about their like for python the could talk about either: the language itself or the culture/ecosystem around the language. Some inherent to the language, so a quirk of it's history.
This applies as much to natural language. You might hear someone love the sound of Spanish or French, or praise the regularity of Latin spelling, or love Greek for the wealth of ancient, influential texts that it gives access to.
In the case of python you get a lot of praise from both angles. People love the language for it's ecosystem, sure; but also for how it does white spacing, its brevity, the specifics of its typing, where it does and dosn't need parentheses, REPLability, etc.
I am not offended that you think I may not program. That is fine. (I mean less than some, more than others. I have coded up the examples I brought up.)
But you haven't responded to the argument. If someone urges you to use Racket, and you have task in front of you (say, put up a website), it sort of matters whether Racket has a framework more than if it has brackets, indents or curly braces.
The amount of code available, and its quality, is one aspect of a language's ecosystem that most people take into account when discussing its merits.
It's quite possible to also discuss the language itself independently of how much code is available for it, while acknowledging that it's an important factor for many people.
Flask equivalents exist in every language. Tensorflow bindings are available in many. Pandas is more specific to Python, but using dataframes in Python vs R vs Julia each has a different feel.
I've found that Typed Racket is still fairly hard to use. For example, type classes aren't there, and type polymorphism interacts poorly with general-purpose functions like "equal?". I would guess that ML or Haskell code translates fairly naturally, but there aren't the guide-rails those languages have to keep you in the subset of the language that type-checks well.
Readit News
Lots of "batteries" are included in TXR Lisp: object system with static and instance slots, single inheritance, GC finalization and a form of RAII, exception handling, delimited continuations, byte code compiler for virtual machine, file compilation, macros (of course), regexes, built-in lazy lists, a fantastic set of macros for partial application, a comprehensive declarative FFI, good amount of POSIX wrapage built-in, a decent REPL with history, completion, multi-line editing, visual copy-paste, ...
TXR Lisp has a concept similar to CL's "generalized places" implemented differently. It supports Lisp-1-style evaluation (higher order functions used without a funcall operator), in the middle of a Lisp-2 substrate. Traditional list operations are generalized so they work on other sequences; you can mapcar over a string and such.
Link: https://nongnu.org/txr
I do a lot of shell based work (Linux commands, Awk, Python, Perl, and Powershell) and Lisp has always interested me, so this may scratch an itch.
How easy is this to get setup? Just a few binaries?
Also, one barrier to me has always been learning lisp as it is very different from most traditional scripting languages. I took a look at your examples and it looks neat, but a little complex. Is there a tutorial document that goes over the main functionality and commands. Like with Linux, knowing grep,cut, sort, uniq, cat, and a few others can take you a long way. I should add that although I haven't coded enough lisp to get competent, I've read two books on Common Lisp and have played with Chez Scheme, Racket, and Clojure...so I get the concepts with CONS/CAR/CDR...macros...etc.
From sources, just unpack, ./configure (with --prefix=/your/dest/dir if you don't want it in /usr/local/bin), make, make tests, make install if you're going from source. The dependencies are very low. But one of them is libffi; ./configure doesn't complain about it missing, but you get a build with reduced functionality (which doesn't pass certain tests).
For every release I make tarball packages for for older Debian 5.x (64 bit), Ubuntu 12 (32 bit), Mac OS 10.7.4 (64) and Solaris 10 (x86, 32 bit) as well as Cygwin (32 and 64).
For native Windows, I make a installer. It registers TXR into your PATH environment variable and associates .txr and .tl file associations.
TXR installations are relocatable: TXR discovers its own path and then finds the satellite files relative to its own location; if you keep the relative locations of the pieces as they are, you can move it anywhere in your filesystem. That's also useful for deployment. In the TXR executable there is a 128 byte space where you can embed command-line arguments to the executable. There is a helper function for this called save-exe. https://www.nongnu.org/txr/txr-manpage.html#N-02850687
If you've read books on Common Lisp, you're in pretty good stead. My own Lisp background is heavily Common Lisp, and the CL influences in TXR Lisp are clear; yet it's not CL and will probably mess with your head if you know a bit of CL in ways that it wouldn't if you didn't know CL. The reference manual has some dialect notes in relation to CL, far and few between though.
I don't have a newbie tutorial, unfortunately; I'm hoping someone else will write that. Keeping the documentation complete and at a good level of quality has been a big effort; it's pushing close to 700 pages now if PDF-ed.
I have a private TODO list, in point form, that is over 1300 lines long, so there is no shortage of stuff to do.
There was never supposed to be Lisp dialect, object system, FFI, or any of those things; it really was just a specialized text extraction tool.
A decade and 5200 commits later, it's interesting that I'm the only contributor (of any logic). There have only been doc fixes.
I also didn't think that a decade later, there would be more than 5200 commits, almost all of them from only me. There have been only some half dozen contributions from outsiders: most of them documentation issues, and something in the configure script. Third party code I've brought in, of which there isn't much, I took over as if they were my own. I've kind of gotten used to it being that way; if that ever changes, it will be big adjustment from "everything here is mine".
Deleted Comment
Python and Hy have 100% interoperability. Python can call Hy out of the box, and Hy can call Python out of the box.
Any Python programmer can pick up Hy basics in minutes, and will be productive in a matter of hours. If you are an intermediate level Python programmer and have experience with Lisp macros, you will be writing macros in no time as well. Writing macro wrappers for numerical/scientific applications can increase your code's signal to noise ratio by quite a lot.
docs: http://docs.hylang.org/en/stable/
github: https://github.com/hylang/hy
get it:
> Alas, Hy didn't really reach my dream. That macro expansion made debugging a nightmare as Hy would lose track of where the line numbers are; it wasn't until that when I really realized that without line numbers, you're just lost in terms of debugging in Python-land. That and Python didn't really have the right primitives; immutable datastructures for whatever reason never became first class, meaning that functional programming was hard, "cons" didn't really exist (actually this doesn't matter as much as people might think), recursive programming isn't really as possible without tail call elimination, etc
As far as the lack of functional programming semantics go (no tail call elimination is very sad indeed), there is no denying that either, but considering the applications I am recommending (c/c++ inter-op with python wrappers), you are hardly working in a functional domain in the first place.
Would love to know more about your experiences with doing Racket FFI. How hard were the bugs around GC to discover and workaround? What were you building in Racket?
https://github.com/clasp-developers/clasp
https://www.youtube.com/watch?v=mbdXeRBbgDM
However, Python does have CLIF[1], which is the nicest solution for calling into C++ that I'm aware of.
[1] https://github.com/google/clif
Deleted Comment
To be honest though, I think the biggest missed opportunity in racket is the single dispatch object system, used particularly by racket/gui. For some reason the core racket folks weren't smitten with CLOS and didn't go the Guile route of implementing something similar to it (Guile has GOOPS, which is related to Tiny-CLOS.)
A runtime multiple dispatch stub is available as a lib, though (https://docs.racket-lang.org/multimethod/index.html)
The Racket class system has seemed fine to me for the GUI tookit (except for me not liking typing `send`). But if you find a limitation for that purpose, please post about it on the `racket-users` email list. The core developers are there. https://groups.google.com/forum/#!forum/racket-users/
FWIW, Racket (when it was still PLT Scheme) got a tiny-CLOS-alike around the time that Guile got GOOPS: Eli Barzilay made Swindle: https://docs.racket-lang.org/swindle/
CLOS is neat, and I'd be interested to see someone make a CLOS (maybe including a MOP?) for Racket, atop the modern Racket `struct` and other features. (If you tune it, try tuning for the tentatively forthcoming Chez backend to Racket, rather than for the current JIT. IIRC, Swindle was written before the JIT.)
In my experience, python is used for Tensorflow, or Pandas, or Django, or Flask, or pytorch or something else that runs on top of it. Sometimes it is even more specialized and I need a wrapper for an API to let me talk to some web data. Maybe I need a crawler/scraper and a parser. There is a specialized language on top of the language.
So when someone says, oh this language is better with objects, or has some syntax thing or the other, or I can reason about it I am left confused.
Its like if I were talking to a professional shoe designer and I ask for hiking boots and they tell me that they're really into having at least two tones to offset the lace and the heels or something.
What am I missing? I want to reason about the language too, but doesn't that pale in comparison to being able to run a specialized library?
However, writing your own Tensorflow interface would take several human lifetimes to get it right, and Google already has provided it. So it seems that is not the part you would re-write no matter how good the language is.
Is this your experience of using python or reading about it? I don't know your background so I apologize for making some assumptions about the source of your confusion. It sounds like you don't have a ton of experience programming, so let me start with a broadly: Python is a language in sense that English is a language, but these "something else that runs on top of it", are more like specialized vocabulary or jargon than "languages on top of the language".
English gives you the grammar/structure/spelling to communicate; it's a foundation. But it also gives you general vocabulary; adjectives and adverbs blend and interact with any new vocabulary that might come into play. It doesn't matter if its a poem, or novel, or a technical documentation, or a text book, there is still a lot of English-ness to it.
In the same way, Python as a language is still the substrate that each of those tools (Tensorflow, or Pandas, or Django, or Flask) are interacted with. I agree with what you're getting at, that maybe the tools are more important than the language. When people talk about their like for python the could talk about either: the language itself or the culture/ecosystem around the language. Some inherent to the language, so a quirk of it's history.
This applies as much to natural language. You might hear someone love the sound of Spanish or French, or praise the regularity of Latin spelling, or love Greek for the wealth of ancient, influential texts that it gives access to.
In the case of python you get a lot of praise from both angles. People love the language for it's ecosystem, sure; but also for how it does white spacing, its brevity, the specifics of its typing, where it does and dosn't need parentheses, REPLability, etc.
But you haven't responded to the argument. If someone urges you to use Racket, and you have task in front of you (say, put up a website), it sort of matters whether Racket has a framework more than if it has brackets, indents or curly braces.
It's quite possible to also discuss the language itself independently of how much code is available for it, while acknowledging that it's an important factor for many people.
But if you're talking about Racket, I would want to know what you can do with it. Does it have a data science library? A web app framework?
Deleted Comment
EDIT: I mean in addition to Typed Racket
Deleted Comment