I have being using Trial[1] for the past few weeks to test out game development in Common Lisp, and have been having a great time. Being able to alter (almost) all aspects of your game while it's running is a blessing.
Lisp languages seem well-suited for building games. The ability to evaluate code interactively without recompilation is a huge deal for feature building, incremental development, and bug-fixing. Retaining application state between code changes seems like it would be incredibly useful. Common Lisp also appears to be a much faster language than I would have blindly assumed.
The main downside for me (in general, not just for game programming) is the clunkiness in using data structures - maps especially. But the tradeoff seems worth it.
One of the downsides is that implementations like SBCL have a deep integration and need things like a well performing GC implementation - to get this running on specialized game hardware is challenging. The article describes that. Getting over the hurdle of the low-level integration is difficult. The reward comes, when one gets to the point, where the rapid incremental development cycles of Common Lisp, even with connected devices, kicks in.
For the old historic Naughty Dog use case, it was a development system written in Common Lisp on an SGI and a C++ runtime with low-level Scheme code on the Playstation.
> Common Lisp also appears to be a much faster language than I would have blindly assumed.
There are two modes:
1) fast optimized code which allows for some low-level stuff to stay with Common Lisp
2) unoptimized, but natively compiled code, which enables safe (-> the runtime does not crash) interactive and incremental development -> this mode is where much of the software can run nowadays and which is still "fast enough" for many use cases
If that's your main downside, that's pretty good, since clunkiness is in many ways fixable. Personally with standard CL I like to use property lists with keywords, so a "map literal" is just (list :a 3 :b 'other). It's fine when the map is small. The getter is just getf, setting is the usual setf around the getter. There's a cute way to loop by #'cddr for a key-and-value loop, though Alexandria (a very common utility library) has some useful utils for looping/removing/converting plists as well.
If typing out "(list ...)" is annoying, it's a few lines of code to let you type {:a 3 :b 4} instead, like Clojure. And the result of that can be a plist, or a hash table, or again like Clojure one of the handful of immutable map structures available. You can also easily make the native hash tables print themselves out with the curly bracket syntax.
(On the speed front, you might be amused by https://renato.athaydes.com/posts/how-to-write-slow-rust-cod... But separately, when you want to speed up Lisp (with SBCL) even more than default, it's rather fun to be able to run disassemble on your function and see what it's doing at the assembly level, and turn up optimization hints and have the compiler start telling you (even on the individual function level) about where it has to use e.g. generic addition instead of a faster assembly instruction because it can't prove type info and you'll have to tell it/fix your code. It can tell you about dead code it removed. You can define stack-allocation if needed. Simple benchmarking that also includes processor cycles and memory allocated is available immediately with the built-in time macro...)
> The ability to evaluate code interactively without recompilation
SBCL and other implementations compile code to machine code then execute it. That is to say, when a form is submitted to the REPL, the form is not interpreted, but first compiled then executed. The reason execution finishes quickly is because compilation finishes quickly.
There are some implementations, like CCL, with a special interpreter mode exclusively for REPL-usage.[1] However, at least SBCL and ECL will compile code, not interpret.
There are 1980's papers about Lisp compilers competing with Fortran compilers, unfortunately with the AI Winter, and the high costs of such systems, people lost sight of it.
There are some libraries that make maps and the like usable with a cleaner syntax. You too could make some macros of your own for the same purpose, if syntax is the concern
This is super neat - SBCL is an awesome language implementation, and I've always wanted to do CL development for a "real" game console.
I'm also surprised (in a good way) that Shinmera is working on this - I've seen him a few times before on #lispgames and in the Lisp Discord, and I didn't know that he was into this kind of low-level development. I've looked at the guts of SBCL briefly and was frightened away, so kudos to him.
I wonder if SBCL (+ threading/SDL2) works on the Raspberry Pi now...
I'm not doing the SBCL parts, that's all Charles' work that I hired him for. My work is the portability bits that Trial relies on to do whatever and the general build architecture for this, along with the initial runtime stubbing.
Thanks to the author for the fascinating and detailed write up. It feels like a lot of the time this level of detail around the specifics of 'blessed' (not homebrew) console porting are only revealed years after the end of the consoles lifetime.
As an aside, reading about this kind of deeply interesting work always makes me envious when I think about the rote software I spend all day writing :)
At least back when I was working these "blessed" tools were usually a tad hacked together, modern homebrew toolchains for many older platforms are better except for debugging support (since the devkits for the machines usually had better hooks available but also avoiding the entire GDB focus).
Having been in both worlds, i'm not entirely sure there's that much to be envious of.
as I was just sitting down to another day of ruby on rails (that I am grateful for!) I was thinking.. I wonder what hobby/open source projects could use some of my attention later..
.. what projects my attention could use later .. :D
You cannot publish games with homebrew, it has to use the official SDK. Besides that, almost nobody has a jailbroken Switch, so it would make it extremely hard to play any games on anything but an emulator.
This isn't really my scene so I don't know the details, but I remember reading that the first 10+ million Switches produced have an unpatchable bootloader exploit. I'm sure you're correct that almost nobody actually has a hacked console, but my understanding is that they're readily available for people who want one.
You don’t let your kids jailbreak their Switch. Because it’s a damn online system, so any leaked info and Nintendo can brick the Switch. And their game states are far too valuable for the kids for that.
Context: Naughty Dog used a custom Lisp-alike (GOAL) to build the Jak & Daxter series on PS2. They left enough debugging information in that it was possible to reverse engineer. The OpenGOAL project has done so, and these games can now be run on all platforms that their GOAL compiler gets ported to (x86 for now AFAIK). Would be cool to port this to the Switch.
I've just bought Kandria. I'm not much of a game player so I probably won't get much play out of it, but Shinmera is clearly pushing the bounds of the Lisp world, and that's something to support.
I wish the likes of Nintendo and Sony themselves finance such efforts. I mean it's one another way to create games (IP) for your console, what possibly could be the downside of starting something similar to Github Accelerator for your platform?
Because it's well established that game developers can and will jump through whatever hoops the platform holder demands at their own expense, they don't have the leverage to be picky about the technical details when deciding which platforms to ship on. Nintendo doesn't need to create new incentives to release on the Switch when they already have the biggest incentive of all: 140+ million units sold, and a high attach rate.
At least there isn't as much hoop jumping as there used to be, since the systems have all converged on using commodity CPU and GPU architectures with at most minor embellishments.
Yeah, also whatever they would build, the other platform vendors won't choose the same thing, and it wont be the exact variant of lisp or w/e that even the few nice developers would want.
I wish vendors would be just more supportive of different llvm tool chains. Rust isn't even well supported.
Readit News
I hope this port succeeds.
[1]: https://github.com/Shirakumo/trial
The main downside for me (in general, not just for game programming) is the clunkiness in using data structures - maps especially. But the tradeoff seems worth it.
For the old historic Naughty Dog use case, it was a development system written in Common Lisp on an SGI and a C++ runtime with low-level Scheme code on the Playstation.
> Common Lisp also appears to be a much faster language than I would have blindly assumed.
There are two modes:
1) fast optimized code which allows for some low-level stuff to stay with Common Lisp
2) unoptimized, but natively compiled code, which enables safe (-> the runtime does not crash) interactive and incremental development -> this mode is where much of the software can run nowadays and which is still "fast enough" for many use cases
If typing out "(list ...)" is annoying, it's a few lines of code to let you type {:a 3 :b 4} instead, like Clojure. And the result of that can be a plist, or a hash table, or again like Clojure one of the handful of immutable map structures available. You can also easily make the native hash tables print themselves out with the curly bracket syntax.
(On the speed front, you might be amused by https://renato.athaydes.com/posts/how-to-write-slow-rust-cod... But separately, when you want to speed up Lisp (with SBCL) even more than default, it's rather fun to be able to run disassemble on your function and see what it's doing at the assembly level, and turn up optimization hints and have the compiler start telling you (even on the individual function level) about where it has to use e.g. generic addition instead of a faster assembly instruction because it can't prove type info and you'll have to tell it/fix your code. It can tell you about dead code it removed. You can define stack-allocation if needed. Simple benchmarking that also includes processor cycles and memory allocated is available immediately with the built-in time macro...)
SBCL and other implementations compile code to machine code then execute it. That is to say, when a form is submitted to the REPL, the form is not interpreted, but first compiled then executed. The reason execution finishes quickly is because compilation finishes quickly.
There are some implementations, like CCL, with a special interpreter mode exclusively for REPL-usage.[1] However, at least SBCL and ECL will compile code, not interpret.
[1] https://github.com/Clozure/ccl/blob/v1.13/level-1/l1-readloo...
Dead Comment
I'm also surprised (in a good way) that Shinmera is working on this - I've seen him a few times before on #lispgames and in the Lisp Discord, and I didn't know that he was into this kind of low-level development. I've looked at the guts of SBCL briefly and was frightened away, so kudos to him.
I wonder if SBCL (+ threading/SDL2) works on the Raspberry Pi now...
And, as mentioned, *her :)
https://reader.tymoon.eu/article/436
As an aside, reading about this kind of deeply interesting work always makes me envious when I think about the rote software I spend all day writing :)
Having been in both worlds, i'm not entirely sure there's that much to be envious of.
as I was just sitting down to another day of ruby on rails (that I am grateful for!) I was thinking.. I wonder what hobby/open source projects could use some of my attention later..
.. what projects my attention could use later .. :D
I'm curious what the rationale here was for using the official SDK, rather than the unencumbered "homebrew" ones[0].
As a complete guess, maybe Nintendo doesn't let you officially publish games built using 3rd party SDKs?
[0] https://switchbrew.org/wiki/Setting_up_Development_Environme...
This isn't really my scene so I don't know the details, but I remember reading that the first 10+ million Switches produced have an unpatchable bootloader exploit. I'm sure you're correct that almost nobody actually has a hacked console, but my understanding is that they're readily available for people who want one.
This doesn't surprise me much, but does Nintendo state this explicitly anywhere public?
If Nintendo chose to sign an application developed using a 3rd party toolchain, there's no technical reason why it couldn't run on retail consoles.
Context: Naughty Dog used a custom Lisp-alike (GOAL) to build the Jak & Daxter series on PS2. They left enough debugging information in that it was possible to reverse engineer. The OpenGOAL project has done so, and these games can now be run on all platforms that their GOAL compiler gets ported to (x86 for now AFAIK). Would be cool to port this to the Switch.
At least there isn't as much hoop jumping as there used to be, since the systems have all converged on using commodity CPU and GPU architectures with at most minor embellishments.
I wish vendors would be just more supportive of different llvm tool chains. Rust isn't even well supported.
That is why we cannot have nice things.