I've been engaged in a 1.5 year long (so far) project to port the entirety of GJS's "scmutils" package over to Clojure, and the erratic behavior of MIT Scheme over Rosetta has been a pain I've consigned myself to for months. I keep an old machine handy when I need to test functions that can't work on the M1.
I am SO HAPPY to see this work! Major timesaver for me and anyone looking to run the executable versions of Functional Differential Geometry[1] and Structure and Interpretation of Classical Mechanics[2] in the original language.
I have to ask, how far along is this project? Is it good enough to run all of the examples in the books? I have run into the issue where the provided compiled version of scmutils from GJS' website doesn't run on recent versions of MIT Scheme (version 11 and up) and there's not much info on compiling it yourself.
It is good enough! Almost all code forms from the book live in the tests (see the FDG directory[0], for example), and there are a few nice environments like Nextjournal[1] where everything from the books works in the browser.
The Clojure port is quite fast, faster than the original for all benchmarks GJS has sent me, and more fleshed out. (That will change, as I've been pushing bugfixes and performance improvements back upstream as I go, as a meager gift to GJS for making this huge, amazing library in the first place.)
I actually wrote to GJS this morning asking for instructions on how to compile the original "scmutils", since I have the same problem. He responded saying he'll get back to me this afternoon, so I'll post here once I have details.
If you are still interested in getting the books going with MIT-Scheme, I put a decent amount of work into the exercises using the original codebase here[2], including a dockerized version of mit-scheme[3] and the scmutils package[4] that might be useful.
Ok, but how do you compile it? When you pull the repo you need to run autoconf to create the configure script but that tells me "This script needs an existing MIT/GNU Scheme installation to function".
What's stopping people from just compiling Scheme for ARM? The website has a separate aarch64 download it seems, so why not patch that instead of relying on Rosetta2?
The vfork/fork issue and the compiler upgrade issue don't seem to be too problematic to work around, so there must be some kind of ARM limitation that's preventing Scheme from working, but what?
MacOS on the M1 processor is the first to use, and require, the W^X bit in memory, meaning that pages of memory are either writable, or can be executed from, but not both. MIT Scheme's front page says this is fundamentally incompatible with their design, and therefore it won't build. When running in the emulator, this requirement would be relaxed for compatibility reasons.
There is an escape hatch for writing JIT compilers (essentially what MIT Scheme is in this case), described here https://developer.apple.com/documentation/apple-silicon/port... although it's fairly cumbersome and would almost certainly require a lot of extra, MacOS specific code. I assume that's why no-one has bothered so far to port it.
From perusing the source of the MIT/GNU Scheme compiler, I suspect that “only” two changes are needed to support W^X:
- Compiled code needs to be allocated separately from Scheme objects. It can still be garbage collected and such - they will probably need to make a separate set of allocation functions for code vs. data. The closure/function objects can be made to point to the code, or, if they don’t need to be written often, simply allocated wholly from the “code” pages.
- Before modifying any of the code (e.g. to patch addresses after GC relocation), a system-specific hook function will need to be called to set the permissions to RW. They already call an I-cache flush function after each modification, so this shouldn’t be too bad.
Some of the necessary changes are already sketched out in cmpint.txt. And, sooner or later, they’re going to have to make these changes: OpenBSD already enforces W^X (but provides a workaround), and MIT/GNU Scheme already applies a paxctl workaround to gain W|X on NetBSD.
Wow really? A common intro to security exercise (think CTFs and university courses) is to write increasingly complicated C programs that leverage W&X. Classic buffer overflow into the stack kind of stuff. On M1 it’s now impossible to exploit even a self-compiled toy in this way?
I mean if they pass the correct flags they get memory that can be toggled rapidly between X and W mode - or is MIT Scheme mixing data and code in the heap and so actually requiring RWX?
I find it a bit disingenuous to say that it runs on apple silicon if you need to modify source code. Also it's not because it compiles and starts that it's fully functional.
The main MIT Scheme page say that it's not possible and need significant efforts, so I would be curious to get a description as to why one claim it's impossible while the other show that it compiles and starts.
Are the original authors too much against M1/Apple and justify themselves ? Or does compiling on M1 sort-of works until you hit more complex features that will crash or misbehave ?
It's also a bit disingenuous to say it's "on Apple Silicon" when you're running it through a translation layer that won't exist in a few year's time. I'd wager the reason why the GNU folks say it doesn't run on ARM is because... it doesn't. Running it as an x86 program is mandatory, apparently.
Apple Silicon has a (mostly) hardware translation layer, which this software is running on.
There's a special aarch64 build of the software available, so it clearly runs on ARM. Perhaps there's some kind of issue specifically on macOS that makes the existing ARM port incompatible with Apple's ARM implementation?
I got MIT Scheme running on my M1 MacBook Pro about 6 months ago when I bought the book "Software Design for Flexibility" and although I can't find my notes for that, I think I remember building from source natively, not via Rosetta - but I may remember incorrectly.
I also remember it taking a while to get Gerbil Scheme running on M1.
Readit News
I am SO HAPPY to see this work! Major timesaver for me and anyone looking to run the executable versions of Functional Differential Geometry[1] and Structure and Interpretation of Classical Mechanics[2] in the original language.
[0] https://github.com/sicmutils/sicmutils [1] https://github.com/sicmutils/fdg-book [2] https://github.com/sicmutils/sicm-book
The Clojure port is quite fast, faster than the original for all benchmarks GJS has sent me, and more fleshed out. (That will change, as I've been pushing bugfixes and performance improvements back upstream as I go, as a meager gift to GJS for making this huge, amazing library in the first place.)
I actually wrote to GJS this morning asking for instructions on how to compile the original "scmutils", since I have the same problem. He responded saying he'll get back to me this afternoon, so I'll post here once I have details.
If you are still interested in getting the books going with MIT-Scheme, I put a decent amount of work into the exercises using the original codebase here[2], including a dockerized version of mit-scheme[3] and the scmutils package[4] that might be useful.
- [0] https://github.com/sicmutils/sicmutils/tree/main/test/sicmut...
- [1] https://nextjournal.com/try/samritchie/sicmutils/
- [2] https://github.com/sicmutils/sicm-exercises
- [3] https://hub.docker.com/r/sritchie/mit-scheme
- [4] https://hub.docker.com/r/sritchie/mechanics
The vfork/fork issue and the compiler upgrade issue don't seem to be too problematic to work around, so there must be some kind of ARM limitation that's preventing Scheme from working, but what?
There is an escape hatch for writing JIT compilers (essentially what MIT Scheme is in this case), described here https://developer.apple.com/documentation/apple-silicon/port... although it's fairly cumbersome and would almost certainly require a lot of extra, MacOS specific code. I assume that's why no-one has bothered so far to port it.
- Compiled code needs to be allocated separately from Scheme objects. It can still be garbage collected and such - they will probably need to make a separate set of allocation functions for code vs. data. The closure/function objects can be made to point to the code, or, if they don’t need to be written often, simply allocated wholly from the “code” pages. - Before modifying any of the code (e.g. to patch addresses after GC relocation), a system-specific hook function will need to be called to set the permissions to RW. They already call an I-cache flush function after each modification, so this shouldn’t be too bad.
Some of the necessary changes are already sketched out in cmpint.txt. And, sooner or later, they’re going to have to make these changes: OpenBSD already enforces W^X (but provides a workaround), and MIT/GNU Scheme already applies a paxctl workaround to gain W|X on NetBSD.
I suppose the wait is on for someone to rewrite the JIT engine to be compatible with Apple's implementation of ARM.
The main MIT Scheme page say that it's not possible and need significant efforts, so I would be curious to get a description as to why one claim it's impossible while the other show that it compiles and starts.
Are the original authors too much against M1/Apple and justify themselves ? Or does compiling on M1 sort-of works until you hit more complex features that will crash or misbehave ?
There's a special aarch64 build of the software available, so it clearly runs on ARM. Perhaps there's some kind of issue specifically on macOS that makes the existing ARM port incompatible with Apple's ARM implementation?
I also remember it taking a while to get Gerbil Scheme running on M1.
https://github.com/racket/ChezScheme/