One design decision they made is that outputs are not stored. This means these notebooks are not suitable replacement for heavy computation routines, where the notebook is a record of the final results. Other people are not expected to run minutes/hours long computation to see what the author intended.
You can work your way around it by storing the results in a separate file(s), and writing the boiler plate to let the reader load the results. Or they let you export to ipynb - which is still sharing two files.
Presumably the reason for this decision is making git diffs short. But to me the solution is to fix git diff to operate on JSON nicely, rather than changing the entire notebook format.
The `export` command can generate a rendered artifact if that's what you're after but there is also another avenue here, have you seen the caching feature? The one that caches to disk and persists?
This can automatically store the output of expensive functions, keeping the previous state of the cells in mind. If a re-compute ever needs to happen it will just load it straight from cache.
Another option is to run in lazy mode, documented here:
This will prevent the notebook from rerunning cells by accident.
We're thinking about adding features that would make marimo great for running long running batch work but there's not a whole lot I can share about it yet. If you have specific thoughts or concerns though, feel free to join our discord!
The caching is a very nice feature, and will stop me from keeping my computer running for days/weeks while I work on a notebook.
If I understand it correctly, `@mo.persistent_cache(name="my_cache")` creates a binary file `my_cache` that I should commit as well if I don't want others to repeat the computation?
This kinda solves the problem, except for having two files per notebook, and that marimo notebooks are no longer viewable with output on github directly.
> This will prevent the notebook from rerunning cells by accident.
Is that really what's wanted? If there's some cell that I need to run twice for some reason that I tried debugging but wasn't able to figure out why, or for debug just run cells 1-5 in order but on specific other (prod) systems, skip cell 4 and run 3 before 2. Now, arguably well written software would handle those things automatically, but we're not talking about battle hardened software that's had an SRE team vigorously refactor it until it's been proven suitable for such purpose by having been on call for it for months off not years. We're talking about notebooks, which have their time and place, but the entire point, I would argue, is to make it easier run notebooks in production without the added overhead of said SRE team. And in that world, the reality is the PhD is gonna have some things they know they should fix, but it's easier to just comment out the unneeded cells on the prod and hit run all, so shouldn't the tools better support that use case (by saving what was actually run and offering to rerun that) over caching based on input string and hoping for the best?
> One design decision they made is that outputs are not stored
This is not quite true. Outputs are not stored...in the Python file*. marimo does store outputs in the `/__marimo__` folder with settings enabled.
> writing the boiler plate to let the reader load the results.
Therea are some primitives to do this for you, such as mo.persistent_cache. This can be an annotation or 'with' block. It intelligently knows when either the source code or inputs change.
The plan is to take this one step further than storing just the output. Because marimo knows the dependencies of each cell, in a future version, it will store each output AND let you know which are stale based on code changes. This is being built by Dylan (from the blog) and inspired by Nix.
It’s true that we don’t store outputs in the file format. This is the main tradeoff, as discussed in the blog. But that doesn’t mean marimo notebooks aren’t suitable for heavy computation.
marimo lets you automatically snapshot outputs in an auxiliary file while you work. We also have a persistent cache that lets you pick up where you left off — saving not just outputs but also computed data.
> Other people are not expected to run minutes/hours long computation to see what the author intended
Arguably this is a good thing. You shouldn't distribute things you can't prove have the same results and one way to do that is to require others to run the same computations.
You lose out on so many use cases without the stored output. The first that comes to mind is all of the learning resources that are now presented in notebooks.
Most learners do not need to fact check the instructor, but do want to see the operations which were run. Those that are curious can run/edit the notebook themselves.
Edit: The JupyterBook ecosystem (https://executablebooks.org/en/latest/gallery/) as an example of what is possible with stored plots/calculations. Most learners are just going to follow along with the material, but being able to optionally play with the data is the super power of the platform with minimal friction.
It's hard for me to imagine the use case where this is appropriate.
I have looked at marimo several times, and while it's great for interactive computing, and it has a fantastic team, it's not a replacement for notebooks and I find their use of the term "notebook" confusing. As a scientist, I don't understand what use case they are exploring, but I do know it's not the use case where Jupyter was created and it's not my current use case for Jupyter on teams I work with.
> You shouldn't distribute things you can't prove have the same results
Why not? Can you expand on this because I don't see why this is not a good thing.
Besides if you distribute your code alongside your output, aren't you providing that proof anyway? People can run your code and see they are getting the same result.
This is the top reply here but I completely disagree with it. Storing the output of running your source file back into your source file is one of the design decisions in Jupyter which I have always considered abhorrent and am very happy to see fixed by Marimo.
i wanted to like marimo, but the best notebook interface i've tried so far is vscode's interactive window [0]. the important thing is that it's a python file first, but you can divide up the code into cells to run in the jupyter kernel either all at once or interactively.
Spyder also has these, possibly for longer than vscode [0]. I don't know who had this idea first but I remember some vim plugins doing that long ago, so maybe the vim community?
This is also where I have landed. Gives you all of your nice IDE tooling alongside the REPL environment. No need for separate notebook aware code formatters/linters/etc. That they version cleanly is just the cherry on top.
Looks very interesting. Could you elaborate on why you prefer this over the .ipynb notebook interface built into VS Code? The doc you linked mentions debugging, but I have found that the VS Code debugger is already fairly well-integrated into .ipynb notebooks. Is it mainly the improved diffing and having a REPL?
my impetus for exploring it was that vim modal editing and keyboard navigation is just really clunky in the notebook integration.
whether or not it's better for you depends on your use case for notebooks — i use them mostly for prototyping and exploratory data analysis so separating the code from the output might be more convenient for me than for you
Agreed, I find this to be a super productive environment, because you get all of vscode's IDE plus the niceties of Jupyter and IPython.
I wrote a small vscode extension that builds upon this to automatically infer code blocks via indentation, so that you don't have to select them manually: [0]
You can have a plaintext file which is also the program which is also the documentation/notebook/website/etc. It's extremely powerful, and is a compelling example of literate programming.
I'm writing this comment as I write a literate README.org that documents and tangles a PoC program I'm writing for an upcoming big initiative at my job.
You can do most work in org instead of editing programs directly, and it'll save time and produce good documentation. Can't recommend it enough.
This approach has the recently developed benefit of being really great as context stores for interacting with LLMs.
One (not a great) workflow I have, is that I use notebooks as quicks UIs to visualize some results.
1. Run a simulation that outputs results to some file
2. Load results in a notebook and do some quick processing + visualization
Very often, I want to compare quickly between 2 different runs and end up copying down the cell with visualization, then just re-run the data load + processing + visualization and compare them.
My understanding is that this would not be possible with marimo, since it will re-run automatically the cell with my previous data right ?
I have had a similar situation and my "hack" for this was to start the same notebook twice and have two tabs open. This worked for something things ...
Other times I just bit the bullet and made two forms on two variables so that everything would fit in a single notebook. By having two variables that contain all the inputs you can make sure that only the cells update that need to update. It takes a bit more effort but makes a lot of sense for some apps that you want to share with a colleague.
If you share more details about your setup I might be able to give better advice/think along more.
It may be preferable to create a variable tied to a UI element that can be used as a toggle to view each analysis.
choice = mo.ui.dropdown(['train', 'split')
data = load(choice.value)
processed = process(data)
visualize(processed)
This way, you can toggle between just more than two if needed. If you need to see both at once, you'd want to refactor the processing and visualizing step into functions, and then just duplicate the finals cell(s).
marimo has a multi-column mode, so you can view them side-by-side
> When working with Jupyter, too often you end up with directories strewn with spaghetti-code notebooks, counting up to Untitled12.ipynb or higher. You the notebook author don’t know what’s in these notebooks
This is such a small UX thing but it’s so damn important. The simple fix is to not auto-name notebooks untitled-# when the user clicks new notebook just ask the name straight away, if they can’t name it don’t create it. It might add the smallest amount of friction to the UX, but it’s so damn important.
Also the choice of json as the file format is just plain wrong. Why the project hasn’t just abandoned that entirely and done a json-#python back and forth when writing to file is beyond me. There are extensions that do this, but that’s a really clunky interface, and while I can set it up for myself it’s difficult to force upon others in a corporate environment.
Great to see someone is taking the seemingly small things up, because they mean a world of difference to the overall ecosystem.
> The simple fix is to not auto-name notebooks untitled-# when the user clicks new notebook just ask the name straight away, if they can’t name it don’t create it.
The even simpler fix is to just not name them until the user does. That's the way other programs work. If you create a new document in a word processor, it will say "Untitled" at the top of the window, but it doesn't create a file called untitled.doc on disk until you do "Save as" and choose a filename. It has always irritated me that Jupyter insists on having an on-disk file right from the beginning.
It would be neat if it worked similar to ChatGPT sessions which give themself a name based on the conversation. You could have a small local model that gives a default name automatically.
It would also be great to have a decent search across notebooks in a project so I could quickly find an old function.
I develop an open source notebook widget. Working with marimo has been a joy compared to developing on top of any other notebook environment.
The team is responsive and they care about getting it right. Having a sane file format for serializing notebooks is an example of this. They are thinking about core problems. They are also building in the open.
The core jupyter team is very unresponsive and unfocused. When you have a bug, you need to figure out which one of many many interelated projects caused the bug, issues go weeks without a response. It's a mess.
Then there are the proprietary notebook like environments. VSCode notebooks, and google colab in particular. They frequently rely on opaque undocumented APIs and are also very unresponsive.
I find it a baffling that the popularity of Jupyter and successes of notebook analysis in science hasn’t brought a change in Python to better support this user base. Packaging has (slowly) progressed and Uv nicely made the experience smooth, fast and above all coherent. Yet the Python runtime and parser are the same as ever. The ipynb notebook format and now Marimo’s decorator approach had to be invented on top. Python might never attain the heights of Lisp’s REPL driven development, yet I wonder if it couldn’t be better. As much I enjoy using Jupyter it’s always been something tacked on top of infrastructure that doesn’t want to accommodate it. Thus you need to take care of cell order yourself or learn to use a helper tool (Jupytext, Nbdev).
Me, I’d have support in the parser for a language structure or magic comment that points the cell boundaries. I would make dynamic execution of code a first party feature with history tracking and export of all the code sent into the runtime through this. Thus what the runtime saw happen could be committed over what user thought they did. Also, a better, notebook aware evaluation with extension hooks for different usage methods (interactive, scripting, testing).
I have no solution to ipynb JSON problem. I do think it is bad that our seemingly only solution for version control can manage only simple text, and users of all the other format have to adapt or suffer.
Readit News
You can work your way around it by storing the results in a separate file(s), and writing the boiler plate to let the reader load the results. Or they let you export to ipynb - which is still sharing two files.
Presumably the reason for this decision is making git diffs short. But to me the solution is to fix git diff to operate on JSON nicely, rather than changing the entire notebook format.
The `export` command can generate a rendered artifact if that's what you're after but there is also another avenue here, have you seen the caching feature? The one that caches to disk and persists?
https://docs.marimo.io/guides/expensive_notebooks/?h=cache#d...
This can automatically store the output of expensive functions, keeping the previous state of the cells in mind. If a re-compute ever needs to happen it will just load it straight from cache.
Another option is to run in lazy mode, documented here:
https://docs.marimo.io/guides/expensive_notebooks/?h=cache#l...
This will prevent the notebook from rerunning cells by accident.
We're thinking about adding features that would make marimo great for running long running batch work but there's not a whole lot I can share about it yet. If you have specific thoughts or concerns though, feel free to join our discord!
https://marimo.io/discord?ref=nav
If I understand it correctly, `@mo.persistent_cache(name="my_cache")` creates a binary file `my_cache` that I should commit as well if I don't want others to repeat the computation?
This kinda solves the problem, except for having two files per notebook, and that marimo notebooks are no longer viewable with output on github directly.
Is that really what's wanted? If there's some cell that I need to run twice for some reason that I tried debugging but wasn't able to figure out why, or for debug just run cells 1-5 in order but on specific other (prod) systems, skip cell 4 and run 3 before 2. Now, arguably well written software would handle those things automatically, but we're not talking about battle hardened software that's had an SRE team vigorously refactor it until it's been proven suitable for such purpose by having been on call for it for months off not years. We're talking about notebooks, which have their time and place, but the entire point, I would argue, is to make it easier run notebooks in production without the added overhead of said SRE team. And in that world, the reality is the PhD is gonna have some things they know they should fix, but it's easier to just comment out the unneeded cells on the prod and hit run all, so shouldn't the tools better support that use case (by saving what was actually run and offering to rerun that) over caching based on input string and hoping for the best?
This is not quite true. Outputs are not stored...in the Python file*. marimo does store outputs in the `/__marimo__` folder with settings enabled.
> writing the boiler plate to let the reader load the results.
Therea are some primitives to do this for you, such as mo.persistent_cache. This can be an annotation or 'with' block. It intelligently knows when either the source code or inputs change.
The plan is to take this one step further than storing just the output. Because marimo knows the dependencies of each cell, in a future version, it will store each output AND let you know which are stale based on code changes. This is being built by Dylan (from the blog) and inspired by Nix.
marimo lets you automatically snapshot outputs in an auxiliary file while you work. We also have a persistent cache that lets you pick up where you left off — saving not just outputs but also computed data.
Many of our users do very heavy computation.
https://docs.marimo.io/guides/expensive_notebooks/
Arguably this is a good thing. You shouldn't distribute things you can't prove have the same results and one way to do that is to require others to run the same computations.
Most learners do not need to fact check the instructor, but do want to see the operations which were run. Those that are curious can run/edit the notebook themselves.
Edit: The JupyterBook ecosystem (https://executablebooks.org/en/latest/gallery/) as an example of what is possible with stored plots/calculations. Most learners are just going to follow along with the material, but being able to optionally play with the data is the super power of the platform with minimal friction.
I have looked at marimo several times, and while it's great for interactive computing, and it has a fantastic team, it's not a replacement for notebooks and I find their use of the term "notebook" confusing. As a scientist, I don't understand what use case they are exploring, but I do know it's not the use case where Jupyter was created and it's not my current use case for Jupyter on teams I work with.
Why not? Can you expand on this because I don't see why this is not a good thing.
Besides if you distribute your code alongside your output, aren't you providing that proof anyway? People can run your code and see they are getting the same result.
0: https://code.visualstudio.com/docs/python/jupyter-support-py
[0] https://docs.spyder-ide.org/current/panes/editor.html#code-c...
whether or not it's better for you depends on your use case for notebooks — i use them mostly for prototyping and exploratory data analysis so separating the code from the output might be more convenient for me than for you
I wrote a small vscode extension that builds upon this to automatically infer code blocks via indentation, so that you don't have to select them manually: [0]
[0]: https://github.com/kylebarron/vscode-jupyter-python
You can have a plaintext file which is also the program which is also the documentation/notebook/website/etc. It's extremely powerful, and is a compelling example of literate programming.
Decent overview here: https://www.johndcook.com/blog/2022/08/02/org-babel-vs-jupyt...
[edit: better link]
You can do most work in org instead of editing programs directly, and it'll save time and produce good documentation. Can't recommend it enough.
This approach has the recently developed benefit of being really great as context stores for interacting with LLMs.
One (not a great) workflow I have, is that I use notebooks as quicks UIs to visualize some results. 1. Run a simulation that outputs results to some file 2. Load results in a notebook and do some quick processing + visualization
Very often, I want to compare quickly between 2 different runs and end up copying down the cell with visualization, then just re-run the data load + processing + visualization and compare them.
My understanding is that this would not be possible with marimo, since it will re-run automatically the cell with my previous data right ?
I have had a similar situation and my "hack" for this was to start the same notebook twice and have two tabs open. This worked for something things ...
Other times I just bit the bullet and made two forms on two variables so that everything would fit in a single notebook. By having two variables that contain all the inputs you can make sure that only the cells update that need to update. It takes a bit more effort but makes a lot of sense for some apps that you want to share with a colleague.
If you share more details about your setup I might be able to give better advice/think along more.
choice = mo.ui.dropdown(['train', 'split')
data = load(choice.value)
processed = process(data)
visualize(processed)
This way, you can toggle between just more than two if needed. If you need to see both at once, you'd want to refactor the processing and visualizing step into functions, and then just duplicate the finals cell(s).
marimo has a multi-column mode, so you can view them side-by-side
This is such a small UX thing but it’s so damn important. The simple fix is to not auto-name notebooks untitled-# when the user clicks new notebook just ask the name straight away, if they can’t name it don’t create it. It might add the smallest amount of friction to the UX, but it’s so damn important.
Also the choice of json as the file format is just plain wrong. Why the project hasn’t just abandoned that entirely and done a json-#python back and forth when writing to file is beyond me. There are extensions that do this, but that’s a really clunky interface, and while I can set it up for myself it’s difficult to force upon others in a corporate environment.
Great to see someone is taking the seemingly small things up, because they mean a world of difference to the overall ecosystem.
The even simpler fix is to just not name them until the user does. That's the way other programs work. If you create a new document in a word processor, it will say "Untitled" at the top of the window, but it doesn't create a file called untitled.doc on disk until you do "Save as" and choose a filename. It has always irritated me that Jupyter insists on having an on-disk file right from the beginning.
It would also be great to have a decent search across notebooks in a project so I could quickly find an old function.
How you start the marimo notebook, via
`marimo edit must-give-name-to-this-file.py`
is indeed one of my teeny by favourite features of it. When you start a new notebook you're kind of forced to name it immediately.
This is false. Org-mode has had this functionality for over two decades.
https://orgmode.org/
The team is responsive and they care about getting it right. Having a sane file format for serializing notebooks is an example of this. They are thinking about core problems. They are also building in the open.
The core jupyter team is very unresponsive and unfocused. When you have a bug, you need to figure out which one of many many interelated projects caused the bug, issues go weeks without a response. It's a mess.
Then there are the proprietary notebook like environments. VSCode notebooks, and google colab in particular. They frequently rely on opaque undocumented APIs and are also very unresponsive.
Happy to hear it! Got a link to your widget? I am assuming it's an anywidget?
Me, I’d have support in the parser for a language structure or magic comment that points the cell boundaries. I would make dynamic execution of code a first party feature with history tracking and export of all the code sent into the runtime through this. Thus what the runtime saw happen could be committed over what user thought they did. Also, a better, notebook aware evaluation with extension hooks for different usage methods (interactive, scripting, testing).
I have no solution to ipynb JSON problem. I do think it is bad that our seemingly only solution for version control can manage only simple text, and users of all the other format have to adapt or suffer.