Hey, HN. We’re thrilled to release Observable Framework today — a new open-source tool for developing data apps. I highly recommend viewing this example report adapted from our internal dashboard analyzing web logs:
This technique of “just plot everything” (7.6M requests as a scatterplot) has revealed surprising insights we’ve used to optimize our servers and better control traffic. We’re also sharing a more traditional dashboard that visualizes the adoption of our open-source visualization library (and in some ways the successor to D3), Observable Plot:
In addition to releasing Observable Framework, we’ve also made Observable free again for individuals (including private notebooks and databases connectors). Let me know if you have any questions!
You've sponsored some very cool, state of the art tools. I've had friends work at Observable. I want you to succeed.
I tried to get our team to use Observable Notebooks a few years back. The researchers I work with are more comfortable in Python. Clearly that's one of the things you're trying to solve in this release. The other half of that uphill battle was discomfort posting code externally. In some ways you've also mitigated that in this release, but I wonder how sustainable it is.
Small teams eat for free by virtue of being small. Large organizations with trepidation or bureaucracy about using SaaS hosting will self host. That leaves the people in the middle: big enough to need to pay, but small enough to not have institutional problems with external hosting. Moreover, if the Observable bill ever gets much higher than the equivalent on Firebase et. al., the medium guys can self-host too.
How do you anticipate the paid side of the new business to work out? What's the hook (beyond thinking you guys are cool and trying to keep you in business) that gets someone to pay for Observable?
Hi! Some background first: I'm putting together a blog right now using Hugo and D3. I'm a huge fan of D3's infinite flexibility, as seen in some famous scrollytellers [0-1], and I've spent some time experimenting with that format myself [2].
My question is: what does Observable Framework offer for data storytellers who want to blog? Is this meant to go up against Hugo/Jekyll in terms of full-fledged max-efficiency site generation? If not, are there plans to add integrations with other blogging frameworks?
We’re not expressly targeting the blogging use case — we primarily want to support data apps, dashboards, and reports. But Observable Framework is quite flexible and you can use it for a lot of things; the Framework documentation is itself written using Framework, for example. So I would say that if you are working with data and you want an automated process to keep your data up-to-date, or to work with multiple languages (e.g., Python and JavaScript), or if you want to do a lot of interactive visualizations then you should give Framework a go. But we don’t have much built-in affordances for blogging, so you might find some things missing. Feel free to file feature requests! We’d love to hear your ideas, though we’re primarily focused on reporting and data app development for work.
I’m not sure what better integration with other blogging frameworks would look like — like, part of the page is rendered by Framework, but the site as a whole is handled by the blogging framework? Perhaps we could develop Framework’s API further so it could function like a plugin. But this is speculative and not a priority for us currently. If you explore the possibilities here please let us know!
Very impressive and will definitely server many use-cases. However, it's static site with data refresh at build time. Does this mean there cannot be user-based row-level security (i.e. selective access)?
One of the main selling points of the clunky, general purpose drag-and-drop BI tools (Power BI, Tableau etc.) is selective access. This is especially important in larger enterprises and for customer-facing dashboards.
For example, you're an enterprise manufacturing and selling IoT devices and have many different corporate customers. When you build a dashboard you want to make sure that each customer can see the data that belongs to their account and potentially, have further user-based restrictions. Obviously this goes against the idea of creating pre-aggregated datasets and instant loads but it's a massive multi-billion gap that currently is being filled by inferior tools to D3/Plot/Framework. This is something that Observable could develop in the future given what I'm seeing now and considering how relatively close already you are to this. Framework could serve both types of needs - static sites and dynamic, user-based more fully-featured sites for Enterprise needs.
Right, conceptually it’s static files, but we could develop a hybrid approach where the server does additional data processing on-demand. We already offer access control, but we could also serve different data snapshots to different users, or even filter the data snapshots based on the user. It still has to be fast, though.
One question I have is if there's a way to integrate an observable framework project into an existing static site? I see how I could easily add a project as a subdomain, but what if I wanted to interleave a project I make with observable framework into my existing domain and that static site generator I already use for that domain?
By the way, thank you making this. I've been reading and enjoying very much the documentation. It looks like it has huge potential.
That is my point too.
Now that I have tried Observable Framework (and before it D3, Plot, Observable Notebook) I do not think I can propose to change our statically generated site to just use Observable Framework.
I will explore ways to migrate parts of the existing stuff and how integrate new pages generated by Framework...
A Big Bang is not an option for us...
(for anybody I guess, so it looks quite a need...but I understand that it doesn't go in the right business direction for Observable the company)
Thank you. At a minimum, you could iframe pages built with Framework, or have them live alongside your other pages and link to them. Maybe it would be possible to use Framework’s internal API to generate HTML that could be embedded within another static site generator page but we haven’t explored that idea yet.
Thank you Mike for pushing the visualisation envelope for so many years.
Is the new Framework going to support virtualized data access for data sets too large to be sent over network (think of a pivot table that allows to browse huge data warehouse) - it is impossible to prepare entire file upfront, so data queries must happen incrementally with users actions? Or is it completely the other direction from where your vision for Framework is?
If you generate Apache Parquet files you can use DuckDB to make range requests and not download everything to the client. This is pretty magical and allows you to have surprisingly large datasets still queryable at interactive speeds.
But the general idea is to not send everything the client — to be more deliberate and restrictive in what you send, and also what you show. So you probably shouldn’t use this for a general-purpose pivot table that’s trying to show “everything” in your data warehouse and enable ad hoc exploration. You’d instead design more specific, opinionated views, and then craft corresponding data loaders that generate specific pre-aggregated datasets.
This seems nice and the plots look great, but I have a hard time imagining switching to Observable from Plotly since there doesn't seem to be a way to make any plots interactive. By which I mean Zoom and Pan. The nearest point high light feature is nice, but what if I want to zoom in? None of the examples here seem to be able to do that and quick google search doesn't make it seem like that's straight forward. That's not even additional code when I use Plotly, it's just built-in.
There's also the issue of convincing staff to use JS instead of Python which is still just a tough sell. I think everyone on my team (of data scientists) would look at me like I've got two heads if I were to suggest that. Maybe we're not the target demographic though.
I do like the idea of shipping the data straight to the client, but I don't have a lot of confidence in our corporate network doing well and not slowing stuff down. Perhaps the graphics all are sent pre-rendered over the wire though? I'm not sure, but would be cool if Observable figured out a way to side step that issue.
We’ve been focused primarily on the static display of visualizations because that’s what viewers see first, and often that’s often the only thing they see. Relying too heavily on interaction places an onus on the user to find the insights; a good display of data should be opinionated about what it shows and guide the user to what is interesting.
We’re not trying to convince you to switch to JavaScript here — a main value prop of Observable Framework is that you can write data loaders in any language (Python, R, Go, Julia, etc.). So do all your data preparation and analysis in whatever language you like, and then do your front-end in JavaScript to leverage the graphics and interactive compute capabilities of modern browsers. It’s pipes and child_process.spawn under the hood. And you still get instant reactivity when you save changes to your data loaders (when you edit Python) because Framework watches files and pushes new data to the client with reactive hot data & module replacement.
And you can compress (aggregate or filter) the data as much as you like, so it’s up to you how much data you send to the client. For example your data loader could be a minimal CSV file that’s just the numbers you need for a bar chart. Or it could be a Parquet file and you use DuckDB (https://observablehq.com/framework/lib/duckdb) on the client to generate dynamic visualizations.
Observable is much more than its library, plot. You mean to compare plot to plotly.
There are a number of reasons to choose Observable’s plot over plotly, but to address your point, there is no lock-in here with using plot for the view - you can seemingly use any JS library, including plotly, vega, D3, etc., so I don’t think that’s a huge issue.
I agree with your point regarding convincing other scientists to use JavaScript - that was the biggest point of failure for Observable notebook adoption that I saw. (As an anecdote, rather than adopt Observable, my science team @bigtech decided to write a Jupyter -> interactive static site transpiler, so the scientists could continue their work in python). Observable 2.0 seems built on recognizing that friction, and making it so that the it’s much easier for non-js users to collaborate. But the npm dependency will still scare many data folks away.
To anyone from
observable reading: I think getting mass adoption involves making this as seamless for python users as possible. (E.g. something similar to marimo notebooks or evidence). Also: great work!
Though it’s not designed for animation, Observable Plot is just a JavaScript library and it renders fast enough that you can do things like that just by re-rendering. Here are some old notebooks with experiments with audio data hooked up to UI controls:
Congratulations on this release! Your writing at bost.ocks.org, D3, and Observable have been big sources of inspiration over the years, and it’s always exciting to see new ideas from this team.
Is this meant to be a competitor to tools like Tableau or Metabase? Something more dev-friendly and maybe git-versioned as opposed to a configurable SaaS tool?
Hosting & compute — operationalizing/productionizing data apps. Observable Framework is open-source, but our hope is that we offer a compelling complementary paid service for you to host your (typically private) data apps on Observable. We make it easy for you to share data apps securely with your team or customers or clients or whoever, and manage the complexities of keeping your app & data up-to-date with continuous deployment, scheduled builds, access control, collaboration, monitoring, analytics, etc.
While the docs look great, I'm having trouble getting over the hump of starting. It would be great if you had a repo with a started app we could fork and play around with to help us understand everything before diving in from scratch.
Did you try running `npm init @observablehq`? It’ll create a starter app for you with everything you need to get started, as described in the Getting started tutorial.
Another tidbit buried in this announcement is that Observable Framework is 100% vanilla JavaScript syntax — so you get Observable’s reactive runtime without the quirky Observable JavaScript syntax (as in Observable notebooks). And you can use static ES imports from npm or local modules, declare multiple top-level variables in a code block (not just a single named variable per cell), call the built-in display(…) function to put things on the page, etc. It’s a huge relief to have vanilla syntax and greatly improves interoperability. And we’re figuring out how to port these improvements back to Observable notebooks in the near future.
With regard to code edits (rather than UI reactivity), this looks similar to how many web development environments watch the file system for changes and then rebuild and reload the page. Is there more to it?
How are syntax errors reported? Is there support for TypeScript syntax and type-checking? Can a page partially run that has errors in some JavaScript snippets, like a notebook with errors in some cells?
In the examples, there is a “view source” link that goes to GitHub. Understanding the code involves finding the Markdown file and then going back and forth between the published page and the Markdown file, which hopefully correspond to the same version.
It seems like the thing that’s lost compared to notebooks is letting the user see and edit the code in the browser. But I suppose that’s a niche use case for coding tutorials. Not everything needs to be a notebook.
Even so, better built-in “view source” support might be nice, even if it doesn’t allow editing. It doesn’t have to be as prominent as it is in a notebook to be useful.
The “trick” is to structure all code as reactive variables (or nodes, defined as pure functions) within a dataflow graph. So if you replace one variable (by replacing a function with a new import), you then have to recompute any downstream variables that depend on the replaced variable, while cleaning up old variables and updating the display.
Invalid syntax doesn’t prevent other code blocks from running (though if it means a variable is then undefined, that might cause downstream errors). Syntax errors are displayed in the page, and also in the console for the running preview server. We’d like to improve the error display in the console to show more context around where the error occurred, since unlike notebooks the code isn’t immediately adjacent to the output.
And yes, we’re making different tradeoffs, optimizing for data apps and dashboards (more polished presentation) rather than ad hoc exploration in notebooks. So it’s more work to find and edit the code, but conversely it’s a more careful, deliberate process that allows code review, unit tests, continuous integration, etc. And we think that’s appropriate for data apps that are depended on by many people.
But still, a view source link back to your source control would be nice, yes!
I too appreciate this. It is so easy to turn javascript into a DSL that doesn't much look like vanilla javascript. Give me a great API anytime over a DSL except in some very specific cases.
I appreciate the nod to "File over app"[1] in the announcement. It's so cool that a Markdown file with code blocks can be the source for complex data visualizations and dashboards. Interoperability of this kind makes me giddy. I played around with editing an Observable site from Obsidian and it works great[2].
We’ve talked about and shared your “File over app” manifesto so many times internally over the last few months. It’s one of those tweets that gets immortalized as the perfect crystallization of an ethos that we might otherwise have only been able to gesture at vaguely. It gives the ethos weight and clarity and credibility, and it’s such a relief to be able to point to it! I’m very grateful. —an Observable employee
My mind immediately went to how these Dasboards could be integrated in Obsidian, and seeing the `import` dependency graph reflected in Obsidian's graph view.
I love Observable. And, this is a phenomenal approach, untethering Observable from observablehq.com. I'm so excited.
It probably goes without saying that EVERYONE should have a blog, and this approach from Observable means journalists everywhere can now easily create a dynamic and information-driven blog. It isn't a coincidence that Observable came from a guy that did amazing data visualizations at the NYTimes. We are on the precipice of a major power shift back to journalists and away from dubious corporations, and tools like this enable that transition.
(Shameless plug: Svekyll is going towards the same goal. Svekyll is a static site generator inspired by Jekyll. If you want to use Svelte in your blog, check it out. Svekyll bundles the incredible Apache ECharts so you can use echarts in your blog with a few lines of code (and no complicated build process). https://extrastatic.dev/svekyll/svekyll-cli. These are ideas I've been thinking about too.)
There's an almost bewildering amount of interesting ideas buried in this.
Things like data loaders which are ANY script that can output data (as JSON or something else) to standard output. Markdown files with ```js blocks in that get executed. The reinvention of the core Observable notebook to avoid custom syntax.
Yeah data loaders are like a UNIX pipe to a reactive notebook cell (?). There needn’t be any question of “do the data loaders support this or that”; it doesn’t even have a concept of “supporting” beyond supporting stdout… Still thinking through how to understand it myself!!
Data loaders seem like an interesting way to define a multi-language build system without having to write a makefile. Lots of build systems do this, but the boundaries between build steps often isn't as clean and uniform as having a single output per build step and relying on a file naming convention.
It's not truly reactive if you have to do a build to make anything happen. But maybe that doesn't matter, as long as it's reactive during development?
The thing about Markdown code blocks is that they're meant to be read (and therefore rendered—as text). "Markdown files with ```js blocks in that get executed" is just <script>.
I'm super excited to try this out! Couple of questions since I see @mbostock active in the comments.
1. Is the flexibility of languages used in data loaders/backend going to eventually come to the front end/ui? Or will the paradigm always be bring-your-own-language for the data loading but build your dashboard with observablejs/observable plot?
2. Considering ObservableJS is supported by Quarto, can we look forward to Observable Framework integrated with Quarto too? Or is the fact that the latest Quarto version also featured Dashboards more of a competitor to Framework?
3. Saw some comparison to Evidence.dev in the comments. I saw some shades of similarity with the markdown focused dev experience too but I recall Evidence chose Apache Echarts for their main charting library. Any thoughts of the pros/cons of Echarts vs ObservableJS/Plot?
(disclosure: Quarto dev here). I'm a huge Observable fan.
Speaking entirely for myself, this space is so important that I'm thrilled to have more activity rather than less. Quarto's great and Observable's great. I hope folks pick the tool that's best for their use case!
I'm looking forward to learning more about which one makes it easier to see how various possible changes in the data are mapped to legible changes in the visualization.
1. We don’t have immediate plans to bring other languages to the front-end — maybe TypeScript, but that’s just stripping annotations; maybe some WebAssembly. Our idea is to have a clear serializable “membrane” separating your back-end (in any language, running on build on your servers) from your front-end (in JavaScript, running on load in the client). Data loaders produce data during build, which gets handed-off to the client to render. Trying to do data processing on the client is often a frustrating and poor user experience. Likewise trying to render great interactive charts without web technologies is quite limiting!
2. I can’t speak to Quarto’s plans. Observable Framework is open-source so they might pick up some of this stuff. I look at Framework more as an alternative to Quarto than a complement.
3. As the creator of Observable Plot (and D3 before that), I’m a huge fan of visualization grammars! Apache Echarts is a chart typology, and while it’s got a lot of chart types in it, it has no overarching conceptual model of how to represent a visualization. And so it’s not very interesting. But “the proof of the pudding is in the eating” as I say in the post, so I encourage you to look at Observable Plot and decide for yourself if you like both the syntax and the resulting plots. I certainly do!
Leland Wilkinson said it best: “If we endeavor to develop a charting instead of a graphing program, we will accomplish two things. First, we inevitably will offer fewer charts than people want. Second, our package will have no deep structure. Our computer program will be unnecessarily complex, because we will fail to reuse objects or routines that function similarly in different charts. And we will have no way to add new charts to our system without generating complex new code. Elegant design requires us to think about a theory of graphics, not charts.”
From the Observable Framework point of view, you’re very welcome to use Apache ECharts or any other library instead of Observable Plot, since you can import whatever you like and it’s all just JavaScript.
There are two pieces of that example code specific to Framework: the html`` tagged template literal creates a DOM element (see https://github.com/observablehq/htl, also usable outside Framework), and the display function inserts it into the document above the code block (see https://observablehq.com/framework/javascript/display). Note that, whereas Observable Plot takes an options object and returns a DOM element, ECharts instead takes a DOM element and mutates it — but in general they should be equally easy to use in Framework.
Like Plot (and Vega-Lite, another great option), ECharts is also now one of Framework’s built-in “recommended libraries” (see https://observablehq.com/framework/javascript/imports#implic...), meaning that if you reference `echarts` Framework will lazy-load it for you. Adding that was a two-line diff: https://github.com/observablehq/framework/pull/811/files#dif.... But I wanna emphasize that Framework doesn’t have to explicitly “support” a given library for you to use it. “Supporting” in this case just means the convenience of saving you a one-line import statement. But don’t wait for our blessing!! Use whatever.
3. Apache echarts are much more interactive out of the box. The API is indeed clunky, but they’ve got all the chart type and all interactions you might need.
IMHO, Plot in comparison, is very limited in interactivity and even chart types ( there are no heat maps or donuts).
echarts have a huge example library with clear examples and though Plot has it too, the library is not thought out well. You might looks at an example in the Plot Library only to realize later that it’s a D3 example.
On the good side, the API in Plot is much cleaner and easier to work with.
We generally recommend stacked bar charts over pie and donut charts, so we haven’t prioritized those. But you can already implement them using custom marks, and there’s even a hacky way of doing them using Plot’s map projection system.
I don’t understand your comment about the “D3 example.” If you’re looking for Plot examples, you can find them linked from the Plot documentation and the gallery:
Plot is designed to be extended with JavaScript (rather than a non-JavaScript DSL such as Vega-Lite), such as for custom marks and data transforms. So you might occasionally see other libraries being used together with Plot.
I can second a +1 vote for eCharts. I have used them for fast updating charts (10x second) and they work and look great. I've only had 2-3 small niggles with them in 2years of heavy usage.
1. Let's say I got a Sqlite/Duckdb database file on my server. It's got multiple tables and some of them 100M to 150M records. I want to create a plot/table that would have a slider/filter to only bring and show a slice of data at a time. Since it's statically generated data, how is this interactivity achieved? All the possible facets of data filtered by which ever way will be generated? Won't it be huge and how long will it take to generates this static data or is there an actual call back to the server to the duckdb file (I assume it works with .duckdb file too?)
2. If Observable Framework provides the front-end, does it mean I can use any auth library if I want to create a web site with a log in and subscription options?
3. If it's a static web page, does it mean that at any time a user views a chart, they will also be able to go to the Dev Tools and download the file with data that's behind the viz?
4. When (if you can share of course) is the planned release of Plot's interactions: zoom, pan, interactive legend, brush?
5. Deployment - with big parquet, sqlite, csv files, it's impossible to do CI/CO through github or vercel and such. Will your hosting services offer an option to host those files and runtimes to generate them?
2. Sure, it’s all open source, I bet you could make that work. Or `yarn deploy` to Observable and configure sharing there (though it wouldn’t let you charge others).
3. Yup. Which is part of the appeal of model of running data loaders at build time: you can query some private data and viewers would only be able to see the final result set. (The lack of something like this has always been a huge problem for Observable notebooks. You’d make some great query-driven charts and then couldn’t make it public without some awkward manual dance of downloading and re-uploading a file to a fork of the notebook.)
Thank you
I wonder if for #3 there is a way of somehow to keep the data hidden and only let people see the chart without hacking their way to see the underlying data
Came here with similar questions and Cmd-F "DuckDB". See the comment about "data loaders". Seems like a "data loader" would provide most of what you're asking about.
I'm also thinking that a "data loader" combined with duckdb-wasm and arrow would be a pretty nice combination. I imagine that it might not be too difficult to switch two between two implementations of the "data loader" as needed. Switch between reading from a remote system (in your case DuckDB on a server) and DuckDB running locally in the browser (that can interact with its own remote or local data sources).
See the example at https://huggingface.co/spaces/observablehq/fpdn where DuckDB is used both as a data loader (to download and digest 200GB worth of source data into a small 8MB parquet file) and on the client-side to allow the user to do live search queries on the minimized data. Server-side, we're using duckdb-the-binary, and client-side we're using duckdb-wasm.
I am totally psyched up to try this. Observable (1.0) has been a very effective outlet for my research. There is no other platform that could have hosted it and offered me the quick and easy tools to make persuasive visualizations. I was a little concerned when I heard some people got laid off from the company, but it seems like it is still going.
Readit News
https://observablehq.com/framework/examples/api/
This technique of “just plot everything” (7.6M requests as a scatterplot) has revealed surprising insights we’ve used to optimize our servers and better control traffic. We’re also sharing a more traditional dashboard that visualizes the adoption of our open-source visualization library (and in some ways the successor to D3), Observable Plot:
https://observablehq.com/framework/examples/plot/
In addition to releasing Observable Framework, we’ve also made Observable free again for individuals (including private notebooks and databases connectors). Let me know if you have any questions!
I tried to get our team to use Observable Notebooks a few years back. The researchers I work with are more comfortable in Python. Clearly that's one of the things you're trying to solve in this release. The other half of that uphill battle was discomfort posting code externally. In some ways you've also mitigated that in this release, but I wonder how sustainable it is.
Small teams eat for free by virtue of being small. Large organizations with trepidation or bureaucracy about using SaaS hosting will self host. That leaves the people in the middle: big enough to need to pay, but small enough to not have institutional problems with external hosting. Moreover, if the Observable bill ever gets much higher than the equivalent on Firebase et. al., the medium guys can self-host too.
How do you anticipate the paid side of the new business to work out? What's the hook (beyond thinking you guys are cool and trying to keep you in business) that gets someone to pay for Observable?
My question is: what does Observable Framework offer for data storytellers who want to blog? Is this meant to go up against Hugo/Jekyll in terms of full-fledged max-efficiency site generation? If not, are there plans to add integrations with other blogging frameworks?
[0]: http://r2d3.us/ [1]: https://algorithms-tour.stitchfix.com/ [2]: https://vermarish.github.io/big-brother-barometer/
I’m not sure what better integration with other blogging frameworks would look like — like, part of the page is rendered by Framework, but the site as a whole is handled by the blogging framework? Perhaps we could develop Framework’s API further so it could function like a plugin. But this is speculative and not a priority for us currently. If you explore the possibilities here please let us know!
One of the main selling points of the clunky, general purpose drag-and-drop BI tools (Power BI, Tableau etc.) is selective access. This is especially important in larger enterprises and for customer-facing dashboards.
For example, you're an enterprise manufacturing and selling IoT devices and have many different corporate customers. When you build a dashboard you want to make sure that each customer can see the data that belongs to their account and potentially, have further user-based restrictions. Obviously this goes against the idea of creating pre-aggregated datasets and instant loads but it's a massive multi-billion gap that currently is being filled by inferior tools to D3/Plot/Framework. This is something that Observable could develop in the future given what I'm seeing now and considering how relatively close already you are to this. Framework could serve both types of needs - static sites and dynamic, user-based more fully-featured sites for Enterprise needs.
By the way, thank you making this. I've been reading and enjoying very much the documentation. It looks like it has huge potential.
Is the new Framework going to support virtualized data access for data sets too large to be sent over network (think of a pivot table that allows to browse huge data warehouse) - it is impossible to prepare entire file upfront, so data queries must happen incrementally with users actions? Or is it completely the other direction from where your vision for Framework is?
But the general idea is to not send everything the client — to be more deliberate and restrictive in what you send, and also what you show. So you probably shouldn’t use this for a general-purpose pivot table that’s trying to show “everything” in your data warehouse and enable ad hoc exploration. You’d instead design more specific, opinionated views, and then craft corresponding data loaders that generate specific pre-aggregated datasets.
There's also the issue of convincing staff to use JS instead of Python which is still just a tough sell. I think everyone on my team (of data scientists) would look at me like I've got two heads if I were to suggest that. Maybe we're not the target demographic though.
I do like the idea of shipping the data straight to the client, but I don't have a lot of confidence in our corporate network doing well and not slowing stuff down. Perhaps the graphics all are sent pre-rendered over the wire though? I'm not sure, but would be cool if Observable figured out a way to side step that issue.
We’ve been focused primarily on the static display of visualizations because that’s what viewers see first, and often that’s often the only thing they see. Relying too heavily on interaction places an onus on the user to find the insights; a good display of data should be opinionated about what it shows and guide the user to what is interesting.
We’re not trying to convince you to switch to JavaScript here — a main value prop of Observable Framework is that you can write data loaders in any language (Python, R, Go, Julia, etc.). So do all your data preparation and analysis in whatever language you like, and then do your front-end in JavaScript to leverage the graphics and interactive compute capabilities of modern browsers. It’s pipes and child_process.spawn under the hood. And you still get instant reactivity when you save changes to your data loaders (when you edit Python) because Framework watches files and pushes new data to the client with reactive hot data & module replacement.
And you can compress (aggregate or filter) the data as much as you like, so it’s up to you how much data you send to the client. For example your data loader could be a minimal CSV file that’s just the numbers you need for a bar chart. Or it could be a Parquet file and you use DuckDB (https://observablehq.com/framework/lib/duckdb) on the client to generate dynamic visualizations.
There are a number of reasons to choose Observable’s plot over plotly, but to address your point, there is no lock-in here with using plot for the view - you can seemingly use any JS library, including plotly, vega, D3, etc., so I don’t think that’s a huge issue.
I agree with your point regarding convincing other scientists to use JavaScript - that was the biggest point of failure for Observable notebook adoption that I saw. (As an anecdote, rather than adopt Observable, my science team @bigtech decided to write a Jupyter -> interactive static site transpiler, so the scientists could continue their work in python). Observable 2.0 seems built on recognizing that friction, and making it so that the it’s much easier for non-js users to collaborate. But the npm dependency will still scare many data folks away.
To anyone from observable reading: I think getting mass adoption involves making this as seamless for python users as possible. (E.g. something similar to marimo notebooks or evidence). Also: great work!
https://observablehq.com/collection/@skybrian/observable-plo...
How is Observable going to make money off of the framework?
Time to call it quits for https://www.jigdev.com :-D
Godspeed Observable, hope you guys make it big
Observable Notebooks reactivity feels intuitive, much like spreadsheets, but the lack of self-hosting options is no-go Drawback in my work context.
https://observablehq.com/framework/getting-started
If you want more starter apps to look at, you can browse our examples on GitHub:
https://github.com/observablehq/framework/tree/main/examples
How are syntax errors reported? Is there support for TypeScript syntax and type-checking? Can a page partially run that has errors in some JavaScript snippets, like a notebook with errors in some cells?
In the examples, there is a “view source” link that goes to GitHub. Understanding the code involves finding the Markdown file and then going back and forth between the published page and the Markdown file, which hopefully correspond to the same version.
It seems like the thing that’s lost compared to notebooks is letting the user see and edit the code in the browser. But I suppose that’s a niche use case for coding tutorials. Not everything needs to be a notebook.
Even so, better built-in “view source” support might be nice, even if it doesn’t allow editing. It doesn’t have to be as prominent as it is in a notebook to be useful.
https://observablehq.com/@observablehq/how-observable-runs
And the source is here:
https://github.com/observablehq/runtime
The “trick” is to structure all code as reactive variables (or nodes, defined as pure functions) within a dataflow graph. So if you replace one variable (by replacing a function with a new import), you then have to recompute any downstream variables that depend on the replaced variable, while cleaning up old variables and updating the display.
Invalid syntax doesn’t prevent other code blocks from running (though if it means a variable is then undefined, that might cause downstream errors). Syntax errors are displayed in the page, and also in the console for the running preview server. We’d like to improve the error display in the console to show more context around where the error occurred, since unlike notebooks the code isn’t immediately adjacent to the output.
We don’t support TypeScript yet, but there’s a PR (https://github.com/observablehq/framework/pull/129) and we are interested in stronger validation at build time to catch more errors.
And yes, we’re making different tradeoffs, optimizing for data apps and dashboards (more polished presentation) rather than ad hoc exploration in notebooks. So it’s more work to find and edit the code, but conversely it’s a more careful, deliberate process that allows code review, unit tests, continuous integration, etc. And we think that’s appropriate for data apps that are depended on by many people.
But still, a view source link back to your source control would be nice, yes!
[1]: https://stephango.com/file-over-app
[2]: https://twitter.com/kepano/status/1758202572446581025
Deleted Comment
It probably goes without saying that EVERYONE should have a blog, and this approach from Observable means journalists everywhere can now easily create a dynamic and information-driven blog. It isn't a coincidence that Observable came from a guy that did amazing data visualizations at the NYTimes. We are on the precipice of a major power shift back to journalists and away from dubious corporations, and tools like this enable that transition.
(Shameless plug: Svekyll is going towards the same goal. Svekyll is a static site generator inspired by Jekyll. If you want to use Svelte in your blog, check it out. Svekyll bundles the incredible Apache ECharts so you can use echarts in your blog with a few lines of code (and no complicated build process). https://extrastatic.dev/svekyll/svekyll-cli. These are ideas I've been thinking about too.)
Things like data loaders which are ANY script that can output data (as JSON or something else) to standard output. Markdown files with ```js blocks in that get executed. The reinvention of the core Observable notebook to avoid custom syntax.
This is really big.
It's not truly reactive if you have to do a build to make anything happen. But maybe that doesn't matter, as long as it's reactive during development?
Deleted Comment
1. Is the flexibility of languages used in data loaders/backend going to eventually come to the front end/ui? Or will the paradigm always be bring-your-own-language for the data loading but build your dashboard with observablejs/observable plot?
2. Considering ObservableJS is supported by Quarto, can we look forward to Observable Framework integrated with Quarto too? Or is the fact that the latest Quarto version also featured Dashboards more of a competitor to Framework?
3. Saw some comparison to Evidence.dev in the comments. I saw some shades of similarity with the markdown focused dev experience too but I recall Evidence chose Apache Echarts for their main charting library. Any thoughts of the pros/cons of Echarts vs ObservableJS/Plot?
Speaking entirely for myself, this space is so important that I'm thrilled to have more activity rather than less. Quarto's great and Observable's great. I hope folks pick the tool that's best for their use case!
2. I can’t speak to Quarto’s plans. Observable Framework is open-source so they might pick up some of this stuff. I look at Framework more as an alternative to Quarto than a complement.
3. As the creator of Observable Plot (and D3 before that), I’m a huge fan of visualization grammars! Apache Echarts is a chart typology, and while it’s got a lot of chart types in it, it has no overarching conceptual model of how to represent a visualization. And so it’s not very interesting. But “the proof of the pudding is in the eating” as I say in the post, so I encourage you to look at Observable Plot and decide for yourself if you like both the syntax and the resulting plots. I certainly do!
Leland Wilkinson said it best: “If we endeavor to develop a charting instead of a graphing program, we will accomplish two things. First, we inevitably will offer fewer charts than people want. Second, our package will have no deep structure. Our computer program will be unnecessarily complex, because we will fail to reuse objects or routines that function similarly in different charts. And we will have no way to add new charts to our system without generating complex new code. Elegant design requires us to think about a theory of graphics, not charts.”
Since there was a lot of interest in this thread, Mike added a page to the docs with an ECharts example: https://observablehq.com/framework/lib/echarts
There are two pieces of that example code specific to Framework: the html`` tagged template literal creates a DOM element (see https://github.com/observablehq/htl, also usable outside Framework), and the display function inserts it into the document above the code block (see https://observablehq.com/framework/javascript/display). Note that, whereas Observable Plot takes an options object and returns a DOM element, ECharts instead takes a DOM element and mutates it — but in general they should be equally easy to use in Framework.
Like Plot (and Vega-Lite, another great option), ECharts is also now one of Framework’s built-in “recommended libraries” (see https://observablehq.com/framework/javascript/imports#implic...), meaning that if you reference `echarts` Framework will lazy-load it for you. Adding that was a two-line diff: https://github.com/observablehq/framework/pull/811/files#dif.... But I wanna emphasize that Framework doesn’t have to explicitly “support” a given library for you to use it. “Supporting” in this case just means the convenience of saving you a one-line import statement. But don’t wait for our blessing!! Use whatever.
echarts have a huge example library with clear examples and though Plot has it too, the library is not thought out well. You might looks at an example in the Plot Library only to realize later that it’s a D3 example. On the good side, the API in Plot is much cleaner and easier to work with.
https://observablehq.com/plot/marks/rasterhttps://observablehq.com/plot/marks/contourhttps://observablehq.com/plot/marks/cell
We generally recommend stacked bar charts over pie and donut charts, so we haven’t prioritized those. But you can already implement them using custom marks, and there’s even a hacky way of doing them using Plot’s map projection system.
https://observablehq.com/@observablehq/pie-to-donut-chart
I don’t understand your comment about the “D3 example.” If you’re looking for Plot examples, you can find them linked from the Plot documentation and the gallery:
https://observablehq.com/@observablehq/plot-gallery
Plot is designed to be extended with JavaScript (rather than a non-JavaScript DSL such as Vega-Lite), such as for custom marks and data transforms. So you might occasionally see other libraries being used together with Plot.
1. Let's say I got a Sqlite/Duckdb database file on my server. It's got multiple tables and some of them 100M to 150M records. I want to create a plot/table that would have a slider/filter to only bring and show a slice of data at a time. Since it's statically generated data, how is this interactivity achieved? All the possible facets of data filtered by which ever way will be generated? Won't it be huge and how long will it take to generates this static data or is there an actual call back to the server to the duckdb file (I assume it works with .duckdb file too?)
2. If Observable Framework provides the front-end, does it mean I can use any auth library if I want to create a web site with a log in and subscription options?
3. If it's a static web page, does it mean that at any time a user views a chart, they will also be able to go to the Dev Tools and download the file with data that's behind the viz?
4. When (if you can share of course) is the planned release of Plot's interactions: zoom, pan, interactive legend, brush?
5. Deployment - with big parquet, sqlite, csv files, it's impossible to do CI/CO through github or vercel and such. Will your hosting services offer an option to host those files and runtimes to generate them?
Thanks
1. It’s just JavaScript so you can fetch stuff dynamically too (see https://observablehq.com/framework/lib/duckdb). But yeah, only client-side. (Though see https://github.com/observablehq/framework/issues/234.)
2. Sure, it’s all open source, I bet you could make that work. Or `yarn deploy` to Observable and configure sharing there (though it wouldn’t let you charge others).
3. Yup. Which is part of the appeal of model of running data loaders at build time: you can query some private data and viewers would only be able to see the final result set. (The lack of something like this has always been a huge problem for Observable notebooks. You’d make some great query-driven charts and then couldn’t make it public without some awkward manual dance of downloading and re-uploading a file to a fork of the notebook.)
4. I wish I knew! It’s being tracked here https://github.com/observablehq/plot/issues/1711. Lately there’s been a lot more work on Framework naturally but now that that’s out…
5. Another good question. We’re definitely interested in tailoring it more to this sort of use case but lots is TBD!
I'm also thinking that a "data loader" combined with duckdb-wasm and arrow would be a pretty nice combination. I imagine that it might not be too difficult to switch two between two implementations of the "data loader" as needed. Switch between reading from a remote system (in your case DuckDB on a server) and DuckDB running locally in the browser (that can interact with its own remote or local data sources).
edit: welp https://observablehq.com/framework/lib/duckdb