My question is: what does Observable Framework offer for data storytellers who want to blog? Is this meant to go up against Hugo/Jekyll in terms of full-fledged max-efficiency site generation? If not, are there plans to add integrations with other blogging frameworks?
[0]: http://r2d3.us/ [1]: https://algorithms-tour.stitchfix.com/ [2]: https://vermarish.github.io/big-brother-barometer/
I’m not sure what better integration with other blogging frameworks would look like — like, part of the page is rendered by Framework, but the site as a whole is handled by the blogging framework? Perhaps we could develop Framework’s API further so it could function like a plugin. But this is speculative and not a priority for us currently. If you explore the possibilities here please let us know!
Edit: link: https://row64.com/
But in general the use case we’re targeting is a shared data app, dashboard, or report. Not something just for you individually, or something ephemeral (that you look at in real-time during training). For example, Framework would work well for sharing a report or dashboard evaluating the performance of the latest models you’ve built with your team.
Is the new Framework going to support virtualized data access for data sets too large to be sent over network (think of a pivot table that allows to browse huge data warehouse) - it is impossible to prepare entire file upfront, so data queries must happen incrementally with users actions? Or is it completely the other direction from where your vision for Framework is?
But the general idea is to not send everything the client — to be more deliberate and restrictive in what you send, and also what you show. So you probably shouldn’t use this for a general-purpose pivot table that’s trying to show “everything” in your data warehouse and enable ad hoc exploration. You’d instead design more specific, opinionated views, and then craft corresponding data loaders that generate specific pre-aggregated datasets.
Another way to use DuckDB if you're warehouse supports it would be e2e Arrow (no col->row->col overhead).
warehouse --> ADBC --> arrow --> DuckDB
Of course this differs in that you would be reading from the warehouse directly, but in my experience fully pre-aggregating data and then staging it (keeping parquet files on S3 up to date) might solve one issue, but results in unimaginable issues. Perhaps the sweet spot might be something like Iceberg in the middle? warehouse --> iceberg table (parquet) --> DuckDB
> craft corresponding data loadersDo you have an example of this using DuckDB? I'm very interested in seeing an actual implementation of Observable's data loaders combined with DuckDB (or any other SQL DB)
edit: nm, found it. https://observablehq.com/framework/lib/duckdb
edit2: eh, I didn't even realize who i was responding to, lol. The more I read into this, the more I see how this is all heavily based on static files. So the static parquet files thing makes more sense, the solution I added makes little. Although I guess you could add a static iceberg table and interact with its manifest with the duckdb iceberg extension.
By the way, thank you making this. I've been reading and enjoying very much the documentation. It looks like it has huge potential.
There's also the issue of convincing staff to use JS instead of Python which is still just a tough sell. I think everyone on my team (of data scientists) would look at me like I've got two heads if I were to suggest that. Maybe we're not the target demographic though.
I do like the idea of shipping the data straight to the client, but I don't have a lot of confidence in our corporate network doing well and not slowing stuff down. Perhaps the graphics all are sent pre-rendered over the wire though? I'm not sure, but would be cool if Observable figured out a way to side step that issue.
We’ve been focused primarily on the static display of visualizations because that’s what viewers see first, and often that’s often the only thing they see. Relying too heavily on interaction places an onus on the user to find the insights; a good display of data should be opinionated about what it shows and guide the user to what is interesting.
We’re not trying to convince you to switch to JavaScript here — a main value prop of Observable Framework is that you can write data loaders in any language (Python, R, Go, Julia, etc.). So do all your data preparation and analysis in whatever language you like, and then do your front-end in JavaScript to leverage the graphics and interactive compute capabilities of modern browsers. It’s pipes and child_process.spawn under the hood. And you still get instant reactivity when you save changes to your data loaders (when you edit Python) because Framework watches files and pushes new data to the client with reactive hot data & module replacement.
And you can compress (aggregate or filter) the data as much as you like, so it’s up to you how much data you send to the client. For example your data loader could be a minimal CSV file that’s just the numbers you need for a bar chart. Or it could be a Parquet file and you use DuckDB (https://observablehq.com/framework/lib/duckdb) on the client to generate dynamic visualizations.
What are your thoughts on PRQL (integrated with DuckDB)?
Also: sure, I get your point on opinionated viz, but this means someone already "played with the data" to figure out what to viz "cleanly", and then coded that
now what if we want to build a viz to do this first "data playground" allowing to ruffle with the data? we would need some sort of interactivity
https://observablehq.com/collection/@skybrian/observable-plo...
There are a number of reasons to choose Observable’s plot over plotly, but to address your point, there is no lock-in here with using plot for the view - you can seemingly use any JS library, including plotly, vega, D3, etc., so I don’t think that’s a huge issue.
I agree with your point regarding convincing other scientists to use JavaScript - that was the biggest point of failure for Observable notebook adoption that I saw. (As an anecdote, rather than adopt Observable, my science team @bigtech decided to write a Jupyter -> interactive static site transpiler, so the scientists could continue their work in python). Observable 2.0 seems built on recognizing that friction, and making it so that the it’s much easier for non-js users to collaborate. But the npm dependency will still scare many data folks away.
To anyone from observable reading: I think getting mass adoption involves making this as seamless for python users as possible. (E.g. something similar to marimo notebooks or evidence). Also: great work!
https://observablehq.com/framework/getting-started
And plus you can import libraries directly from a CDN rather than needing to use npm or yarn to manage dependencies. (Though we plan on supporting the latter in the future, too.)
https://observablehq.com/framework/javascript/imports
See this example for getting started with Python:
https://github.com/observablehq/framework/tree/main/examples...
But of course we’d love to add more affordances and documentation for other languages. We’re naturally biased towards JavaScript as our focus has historically been on visualization, but I like to think we’re making progress on the polyglot dream.
Time to call it quits for https://www.jigdev.com :-D
Godspeed Observable, hope you guys make it big
How is Observable going to make money off of the framework?
In case you ever forget the difference between complementary and complimentary, that's it right there.
https://observablehq.com/framework/getting-started
If you want more starter apps to look at, you can browse our examples on GitHub:
https://github.com/observablehq/framework/tree/main/examples
Observable Notebooks reactivity feels intuitive, much like spreadsheets, but the lack of self-hosting options is no-go Drawback in my work context.
I tried to get our team to use Observable Notebooks a few years back. The researchers I work with are more comfortable in Python. Clearly that's one of the things you're trying to solve in this release. The other half of that uphill battle was discomfort posting code externally. In some ways you've also mitigated that in this release, but I wonder how sustainable it is.
Small teams eat for free by virtue of being small. Large organizations with trepidation or bureaucracy about using SaaS hosting will self host. That leaves the people in the middle: big enough to need to pay, but small enough to not have institutional problems with external hosting. Moreover, if the Observable bill ever gets much higher than the equivalent on Firebase et. al., the medium guys can self-host too.
How do you anticipate the paid side of the new business to work out? What's the hook (beyond thinking you guys are cool and trying to keep you in business) that gets someone to pay for Observable?
One of the main selling points of the clunky, general purpose drag-and-drop BI tools (Power BI, Tableau etc.) is selective access. This is especially important in larger enterprises and for customer-facing dashboards.
For example, you're an enterprise manufacturing and selling IoT devices and have many different corporate customers. When you build a dashboard you want to make sure that each customer can see the data that belongs to their account and potentially, have further user-based restrictions. Obviously this goes against the idea of creating pre-aggregated datasets and instant loads but it's a massive multi-billion gap that currently is being filled by inferior tools to D3/Plot/Framework. This is something that Observable could develop in the future given what I'm seeing now and considering how relatively close already you are to this. Framework could serve both types of needs - static sites and dynamic, user-based more fully-featured sites for Enterprise needs.
https://observablehq.com/framework/examples/api/
This technique of “just plot everything” (7.6M requests as a scatterplot) has revealed surprising insights we’ve used to optimize our servers and better control traffic. We’re also sharing a more traditional dashboard that visualizes the adoption of our open-source visualization library (and in some ways the successor to D3), Observable Plot:
https://observablehq.com/framework/examples/plot/
In addition to releasing Observable Framework, we’ve also made Observable free again for individuals (including private notebooks and databases connectors). Let me know if you have any questions!