riordan

We haven’t decided that—Substrait[0] is still the IR for query plans across different execution engines—but projects like QueryBrew are a great side effect of what a SOTA planner can accomplish.

[0]: https://substrait.io

Except as the corporate spending card company they see just how much their customers are spending on inference and have made it a cornerstone of their marketing for the last two years.

Their positioning is: You already have all your teams spending routing through us so that through greater transparency to finance you get better spending controls. We now apply the same to inference, which is spread out across all parts of your company in a similar way.

All summed up really well with their tagline “Tokens are money. Save both.”

Don’t think I’ll be using model routing / shared gateway from Ramp any time soon, but the value of this kind of capability is definitely there. And there’s no shortage of vendors aiming to offer this kind of gateway/router functionality. But most of those come from the technology side of the organization. This gets to come from the office of the CFO.

To be fair we were done transcribing and double verifying in about 2015. We’d get a batch of new menus from the imaging team, tweet it and no matter the size it’d be gone in about an hour.

And alligator pears are avocados!

The dataset of menus and items is all hand labeled and verified at menus.nypl.org.

This is why I preferred Mike Migurski’s nickname for PHP: Apachescript; easy to integrate to the web server and next fastest thing to C for Apache.

This absolutely rocks: like an open and usable version of the once beloved and now deprecated Echo Nest Audio Analysis api [0]. There used to be a tool to use this data to manage your own (Spotify-only) music library and while it wasn’t magic, it sure felt like it.

Super excited to try AudioMuse

[0]: https://developer.spotify.com/documentation/web-api/referenc...

11 years ago, NY Magazine published my favorite startup zeitgeist article of the Uber-For-X era: “Let’s, like, demolish Laundry”. This feels like a perfect revisit of that for our darker and more cynical era.

https://nymag.com/news/features/laundry-apps-2014-5/

Seriously - I find myself coming back to read this once every few years because of how riveting the piece is (oh no -just realized the pun)

Also I’ve heard wonderful things about The Great Miscalculation[0], a recently released book about the Citicorp Tower incident

[0]: https://search.worldcat.org/en/title/1458613829

There have been several projects over the past few years to make text embeddings visually explorable (notably Nomic.ai's Nomic Atlas). However, Apple's just released a tool that makes this kind of analysis super accessible and insanely interactive.

Under the hood it's powered by Mosaic[0], a dataviz library built on top of DuckDB that's designed to handle coordinated interactive plots over huge datasets, the kind of thing where you interact with one plot and the rest all respond, which requires going back to the database to recalculate all the aggregations.

I've been fanboying Mosaic for the past year but finally have this to point to as an illustration of what's possible with it.

[0]: https://idl.uw.edu/mosaic

> Polygons, curves, or lines are an afterthought. For example, the polygonToCells function returns a set of cells that are entirely within the polygon. If you want to join, you'd need to also have the set of all cells that entirely contain the polygon. I've never found a reliable way to get that.

With v4 of h3 they (finally) have a clean syntax for this with polygonToCellsExperimental[0].

Now there’s options for

- Cell center is contained in the shape (default) - Cell is fully contained in the shape - Overlapping (covering): Cell overlaps the shape at any point - BBOX: Cell bounding box overlaps shape

Makes life a fair bit easier if you’ve gotta deal with H3 polys. And if you’re working locally, DuckDB Spatial’s r-tree indexing[1] can make for a nice stand-in for PostGIS as a quick point-in-polygon solution without the need to spin up a service.

[0]: https://h3geo.org/docs/api/regions/#polygontocellsexperiment... [1]: https://duckdb.org/docs/stable/extensions/spatial/r-tree_ind...

Chronicling America is great, but as you’ve noticed it’s limited to issues that are in the public domain. It’s not just LC, the National Digital Newspaper Program[1] has been funding hundreds of libraries across all 50 USA states to make it possible since the mid-00s.

Australia’s national library offers Trove[2], which has a huge collection of Australian public domain newspapers.

Most of their repository is likely funded by your tax dollars and is there for the public to use.

[1]: https://www.neh.gov/divisions/preservation/national-digital-... [2]:https://trove.nla.gov.au/newspaper/

It’s good to hear that PWAs are still on Firefox Android, even though they’re out of Desktop.

From my [fairly-out-of-the-loop-for-the-past-few-years] vantage point, Mozilla’s been a lot less invested in the PWA ecosystem since they abandoned their Firefox Phone / Boot2Gecko initiative, which was intended to create a middle tier between the expensive smartphones of the early 2010’s and ubiquitous and cheap feature phones (flip phones, classic Nokia candy bars), and expand access to the web across the world with it.

All the apps were PWAs, which made it simple to build out. Eventually Mozilla stopped the project, but KaiOS became a commercial implementation and it still runs on a fair number of feature phones to this day.

But without that pressure for PWA support in Firefox as a critical mobile feature, it was largely serving as an expensive bookmark launcher in the Firefox code base so that folks could alt-tab to the small number of sites that supported it on their desktops. Not a noble end for Mozilla’s support for what should have been / could have been an incredible leverage point for the web ecosystem and open development.

It’s not. If it holds up well in terms of operating it and extending it when things change, then it’s not “too simple”. You’ve just built on 4 extremely resilient and scalable technologies which have made the kind of “Big Data” hoop-jumping of the past 20+ years rather moot.

You know the drill: SQLite and DuckDB mean you’ve got transactional database and data warehouse that live in the single machine and unless it’s terabytes runs analytics performance comparable to bigquery and snowflake.

No need to write an etl pipeline that runs on a Hadoop cluster, provision that Hadoop cluster, spin up a Hive/Pig instance for analysis. Nah, it fits in *your* computer and is scripted using the same language in the pipelines as in the analysis without a performance cost.

If you need to scale it, it’s not technical scaling, it’s team knowledge scaling (still hard, but not a fundamental stumbling block). So bring in DBT/Dagster (or airflow) and now it’s got supported frameworks that others already know and use.

Likewise; this ticks so many boxes for me, but as a hippy-dippy non-gmail user I too have to second the IMAP/JMAP ask.

Gmail absolutely hits that sweet spot of API capabilities and where the users are, so I can’t fault the project creators (or most every email client business these days) for building first (or exclusively) for it.

That said, seeing Outlook as coming soon on their login page is reassuring that they’re building in a way that won’t tie them to Gmail forever. And while few email providers outside of Fastmail are offering JMAP support, as an API it’s much closer to the degree of functionality expected by anyone building on top of Gmail’s API today. A great new client that gives a big section of the public a better way to “do email” might be what it takes for more services to start offering JMAP.

So hats off to y’all and fingers crossed on incorporating open standards.

This is genuinely awesome. The crossstreet based search is a perfect approach in NYC. Reminds me of how I used to use Google’s sms service to get directions from about 2004-2010.

Sunday service has always been particularly important to the New York City public libraries. Andrew Carnegie’s original deal always was, he would find the construction of the branches the libraries would run the branches, and the city would fund the branches with seven-day service. For a while. the Carnegie branches open seven days a week, even as they had to follow through on cutbacks at the non-Carnegie branches.

But that ship sailed long ago. Very few were still able to offer Sunday service before this:

- NYPL (Manhattan/Bronx/Staten Island): 8/92 sites

- Brooklyn Public Library: 8/66 sites

- Queens Public Library: 2/66

(Yes, there are three separate public library systems for New York City. They pre-date the consolidation of the city and no matter how hard folks have tried every study on consolidating the three systems into a single organization, winds up costing significantly more than the current status quo.)

This is a big step forward, and is also making me nostalgic for the Friend-of-a-Friend (FOAF)[0] blogrolls of the early 00’s. RDF-in-HTML standard that could express recommendations and relationships.

That said, I’m glad to see Webmention adoption. It’s got a much clearer purpose than FOAF (it might’ve been a little too expressive) and fits nicely into the current web ecosystem.

[0]: https://twobithistory.org/2020/01/05/foaf.html

> In this context-- the section in article where it says present data is of virtually zero importance to analytics is no longer true. We need a real solution even if we apply those (presumably complex and costly) solutions to only the most deserving use cases (and not abuse them).

Totally agreed, though where real-time data is being put through an analytics lens is where CDW's start to creak and get costly. In my experience, these real-time uses shift the burden from being about human-decision-makers to automated decision-making and it becomes more a part of the product. And that's cool, but it gets costly, fast.

It also makes perfect sense to fake-it-til-you-make-it for real-time use cases on an existing Cloud Data Warehouse/dbt style _modern data stack_ if your data team's already using it for the rest of their data platform; after all they already know it and it's allowed that team to scale.

But a huge part of the challenge is that once you've made it, the alternative for a data-intensive use case is a bespoke microservice or a streaming pipeline, often in a language or on a platform that's foreign to the existing data team who's built the thing. If most of your code is dbt sql and airflow jobs, working with Kafka and streaming spark is pretty foreign (not to mention entirely outside of the observability infrastructure your team already has in place). Now we've got rewrites across languages/platforms, and leave teams with the cognitive overhead of multiple architectures & toolchains (and split focus). The alternative would be having a separate team to hand off real-time systems to and only that's if the company can afford to have that many engineers. Might as well just allocate that spend to your cloud budget and let the existing data team run up a crazy bill on Snowflake or BigQuery as long as it's less than the cost of a new engineering team.

------

There's something incredible about the ruthless efficiency of sql data platforms that allows data teams to scale the number of components/engineer. Once you have a Modern-Data-Stack system in place, the marginal cost of new pipelines or transformations is negligible (and they build atop one another). That platform-enabled compounding effect doesn't really occur with data-intensive microservices/streaming pipelines and means only the biggest business-critical applications (or skunk works shadow projects) will get the data-intensive applications[1] treatment, and business stakeholders will be hesitant to greenlight it.

I think Materialize is trying to build that Modern-Data-Stack type platform for real-time use cases: one that doesn't come with the cognitive cost of a completely separate architecture or the divide of completely separate teams and tools. If I already had a go-to system in place for streaming data that could be prototyped with the data warehouse, then shifted over to a streaming platform, the same teams could manage it and we'd actually get that cumulative compounding effect. Not to mention it becomes a lot easier to then justify using a real-time application the next time.

[1]: https://martin.kleppmann.com/2014/10/16/real-time-data-produ...

Indeed they did.

I also fondly remember how wonderful the web development deployment workflow was in Coda. This was back before source control was ubiquitous and CI/CD was required for any real production environment. It was the cleanest way I'd seen to go from saving in your test editor and having a clear path to SFTP it to the server. I half recoil in horror and am still in awe of how considered their experience was.

Not gaming related but if you want to know the origins of where everything in computation came from, George Dyson’s “Turing’s Cathedral” is a revelation. It’s the story of computer, told by interviews of the folks who created the architecture of it all. It’s incredible history

It seems like you're putting the text out under CC-4.0-BY-SA [1] (Attribution, Share-Alike), which is great (the updated version of the same copyright for Wikipedia content). However, you're also collecting a TON of structured and relational data, which seems to be the value generated by your editor. Are you planning on keeping that locked up?

[1]: https://creativecommons.org/licenses/by-sa/4.0

One of the under-appreciated aspects of Colaboratory is that it's completely integrated into the Google Drive ecosystem, including multiple real-time users of the same notebook (sharing the same VM). This was a real game-changer for me.

The real-time use-case has a nice wow factor to it; I've used it as a way to pair program for data science problems. The input cells sync in real-time (a la Google Docs), and so too do the output cells when one person runs a cell. And it's nice to be able to leave comment threads on a cell that can be resolved as a form of peer review.

But what made Colab a game-changer for me is how it let me seamlessly put my notebooks and a VM into Google Drive, making anything I put in a notebook accessible to anyone within my organization without needing to set up an environment, be it shared or local.

My last organization was a small rare disease research foundation, and I primarily worked on the fundraising side of the house; it was not a technical organization. When thinking about the longevity of my work, I realized that even the one person managing IT for them probably couldn't set up, let alone justify maintaining a networked Jupyter environment. So rather than ask for that and store all my analyses and small utilities on GitHub, I built everything on top of Google Drive and Collab. Folks were used to using Drive for everything else, so it meant my work was adjacent and discoverable to the team it was pertinent to and they could get access to both the outcomes of prior runs or change a few variables and run it again without me being needed. I left recently and I've still heard from a few former colleagues that they're still using many of the these notebooks and discovering others I'd built on their own time.

For a small data analysis operation in a Google Apps organization, Colaboratory is a godsend.