carlineng

This is really a question of economics. The biggest organizations with the most ability to hire engineers have need for technologies that can solve their existing problems in incremental ways, and thus we end up with horrible technologies like Hadoop and Iceberg. They end up hiring talented engineers to work on niche problems, and a lot of the technical discourse ends up revolving around technologies that don't apply to the majority of organizations, but still cause FOMO amongst them. I, for one, am extremely happy to see technologies like DuckDB come along to serve the long tail.

Not yet, but I believe the DataCouncil staff recorded it and will post it to their YouTube channel sometime in the next few weeks: https://www.youtube.com/@DataCouncil/videos

I just watched the author of this feature and blog post give a talk at the DataCouncil conference in Oakland, and it is obvious what a huge amount of craft, ingenuity, and care went into building it. Congratulations to Hamilton and the MotherDuck team for an awesome launch!

I agree 100% that this needs to be more of a thing. For data engineers building data pipelines, queries are like functions, and table schemas are like types. There needs to be a way to write a query that runs on an abstract interface, rather than an actual table. To do this, most folks rely on string templating in Python or Jinja, which makes the development process really cumbersome. As a result, most teams end up in scenarios where data pipelines are always a big mess of spaghetti SQL, or they are stuck maintaining complex frameworks that abstract away common logic, but are inscrutable to the average user.

I wrote a longer blog post about this recently: https://carlineng.com/?postid=holy-grail-data-engineering#bl...

The argument is more than that -- namely that in addition to having sophisticated AI, Google also controls the OS (Android) and hardware (Pixel). Being able to integrate best-in-class AI at every level of the stack is a tremendous advantage. OpenAI can't do this because they don't control the OS, and will always need to go through an app. Apple can play since they control OS and hardware, but at the moment they appear pretty far behind in the AI aspect.

My favorite discussion of this topic is from David Beazley: https://www.youtube.com/watch?v=5C6sv7-eTKg

He does a wonderful job of taking very dense mathematical notation and explaining it in ways that anyone can understand. He derives the basic concepts of the lambda calculus from the ground up using Python. Super fun to follow along with.

My understanding of Cube is that iterating on the data model requires the user to (1) write SQL to develop a metric (2) edit YAML or JS config to incorporate the new metric (3) issue API request to Cube server and (4) compare results to raw SQL. Am I mistaken? Does Cube offer a smoother way to do this exploration/iteration?

Thanks for the kind words! Admittedly, a lot of the stuff you're asking about is still a work in progress, and we don't have good answers for it all just yet. The upside of that is you have the potential to influence which direction we take next. If you're thinking about building on top of Malloy, join our community Slack channel, and we'd be happy to provide guidance or take your suggestions/feedback! https://join.slack.com/t/malloy-community/shared_invite/zt-1...

Glad you asked! It is certainly usable for writing apps. We publish an npm package, and the Malloy VSCode extension [0] is one such example of an app built on top of it. There's also a demo of a toy CLI app that showcases the simplest possible use of the SDK: https://github.com/malloydata/malloy-demo-bq-cli

Fair critique on the presentation. That's mostly a function of the circles I run in, which is heavily weighted towards data analysts, data engineers and "analytics engineers". But to your point, I think that cohort is unlikely to be early adopters of a tool like Malloy, and that developers are a much easier sell.

[0]: https://github.com/malloydata/malloy-vscode-extension

Semantic layers, also known as metrics layers or “headless BI”, have become a popular topic in the online data community in recent years. For all the hype, the idea hasn't seen much traction. In this blog post, I hypothesize why not, and describe the Malloy language and why I think it has a better chance at succeeding.

I've written quite a bit about a new query language called Malloy: https://carlineng.com/?postid=malloy-intro#blog

It's a semantic layer with an integrated query language that makes a ton of improvements on SQL. Full disclosure, I actually joined the team a couple of weeks ago :)

Would love to! How do I get in touch? My contact info is in my profile.

I'd wager you're right. All the "boring" stuff that's actually very complicated/difficult, and without which no large enterprise will adopt a technology.

MotherDuck has been making the rounds with a big funding announcement [1], and a lot of posts like this one. As a life-long data industry person, I agree with nearly all of what Jordan and Ryan are saying. It all tracks with my personal experience on both the customer and vendor side of "Big Data".

That being said, what's the product? The website says "Commercializing DuckDB", but that doesn't give much of an idea of what they're offering. DuckDB is already super easy to use out of the box, so what's their value-add? It's still a super young company, so I'm sure all that is being figured out as we speak, but if any MotherDuckers are on here, I'd love to hear more about the actual thing that you're building.

[1]: https://techcrunch.com/2022/11/15/motherduck-secures-investm...

> I had a student storm out of my class [1] because I was teaching relational databases instead of web3.

Wow... that exchange is surreal. I wonder if it was some kind of fraternity prank or troll job?

[1] https://www.youtube.com/watch?t=1490&v=uikbtpVZS2s