mousematrix

hey all, another perspective that I have been thinking about is if semantic layers are like ORM for but BI dashboards. Actually, they I think its more than BI dashboards since a similar idea applies to Features. Features in ML land are nothing but a Measure + Entity metadata + TTL. So, really its about higher-order semantics and as we move up the stack, we need richer expression to describe our world.

Feature stores explored here: https://www.xorq.dev/blog/featurestore-to-featurehouse

I think my key takeaway building this is that we need better expression systems and Ibis is a great foundation to build yours..maybe you want to build a language for some other domain etc.

PS: I am one of the authors of bsl and co-founder of Xorq.

I am one of the authors of bsl and founder of xorq.

As opposed to Substrait? I think there is certainly a path to converting to Substrait as one of the outputs for on-the-wire format to send query plans to various engines. YAML as such gives a human-diffable representation that can be checked-in to git. Let me know your thoughts...we are super early so happy to take in feedback...

I think so too...seriously considering changing the name

this is really useful. In our case, we are inspired by a terminal game called Zork, which is an Activision trademark. So, we tried to be creative and replacing Z with X made sense at the time. Of course, we did not want to be like any "dork", hence the Q.

Now I read this tought process above, I think this leaves much to be desired.

great point...honestly this will be our 3rd rebrand, but worth considering. I am just hoping that we are in a different enough space that it could slide....

While we considered Narwhals (It's pretty new) over Ibis, but ultimately we decided to go with Ibis since it supports the SQL backends that we needed.

We have ambitions of supporting alternative APIs like Narwhals in future though, that can map polars API to Ibis's internal representation.

We found Ibis to be super extensible. In xorq, we also support pandas UDFs, so if you know pandas you should be pretty well covered. UDFs are pretty nice way to extend the API.

What sort of operations did you find Ibis was missing when you tried? What about it isn't extensible?

this is an example of an out-of-core processing: https://www.xorq.dev/posts/trino-duckdb-asof-join

Anecdotally, TPC-H 10 TB is pretty doable now a days with DuckDB, so xorq goes as far as your engine may take you...

Looks v. cool, how does gizmo handle concurrency for multiple writes? are there any limits that you have observed?