News for hackers. Stuff that matters.

jacquesnadeau

Apple container is more akin to a replacement for docker or colima (although patterned more like Kata containers where each container is a separate vm as opposed to a bunch of containers in a single vm). It's a promising project (and nice to see Apple employees work to improve containers on macOS).

Hopefully, they can work towards being (1) more docker api compatible and (2) making it more composable. I wrote up https://github.com/apple/container/discussions/323 for more details on the limitations therein.

Originally, I planned to built shai to work really well on top of apple container but ultimately gave up because of the packaging issues.

cool to see ctenv. definitely a similar vibe. thanks for sharing! will look at more closely.

Interesting to see how you incorporated some dockerfile patterns. devcontainer feature-esque.

I'm curious to know if you are using it for the isolation concepts I call "cellular development": https://shai.run/docs/concepts/cellular-development/

I'm one of the creators of shai. Thanks for the callout!

Interesting to see the work on Yolobox and in this space generally.

The pattern we've seen as agent use grows is being thoughtful about what different agents get access to. One needs to start setting guardrails. Agents will break all kind of normal boundaries to try to satisfy the user. Sometimes that is useful. Sometimes it's problematic. (For example, most devs have a bunch of credentials in their local env. One wants to be careful of which of those agents can use to do things).

For rw of current directory, shai allows that via `shai -rw .` For starting as an alternative user, `shai -u root`.

Shai definitely does have the attitude that you have to opt into access as opposed to allowing by default. One of the things we try to focus on is composability: different contexts likely need different resources and shai's config. The expectation is .shai/config.yaml is something committed to the repo and shared across developers.

Arrow supports a union type for heterogeneous columns (we use it for random json in Dremio) and a 128-bit decimal.
PrestoDB is similar to Impala, Hive and other SQL Engines. Each is designed to do distributed SQL processing. Dremio does embed an OSS distributed SQL processing engine (Sabot, built natively on Arrow) as well but we see that as only a means to an end. Our focus is much more on being a bi & data fabric/service.

At the core of this vision are: very advanced pushdowns (far beyond other OSS systems), a powerful self-service UI for managing, curating and sharing data (designed for analysts, not just engineers) and--most importantly--the first open source implementation of distributed relational caching for all types of data. You can see more details about this last part in a deck I presented at DataEngConf early today: https://www.slideshare.net/dremio/using-apache-arrow-calcite...

Arrow is all about in-memory, not long-term persistence. Systems can write it to disk but it is the one in-memory representation to rule them all, not storage/disk. Disk has its own requirements and challenges outside the scope of Arrow.
For those level of operations, the Arrow library itself will probably have something fairly soon.
It's more than an implementation detail because we're also targeting interoperability between multiple separate technologies. One of the key things that the article didn't fully cover is that Arrow serves two purposes: high performance processing and interoperability.

A key part of the vision is: two systems can share a representation of data to avoid serialization/deserialization overhead (and potentially copying in a shared memory environment). This is only possible if the in-memory format is also highly efficient for processing. This allows the processing systems (say Pandas and Dremio) to share a representation, both process against it, and then move the data between each other with zero overhead.

If you shared the data representation on the wire and then each application had to transform it to a better structure for processing, you're still paying for a form of ser/deser. By using Arrow for both processing and interoperation you benefit from near-no-cost movement between systems and also a highly efficient representation to process data (including some tools to get you started in the form of the Arrow libraries).

Dremio is focused on a combination of data access, acceleration and a self-service analyst experience (Tableau/Qlik one-click integration, data curation and data management).

We also invest heavily in our pushdowns. For example, we invested heavily in our Elasticsearch capabilities (support for CONTAINS and Lucene syntax, Painless and Groovy pushdowns, index leveraging, etc) and I'm pretty comfortable saying we're the best in world at exposing Elastic in a SQL context. Similarly for our other sources (join & window pushdowns in Oracle, SqlServer, etc, high speed predicate application in Hadoop, Mongo aggregation pipeline pushdowns including support unwind, etc).

I don't know Querona well (just a cursory site review). They seem much more focused on classic federation. It would also be important to understand when they repackage the Spark source connectors versus where they have built something that pushes down more powerfully (which you typically need for the best performance with these newer data source systems).

People are already using Arrow for high performance GPU processing. FPGA is possible but I'm not sure if anyone is actually doing this today.
One quick note to make on this. Kudu is a storage implementation, (similar to Parquet in some ways). Arrow isn't about persistence and is actually built to be complementary to both Kudu and Parquet.

Also note: Kudu is a distributed process. Arrow and Parquet are libraries that can be embedded into your existing applications.

The Arrow project itself is a set of libraries. One of the things we'll do is try to add more algorithms over time to it so if you want say, a fast arrow sort or arrow predicate application. Full SQL is always far more complex and I can't see the project itself .

The engine inside of Dremio is something we call Sabot (a shoe for modern arrows, see sabot round on wikipedia). We hope to make it modular enough one day to use a library but it isn't there yet.

In regards to your other question re projects/products: Arrow contributors are actively trying to get more adoption of Arrow as an interchange format for several systems. We've had discussions around Kudu (no serious work done yet afaik). Parquet-to-Arrow for multiple languages is now available. Arrow committers include committers from several other projects such as HBase, Cassandra, Phoenix, etc. The goal is ultimately to figure integrations with all.

In most cases, these data storage systems are saddled with slow interfaces for data access. (Think row-by-row, cell-by-cell interfaces.) Arrow, among other things, allows them to communicate through a much faster mechanism (shared memory--or at least shared representation if not node local).

You nailed it. The most exciting part about all of this is being able to move between a "data science" context and a "database" context (and back again) without pain or penalty.
Possible, yes. Performant or pleasant? Maybe not :)
We built Dremio (github.com/dremio/dremio-oss, apache licensed) entirely on top of Apache Arrow specifically for the purposes of creating a high speed analytical capabilities including MOLAP like work as well as other forms of caching/acceleration for analytical workloads. Other products/projects are also starting to adopt a similar technical architecture.
I'm also a developer on Arrow (https://github.com/jacques-n/), similar to WesM. It is always rewarding (and also sometimes challenging) to hear how people understand or value something you're working on.

I think Dan's analysis is evaluating Arrow from one particular and fairly constrained perspective of "if using Arrow and Parquet for RDBMS purposes, should they exist separately". I'm glad that Dan comes to a supportive conclusion even with a pretty narrow set of criteria.

If you broaden the criteria to all the different reasons people are consuming/leveraging/contributing to Arrow, the case only becomes more clear for its existence and use. As someone who uses Arrow extensively in my own work and professionally (https://github.com/dremio/dremio-oss), I find many benefits including two biggies: processing speed AND interoperability (now two different apps can share in-memory data without serialization/deserialization or duplicate memory footprint). And best of all, the community is composed of collaborators trying to solve similar problems, etc. When you combine all of these, Arrow is a no brainer as an independent community and is developing quickly because of that (80+ contributors, Many language bindings (6+) and more than 1300 github stars in just a short amount of time).