hawaiianSpork

How do succinct data structures do with vector operations on cpu?

Not sure if they are succinct, but the Apache arrow format encodes data in several ways that is compact in memory but also allows operations on these structures.

Parquet has been the lakehouse file format of choice for nearly half a decade. But we are starting to see other contenders that are optimized more for lower latency like lance https://github.com/lancedb/lance

I looked for the York Abstract Machine used in the paper and couldn't find anything about it outside of this paper: https://www.cs.york.ac.uk/plasma/publications/pdf/ManningPlu...

It would be nice to play with the code.

I wonder if exposing this as a language server would be helpful?

If you are looking to do data validation from the JVM, you may try Baleen (written in Kotlin): https://github.com/ShopRunner/baleen/

I'm one of the contributors. We created a DSL in the language to describe the data and create tests. You can then use that data description to validate against json, csv, avro... One of the neat things we came up with was the concept of a data trace which is like a stack trace but is a path through the data to a particular error.