News for hackers. Stuff that matters.

A comment on The best way to use text embeddings portably is with Parquet and Polars.

Re:The best way to use text embeddings portably is with Parq... by kernelsanderz on Monday February 24, 2025 @03:20PM (#43164469)
For another library that has great performance and features like full text indexing and the ability to version changes I’d recommend lancedb https://lancedb.github.io/lancedb/

Yes, it’s a vector database and has more complexity. But you can use it without creating indexes and it has excellent polars and pandas zero copy arrow support also.

[ Parent | on HN ]
Re:The best way to use text embeddings portably is with Parq... by daveguy on Monday February 24, 2025 @03:39PM (#43164680)
Since a lot of ML data is stored as parquet, I found this to be a useful tidbit from lancedb's documentation:

> Data storage is columnar and is interoperable with other columnar formats (such as Parquet) via Arrow

https://lancedb.github.io/lancedb/concepts/data_management/

Edit: That said, I am personally a fan of parquet, arrow, and ibis. So many data wrangling options out there it's easy to get analysis paralysis.

[ Parent | on HN ]
Re:The best way to use text embeddings portably is with Parq... by esafak on Monday February 24, 2025 @03:22PM (#43164496)
Lance is made for this stuff; parquet is not.
[ Parent | on HN ]
Re:The best way to use text embeddings portably is with Parq... by 3abiton on Monday February 24, 2025 @06:08PM (#43166036)
How well does it scale?
[ Parent | on HN ]