Re:turbopuffer: Fast Search on Object Storage
by rohitnair on Wednesday July 10, 2024 @01:50PM
(#40929482)
Elasticsearch and OpenSearch already support S3 backed indices. See features like https://opensearch.org/docs/latest/tuning-your-cluster/avail... The files in S3 are plain old Lucene segment files (just wrapped in OpenSearch snapshots which provide a way to track metadata around those files).
[ Parent
| on HN ]
Re:turbopuffer: Fast Search on Object Storage
by francoismassot on Wednesday July 10, 2024 @01:56PM
(#40929561)
But you don’t have fast search on those files stored on object storage.
Re:turbopuffer: Fast Search on Object Storage
by rohitnair on Wednesday July 10, 2024 @02:01PM
(#40929618)
Yes, there is a cold start penalty but once the data is cached, it is equivalent to disk backed indices. There is also active work being done to improve the performance, example https://github.com/opensearch-project/OpenSearch/issues/1380...