stavrospap

Stavros from TileDB here (Founder and CEO). I thought of requesting some feedback from the community on this blog. It was only natural for a multi-dimensional array database like TileDB to offer vector (i.e., 1D array) search capabilities. But the team managed to do it very well and the results surprised us. We are just getting started in this domain and a lot of new algorithms and features are coming up, but the sooner we get feedback the better.

TileDB-Vector-Search Github repo: https://github.com/TileDB-Inc/TileDB-Vector-Search

TileDB-Embedded (core array engine) Github repo: https://github.com/TileDB-Inc/TileDB

TileDB 101: Vector Search (blog to get kickstarted): https://tiledb.com/blog/tiledb-101-vector-search/

Hi folks, Stavros from TileDB here. Here are my two cents on tabular data. TileDB (Embedded) is a very serious competitor to Parquet, the only other sane choice IMO when it comes to storing large volumes of tabular data (especially when combined with Arrow). Admittedly, we haven’t been advertising TileDB’s tabular capabilities, but that’s only because we were busy with much more challenging applications, such as genomics (population and single-cell), LiDAR, imaging and other very convoluted (from a data format perspective) domains.

Similar to Parquet:

* TileDB is columnar and comes with a lot of compressors, checksum and encryption filters.

* TileDB is built in C++ with multi-threading and vectorization in mind

* TileDB integrates with Arrow, using zero-copy techniques

* TileDB has numerous optimized APIs (C, C++, C#, Python, R, Java, Go)

* TileDB pushes compute down to storage, similar to what Arrow does

Better than Parquet:

* TileDB is multi-dimensional, allowing rapid multi-column conditions

* TileDB builds versioning and time-traveling into the format (no need for Delta Lake, Iceberg, etc)

* TileDB allows for lock-free parallel writes / parallel reads with ACID properties (no need for Delta Lake, Iceberg, etc)

* TileDB can handle more than tables, for example n-dimensional dense arrays (e.g., for imaging, video, etc)

Useful links:

* Github repo (https://github.com/TileDB-Inc/TileDB)

* TileDB Embedded overview (https://tiledb.com/products/tiledb-embedded/)

* Docs (https://docs.tiledb.com/)

* Webinar on why arrays as a universal data model (https://tiledb.com/blog/why-arrays-as-a-universal-data-model)

Happy to hear everyone’s thoughts.

Disclosure: I am the author of this article and the Founder/CEO of TileDB. The article touches upon some important and rather thought provoking issues around data management, so I thought to get some feedback from the HN community.

TileDB Embedded is a storage engine like HDF5, with the following differentiators: (1) it is cloud-native, (2) it supports also sparse arrays, (3) it offers rapid updates, (4) it supports data versioning and time traveling built into its format. TileDB Cloud (our cloud SaaS solution) further allows you to see which arrays you own in the cloud and which ones you share with others, along with full access logs. You can also attach arbitrary descriptions and metadata that can search on, even find and access public datasets posted by you or others.

TileDB and Hail are rather complementary. We have customers that use TileDB to store and manage their variants, and Hail to perform GWAS (by exporting from TileDB to Hail format). We are currently designing a tighter integration with Hail. This expands on our vision for a universal data engine that integrates with pretty much everything out there and does not lock you in a single framework (e.g., Spark).

This is an API issue with TileDB-VCF. The core TileDB library supports inserts/appends/overwrites without issues and we just need to expose those operations in the TileDB-VCF APIs. Added to our backlog, thanks!

Stavros from TileDB here. Great description of the genomics use case for TileDB. We'd be interested in learning what limitations you've found. Happy to discuss over email as well ([email protected]).

Folks, apologies, but I think we got a bit side tracked here, TileDB does not suffer from the consistency issues mentioned above.

Here is how TileDB performs a new (potentially concurrent with other reads and writes) write:

- It creates a fragment folder (or "prefix" of a set of objects on S3 - there are no "folders" on S3) which is timestamped and carries a unique UUID. This fragment is self-contained and represents the entire write (e.g., all cells and all attribute values)

- It writes all data objects under the fragment prefix. Note that TileDB never updates, it always writes new immutable objects.

- After all the PUT requests succeed for the data objects, it creates an empty "ok" object.

Here is how TileDB performs a (potentially concurrent with other reads and writes) read:

- It lists the array prefix to get the ok objects

- There are two cases:

1. The ok object is not there for some fragment. That fragment is completely ignored.

2. The ok object is there. Since TileDB writes the ok object last, all the data objects it wrote have been committed and are all visible with GET requests. TileDB reads the data objects only with GET requests (not ListObject requests). Due to S3’s read-after-write consistency model (https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction...), all those objects will be available for reading (now on all S3 regions) with GET and there will be no errors.

Therefore, TileDB follows the eventual consistency model of S3 without any errors and surprises. The user doesn't need to handle anything. Our customers have been using TileDB in production for a long time, storing hundreds of TBs of data on S3, and no consistency issue has ever come up.

Summarizing, what xyzzy_plugh is raising here is that TileDB does not have ACID guarantees. And that is true (we never claimed the contrary) and intentional. We are building a transactional layer outside of the storage engine. The reason is that this transactional layer indeed needs to be a constantly running distributed service, whereas we want the TileDB storage engine to be embeddable and used without performance regression even by applications that do not need ACID (that is, the majority of our data science applications).

Efficient slicing happens because of "tiling", hence the name TileDB. A tile is similar to an HDF5 or Zarr "chunk", or more loosely to a Parquet page. Although totally configurable, tiling is handled solely by TileDB, the user doesn't need to know about it. A tile is the atomic unit of IO and compression. TileDB maintains all the necessary metadata and indexing built into its format and, given a query, it knows how to fetch only the tiles that might include results. The tiles are decompressed in your memory and filtered further for the actual results. The dense array case is rather straightforward. The sparse case is a big differentiator in TileDB and it is quite challenging, especially in the presence of updates. TileDB handles the sparse case via bulk-loaded R-trees for multi-dimensional indexing, and via an LSM-tree-like approach with immutable objects that allows time traveling.

Concerning your point on potential errors occurring on S3, this is addressed by TileDB's immutable object approach. If an error occurs upon some write, there will be no array corruption. Happy to discuss about this topic on a separate thread.

Some related docs:

https://docs.tiledb.com/main/performance-tips/choosing-tilin...

https://docs.tiledb.com/main/basic-concepts/tile-filters#til...

https://docs.tiledb.com/main/basic-concepts/definitions/frag...

Full developer docs here: https://docs.tiledb.com/main/

Specific dataframe examples coming up shortly.

Stavros form TileDB here. Here is a more verbose explanation. Up until before 2.0, TileDB was already powerful for the main applications we targeted at: geospatial and genomics. The support for both dense and sparse arrays and the way it handles data versioning made it quite unique vs. HDF5 and Zarr. But we noticed that most of the data scientists we were working with had a lot of data beyond genomic variants, LiDAR points and rasters. They had tons of dataframes. And they were using at least two storage engines, TileDB for arrays, and Parquet or a relational database for dataframes. If you are in a large organization, this a big pain.

In TileDB 2.0 we made a huge refactoring to support something seemingly simple: dimensions in sparse arrays that can have different types and that could even be strings. This allowed us to model any dataframe as a sparse array, effectively making TileDB act as a primary multi-dimensional index. In relational databases, this means that your data is sorted in an order on disk that favors your multi-column slicing enormously, so range search becomes rapid.

Therefore, what we are telling the community with this release is that you can have dense arrays, sparse arrays, and dataframes in a single embeddable library being integrated with pretty much every data science tool out there, so that data scientists never have to worry about backends, files, updates, or anything other than their scientific analysis. In other words, we believe the future of data science is more science.

Thanks for the post!

Stavros from TileDB here. TL;DR, it's all about fast slicing on multiple columns while supporting updates, locally or in the cloud.

Suppose you serialize your dataframe in HDF5 or Redis. Let your dataframe have schema (Date, Stock, Price). Assume this dataframe is 1 TB long and stored on S3, GCS or Azure (as they are cheap). How would you be able to efficiently perform an average query on Price for a specific Date range and Stock symbol? With HDF5 you would have to download 1 TB (no notion of "fast slicing on variable predicates") and apply the predicates locally. If you stored the dataframe in Parquet (a better choice for this use case), then you would be able to build some logic in your code that uses the Parquet metadata/indexes and prune a lot of unnecessary information (as Spark does). However, Parquet is "one-dimensional", i.e., your pruning would be efficient on Date, but not on Stock (you'd have to "partition" your Parquet files with Spark or Hive and things could get quite complicated). Most importantly, you wouldn't be able to update the Parquet files; you would have to generate new files and build a catalog on top (or use services like Delta Lake) to manage your Parquet files. And this is an extremely cumbersome task.

TileDB abstracts everything for you, while allowing you to slice fast on any number of columns. You just define Date and Stock as "dimensions", and slicing on both those columns becomes uber efficient locally or in the cloud. Effectively, you turn this dataframe into a sparse 2D array. Updates and time traveling are handled by TileDB. You get to use Spark, Dask, MariaDB and PrestoDB as you did before, but there is no need for Hive, Delta Lake or any other cataloging service. Thank you for pointing out the confusion though. We just launched and we have tons of examples coming up.

Thank you! We hear you on the webpage messaging, and we are ramping up our efforts on that front. We are also planning on publishing a series of technical blog posts, aiming to clarify the value of TileDB. Please feel free to reach out to us should you or your students need any further information.

Hi all, we'd love to hear your thoughts about TileDB (https://github.com/TileDB-Inc/TileDB), which offers the key advantages of both HDF5 and Zarr, such as chunked dense arrays, numerous compression filters and native cloud support. But TileDB is more as it also supports sparse arrays and integrates with more languages, databases and data science tools. If you'd like your data to be readable by multiple languages and tools or need array versioning, it is worth taking a look.

Docs: https://docs.tiledb.com/developer/

Website: https://tiledb.com/

Earlier HN post: https://news.ycombinator.com/item?id=15547749

Disclosure: I am a member of the TileDB team.

TileDB, Inc. | Full-Time | REMOTE | USA | Greece | https://tiledb.com

TileDB, Inc. is a data management company spun out of Intel Labs and MIT to help Data Science teams make faster discoveries, by giving them a more natural way to store, analyze and share large sets of diverse data, so that they can stop wasting time working around performance limitations, inadequate data storage formats, and unfamiliar tooling. TileDB is comprised of a new multi-dimensional array data format, a fast embeddable, open-source C++ storage engine with numerous APIs and Data Science tooling integrations, and a cloud service for easy data management and serverless computations, all architected to solve common pain points faced by data scientists.

Website: https://tiledb.com

GitHub: https://github.com/TileDB-Inc

Docs: https://docs.tiledb.com

Blog: https://medium.com/tiledb

Our headquarters are located in Cambridge, MA and we have a subsidiary in Athens, Greece. We offer the ability to work remotely, but the candidates must reside either in the US or in Greece. US candidates must be US citizens, whereas Greek candidates must be Greek or EU citizens.

We have several open positions aimed at increasing TileDB’s feature set, growth and adoption. You will have the opportunity to work on innovative technology that creates impact on challenging and exciting problems in Genomics, Geospatial, Time Series, and more. A few features on the roadmap include enhancing our TileDB Cloud offering by optimizing our serverless framework, adding ML pipelines, and integrating with JupyterLab.

We are primarily seeking:

- Developer Advocate

- Head of Marketing

- C++ Engineer

- R Engineer

- Backend Engineer

- Python Engineer

Apply today at https://tiledb.workable.com !

TileDB, Inc. | Full-Time | Cambridge, MA, USA | Athens, Greece | REMOTE (USA only) | https://tiledb.io

TileDB is a disruptive technology for storing and managing enormous volumes of structured data, adopting the best ideas from columnar and spatial database research to support fast updates, compression, and interoperability with scalable cloud object storage backends. TileDB efficiently stores data generated from variety of domains (genomics, finance, imaging, geospatial, etc) in a novel unified format as sparse or dense multidimensional arrays. Users can efficiently access this data via a growing number of language APIs and interfaces (C/C++, Python, R and SQL data engines).

TileDB, Inc. has raised $4M in seed funding over the past 18 months. We are looking to aggressively expand our team. The new members of TileDB will help us build out a solution to enhance cloud interoperability, bring scalable computations to data stored in TileDB more easily, and improve areas such as data consistency, access control and sharing on the cloud through a managed service.

We are primarily seeking:

- Frontend engineers (UI / UX with Vue.js)

- Backend engineers (Go / K8S)

- Scientific R programmers

TileDB GitHub organization: https://github.com/TileDB-Inc

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

TileDB recent blog post: http://bit.ly/2SjkeYR

TileDB talk at PyData 2018: http://bit.ly/2BfbyJ3

Our headquarters are located in Cambridge, MA. The candidates must be US citizens or permanent residents located in the US, or Greek citizens to be located at our subsidiary in Athens, Greece.

Apply today at https://tiledb.workable.com !

Hey HN, Stavros from TileDB here. We are excited to announce this newest round of funding to help us continue the development on TileDB! There was some excellent discussion about TileDB on HN previously here: https://news.ycombinator.com/item?id=15547749. We are happy to answer any questions.

TileDB, Inc. | Full-Time | Cambridge, MA or REMOTE (US) | https://tiledb.io

TileDB is a disruptive technology for storing and managing enormous volumes of structured data, adopting the best ideas from columnar and spatial database research to support fast updates, compression, and interoperability with scalable cloud object storage backends. TileDB efficiently stores data generated from variety of domains (genomics, finance, imaging, geospatial, etc) in a novel unified format as sparse or dense multidimensional arrays. Users can store their data in a unified namespace, while being able to efficiently access this data via a growing number of language APIs and interfaces (C/C++, Python, R and SQL data engines).

TileDB, Inc. has raised $4M in seed funding over the past 18 months. We are looking to aggressively expand our team. The new members of TileDB will help us build out a solution to enhance cloud interoperability, bring scalable computations to data stored in TileDB more easily, and improve areas such as data consistency, access control and sharing on the cloud through a managed service.

We currently have open positions in the following areas:

- Backend / frontend / full-stack engineers (Vue.js / Go / K8s)

- Experts in databases / database engines (C++ / Java / Scala)

- Scientific programmers (Python, R)

TileDB GitHub organization: https://github.com/TileDB-Inc

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

Our headquarters are located in Cambridge, MA. The candidates must be US citizens or permanent residents located in the US.

Apply today at https://tiledb.workable.com !

TileDB is a disruptive technology for storing and managing enormous volumes of structured data. TileDB efficiently stores structured data from a variety of applications (genomics, finance, imaging, LiDAR, etc) in a novel unified format as sparse or dense multi-dimensional arrays. Users store their data in a single place, while being able to efficiently access it via a growing range of languages and environments (C/C++, Python, R, SQL database engines). TileDB adopts the best ideas from columnar and spatial database research, supporting fast updates, excellent compression, and interoperability with cloud object storage backends.

TileDB, Inc. has just closed a new $3M financing round (official announcements coming up soon), having raised $4M in total seed funding over the past 18 months, and is looking to aggressively expand its team. The new members of TileDB will help us build out a solution to enhance cloud interoperability, compute on TileDB data more easily, and improve areas such as data consistency, access control and sharing. We are looking for backend/frontend engineers, experts in cloud technologies (AWS, Azure, Google Cloud), database engines, scientific programming (Python, R), and bioinformatics data engineering.

TileDB GitHub organization: https://github.com/TileDB-Inc

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

Our headquarters are located in Cambridge, MA. The candidates must be US citizens or permanent residents located in the US.

Apply today at https://tiledb.workable.com !

Hi there. Unfortunately, we are currently considering only US residents, but please check back in the future as we grow our team.

TileDB, Inc. | Senior Software Engineer | Cambridge, MA or REMOTE (US) | tiledb.io

Come join TileDB, Inc. in developing the future of big data storage. For emerging and growing fields such as genomics, earth science, imaging, and financial analysis, the volume of data is increasing at an astounding rate, and TileDB is meeting the challenge head on. Data in these areas is well represented as multidimensional dense or sparse arrays, and the open-source TileDB data management software is a novel, cutting edge solution for array storage.

The company closed a $1M seed in May 2017 led by Intel Capital and Nexus Venture Partners, and is raising a new round of funding in the upcoming weeks.

Our team combines software engineering expertise with domain knowledge to develop the open-source TileDB array data management software, and we are looking for talented and motivated engineers for the following positions:

- Full Stack Web Developer: Production experience with Kubernetes, Payment gateways, and React or Vue.js for frontend work

- Spark Software Engineer: Deep experience with Spark, Scala, Java, native JVM extensions / JNI, Maven packaging and the SBT build system

- Bioinformatics Software Engineer: Strong C++ skills, experience with R or Python, FastQ, VCF, BAM formats, bcftools, htslib, GATK, Hail

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

TileDB GitHub organization: https://github.com/TileDB-Inc

Our headquarters are located in Cambridge, MA. To cope efficiently with the different time zones and hiring processes, priority will be given to candidates that are located in the US, and are US citizens or permanent residents.

Apply at https://tiledb.workable.com

TileDB, Inc. | Senior Software Engineer | Cambridge, MA or REMOTE (US) | tiledb.io

TileDB, Inc. leads the development of the open source TileDB array data management software. The company closed a $1M seed in May 2017 led by Intel Capital and Nexus Venture Partners (http://www.businesswire.com/news/home/20171019005449/en), and is looking to raise a Series A round in the upcoming months.

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

TileDB GitHub repo: https://github.com/TileDB-Inc/TileDB

Array data volumes are increasing in genomics, earth science, imaging, and other sensing applications, and TileDB is meeting the challenge head-on. We are a small distributed team looking to aggressively adapt TileDB to better take advantage of distributed storage and compute backends in the hybrid local-and-cloud domain.

We are primarily looking for someone to help us transition scientific data storage and analysis to the cloud. Anyone who has experience in the following areas is welcome to apply:

  - Cloud object storage and compute
  - Spark / Arrow integration
  - Scalable REST server / services and APIs
  - SaaS services around access control, data sharing, and encryption

Additionally, we would be interested in a candidate with experience in one or more of the following areas: Scientific data storage / analysis, modern C++ (C++11 and later), parallel and/or distributed programming, compute or I/O performance optimization, genomic data formats / libraries (such as bam, vcf, htslib, bcftools, etc.), encryption / security.

Our headquarters are located in Cambridge, MA. To cope efficiently with the different time zones and hiring processes, priority will be given to candidates that are located in the US, and are US citizens or permanent residents.

Apply at https://tiledb.workable.com

Contact us at [email protected] with questions.

TileDB, Inc. | Senior Software Engineer | Cambridge, MA or REMOTE (US) | tiledb.io

TileDB, Inc. leads the development of the open source TileDB array data management software. The company closed a $1M seed in May 2017 led by Intel Capital and Nexus Venture Partners (http://www.businesswire.com/news/home/20171019005449/en), and is looking to raise a Series A round in the upcoming months.

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749

TileDB GitHub repo: https://github.com/TileDB-Inc/TileDB

We are a small distributed team looking to aggressively adapt TileDB to better take advantage of distributed storage and compute backends in the hybrid local-and-cloud domain.

We are primarily looking for someone to build access control / security into our product, however anyone who has the experience in the following areas is welcome to apply:

  - S3 object storage / AWS Batch / AWS Lambda
  - Azure blob storage / Azure Functions
  - Google cloud storage / Google Cloud Functions
  - Spark / Arrow integration
  - Scalable REST server / service and API
  - SaaS services around access control and encryption

Additionally, experience in any of the following would be a plus: Scientific data storage / analysis, Modern C++ (C++11 and later), Parallel and/or distributed programming, Compute or I/O performance optimization, Scalable object storage, Java / Spark ecosystem, Encryption / secure systems.

Our headquarters are located in Cambridge, MA. To cope efficiently with the different time zones and hiring processes, priority will be given to candidates that are located in the US, and are US citizens or permanent residents.

Apply at https://tiledb.workable.com

Contact us at [email protected] with questions.

TileDB, Inc. | Senior Software Engineer | FULL-TIME | Cambridge, MA or REMOTE (US) | tiledb.io

TileDB, Inc. leads the development of the open source TileDB array data management software. The company closed a $1M seed in May 2017 led by Intel Capital and Nexus Venture Partners (http://www.businesswire.com/news/home/20171019005449/en), and is looking to raise a Series A round in the upcoming months.

TileDB has been featured on HN: https://news.ycombinator.com/item?id=15547749.

TileDB GitHub repo: https://github.com/TileDB-Inc/TileDB

Array data volumes are increasing in genomics, earth science, imaging, and other sensing applications, and TileDB is meeting the challenge head-on. We are a small distributed team looking to aggressively adapt TileDB to better take advantage of distributed storage and compute backends in the hybrid local-and-cloud domain. We are currently looking to expand our team with someone experienced in one of the following areas:

  - S3 object storage / AWS Batch / AWS Lambda
  - Azure blob storage / Azure Functions
  - Google Cloud Storage / Google Cloud Functions
  - Spark / Arrow integration
  - Scalable REST server / service and API
  - SaaS services around access control and encryption

Additionally, experience in any of the following would be a plus:

  - Scientific data storage / analysis
  - Modern C++ (C++11 and later)
  - Parallel and/or distributed programming
  - Compute or I/O performance optimization
  - Scalable object storage
  - Java / Spark ecosystem
  - Encryption / secure systems

Our headquarters are located in Cambridge, MA. To cope efficiently with the different time zones and hiring processes, priority will be given to candidates that are located in the US, and are US citizens or permanent residents.

Apply at https://tiledb.workable.com

Contact us at [email protected] with questions.

By "very competitive" we mean that we will match any other offers you could get for such a position, provided that we believe that you are an ideal fit (I haven't seen a range on HN today that we cannot match). The base salary will depend on the person (skills and experience) and the place of residence.

Sorry about that. The correct email is "[email protected]". Thanks.