Re:Databricks response to Snowflake's accusation of lacking ...
by sanketsarang on Tuesday November 16, 2021 @12:37AM
(#29237260)
I did work on making a database myself, and I must say that querying 100TB fast, let alone storing 100TB of data, is a real problem. Some companies (very few) don't have much choice but to use a DB that works on 100TB. If you do have small data, then you have a lot of options. But if your data is large, then you have very few options. So it is correct to be competing on how fast a DB can query 100TB of data; while at the same time being slow if you have just 10GB of data. Some databases are designed only for large data, and should not be used if your data is small.
[ Parent
| on HN ]
Re:Databricks response to Snowflake's accusation of lacking ...
by doppelganger1 on Tuesday November 16, 2021 @06:04AM
(#29239154)
The larger your data, the more that indexing and maintaining them hurt you. This is why they do much better at larger datasets vs small data sets. It’s all about trade offs.
To overcome this, they make use of cache and if the small data is frequently accessed, the performance is generally pretty good and acceptable for most use cases.
Re:Databricks response to Snowflake's accusation of lacking ...
by geoduck14 on Tuesday November 16, 2021 @04:03PM
(#29246322)
Did anyone else notice the surge of brand new accounts that are appearing on these discussions of Databricks with pro-Databrick opinions?
If we had access to IP address of the posters, I sure would be interested in looking at correlation among them.
Re:Databricks response to Snowflake's accusation of lacking ...
by doppelganger1 on Thursday November 18, 2021 @12:40PM
(#29268162)
What about my comment above is pro-Databricks? Snowflake works the same way. So do most large scale DW insert Exadata, Netezza, etc...
Does anyone else notice people questioning common sense?