News for hackers. Stuff that matters.
Re:Databricks is an RDBMS by kfk on Monday February 01, 2021 @04:01PM (#25993879)
These are great innovations but can we please take a moment to realize 99% of companies are still stuck with a blend of excels, access and sql servers? Why is adoption of this new tech so poor? Maybe it has something to do with the amount of confusion all the sales pitches about data lakes, big data, ai and company are generating
Re:Databricks is an RDBMS by phoe-krk on Monday February 01, 2021 @04:11PM (#25994002)
> Why is adoption of this new tech so poor?

Because it's unnecessary to those 99% of companies. If company data fits in an Excel spreadsheet, Access database, or a single MySQL/Postgres instance, then introducing all this new tech with all of the associated costs and little return gain is a net loss.

Re:Databricks is an RDBMS by kfk on Tuesday February 02, 2021 @02:37AM (#25998263)
Some of this new tech is simpler to run than MySQL/Postgres like S3/Spectrum. Excel is ok for last mile type of analytics but it doesn't work if you are looking for serious insights. Excel also generates a significant amount of unnecessary manual work and data quality issues. It's not so black and white, what I am saying is that the analytics tech industry direction doesn't seem to address the real issues.
Re:Databricks is an RDBMS by phoe-krk on Tuesday February 02, 2021 @02:49AM (#25998316)
Sure. It's possible to create something that is both good for small data and has a better feature set than Excel/Access/etc., but then you do not have the other main asset of those databases: popularity. It's relatively very easy to find people skilled in these technologies, whereas new technologies need to come either with advocates capable of teaching others on their own or programmers already skilled with them. Otherwise, teaching people new technical skills is costly.
Re:Databricks is an RDBMS by baskire on Tuesday February 02, 2021 @02:23AM (#25998191)
Might even see negative gain.

Often time the query perf per machine goes down as you move from a single node, indexed, postgresql To a multi node unstructured data.

If you don’t have so much data or unstructured data. Why use the more complex and generally slower per FLOP solution.

Re:Databricks is an RDBMS by t0mas88 on Monday February 01, 2021 @04:03PM (#25993916)
Because most data isn't "big" data. If it fits in Excel on a laptop, why bother to roll out a distributed data lake system like Spark with all the associated ops work.
Re:Databricks is an RDBMS by noxvilleza on Monday February 01, 2021 @06:04PM (#25995366)
... and if it doesn't fit on Excel on a laptop, the natural path is just some database running on a desktop, then a server, then a beefier server. It's actually amazing how far just something like postgres will scale for your data needs.
Re:Databricks is an RDBMS by hztar on Monday February 01, 2021 @04:27PM (#25994204)
The ones I have been talking to stick to their Excel because their little part of the puzzle can solved by Excel. These technologies usually demand data to be collected, but the amount of incentive to do so is low in any classic balkanized F500 organization.
Re:Databricks is an RDBMS by magicalhippo on Tuesday February 02, 2021 @05:19AM (#25999237)
Most of our customers have SQL databases in the 5-20GB range. The largest one is around 100GB. Most of them self-host.

What would "this new tech" bring us and our customers, and would it be enough to justify the cost of porting?

Re:Databricks is an RDBMS by Draken93 on Tuesday February 02, 2021 @03:48AM (#25998653)
There are two scenarios

1. It is not necessary. For smaller datasets excel or sql servers are sufficient.

2. The small dataset has grown over the years and a other solution would be beneficial. But people like the things they are familiar with. They do not want to have the additional work, of learning new tools. Who ever wants to change the existing structures, will usually face high resistance.

In the second case, companies often accept the issues of the old solution and they tell themself it is the right way.

Re:Databricks is an RDBMS by QuesnayJr on Monday February 01, 2021 @04:17PM (#25994086)
I actually think the effect of all of this hype will move people past the Excel/Access era. There are simple analyses every company could do with R or scikit-learn that would save or make them money, and they just don't know how. Someone with AI expertise is over-qualified to do this, but they are at least qualified.
Re:Databricks is an RDBMS by vmsp on Monday February 01, 2021 @05:16PM (#25994835)
I know Excel runs the world but are there really that many people using Access?
Re:Databricks is an RDBMS by Tarq0n on Monday February 01, 2021 @10:47PM (#25997257)
In very technology deficient enterprise organizations it's sometimes the only query tool people are allowed access to.
Re:Databricks is an RDBMS by phoe-krk on Tuesday February 02, 2021 @02:50AM (#25998323)
> are allowed access to.

I enjoyed the pun.

Re:Databricks is an RDBMS by fukmbas on Monday February 01, 2021 @10:15PM (#25997095)
Because 99% of new tech is bullshit. It changed too much, no one can keep up. These old stacks are still around because they're proven to get the job done.