Re:Finance Pros Say You’ll Have to Pry Excel Out of Their Co...
by sgt101 on Friday December 01, 2017 @07:32AM
(#15823429)
Sometimes, but it depends on what you are doing with a data set - even if it's 1.75Gb : from the link...
"Since the problem is basically just to look at the result lines of each file and aggregate the different results" which is why this works on this data for this problem. Many problems are not efficient to solve in this way, for example searches through large products and expansions, especially where the evaluation function is costly.
Some problems are data bound - you are handling 100'sTb of data, hadoop can be a good fit.
Some problems are CPU bound, and embarrassingly parallel - you require massive parallelism to get through the core seconds that you need to solve the problem. If you need 1 million core seconds and you have 1000 cores, and a way of getting the 1000 cores to do 1/1000th of the work... all is good.
Some problems are memory bound. You can build single space machines with many TB memory now, also you can cache with SSD's and optanes more than in the past, but Hadoop does offer a utility for these problems.
But in a corporate all problems are admin and discovery bound. If your dataset is what I need and I don't know about it, can't find it and anyway if I did then you won't give it to me and I can't unlock it legitimately; I am screwed. If it's a proper corporate asset then I've got a fighting chance.
"Since the problem is basically just to look at the result lines of each file and aggregate the different results" which is why this works on this data for this problem. Many problems are not efficient to solve in this way, for example searches through large products and expansions, especially where the evaluation function is costly.
Some problems are data bound - you are handling 100'sTb of data, hadoop can be a good fit.
Some problems are CPU bound, and embarrassingly parallel - you require massive parallelism to get through the core seconds that you need to solve the problem. If you need 1 million core seconds and you have 1000 cores, and a way of getting the 1000 cores to do 1/1000th of the work... all is good.
Some problems are memory bound. You can build single space machines with many TB memory now, also you can cache with SSD's and optanes more than in the past, but Hadoop does offer a utility for these problems.
But in a corporate all problems are admin and discovery bound. If your dataset is what I need and I don't know about it, can't find it and anyway if I did then you won't give it to me and I can't unlock it legitimately; I am screwed. If it's a proper corporate asset then I've got a fighting chance.