News for hackers. Stuff that matters.
Re:TileDB: Storing massive dense and sparse multi-dimensiona... by _pmf_ on Thursday October 26, 2017 @10:52AM (#15559318)
> Python, R, Matlab and Excel

Mentioning Matlab and Excel immediately puts the product in the category "they know what they are doing" as opposed to "another group of sophomores trying to reinvent data science".

I'm still waiting for a raw data dump with "avoid copies at all cost" access for very raw, very verbose vehicle and manufacturing data that is in 99.9 per cent never accessed, but must be analyzed when errors are detected late in the process. I.e. there's practically no transformation to apply during storage, but upon access, transformation must be done.

If TileDB is kind of like a more structured, more low-level struct oriented Redis, it is a very welcome addition.

Re:TileDB: Storing massive dense and sparse multi-dimensiona... by srean on Thursday October 26, 2017 @01:51PM (#15561301)
Apart from a diminishing population of old farts and a clutch of national labs and universities that Mathworks keeps well greased, is MATLAB even relevant anymore ?
Re:TileDB: Storing massive dense and sparse multi-dimensiona... by jakebol on Thursday October 26, 2017 @04:39PM (#15562795)
Jake from TileDB, Inc. Engineering as a discipline is conservative (for good reason). The tools and the processes they use change slowly. Matlab is still entirely relevant both in the sciences and in industry. There are people with huge amounts of domain knowledge (who may only know Matlab or Excel) that are increasingly called upon to analyze and interpret larger amounts of data. These "old farts" are the people engineering, designing, and debugging our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about. There is tremendous value in building bridges across communities and across generations here.

I say this as someone who helped in small ways to develop open source alternatives to Matlab.

Re:TileDB: Storing massive dense and sparse multi-dimensiona... by StanSeltser on Thursday October 26, 2017 @11:21PM (#15565453)
Stanislav, Seltser, Petacube agree with Jake. Matlab still has some unique features to it which many open source projects (eg python) don't have - eg FPGA integration, System Modeling and Simulation. Combine that with decent language, nice debugger and many strong industry-specific solutions its good bet its will be around for a while. The reasons people dump matlab is not that its overpriced but due to the lack of integration with big data systems and the fact that matlab license cost at large scale becomes untenable.Plus number of industries using these unique matlab features is relatively small.
Re:TileDB: Storing massive dense and sparse multi-dimensiona... by srean on Thursday October 26, 2017 @05:00PM (#15562951)
> These "old farts" are the people engineering and designing our modern world. Empowering people with domain knowledge to answer data driven questions is what the democratization of data science is all about.

BTW I don't begrudge that TileDB has MATLAB support.

I see plenty of modern scientific/engineering workload in my day job. From what I see around myself it is usually a handful of people set in their ways that are holding others back by keeping a ridiculously overpriced tool alive that has comparable if not better alternatives. I gather from comments that it is different in Europe.

Re:TileDB: Storing massive dense and sparse multi-dimensiona... by curious12345 on Thursday October 26, 2017 @05:01PM (#15562963)
So Jake, how are the Julia bindings coming along?
Re:TileDB: Storing massive dense and sparse multi-dimensiona... by bigger_cheese on Thursday October 26, 2017 @07:20PM (#15564120)
It is used pretty extensively in my industry (industrial manufacturing). There has been some talk about R but not sure it will be widely adopted.

You are right that engineers are conservative which no doubt plays some role. But I'd say technical debt and legacy code plays much more of a role.

Most Engineers (Chem, Mech, Matls etc) are not exposed to code during university. Maybe it is changing now but it is slow process. Often first exposure comes when the engineer enters industry and is asked to work on existing model - usually under supervision of a senior engineer. You learn whatever the language the senior engineer knows and that is typically what they learnt from similar mentorship - it is often Fortran or C/C++. Our Industrial Process has not changed dramatically in last 30 years. So once efficiently written and accurate simulation code exists very little reason to rewrite (if given the choice to rewrite an existing model in whatever the current language du jour is or continue to hack on something that already exists most engineers - at least the ones I know, would chose to hack on existing code). For pretty much this reason there is a heap of Fortan that is still alive in my org (with roots that can be traced back to the '80's). I think it's probably worth mentioning a lot of engineers don't approach code thinking about algorithms - it's equations i.e. I need to write something to solve Bernoulli's equation - or Ergun's equation or similar. Linear programming (i.e solving simultaneous equations) is the other main reason to write code and hey MATLAB does this pretty well...

The other reason for being "gunshy" about new technology at least in my org is that a lot of the senior engineering people still remember when we got bitten by investing in Microsoft "Stack" in the late 90's. There were a lot of Modelling done with VB6 and Access which tuned out to be a technology dead end. Access databases in particular have plagued our org in many cases multiyear effort to migrate something out of Access. Open source probably protects against this but everyone wants to be sure any new technology will still be around in 15 years time. I think that is why people in my org are finally starting to look at R it has passed that initial hurdle and there is some confidence it is not just a fad...

Re:TileDB: Storing massive dense and sparse multi-dimensiona... by jeff_science on Thursday October 26, 2017 @07:14PM (#15564084)
Yeah, those deep-pocketed government lab and university researchers are clearly the dominant revenue source for MathWorks. Everybody in the corporate world is super nimble and moved on to Numpy as soon as they could.
Re:TileDB: Storing massive dense and sparse multi-dimensiona... by _pmf_ on Thursday October 26, 2017 @03:55PM (#15562425)
Pervasive in the German automotive industry.
Re:TileDB: Storing massive dense and sparse multi-dimensiona... by StanSeltser on Thursday October 26, 2017 @10:59PM (#15565353)
Stanislav Seltser, Petacube _pmf: think of distributed persistent numpy arrays as opposed to key-value pairs in Redis. The idea you will not pull data to do computation, you will push your code in. pretty good for archiving due to compression