News for hackers. Stuff that matters.
Re:Cloudera and Hortonworks merge by jamesblonde on Wednesday October 03, 2018 @06:03PM (#18135220)
We have a startup, Logical Clocks (www.logicalclocks.com), that just raised money to sell our data-science next-generation Hadoop stack. It has a new verison of HDFS called HopsFS (Nvme storage, distributed metadata) and support for GPUs in YARN. And distributed tensorflow, Spark, Airflow, Flink.

So, what does that mean for us? I am shellshocked. I expect prices to increase (good for us). What else should we expect?

Re:Cloudera and Hortonworks merge by agibsonccc on Wednesday October 03, 2018 @06:57PM (#18135497)
Both Horton and Cloudera require a huge number of partner resellers and consultants. What horton and cloudera have isn't just software, but brand, an install base, and the ability to back it up.

They merged partially because they were both being cannibalized by services revenue they couldn't get rid of. Now they are struggling to move to the cloud (see Atlas now)

I'm not sure yet another hadoop distro with a bunch of 1 off tooling that is supposedly faster is the answer. Why is all that stuff even needed?

Re:Cloudera and Hortonworks merge by jamesblonde on Wednesday October 03, 2018 @06:58PM (#18135507)
Adam, what about Java for deep learning? Is that needed?
Re:Cloudera and Hortonworks merge by agibsonccc on Wednesday October 03, 2018 @07:07PM (#18135549)
According to my customers and users yes? Granted, we do a lot more than just that though.

Dl4j itself has a decent sized user base. Ranging likely from your phone maker to your bank and retail store.

We have our own software distro too which is why I'm commenting on this. We don't try to boil the ocean with a bunch of tech though.

There's a whole new crop of companies focusing on solving bits of the ML problem well rather than trying to do storage and god knows what else.

My point here about you guys is you're trying to compete in what is largely a commodity market. People don't need all this stuff. Simplicity won here. It's not about better tech.

You guys have the same pitch MapR does and largely the same problem: Better tech is only part of the problem with adoption. You need customers, users, and a clear business model when going to market.

Cloudera and Horton ran one playbook that at least somewhat worked (it got them public) and now they can focus on competing with the cloud vendors, which made the right decision and just made commonly used software easy to use.

Re:Cloudera and Hortonworks merge by jamesblonde on Wednesday October 03, 2018 @07:13PM (#18135581)
We don't. We are the only on-premise vendor with proper support for GPUs and python. We are a full data science stack, backed by Hadoop. We even have Kubernetes for model serving. And python in the cluster (with conda environments). And we have customers and funding. Nothing like MapR. And none of the legacy mapreduce crap.
Re:Cloudera and Hortonworks merge by agibsonccc on Wednesday October 03, 2018 @07:16PM (#18135601)
You bundle way more than they do and on top of that have your own file system just like mapr does.

Your pitch is still about differentiated tech, not a large install base, a differentiated business model

and something related to people like a good partner ecosystem.

Your pitch here requires tons of services.

People don't know how to use all of this stuff especially on prem.

It takes more than just code to build a business.

I say this as someone who's been doing this since 2013. It's not easy.

Re:Cloudera and Hortonworks merge by jamesblonde on Wednesday October 03, 2018 @07:28PM (#18135676)
Ok, now you're changing your angle. Differentiated tech is what we are all about - that is ok by me (for now).

If you want to train DNNs on a hundred GPUs today on-premise on TensorFlow, come to us, we can do it. They can't.

Re:Cloudera and Hortonworks merge by agibsonccc on Wednesday October 03, 2018 @07:39PM (#18135731)
I don't think I changed my angle here? I'm still addressing

the same point. Tech doesn't matter. Simplicity does.

Even in our own product line, we only do a small

subset of this. We don't even require a cluster

to run. We also work with tech that people use.

You are currently competing with horovod

and kubeflow. eg: "competing with free"

You need more than that to survive.

Generally, that comes down to services.

Re:Cloudera and Hortonworks merge by ironchef on Wednesday October 03, 2018 @08:02PM (#18135895)
A slightly different wording is it comes to using the tech... which is either through services or through adoption / operationalization (which... with complexity... is very difficult)
Re:Cloudera and Hortonworks merge by agibsonccc on Wednesday October 03, 2018 @09:12PM (#18136261)
Exactly!
Re:Cloudera and Hortonworks merge by xapata on Thursday October 04, 2018 @10:49AM (#18140380)
Who needs to do that? Not many companies. Deep Learning is overhyped, just as MapReduce was.
Re:Cloudera and Hortonworks merge by HenryR on Wednesday October 03, 2018 @11:00PM (#18136801)
I am surprised that anyone is trying to differentiate on storage at this time, precisely when that's the part of the stack that's being cannibalized by the cloud vendors (look at the rate of innovation in HDFS over time; the effort is going elsewhere). Are you just targeting on-premise clusters, or is there some differentiation planned for the cloud as well?
Re:Cloudera and Hortonworks merge by jamesblonde on Thursday October 04, 2018 @03:58AM (#18137986)
We think that there is a niche for a higher performance dist FS than S3. We have integrated NVMe hardware with our HDFS implementation (HopsFS) and made its metadata layer distributed. NVMe means you can, for example, work with datasets with millions of files for deep learning - instead of having to munge them into parquet files because your FS is slowing down your machine learning pipeline.

Reference: https://www.logicalclocks.com/millions-and-millions-of-files...

We have also redesigned the stack around our distributed metadata layer.

We are primarily targeting on-prem right now, but HopsFS would be the fastest DFS in the cloud if you ran it there today.

Re:Cloudera and Hortonworks merge by Game_Ender on Thursday October 04, 2018 @07:57AM (#18139044)
How does HopsFS compare to Lustre?
Re:Cloudera and Hortonworks merge by SpaceManNabs on Wednesday October 03, 2018 @07:13PM (#18135580)
You support Flink too? Nice! I am glad to see more and more people eyeing it as a viable framework for streaming.

Do you support Kubernetes too?

Re:Cloudera and Hortonworks merge by jamesblonde on Wednesday October 03, 2018 @07:14PM (#18135588)
Yes, but for model serving. Not for Flink or Spark. That's on YARN, for now.