News for hackers. Stuff that matters.
Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by m0zg on Wednesday July 17, 2019 @04:12AM (#20457495)
I get that Rust is exciting, but you aren't really building this for engineers, right?

When embarking on a non-trivial project, it's good to know who the prospective end user is. Helps shape the product requirements. Looking at your feature list for V1, it can easily take a team of 5-7 excellent engineers with strong, relevant domain expertise a couple of years to do most of the items on it well if your end user is "anybody". Some items on it (such as distributed query optimization) are open research problems. Some (distributed query execution, particularly if you want to support high performance multi-way joins) can take years all on their own.

If it's something more specialized, it could take a lot less time and effort. I say this as someone who's "been there done that" (at Google). We had to support a broad range of use cases, but we also had some of the best engineers in the world, some with decades of experience in the problem domain, and it still took years to get the product (BigQuery) out the door. This was, to a large extent, because the requirements were so amorphous. First version of what later became BigQuery was built in a few months by a much smaller team, because it was targeted at a much more specialized use case: querying structured application logs in (almost) realtime.

It's much easier to not try to be all things to all people.

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by andygrove on Wednesday July 17, 2019 @04:24AM (#20457544)
Some great points there.

I have multiple goals here.

Goal #1 is to demonstrate how Rust can be effective at data science / analytics and evangelize a bit.

Goal #2 is to have fun trying to build some of this myself and improve my skills as a software engineer.

Goal #3 (getting a little more vague now) is to ensure that in some hypothetical future I can get a job working with Rust instead of JVM/Apache Spark.

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by m0zg on Wednesday July 17, 2019 @04:34AM (#20457591)
Got it. Fun side project then! Good luck. Rust does seem like a good fit for something like this.

I think the hard-ass responses in the thread (all unwarranted) simply misunderstand the goals.

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by cauterize on Wednesday July 17, 2019 @12:32PM (#20461285)
FWIW, I'm keeping your name in the back of my head for when my current company inevitably fails at their big data revolution. Biggest product produces 10-15 PB/day that must be processed. Our existing solution is 20 years old and relies on navigating C code with function line counts on the orders of 10k. Rust would've saved me a lot of time from living in gdb.

Really enjoyed the article. Keep it up!

Re:Ballista: Distributed Compute with Rust, Apache Arrow, an... by thundergolfer on Thursday July 18, 2019 @03:19AM (#20467518)
> First version of what later became BigQuery was built in a few months by a much smaller team, because it was targeted at a much more specialized use case: querying structured application logs in (almost) realtime.

Is there more about this story that you could link me to? Or would reading the whitepaper provide most of the interesting detail of how a log query engine became a general-purpose query product?