glapark - Hackdot

glapark

Re:Launch HN: Data Mechanics (YC S19) – The Simplest Way to Run Apache... on Monday May 11, 2020 @03:53PM (#23146229)

Shuffling in Spark works well for small datasets, but is not reliable for large datasets because fault tolerance in Spark is incomplete. For example, check this Jira:

https://issues.apache.org/jira/browse/SPARK-20178

So, if your problem was mainly due to shuffle-heavy workload, then I guess no managed Spark service would be able to alleviate/eliminate it by automatic parameter tuning. In other words, your pain might be due to a fundamental problem in Spark itself.

IMO, Spark is great, but its speed is no longer its key strength. For examples, Hive is much faster than SparkSQL these days.

Re:Ask HN: More open than open source products on Friday January 17, 2020 @01:36PM (#22077653)

I guess my question was a bit unclear. We state upfront that our product is not open source because it is shipped with a proprietary part. It just builds on an open-source project.

Ask HN: More open than open source products

Posted by glapark on Friday January 17, 2020 @01:25PM
from the ask-hn dept.

We are planning to start a business in the near future around a product (a data warehouse system) that we have been building over the years. Our product builds on an Apache-licensed open source project by replacing its core component with a proprietary engine.I would like to ask HN about the licensing model that we are considering. Tentatively we call it as 'more open than open source products'...

( Read More... | 4 comments | 2 points | on HN )