Re:FastSpark: A New Fast Native Implementation of Spark from...
by sandGorgon on Thursday October 31, 2019 @03:26PM
(#21412635)
Could you talk about Prefect ? we are in the process of moving from Spark to Dask. I have never heard of prefect. what do you use it for ?
[ Parent
| on HN ]
Re:FastSpark: A New Fast Native Implementation of Spark from...
by missosoup on Thursday October 31, 2019 @07:45PM
(#21414972)
tl;dr we use it for a similar set of tasks that one would use Airflow for.
Unlike Airflow, this lends itself to microbatching and streaming. Plus a bunch of housekeeping items ticked off that Airflow never got around to. With a bit of devops engineering time, you can have perfect manage the size of your worker cluster on k8s and scale it up/down with ingest demand, etc.
I'll say one thing though. The Perfect website used to be a lot more technical and explicit about what it is and isn't. Now it's mostly sales gobbledegook. Maybe not a good sign. I've seen this happen before with dremio.
Re:FastSpark: A New Fast Native Implementation of Spark from...
by sandGorgon on Friday November 01, 2019 @12:00AM
(#21416176)
This is super interesting!
Do you run dask on k8s ? I have been concerned that dask does not leverage kubernetes HPA for autoscaling...but instead chooses to run an external scheduler.
How has your experience been ?