News for hackers. Stuff that matters.
Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Thursday October 31, 2019 @07:45PM (#21414972)
tl;dr we use it for a similar set of tasks that one would use Airflow for.

Unlike Airflow, this lends itself to microbatching and streaming. Plus a bunch of housekeeping items ticked off that Airflow never got around to. With a bit of devops engineering time, you can have perfect manage the size of your worker cluster on k8s and scale it up/down with ingest demand, etc.

I'll say one thing though. The Perfect website used to be a lot more technical and explicit about what it is and isn't. Now it's mostly sales gobbledegook. Maybe not a good sign. I've seen this happen before with dremio.

Re:FastSpark: A New Fast Native Implementation of Spark from... by sandGorgon on Friday November 01, 2019 @12:00AM (#21416176)
This is super interesting!

Do you run dask on k8s ? I have been concerned that dask does not leverage kubernetes HPA for autoscaling...but instead chooses to run an external scheduler.

How has your experience been ?