News for hackers. Stuff that matters.
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Thursday October 31, 2019 @01:37PM (#21411455)
Nice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)
Re:FastSpark: A New Fast Native Implementation of Spark from... by aloknnikhil on Thursday October 31, 2019 @02:33PM (#21412088)
Spark is specifically useful for querying streaming data. How would a distributed database help with that? You'd have to build your own stream executor on top of that.
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Friday November 01, 2019 @12:59AM (#21416388)
Agreed. at the same time, Building stream execution pipeline is not rocket science. I am not saying modern distributed SQL Databases are exact replacement or clones of Spark. I am saying with little more help from the application server they are much more capable than Spark.

You can use the following options individually or in combination.

Option 1 : Pipeline DB extension (PostgreSQL)

Option 2 : Service broker in commercial SQL databases or building PUSH/PULL queue if not supported. There are many libraries in each programming language which tries to do that. Also see option 4.

Option 3 : Using CDC or Replication for synchronous or asynchronous streamed computation on single or multi node cluster

Option 4 : Transducers. For example, you can compose many sql functions or procedures to act on a single chunk of data instead of always doing async streamed computation after each stage of transformation.

Re:FastSpark: A New Fast Native Implementation of Spark from... by dominotw on Thursday October 31, 2019 @04:29PM (#21413384)
isn't all data streaming data? whats so specail about streaming data.
Re:FastSpark: A New Fast Native Implementation of Spark from... by mvitorino on Thursday October 31, 2019 @02:31PM (#21412072)
How would you efficiently keep continuously updated complex metrics (aggregations, windowed functions, etc) calculated on top of unbounded/streaming data using a database? I'm not saying that Spark is the ideal solution, but there are a set of problems that require tools such as Spark.
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Thursday October 31, 2019 @02:51PM (#21412287)
One example could be PipelineDB extension for PostgreSQL

http://docs.pipelinedb.com/introduction.html

Re:FastSpark: A New Fast Native Implementation of Spark from... by bsg75 on Thursday October 31, 2019 @03:01PM (#21412381)
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Friday November 01, 2019 @01:04AM (#21416409)
I know. But at the same time, 1.0 is pretty much capable. Give it a try and you will realize it.
Re:FastSpark: A New Fast Native Implementation of Spark from... by madenine on Thursday October 31, 2019 @01:49PM (#21411612)
depends what you’re doing. For querying large datasets? 100% with you.

For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out

Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Thursday October 31, 2019 @02:03PM (#21411762)
What does spark win at exactly?

Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.

Re:FastSpark: A New Fast Native Implementation of Spark from... by sandGorgon on Thursday October 31, 2019 @03:26PM (#21412635)
Could you talk about Prefect ? we are in the process of moving from Spark to Dask. I have never heard of prefect. what do you use it for ?
Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Thursday October 31, 2019 @07:45PM (#21414972)
tl;dr we use it for a similar set of tasks that one would use Airflow for.

Unlike Airflow, this lends itself to microbatching and streaming. Plus a bunch of housekeeping items ticked off that Airflow never got around to. With a bit of devops engineering time, you can have perfect manage the size of your worker cluster on k8s and scale it up/down with ingest demand, etc.

I'll say one thing though. The Perfect website used to be a lot more technical and explicit about what it is and isn't. Now it's mostly sales gobbledegook. Maybe not a good sign. I've seen this happen before with dremio.

Re:FastSpark: A New Fast Native Implementation of Spark from... by sandGorgon on Friday November 01, 2019 @12:00AM (#21416176)
This is super interesting!

Do you run dask on k8s ? I have been concerned that dask does not leverage kubernetes HPA for autoscaling...but instead chooses to run an external scheduler.

How has your experience been ?

Re:FastSpark: A New Fast Native Implementation of Spark from... by ai_ja_nai on Thursday October 31, 2019 @02:33PM (#21412098)
Very interesting. Can't find references to "Perfect", though; could you please point to a link?
Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Thursday October 31, 2019 @02:35PM (#21412120)
https://www.prefect.io

Not the most SEO-friendly choice of name. Great product though.

Re:FastSpark: A New Fast Native Implementation of Spark from... by vamega on Thursday October 31, 2019 @10:16PM (#21415739)
Are you using their cloud product? The core/open source product doesn't have a way to persist schedule data.
Re:FastSpark: A New Fast Native Implementation of Spark from... by maximente on Thursday October 31, 2019 @03:39PM (#21412769)
looks like dask is python-only, so it's a nonstarter (loser) for already existing JVM code that runs on spark
Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Thursday October 31, 2019 @07:37PM (#21414919)
Spark stacks inevitably end up with PySpark though. It's rework for people who already committed to Spark, sure. And for bigger projects that committed to Spark this change isn't justifiable. But for a greenfield project, choosing Spark is just silly today.
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Thursday October 31, 2019 @02:49PM (#21412270)
Say for an example, I am using PostgreSQL 12 + CitusDB extension

Data cleaning -> PL/SQL and various inbuilt functions for the transformation of data (or new UDF if required at all)

Processing -> PostgreSQL Parallel processing on the local node and Citus DB extension for distributed computing and sharding

Analytics -> Many options here. Materialized views OR Triggers OR Streaming computation with PipelineDB extension OR Using Logical replication for stream computation

ML -> PG support variety of statistics functions. It also supports PL/R and PL/Python extension to interface with ML libraries.

Also, there are various kinds of Foreign Data Wrappers supported by PG - https://wiki.postgresql.org/wiki/Foreign_data_wrappers

Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Thursday October 31, 2019 @03:09PM (#21412460)
Yeah that's not going to work for what people call analytics workloads today.

PG is great but it's not suitable to be a feature store and sure as hell not suitable to fan out ML workloads. In a modern ML stack, PG might play the role of the slow but reliable master store that the rest of the ML pipeline feeds off.

Re:FastSpark: A New Fast Native Implementation of Spark from... by riku_iki on Thursday October 31, 2019 @04:00PM (#21413069)
> hell not suitable to fan out ML workloads

depends on the scale? Not everyone processes petabytes of data.

> PG might play the role of the slow

You have any benchmark in your hand to support this? I believe highly optimized C code in PG can be significantly faster than Scala inside Spark.

Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Thursday October 31, 2019 @08:02PM (#21415066)
> I believe highly optimized C code in PG can be significantly faster than Scala inside Spark.

There's no question about this. If you can express your task in terms of PG on a single instance, then you probably should.

When you get to more complex tasks, like running input through GloVe and pushing ngrams to a temporal store, PG offers very little - which is fine, it's not at all what PG is designed for. Inter-node IO eclipses single node perf, which is why Spark is used despite being a terribly inefficient thing (although in the case of Spark, it's so inefficient that for interim sized workloads you'd actually be better off vertically scaling a single node and using something else). PG won't help at all with these tasks.

Also, that smorgasbord of extensions GP listed isn't offered by any cloud vendor as a managed service afaik, meaning you must roll and manage your own. Depending on your needs, that might be a show stopper.

Re:FastSpark: A New Fast Native Implementation of Spark from... by riku_iki on Thursday October 31, 2019 @09:59PM (#21415667)
> like running input through GloVe and pushing ngrams to a temporal store

why exactly you think PG will not do this well?

Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Friday November 01, 2019 @06:28AM (#21417467)
Tell me how you'd do it and I'll tell you why it won't work :)
Re:FastSpark: A New Fast Native Implementation of Spark from... by riku_iki on Friday November 01, 2019 @10:14AM (#21419054)
gloves are stored in table: token -> vector. Function tokenizes text and store in another table: texd_id, token

Then you join first and second table.

Also, I think typical scenario is to resolve embeddings in your model code or data input pipeline.

Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Friday November 01, 2019 @11:37AM (#21419926)
> Also, I think typical scenario is to resolve embeddings in your model code or data input pipeline.

Correct. PG has no place in this workload other than being the final store for the model output. And even then, you'd be using a column store like Redshift or Clickhouse. PG not even suitable for the ngram counters because its ingest rates are way too slow to keep up with a fanned out model spitting out millions of ngrams per second in addition to everything else going on in the pipeline.

You -could- probably do it all in PG. But that'd be a silly esoteric challenge exercise and not something anyone would try on a project. I am sure you recognise that.

Re:FastSpark: A New Fast Native Implementation of Spark from... by riku_iki on Friday November 01, 2019 @12:01PM (#21420227)
I would say "fanned out model spitting out millions of ngrams per second" is much more unusual exercise comparing to using PG for ETL workload.
Re:FastSpark: A New Fast Native Implementation of Spark from... by missosoup on Friday November 01, 2019 @12:49PM (#21420901)
A typical twitter post will have about 50 2/3/4-grams. Let's ignore skipgrams. The twitter decahose will throw about 600 of these at you per second. That's 30k barebones ngrams per second to keep with the decahose.

But you have a year worth of historical data that you want to work with. If you're able to process 1m ngrams per second, it'll take a couple of days to get through that. You probably want to get closer to 10m/s if you're tweaking your model and want to iterate reasonably quickly. Of course there's ways to optimise all that and batch it and whatnot, but basically any big data tasks with the need to work on historical data and iterate on their models, quickly end up with kafka clusters piping millions of messages per second to keep those iteration times productive.

Ultimately this post is about Spark, and the comment that started this was someone listing PG 'replacements' for traditional ML pipeline components. If you need Spark, you're at scales where PG has no place.

Re:FastSpark: A New Fast Native Implementation of Spark from... by riku_iki on Friday November 01, 2019 @01:21PM (#21421345)
That's why I mentioned scale in my first comment. For sub-TB datasizes with 16 cores CPU and NVME raid (you can get such machine for less than $1k nowdays) PG will be just fine.

Also in typical ML pipeline as I mentioned you can generate ngrams in input function of your model (Dataset API in TF), you don't need to store it somewhere.

Re:FastSpark: A New Fast Native Implementation of Spark from... by outside1234 on Thursday October 31, 2019 @04:42PM (#21413527)
I see you haven't worked with a truly titantic amount of data then :)
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Friday November 01, 2019 @01:00AM (#21416390)
I see you assume too much. :)
Re:FastSpark: A New Fast Native Implementation of Spark from... by nxpnsv on Thursday October 31, 2019 @02:11PM (#21411840)
If you want a more complicated transform on lots of data your fancy sql won’t help.
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Thursday October 31, 2019 @02:54PM (#21412306)
Please give me an example. I can't think of any transform which cannot be done by using SQL or inbuilt Functions or new UDF.
Re:FastSpark: A New Fast Native Implementation of Spark from... by brokensegue on Thursday October 31, 2019 @04:39PM (#21413483)
train a set of sklearn models one each per a random partition of the data (computed distributed). then combine all those models using averaging and evaluate them all against an even larger dataset. how do you do that in SQL
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Friday November 01, 2019 @01:19AM (#21416464)
Sharding the table can help scale the problem across many machines and as I mentioned earlier you can use PL/R or PL/Python language extension to lift all sorts of ML functions to SQL functions.
Re:FastSpark: A New Fast Native Implementation of Spark from... by brokensegue on Friday November 01, 2019 @11:18AM (#21419722)
I'm unfamiliar with PL/Python. Can you have a Python object be the returned value of a sql query? Because that's a requirement of my example.
Re:FastSpark: A New Fast Native Implementation of Spark from... by nxpnsv on Friday November 01, 2019 @09:32AM (#21418551)
It’s also possible to do a lot in excel, it is just not always the best tool for the job.
Re:FastSpark: A New Fast Native Implementation of Spark from... by ai_ja_nai on Thursday October 31, 2019 @04:29PM (#21413394)
Spark != SQL

It's also graph analysis and ML models.

Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Friday November 01, 2019 @01:24AM (#21416476)
Graph analysis -> Recursive common table expression (https://www.postgresql.org/docs/current/queries-with.html)

ML models - I already mentioned how to uplift R and Python functions to SQL function. even if you are not using PostgreSQL many other databases help you with uplifting and interfacing with existing ML libraries through FFI

Re:FastSpark: A New Fast Native Implementation of Spark from... by RocketSyntax on Thursday October 31, 2019 @03:18PM (#21412555)
distributed in-memory computing for massive datasets to big to fit into vertically scaled memory. generic tabular files, not tables. delta lake.
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Friday November 01, 2019 @01:01AM (#21416400)
Yes those features help and all of the distributed SQL databases have data and query cache.
Re:FastSpark: A New Fast Native Implementation of Spark from... by RocketSyntax on Friday November 01, 2019 @09:34AM (#21418572)
Then once the subset of data is in distributed memory... you hittem w pyspark and all compatible libraries.