Why old-school PostgreSQL is so hip again

Re:Why old-school PostgreSQL is so hip again by sidlls on Monday December 11, 2017 @12:16PM (#15898455)

Postgres is likely suitable for 90% (or more) of the crap "engineers" currently fall all over themselves to implement Kafka, Spark, and whatever other bizzword laden "big data" or "machine learning" resume driven development for.

It's not a niche.

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by 3pt14159 on Monday December 11, 2017 @12:23PM (#15898518)

I'm still trying to find an engineer that can explain to me why they use Redis for job queuing over Postgres.

Redis for HTML caching: Cool.

Redis for queueing: Are you really sure you need the performance gain? Really? You can do 10k writes per second with Postgres (100k per second with COPY). If you need more than that fine, but for 99% of web apps out there it's just one more moving piece that probably doesn't get included in your backups.

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by stevenwoo on Monday December 11, 2017 @12:34PM (#15898604)

I was looking at that choice for my latest project and the gem recommended (resque?) only using Redis for the job queuing at the benefit of much higher performance over something like delayed_jobs using postgresql in jobs. It didn't seem worth it as job queueing was not the main feature of the server. I suppose if one were doing some sort of real time feature where you had to queue things off a Rails API response thread it would make sense but wasn't doing that for now. But just judging by the number of downloads, the Redis based job queue is possibly more popular for some reason.

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by rickbergfalk on Monday December 11, 2017 @06:44PM (#15901855)

Proper queuing in RDBMSs can be tricky.

2ndQuadrant has a good write up of what is required to make sure you don't fall into any traps: https://blog.2ndquadrant.com/what-is-select-skip-locked-for-...

Now if we're just talking about persisting data to somewhere - postgres COPY is pretty amazing.

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by orangejewce on Monday December 11, 2017 @12:31PM (#15898584)

Spark is not a buzzword if you do analytics, it's the preferred platform, especially over native Hadoop. Honestly most developers have no idea what the heck Spark is or how to use it. NoSQL/Kafka yes, Analytics/Machine Learning is still far too complicated for most to stand up on their own.

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by sidlls on Monday December 11, 2017 @09:09PM (#15902524)

Spark has a place: at large scale. For 100s of GB to a few TB of data PostgreSQL works very well. At least, it does for my team. I don't want Spark, Kafka, NoSQL or any other modern fad near my team's data. It's just not appropriate.

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by jack9 on Monday December 11, 2017 @12:54PM (#15898752)

Kafka was useful in a project because of the semantics for what amounts to writing to file from network sockets.

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs. If you want a common format to trade with buddies, it's useful now. I expect something else will come along to replace that fad tech.

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by tabacof on Monday December 11, 2017 @01:10PM (#15898921)

>Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs.

Can you expand on that?

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by mdekkers on Monday December 11, 2017 @03:00PM (#15899883)

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs.

You gave a typo there.

Spark is a terribly inefficient solution to any known/stable data processing or analytics jobs I have ever come across.

There, fixed that for you.

[ Parent | on HN ]

Re:Why old-school PostgreSQL is so hip again by jack9 on Thursday December 14, 2017 @07:26PM (#15928737)

I don't think that's fixed, because it's not what I assert.

Any job that is predictable, can be done faster and cheaper with some C (or Go) and ad-hoc delegation to load balanced VMs. Spark is difficult to optimize for consistent processes, poor on resource usage, and requires specialized knowledge. Just pay for someone who has done a little embedded programming and stop creating buzzword jobs because a prototype went to production without comparison.

[ Parent | on HN ]