News for hackers. Stuff that matters.
Re:Building a recommendation engine inside Postgres with Pyt... by wallace01 on Tuesday October 26, 2021 @05:17PM (#29006367)
Enjoyed that read but to be honest I always had mixed feelings about PSQL Extensions. Logic in a database is wrong
Re:Building a recommendation engine inside Postgres with Pyt... by snissn on Tuesday October 26, 2021 @05:32PM (#29006560)
It’s so amazing to have the ability to enhance your database with custom methods. Keeps your data model really well organized across your infrastructure
Re:Building a recommendation engine inside Postgres with Pyt... by systemvoltage on Tuesday October 26, 2021 @06:17PM (#29007067)
That's the job of an API IMO.
Re:Building a recommendation engine inside Postgres with Pyt... by uoaei on Tuesday October 26, 2021 @07:19PM (#29007651)
An API makes sense for this case when you won't be throwing away a lot of the raw data in favor of the processed/transformed data.

But doing a lot of these operations on-server makes sense when there's a significant volume of highly parallelizable transformations which need to be done on the data before it's usable.

Of course the best solution is likely to be a happy medium between the two, where simple low-level transformations are done on-server and the rest of the data preparation is done as the data is transferred to the client.

Re:Building a recommendation engine inside Postgres with Pyt... by mrbungie on Tuesday October 26, 2021 @07:55PM (#29007977)
Not in data processing. There are cases where you have so much data that is better to bring computation to data rather than the other way around.
Re:Building a recommendation engine inside Postgres with Pyt... by FridgeSeal on Tuesday October 26, 2021 @09:52PM (#29008730)
Having dealt with the fallout of a database that’s had hundreds of stored procedures and custom code stapled into it: never again.

It was so hard to debug, so hard to safely maintain these procs, and a general nightmare

Re:Building a recommendation engine inside Postgres with Pyt... by arthurcolle on Tuesday October 26, 2021 @06:08PM (#29006981)
Do what thou wilt shall be the whole of the Law
Re:Building a recommendation engine inside Postgres with Pyt... by CameronNemo on Tuesday October 26, 2021 @05:42PM (#29006692)
Can you expand on why you think it is wrong or problematic?
Re:Building a recommendation engine inside Postgres with Pyt... by FridgeSeal on Tuesday October 26, 2021 @06:49PM (#29007389)
Data and its structure outlives the developer, and often the application.

Tightly coupling a lot of the application-specific compute in with how the day is stored and accessed says you up for even more difficulties when you need to debug, scale, migrate storage/compute or evolve your application faster or more radically than your data organisation.

Re:Building a recommendation engine inside Postgres with Pyt... by systemvoltage on Tuesday October 26, 2021 @06:14PM (#29007033)
Database is the last thing that scales usually so if you put a bunch of computational load on it besides queries, you've set yourself up for scaling/sharding sooner than later.
Re:Building a recommendation engine inside Postgres with Pyt... by lowwave on Tuesday October 26, 2021 @06:54PM (#29007433)
Not always! If computation involves math over set of records, then Postgres is great for that. Have the operation inside the db reduce connection pooling on the application level.
Re:Building a recommendation engine inside Postgres with Pyt... by rbanffy on Tuesday October 26, 2021 @08:48PM (#29008362)
> you've set yourself up for scaling/sharding sooner than later.

OTOH, if you can pre-process the data (or shards) in the server (or servers), you won't need to move it across the network for processing.

Hard problems are hard and one of the reasons databases outlive their systems and developers is because moving that data is a Royal Pain.

Re:Building a recommendation engine inside Postgres with Pyt... by jamesfinlayson on Tuesday October 26, 2021 @07:41PM (#29007867)
I remember a friend telling me in a previous job me they loaded the database with stored procedures because it ran on the most powerful server when the product launched. And then the database server hit its capacity very quickly.
Re:Building a recommendation engine inside Postgres with Pyt... by rbanffy on Tuesday October 26, 2021 @08:44PM (#29008343)
Reminds me of why I learned to program in PostScript - the PostScript printer was, by far, the fastest computer I had access to - also it had a ton of memory and its very own SCSI hard disk.

Plotting a Mandelbrot on the Mac would take a lot longer, even in C, than just making the printer do it.

Usually overnight, because the program took a couple hours to run most of the time.