News for hackers. Stuff that matters.
Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @12:10PM (#22376561)
Yes, this was inspired by that, and the fact that our database engineer went "I'm not putting that much garbage in my database"!

And I figured, yeah, that's still a lot of data, and it's massively misusing several different tools. Why not create something that is actually designed for the task?

Re:BlurHash: Algorithm to generate compact representation of... by mark242 on Thursday February 20, 2020 @12:45PM (#22376898)
the fact that our database engineer went "I'm not putting that much garbage in my database"!

This is really disingenuous-- a DBA should not be a gatekeeper of what gets put into a database. Storing 200 bytes of extra data per row costs you absolutely nothing. Even if you assume 100 million photos, that's 20GB of data. That's a little over $2 of EBS cost per month. Who cares.

Re:BlurHash: Algorithm to generate compact representation of... by penagwin on Thursday February 20, 2020 @04:44PM (#22378941)
> a DBA should not be a gatekeeper of what gets put into a database. Storing 200 bytes of extra data per row costs you absolutely nothing.

To be fair, isn't it the DBA's job to gatekeep the database? The application programmers and business managers don't have the in-depth understanding to know how every individual change will effect the database. Of course they should follow best practices, but it's not really their job to know it inside and out - that's what the DBA does.

You don't know that 200 bytes per row would cost nothing - that's the DBA's job to understand it's cost. And we aren't talking about the amount of data, but without an in-depth understanding of what tables they were adding it to, how they're indexed, cached, etc, then you don't know the real cost. That 200 bytes might mean less can be stored in memory, if this is a frequently fetched table then that could be bad.

That's not to say the DBA made the right call, but that we don't have enough information to know if it was.

Re:BlurHash: Algorithm to generate compact representation of... by JoshTriplett on Thursday February 20, 2020 @01:25PM (#22377266)
Unless you have some clever storage mechanism (such as a column-oriented database or a separate table with a join), 200 extra bytes directly in each row of a frequently used table makes that table less efficiently use disk and memory caches. Consider if the entire rest of that row takes up 100 bytes, and now it takes up 300. You now can fit a third as many rows in each disk block or memory page.
Re:BlurHash: Algorithm to generate compact representation of... by magicalhippo on Thursday February 20, 2020 @02:26PM (#22377786)
This discussion reminds me of when I read in our database server's documentation something along the lines of "for very wide tables (more than 10 columns)" and I literally had to laugh out loud, as the main table at work is over 500 columns wide...

Gotta love organic db schemas with long history...

Re:BlurHash: Algorithm to generate compact representation of... by hinkley on Thursday February 20, 2020 @04:08PM (#22378645)
> the main table at work is over 500 columns wide...

This is horrifying, but at least not as horrifying as the public sector database I once had to work with that predated proper support for foreign keys, so there was a relationship table that was about 4 columns wide and must have had more rows than the rest of the database combined.

Even the database they had moved this schema to struggled with that many join operations.

Re:BlurHash: Algorithm to generate compact representation of... by socceroos on Thursday February 20, 2020 @01:45PM (#22377471)
This is so true. Fitting rows in the page was a big win for query times in my case.
Re:BlurHash: Algorithm to generate compact representation of... by ska on Thursday February 20, 2020 @02:11PM (#22377675)
Hence columnar stores.

But parent is right that the choice of what shouldn't be coming from your DBA; they should be more interested in how.

Re:BlurHash: Algorithm to generate compact representation of... by JoshTriplett on Thursday February 20, 2020 @02:17PM (#22377729)
It is the DBA's job to advise, and that includes advice like "this isn't a good idea", or "could you do something else instead", or "this will require XYZ rearchitecting, which will affect these other factors/queries/etc". The DBA isn't the final decision-maker, but they're a critical advisor.
Re:BlurHash: Algorithm to generate compact representation of... by ska on Thursday February 20, 2020 @02:32PM (#22377825)
Yes of course, but it's the business need that should drive that discussion, not the DB architecture. "I'm not putting X in my database" is very different than "Well, if we need to do Y, I think we'll have to change A, B & C". Up to others to decided if it is worth it or not.
Re:BlurHash: Algorithm to generate compact representation of... by JoshTriplett on Thursday February 20, 2020 @02:48PM (#22377944)
It's also completely reasonable to say "We shouldn't do Y". The response to everything shouldn't be to go immediately into solution space and list off trade-offs on the assumption that Y is happening. With people you trust for both their integrity and their expertise, "we shouldn't do Y" is a reasonable shorthand for "there are a pile of valid reasons we shouldn't do Y, and you and I both know I can list them for you off the top of my head, and it might well be possible anyway but it would take a lot more work and trade-offs than you might expect".

That isn't the end of the conversation; if the response to "we shouldn't do Y" is "it'll be a pain if we can't do Y", then you can explore solution space together, where that solution space includes both trade-offs to support doing Y and alternatives to doing Y. Use experts for their expertise, rather than the equivalent of "I don't care, make it happen".

Or, in short: yes, of course the DBA doesn't unilaterally dictate the architecture, but maybe listen to their expertise anyway.

Re:BlurHash: Algorithm to generate compact representation of... by ska on Thursday February 20, 2020 @02:53PM (#22377980)
As far as I can tell, we are agreeing.
Re:BlurHash: Algorithm to generate compact representation of... by DagAgren on Thursday February 20, 2020 @03:11PM (#22378120)
You should also consider that that was a loose, humorous description of what happened, and not something really worth analysing to the last letter.
Re:BlurHash: Algorithm to generate compact representation of... by ska on Thursday February 20, 2020 @03:17PM (#22378176)
Fair. I have multiple times run into that attitude seriously, otoh.
Re:BlurHash: Algorithm to generate compact representation of... by hinkley on Thursday February 20, 2020 @04:00PM (#22378572)
It is still my very firm belief that the primary motivation for the use of ORMs is backlash against gatekeeping DBAs. People keep looking for (literal) tools that keep them from having to deal with a (figurative) tool who won't let them use the damned database that belongs to the business to support business initiatives.

[edit: and that almost all network traffic is now L7 routed because of gatekeeping firewall administrators in the late 90's, so everything had to go over :80 or :443]

Re:BlurHash: Algorithm to generate compact representation of... by Spivak on Thursday February 20, 2020 @05:10PM (#22379136)
But that tool is also paid by the business to ensure that the database stays up and performant.

Seems kinda rude to ignore the motivations and incentives of the people who are responsible when problems happen.

Yes devs and infrastructure are often at odds professionally by design. Making it personal ignores the reason that different parts of the app have different owners that must agree to make changes.

“Those damn developers won’t just let me push my hotfix to master and make me have to do a code review when our severs are getting hammered.”

Re:BlurHash: Algorithm to generate compact representation of... by hinkley on Thursday February 20, 2020 @05:26PM (#22379221)
This only indicates that you have worked with a better class of DBA than I have.

"Why are we creating new table and foreign key relationships for this single bit of data?"

"Oh, because we aren't allowed to add any columns to <the appropriate table>"

"Wait. None?"

"None."

(Only the last conversation I had of this sort involved MySQL, which was notoriously bad at adding columns to live tables)

The cost of 5 extra joins for all business activities is going to outweigh whatever you think you have going on with that original table.

I think in some ways the situation was improved by expanding developer responsibilities to include these tasks rather than having a dedicated role. When you have to deal with the consequences of your own decisions instead of someone else being responsible, some activities won't be done at all, while others will go more smoothly. Without the database being personified, it's more of a doing something for somebody (else) instead of doing it to somebody.

Re:BlurHash: Algorithm to generate compact representation of... by streb-lo on Thursday February 20, 2020 @12:50PM (#22376935)
> disingenuous

Don't think that's the correct adjective here.