News for hackers. Stuff that matters.
Re:Design better databases by skrebbel on Thursday April 07, 2016 @02:29PM (#11449440)
I'm daft. What's wrong with using uuids as primary keys everywhere? I'd appreciate it if you could elaborate :)
Re:Design better databases by mvitorino on Thursday April 07, 2016 @03:05PM (#11449731)
The problem is that most uuids are generated in a way that, when sorted alphabetically as strings, would have a random order.

Example: t0: 7458e3a9-716b-4352-b2e4-b5b67d0c089b t1: 4d8d753c-1777-439d-8725-b093b1bd8430

Using this as a PK in any relational database will mean the rows are stored in the clustered index order, which causes extreme fragmentation because the db engine constantly has to find "holes" in the data pages to insert more recent records instead of adding those at the bottom of the table as would happen with any continuously increasing key.

Re:Design better databases by ww520 on Thursday April 07, 2016 @03:44PM (#11450050)
This is a non-issue. Most db engines just append new data rows at the end of the data table and assign an internal row ID for it. The keys (UUID in this case) are stored in the separate index pages using B+ tree, which searches random key (UUID) or sequentially incremented key equally well.

UUID key is a problem only if your main query is a range query on the PK of a clustered indexed table. If your main query is a range query, you should pick something that can be sorted in the range anyway.

Re:Design better databases by mvitorino on Thursday April 07, 2016 @05:02PM (#11450693)
A lot of times when a UUID is used, it's not generated in the db, such as when you need to generate guaranteed unique IDs client side. Also 4 bytes vs 16 is not "equally well" in terms of performance. Note that the the diagram linked at the root of this thread is encoding uuids as strings which means 32-36 bytes instead of 16.
Re:Design better databases by ddorian43 on Thursday April 07, 2016 @05:08PM (#11450741)
Actually only postgresql does it this way (by storing data in the heap and not in the primary index). Mysql(innodb),mssql,oracle uses the store the row in the primary-key.

Edit: I'm ~wrong, see below.

Re:Design better databases by ww520 on Thursday April 07, 2016 @05:51PM (#11450995)
Only Innodb does it by default. The others use heap table by default.

MSSQL allows clustered indexed table as an option to order the physical storage of rows. Oracle has index-organized table as an option.

Edit: They don't use clustered indexed table by default because record insertion is very expensive since clustered index forces the table to store the records contiguously in the index's order. Also Innodb is not truely clustered indexed. It only stores records contiguously for one page at the B+ Tree leaf level. Records in different pages are scattered all over even if the index values are sequential.

Re:Design better databases by Jweb_Guru on Thursday April 07, 2016 @06:14PM (#11451127)
It's pretty variable, but Postgres definitely is not the only one that does it that way (or even close).
Re:Design better databases by mvitorino on Thursday April 07, 2016 @04:59PM (#11450668)
Not the case with MS SQL Server for example (and there are probably several others). Physical row ordering (at least up to 2008 edition as far as I can recall) coincides with clustered index.
Re:Design better databases by wiredfool on Thursday April 07, 2016 @03:20PM (#11449853)
I don't believe that the Primary Key affects the physical ordering of records in PostgreSQL.

Primary Key is implemented as a unique index coupled with a not null constraint. It's essentially syntactic sugar.

Re:Design better databases by mortehu on Thursday April 07, 2016 @03:29PM (#11449940)
To elaborate, even if you do call CLUSTER[1] to enforce a physical ordering, the primary key is not the default (there is no default).

1. http://www.postgresql.org/docs/9.5/static/sql-cluster.html

Re:Design better databases by gnaritas on Thursday April 07, 2016 @03:16PM (#11449821)
That would be true except for that fact that database designers know this, and offer a way to generate sequential GUID's; so it's actually not a problem for that reason. The only real downside is performance, an int key performs better.
Re:Design better databases by mvitorino on Thursday April 07, 2016 @05:04PM (#11450713)
A number of scenarios that require UUIDs as keys, involve generation client side where such guarantees may not be available. In my view, UUIDs should not be used unless there is a strong reason for it (such as making keys not guessable). Most people will implement them incorrectly/inefficiently.
Re:Design better databases by jsmthrowaway on Thursday April 07, 2016 @04:03PM (#11450193)
> an int key performs better.

A UUID is a 128-bit integer. That people do not store, generate, or interact with them that way is the bug.

I'd kill for a 128-bit architecture so a UUID compare would be a single instruction, and it's too bad consensus is that we don't really need it.

Re:Design better databases by mortehu on Thursday April 07, 2016 @05:52PM (#11451004)
> I'd kill for a 128-bit architecture so a UUID compare would be a single instruction

Well, there's PCMPESTRI on Intel.

http://www.felixcloutier.com/x86/PCMPESTRI.html

Re:Design better databases by jsmthrowaway on Thursday April 07, 2016 @06:30PM (#11451218)
Yeah, I just said this in another comment, but SSE is my weak point; I stopped writing assembly and dealing with this stuff before SSE was even introduced.

That looks basically like what you need, though, yeah. Cool.

Re:Design better databases by gnaritas on Thursday April 07, 2016 @04:14PM (#11450275)
An int is either 4 or 8 bytes, it's still going to perform better no matter how you treat the GUID.

edit: bytes

Re:Design better databases by jessedhillon on Thursday April 07, 2016 @05:20PM (#11450821)
> An int is either 4 or 8 bits

This perfectly summarizes the bitwise-innumeracy of the argument against UUIDs. Even without 128 bit word sizes, most UUIDs are going to be non-matches in their lower bits, assuming a random distribution. There's no computational efficiency gained in the vastly wide critical path here.

Re:Design better databases by gnaritas on Thursday April 07, 2016 @05:30PM (#11450887)
> most UUIDs are going to be non-matches in their lower bits, assuming a random distribution

But you can't assume a random distribution because anyone using UUID's for database keys isn't using random UUID's, they're using sequential ones.

Re:Design better databases by jsmthrowaway on Thursday April 07, 2016 @05:34PM (#11450908)
Which would still make the low-order quadword not match.
Re:Design better databases by gnaritas on Thursday April 07, 2016 @05:44PM (#11450958)
It's a database though, you don't get to control how comparisons are done. Guid's are larger than ints, so using them as keys or joining on them is going to be slightly slower (not that it makes them bad, I'm not against GUID's). Just as using a bigint will be slower than an int.
Re:Design better databases by jsmthrowaway on Thursday April 07, 2016 @06:26PM (#11451192)
If I were implementing the uuid type in Postgres and did anything other than `if(unlikely(LO(uuid1) == LO(uuid2)) && unlikely(HI(uuid1) == HI(uuid2)))` I'd consider myself doing the wrong thing, personally. Two 64-bit compares, two flag checks, and a branch, which is about as fast as it's going to get in the absence of a 128-bit architecture.

You could actually vectorize the compares in SSE, I believe (but I stopped writing assembly before SSE existed and don't know much about it at a low level). This is why it's so critical to treat a UUID as an integer instead of a string, and I see a lot of code that does the latter.

Re:Design better databases by gnaritas on Thursday April 07, 2016 @08:24PM (#11451749)
While all of that is true, it's really not relevant to a programmer deciding on which to use as a key; he has no control over any of those things.
Re:Design better databases by jandrese on Thursday April 07, 2016 @04:33PM (#11450467)
I hope you mean bytes. On a 128bit architecture that would be a single instruction compare, just like a 32 or 64 bit compare. Performance would be roughly the same (minus maybe a cache miss because you blew it out with your fat ints).
Re:Design better databases by dhd415 on Thursday April 07, 2016 @03:23PM (#11449883)
Every sequential GUID generation feature provides guarantees only per session. In other words, when you bring your database down to upgrade or patch it and then bring it back up, your GUID sequence starts over again at some arbitrary value which is likely to overlap with your existing GUIDs.
Re:Design better databases by gnaritas on Thursday April 07, 2016 @04:12PM (#11450258)
> your GUID sequence starts over again at some arbitrary value which is likely to overlap with your existing GUIDs.

I don't believe that's true. You're right that they start the session over, but it'll still be a fresh sequence that doesn't overlap anything generated in the past.

Re:Design better databases by enobrev on Thursday April 07, 2016 @03:56PM (#11450128)
So you're saying it's specifically less efficient for a write-heavy tables, correct?

I generally prefer auto-incremented integers, but UUIDs are very useful for client-generated records (like for an app/website that's built for offline usage).

Re:Design better databases by acscott on Thursday April 07, 2016 @04:34PM (#11450473)
It's also bigger than necessary when a sequence will do. Large PKs mean larger indexes. For very large databases or very high volume every little inefficiency adds up.

As I always say, if we don't split hairs now, we'll be splitting heads later.

Re:Design better databases by Retra on Thursday April 07, 2016 @06:35PM (#11451247)
I don't understand this... How does the sort order in the index have anything to do with the arrangement of the table? If you build an index over the UUIDs, it's just going to refer to rows in the table by their internal offsets.
Re:Design better databases by dspillett on Friday April 08, 2016 @05:38AM (#11453555)
>How does the sort order in the index have anything to do with the arrangement of the table?

If you have a clustered index based upon that column (and in systems that support this it is common for the primary key to be the clustered index for the table) then the physical layout of the data will follow this column.

See http://use-the-index-luke.com/blog/2014-01/unreasonable-defa... as a good run down of the whys and wherefores of clustered indexes.

Re:Design better databases by slagfart on Thursday April 07, 2016 @07:38PM (#11451567)
Having a sort order is immensely preferable when you're joining multiple sequential rows at once. The optimiser can create a pseudo-partition for your join, and constrain its reads to that. Especially on spinning rust (or I/O constrained systems), you're also reading multiple sequential pages in a stream off disk, which gives a big speed boost compared to random reads.

Without that ordering, the index is 'dumb', and has to find all your rows one by one in its structure. Admittedly it's not as much of an issue for SSDs, but it does still impact performance.

Re:Design better databases by nickpeterson on Thursday April 07, 2016 @03:02PM (#11449706)
First there is the question of whether one should have surrogate keys at all, some oppose surrogate keys because: They make tables more difficult to reason about. They encourage structures that involve large amounts of joins.

Others embrace surrogate keys because: They make joins between tables easier (usually one field rather than a compound key where one might accidentally not include all the join predicates). They save space since an integer is almost always smaller than a compound key.

So if you are in the camp that favors surrogate keys, then consider these points:

Some quick downsides for UUIDs: If your table has a clustered index, UUIDs are generally not created in a regular order, and so you'll be constantly inserting into a part of your index rather than the end. Some databases allow sequential UUID generation to help mitigate this. UUIDs are usually 16 bytes, but an int is usually 1-8 bytes. Also keep in mind you pay this price multiple times since the primary key (generally) becomes the foreign key in other tables.

Some upsides for UUIDs: If you have a process that has no master, being able to tie disparate things together with a shared UUID is very useful sometimes.

Often developers use UUIDs/GUIDs because: Their app wants to perform a 'Create' operation and they want to generate the key in advance.

This can usually be mitigated by creating a routine that generates a range of reserved surrogate integer keys, and gives a starting value to the application layer. Then the application layer can use those values with knowledge they won't already exist. For instance, the app layer say, "I'm going to make 100 customers", and the response comes back "10232". Meaning that 10132-10232 is reserved for that process to use for customer primary keys. This allows larger (chunkier) requests which can be interrelated without constantly making requests to the Database layer. One could extent this to be a true cache for entity reserved numbers that gets dolled out as needed by the app layer.

Re:Design better databases by ww520 on Thursday April 07, 2016 @04:16PM (#11450298)
UUID as PK is perfectly fine; however, as with any design decision it really depends on your needs and weighing alternatives.

Some popular ways to do PK: natural key, sequentially generated key, and UUID. Personally I would prefer natural key if I can find an immutable natural key for the table. However, natural keys are hard to find. The food example in this case doesn't have natural key. Also if the natural key requires a compound key, it's just not worth the pain.

Sequential auto generated key is good when you need to hand it off to users, like order id or ticket number. It's short and simple and it's auto generated. The downsize is when you migrate databases, you need to seed the new database carefully or it would start from beginning again. Also in record creation, you need to do extra read to get back the newly generated key.

UUID is mostly worry free. It can be generated anywhere and doesn't need to be in the database. For a setup with distributed databases, I would use UUID just to have global uniqueness. For an offline app, I would use UUID to create the data records locally and later sync them with the main database. UUID is good when it's used internally and not exposed to the users.

Re:Design better databases by jon-wood on Thursday April 07, 2016 @04:26PM (#11450396)
Sequential IDs can also leak business information when exposed to customers such as how many orders your taking - an interested party can place an order at 10am and another at 11am. Once you've done that you can compare the IDs and know how many orders were taken that hour.
Re:Design better databases by irl_zebra on Thursday April 07, 2016 @04:41PM (#11450529)
Why should UUIDs not be exposed to users? Because of the their unwieldy look, or because of security concerns?

I ask because I have an app where previously I was accessing a particular data point via pk, and the user saw the pk in the url bar. But it could expose user/site data as any user could access that pk, or just guess at the next sequentially generated pk. I switched to uuid and now it's a little ugly in the url bar, but no data is exposed.

Re:Design better databases by ww520 on Thursday April 07, 2016 @04:50PM (#11450611)
I meant the key handed off to the user for human interaction. As a user I would prefer order number 7356 than 7458e3a9-716b-4352-b2e4-b5b67d0c089b.

If there's no human interaction with the key, UUID is perfectly fine.

Re:Design better databases by mhd on Friday April 08, 2016 @03:32AM (#11453152)
I've seen scenarios where UUIDs were used intentionally for this, to prevent users who get a full export of table data to reference the UUID. If it's just a sequence of small integers they might them to more permanent and intrinsically tied to the data, whereas UUIDs seem more like computing artifacts.

Yeah, I know, this could easily fixed by not friggin' showing surrogate keys to those users in the first place, but, well, data integration is ugly business.

Re:Design better databases by irl_zebra on Thursday April 07, 2016 @07:10PM (#11451433)
Gotcha, just wanted to make sure I wasn't missing something. Makes total sense, thanks!
Re:Design better databases by radagast on Friday April 08, 2016 @04:29AM (#11453338)
> But it could expose user/site data as any user could access that pk, or just guess at the next sequentially generated pk.

Using a UUID instead of a sequentially rolling integer ID isn't solving your problem, you're just doing security through obscurity. The real solution is implementing real authentication & authorization - not making the key harder to guess.

Re:Design better databases by naasking on Friday April 08, 2016 @10:37AM (#11454895)
> Using a UUID instead of a sequentially rolling integer ID isn't solving your problem, you're just doing security through obscurity.

A common sentiment, but not true if your id is cryptographically random. It amounts to capability security which is the right approach to security if used comprehensively.

Re:Design better databases by Scirra_Tom on Thursday April 07, 2016 @06:39PM (#11451269)
Came across an interesting case against using UUIDs the other day, in the UK it's a legal requirement that VAT invoices are sequential: http://www.hmrc.gov.uk/manuals/vatrecmanual/vatrec5020.htm

Using random UUIDs as an invoice number wouldn't cut it, you'd have to have a sequential ID in some form. You could of course have both but that's adding unnecessary complexity.

Another example, if you are designing a website and lets say give each blog post a UUID, a URL of:

website.com/blog/my-post-34

Looks better than

website.com/blog/my-post-d173affb-6bb8-4435-bef1-4e29409dff4c

Unique sequential Id's also give some clues about the relative age of rows (eg ID 100 is older than ID 50). This doesn't come up often but can sometimes be useful.

Regardless, I always find using a unique int ID for pretty much every record makes your job 10x easier and I fail to see why you would ever want to do anything different. If you want to 'hide' the ID in special cases (eg order numbers) better to generate a unique random int ID.

Re:Design better databases by xrstf on Friday April 08, 2016 @06:18AM (#11453684)
Sequential order IDs are also mandatory in Germany. At least I remember that we had to make sure in one project that the order IDs had no gaps and were sequential.
Re:Design better databases by clavalle on Thursday April 07, 2016 @02:34PM (#11449480)
There is no way to enforce data integrity between related tables by virtue of the data model for one thing.
Re:Design better databases by gcr on Thursday April 07, 2016 @02:52PM (#11449619)
I don't get this argument. Foreign key constraints are part of the schema.

What should the primary key be if not a UUID?

Re:Design better databases by clavalle on Thursday April 07, 2016 @06:23PM (#11451172)
A natural key. It is not always possible, but when it is it should be used.
Re:Design better databases by cortesoft on Thursday April 07, 2016 @03:15PM (#11449815)
an integer id?
Re:Design better databases by billybugmaster on Thursday April 07, 2016 @02:51PM (#11449614)
isn't that what the fk are for? food_uuid for example. I concede I wouldn't call it uuid but I don't see a problem keeping the data straight.
Re:Design better databases by jtmarmon on Thursday April 07, 2016 @02:43PM (#11449548)
?? foreign keys?
Re:Design better databases by kawera on Thursday April 07, 2016 @03:16PM (#11449819)