For the record: the valid chars string is 62 characters, so naively using a modulo on a random byte will technically introduce a bias (since dividing 256 values by 62 leaves a remainder). I don't expect it to really matter here, but since you're putting in the effort of using crypto.randomBytes I figured you might appreciate the nitpick ;).
Melissa E. O'Neill has a nice article explaining the problem is, and includes a large number of ways to remove the bias as well:
https://www.pcg-random.org/posts/bounded-rands.html
(in this case adding two more characters to the validChars string would be the easiest and most efficient fix, but I'm not sure if that is a possibility here)
Indeed, there's no reason you couldn't just add "_" and "-" or "." as well to complete the set. Your identifier will still be URL-safe. I've been using this type of encoding for years [1] for these kinds of ids to use in URLs.
https://sourceforge.net/p/sasa/code/ci/default/tree/Sasa.Web...
That said, iOS and Android text selection have gotten worse recently, IMO.
As a developer I will be exposed to ids being displayed in code, terminals, browsers of various sorts, database editors, json dumps, text editors, api responses, chat messages, you name it.
If all you need is unpredictability, a minor bias is sub-optimal but not disastrous. The 256 % 62 bias used here should reduce the min-entropy per character by 5%. And you can easily minimize the bias by using larger integer than a single 8-bit byte. There few algorithms where minor biases cause a disaster, like DSA.
Imagine you prefix all customer IDs with `cus_`, but at some point decide to rename Customer to Organization in your codebase (e.g. because it turns out some of the entities you are storing are not actually customers). Now you have some legacy prefix that cannot be changed, is permanently out of sync with the code and will confuse every new developer.
Yes you can.
You can support the old cus_<ID> prefix as well as the new org_<ID> prefix, but always return org_<ID> from now on
Though I believe they mostly do it because their IDs are sequential, so without prefix you wouldn't easily notice if you use the wrong kind of id. They also only apply prefixes at the api boundary when they base36 encode the IDs, the database stores integers
It's generally a good idea to drop vowels for this reason.
We never had a similar issue with our random numbers/letters/reset passwords or anything like that which don't have any kind of "dont return profanity" protections. Though I agree, someone getting a randomly generated customer portal url or something containing fuck or similar would look bad. Our cloudfront or something (or was it main public facing s3 bucket? can't remember) starts with "gay" and was never picked up on.
Many in real life do refer to themselves as gitano or Gypsy and would ask for others to refer to them as such.
Of course it's very easy to find an article saying otherwise and then using that as the end of discussion for Wikipedia editors.
Better to be careful and let any individuals or communities tell you what they want. I have Roma connections in my family and at one point the word we’d use is ‘gypsy’. But, because I’m not Roma myself, if I came across some other group I wouldn’t assume I’m just allowed to say it to them.
For the sake of mass communication where you can’t really know your receiver, you have to do your best to just communicate whatever you need to (i.e. ‘a single approach’). Choosing to use a word that is ambiguous as to whether it is a slur is a bit unwise. I think it is probably unwise to do the same in personal interactions.
That attitude defaults to better at violence in a particular context gets to impose their will. Or whoever has the security forces to back them out of an inferiority situation.
You miss the part when I can arbitrarily warn you about a lot of things myself and then use any interpretation of rule breaking on your part to attack you.
I know it may sound harsh but this is where many end up going so let's make it explicit.
Locally (north west england) people generally use "traveller" as a description ... but there are definitely people who use that as a slur.
Language be like that sometimes.
Comes from the ye olde paper-based blogs they call newspapers. When an article is being put together it’s given a short name, sort of like a project name. This name would remain the same throughout the article’s life - from reporter through to editor - it left its trail through the process. Like a slug.
>The origin of the term slug derives from the days of hot-metal printing, when printers set type by hand in a small form called a stick. Later huge Linotype machines turned molten lead into casts of letters, lines, sentences and paragraphs. A line of lead in both eras was known as a slug.
Apologies for the wetbrain hallucination, HN!
5. Print. a. a thick strip of type metal less than type-high. b. such a strip containing a type-high number or other character for temporary use. c. a line of type in one piece, as produced by a Linotype. 8. Journalism. a. a short phrase or title used to indicate the story content of a piece of copy. b. the line of type carrying this information.
Got to the Wikipedia page https://en.wikipedia.org/wiki/Slug_(publishing) which could possibly support the slimy conclusion of "it's a trail through the process" but that article has an etymology section that refers to the metal slug
I guess it could mean both depending on whether you're looking for the meaning of the word or the meaning of the concept but I didn't find any other slimy grub references (via an admittedly limited double check)
There is a similarly suspicious pattern in some of their longer identifiers, invoices for example at 24 characters (ca.143 bits) all seem to have a first byte of "00000010".
Even in the article you've linked to, look closely at the IDs:
pi_3LKQhvGUcADgqoEM3bh6pslE
pm_1LaXpKGUcADgqoEMl0Cx0Ygg
cus_1KrJdMGUcADgqoEM
card_1LaRQ7GUcADgqoEMV11wEUxU
Based on these values it seems there's a metadata preamble in the upper bits of the supposedly "random" value, and it's quite possible that some have an embedded timestamp, possibly timeshifted, and a customer reference, as well as a random part, and who knows maybe there's a check digit as well.
It's possible - albeit this is not analytical but more of a guess - that the customer ID includes an epoch-ish timestamp followed by randomness or (worst case, left field) is actually a sequence ID that's been encrypted with a 64-bit block cipher and 32 bits of timestamp as the salt, or something similar (pro tip: don't try that at home).
My view is that either Stripe's engineering blog is being disingenuous with the truth of their ID format, or they're using a really broken random value generator. If the latter, I hope it's only in scope of their test/example data.
https://dev.to/stripe/designing-apis-for-humans-object-ids-3...
function makeSlug(length: number): string {
const alphabet = "0123456789abcdefghjkmnpqrstvwxyz";
let result = "";
for (let i = 0; i < length; i++) {
result += alphabet[crypto.randomInt(alphabet.length)];
}
return result;
}This can be solved in user space by regenerating if the character sequences are detected, but this a) skews the distribution, and b) potentially takes time, especially when the ID generator is made to not be “too fast”. I want to generate a single ID that passes the blocklist in a timeframe that is not too fast, if that makes sense.
Is there an ID generator that takes this into consideration?
"It's one-three-oh-dee-ee-el. Yes, I'm sure, EL as in elephant."
In other words, asking a customer/consumer to be able to recite something in phonetics is not realistic in most cases.
Fortunately the code already takes this into consideration and removes ambiguous characters.
I also ask them to check my readback of key information they have given me and vice-versa; usually that works well.
(No, it isn't, if you look closely enough.)
Then its simple for support to say "red is one, green is ell". But you can't just add a color to a character, because copy paste/rich formatting don't work everywhere, or even transfer well...
Alternatively, if you use ⓪ and ⒈ it matters less if the user says "at" or "zero", and more that they didn't say "oh" or "one".
(Unicode has its strengths. Making up replacement characters isn't one.)
https://github.com/kstenerud/safe-encoding/blob/master/safe3...
Notably, confusable characters are interchangeable when being ingested (although a machine encoder MUST always produce canonical output). https://github.com/kstenerud/safe-encoding/blob/master/safe3...
So a user can confuse 1 for l, 0 for o, I for l, u for v, uppercase, lowercase etc, or the agent can say any of those over the phone, and it won't matter.
Crockford also incorporates error correction, which is unnecessary in modern systems since the underlying protocols do that already.
It's obvious why the safe64/safe80/safe85 cannot do this, but is there a reason why the safe16 version doesn't have the same features?
I have a note from a few years ago that 367CDFGHJKMNPRTWX may be a sufficiently unambiguous alphabet. Drop the one you like the least (probably N) to obtain a faux hex encoding.
Multiple calls to a randomness generator can be expensive, and a waste of entropy; production-scale random string generators should still respect this and ask for a block of bytes, then encode them, but with bias correction. You're off the hook in this case, I think node's implementation of randomInt is doing exactly that for you and conserving remaining entropy in a cache.
This means you can store them in the db not as a string but as a uuid, which is a lot more performant. You also get time stamping for free.
Especially if you want to use this thing as an index in a database, you'll run into problems where you try doing middle insertions frequently, which causes fragmentation.
The solution to this problem is making the higher order characters time sorted [1]. You don't need to go all out like uuid, you can have a pretty low resolution. It's more important that new insertions tend to be on the same page. If you have a low frequency insertions then minute resolution is probably good enough. (Minutes since 2000 is an easy calculation).
To implement that here, I'd suggest looking at how base64 or 85 encoders work and use that instead of repeated mods. You can then dedicate the upper bits to a time component and the lower bits can remain random. [2]
[1] https://vladmihalcea.com/uuid-database-primary-key/
[2] https://github.com/mklemm/base-n-codec-java/blob/master/src/...
1. An auto-incremented 64-bit (unless you have a good reason, in which case 32-bit is fine) primary key, used internally for foreign key relations. This will generally result in less index bloat on associated tables, and fast initial inserts.
2. A public-facing random string ID. Don't use this internally (other than in an index on the table it's defined for), since it's large. But this should be the only key you expose to end-users, to prevent leaking data via the German Tank Problem: https://en.wikipedia.org/wiki/German_tank_problem
Only create the second key if this is data you're exposing to users, of course — for data that's only used internally, just use the 64-bit auto-incremented PK and skip the added index bloat entirely.
One benefit of a random id is if you are working with more complex data models it can make creating those easier/faster. Instead of having a centralized location to get new ids from (the DB) you can create ids on the fly from the application which can turn the write into a single action from the application rather than a dance of inserting the main table, getting the new id, then inserting to the normalized tables.
Not great if you are spending 1 minute on the first lookup just to do the mapping.
Worst case you can cache, as others have mentioned, but TBH I think you don't need to for a query that simple and should save the cache space for something more complicated. Most SQL DBs are excellent for ready-heavy workloads; it's writes that tend to make them fall over.
[1] https://dev.to/wrongbyte/cryptography-basics-breaking-repeat...
I got some great DB cache/index performance from ULIDs with a bit of work to order the ULID timestamp bits in the way the DB's 128-bit column "uuid" sort best supported.
Now that UUIDv7 is standardized we should hopefully see good out-of-the-box collation for UUIDv7 in databases sooner rather than later.
Imagine, for example, that you picked DES and "kangaroo" as the secret several years back. You are now pinned to an algorithm and key with known security problems and a weak key.
Create 1000 users very quickly in a row and you now have 1000 samples of enc(i) all the way up to enc(i+1000) that can help you break the algorithm.
Not saying it will be easy but it surely lowers the guarantees given by the encryption. Someone better at crypto can probably quantify this risk better than me.
Another hack for advanced active-active situations where you may need to route events before replication completes: encoding the author shard / region in the lower order bytes.
There are lots of interesting primary key hacks for dealing with physical or algorithmic complications.
Because of the random portion of the key, that means you'll get good distribution so long as the distribution algorithm isn't something stupid like relying solely on the highest order bits.
Whether or not that's bad fully depends on your platform and the number of writes you do. If you're using a massively distributed database like Datastore, Spanner etc, you want random keys as to avoid hot spots for writes. They produce contention.
One solution to that is having more complex keys. For example, in one of our more contentious tables the index includes an account id (32bit int) and then the id of the entity being inserted. This causes inserts for a given account to still be contiguous (resulting in less fragmentation) while not creating a writing hotspot since those writes are distributed across various clients.
I don't know how you can successfully maintain or develop software without developing an understanding of the underlying domain. I've seen devs try that route and the quality of their work has never been high.
* not the same as your internal DB row primary keys, which in Postgres should usually be bigserial
Yes, in that there are DB technologies not built in a fashion where records are stored in a sorted order of some fashion. No, in that they are very much not common technologies. Most databases, relational, non-relational, etc have some form of a B-Tree at their core somewhere.
MySQL docs claim that a hash index is much faster if you only need kv lookups, so it seems like uuid4 with hash index would be suitable. Never tried it though, and can't say whether it's faster than using a btree with uuid7. Seems like in theory it would be.
Hash maps should generally always have faster lookups than Btree based structures. However, they'll have slower writes especially when contested. A key issue hash tables have to deal with is what happens when a remapping needs to happen. For example, when 2 keys have the same hash. In that case, locking becomes a lot more messy.
For a btree this is simpler. It's built to be able to handle reshuffling and rebalancing in a way that's semi easy to have fine grained locks around.
Edit:
Other honorable mentions: ObjectID's as used by MongoDB which contain the creation timestamp. Also Discord's snowflakes (inspired by Twitter's iirc), which also contain the creation timestamp.
Case in point, the parent poster's base64 ID is 14 characters long. When encoded as base32 that's still only 17 characters (or 19 in base16), and now you have completely gotten rid of all notion of casing, which is annoying to communicate verbally.
It's immediately clear, when I see an ID in a log somewhere or when a customer sends me an ID to debug something, to which customer, system and entity such an ID belongs.
UUIDv7 is monotonic, so it's nice for the database. Those IDs are not as 'human-readable' for the average Joe, but for me as an engineer it's a bliss.
Often I also encode ID's I retrieve from external systems this way: `uri:3rd_party_vendor:system_name:entity_type:external_id` (e.g. `uri:ycombinator:hackernews:item:40580549:comment:40582365` might refer to this comment).
Let me guess - you're a Java developer, right?
For numeric values, making them all a multiple of 11 is a simple way to catch all single digit errors or single transpositions.
From this there is many possibilities, but for example, let’s consider only a base ten. Starting with vowels order o, i, e, a, u with mnemonic o, i graphically close to 0, 1 and then cyclically continue the reverse order in alphabet (<-a, <-e, <-i, <-o*, |-u). We now only need two consonants for the two series of 5 cardinals in our base ten, let’s say k and n.
So in a quick and dirty ruby implementation that could be something like:
$digits = %w{k n}.product(%w{o i e a u}).map{it.join('')}
def euphonize(number) = number.to_s.split('').map{$digits[it.to_i]}.join('-')
euphonize(1234567890) # => "ki-ke-ka-ku-no-ni-ne-na-nu-ko"
def numerize(euphonism) = euphonism.split(?-).map{$digits.find_index(it)}.map{it.to_s}.join.to_iAnd in most cases I think you're also better off just using a uuid and encoding its bytes as base 32, in which case you're basically doing type id's [1]. If the "slug" portion of the id actually encodes a uuid, then it gives you the option to store it in your database using an appropriate uuid type. This will make your database much happier than using long string PK's.
In my databases, I often prefer integer primary keys for performance reasons. On the other hand, I don't want to expose my primary keys because they are easy to guess.
Recently I've been playing with Rust, and ended up publishing a library to encrypt IDs in they way I like:
If you use JavaScript/TypeScript, you can make them like this: