News for hackers. Stuff that matters.
Re:Twitter's UID associated to each tweet will very soon exc... by drawkbox on Tuesday June 09, 2009 @03:01PM (#649674)
Always use a UUID or GUID (same thing) in todays web applications. When we get to terrabytes of data as the norm for apps int based keys are going to seem so ancient.

Sure it takes more room but space it cheap, and it makes syncing and distributed databases so much easier (not tied to one machine or cluster)

Re:Twitter's UID associated to each tweet will very soon exc... by dpcan on Tuesday June 09, 2009 @11:10AM (#649204)
I figure many twitter apps are written with SQLite as their database (in AIR, etc) so the Integer, Primary Key, can get up to 9223372036854775807

http://www.sqlite.org/faq.html

What is the point of creating a website like this?

Re:Twitter's UID associated to each tweet will very soon exc... by lallysingh on Tuesday June 09, 2009 @08:43AM (#648945)
Anyone notice the footnote on the estimated time of crossing?

"3 the average tweets per second and the Twitpocalypse date are a rough approximation, please do not schedule any vacation around that date"

Re:Twitter's UID associated to each tweet will very soon exc... by ivankirigin on Tuesday June 09, 2009 @08:45AM (#648949)
Maybe I'm just paranoid, but I index the tweet id's in Tipjoy with a string: tweetId = models.CharField(max_length=50, db_index=True)
Re:Twitter's UID associated to each tweet will very soon exc... by Steve0 on Tuesday June 09, 2009 @07:09AM (#648874)
Meh, anyone who's coding for this platform must have put those id's in a unsigned integer, wouldn't they?
Re:Twitter's UID associated to each tweet will very soon exc... by patio11 on Tuesday June 09, 2009 @08:54AM (#648960)
I suspect for a lot of people coding against Twitter the notion of integers being signed or unsigned is about as foreign to their experience as pointer arithmetic. I do not consider that a bad thing.

irb > "2,147,483,647".gsub(",","").to_i + 1

2147483648

(Forgive the extra code necessary to strip out the commas. The Windows irb console leaves a little to be desired in terms of midline editing.)

Re:Twitter's UID associated to each tweet will very soon exc... by there on Tuesday June 09, 2009 @12:32PM (#649334)
you know ruby supports _ as a comma separator, right?

irb(main):002:0> 2_147_483_647 + 1

=> 2147483648

Re:Twitter's UID associated to each tweet will very soon exc... by tjogin on Tuesday June 09, 2009 @08:03AM (#648910)
It's such a low barrier of entry and almost non-existant learning curve to code for Twitter, so I expect quite a lot of services to have made that mistake. Not the big ones, though.
Re:Twitter's UID associated to each tweet will very soon exc... by cl3m on Tuesday June 09, 2009 @05:32AM (#648790)
Re:Twitter's UID associated to each tweet will very soon exc... by pierrefar on Tuesday June 09, 2009 @05:56AM (#648810)
It's actually sooner. They assume 83 tweets per second (reference 2) but the number is 200 tweets per second: http://friendfeed.com/scobleizer/50e673d8/some-stats-from-tw...
Re:Twitter's UID associated to each tweet will very soon exc... by m_eiman on Tuesday June 09, 2009 @06:00AM (#648819)
From the link:

"Twitter is seeing about 200 tweets per second, during peak loads. - Robert Scoble"

Note "during peak loads", so 83 is probably not too far off.

Re:Twitter's UID associated to each tweet will very soon exc... by pierrefar on Tuesday June 09, 2009 @06:47AM (#648857)
Fair enough. Do we have any solid references of the actual average rate of tweets?
Re:Twitter's UID associated to each tweet will very soon exc... by m_eiman on Tuesday June 09, 2009 @07:19AM (#648883)
I haven't seen one, but it's easy enough to get it. Just check the UID of the latest tweet on the public timeline, wait a week and check again.
Re:Twitter's UID associated to each tweet will very soon exc... by jsdalton on Tuesday June 09, 2009 @08:59AM (#648977)
They must read HN. It now says:

2 at a rate of 151 tweets per seconds

Re:Twitter's UID associated to each tweet will very soon exc... by IsaacL on Tuesday June 09, 2009 @01:07PM (#649408)
"Values updated every 5 minutes". Looking at the source (I don't actually know JavaScript, but it seems pretty simple) I'm assuming that means they get the current TwitID AND the current rate of increase.
Re:Twitter's UID associated to each tweet will very soon exc... by DrJokepu on Tuesday June 09, 2009 @06:06AM (#648823)
In the age of multicore 64 bit CPUs, they're not using signed 32 bit integers as primary key, do they?
Re:Twitter's UID associated to each tweet will very soon exc... by thristian on Tuesday June 09, 2009 @06:32AM (#648840)
To be fair, they say "For some of your favorite third-party Twitter services not designed to handle such a case..." rather than accusing Twitter themselves of primary-key ineptitude.
Re:Twitter's UID associated to each tweet will very soon exc... by domdefelice on Tuesday June 09, 2009 @06:32AM (#648839)
I think the problem is not in twitter itself but in the services using their APIs.
Re:Twitter's UID associated to each tweet will very soon exc... by ks on Tuesday June 09, 2009 @01:08PM (#649411)
Twitter is using an unsigned 64-bit bigint according to this:

http://twitter.com/twitterapi/status/2048659057

Re:Twitter's UID associated to each tweet will very soon exc... by charlesju on Tuesday June 09, 2009 @04:20PM (#649898)
I'm going to put money down that this will not affect anything.
Re:Twitter's UID associated to each tweet will very soon exc... by TweedHeads on Tuesday June 09, 2009 @08:37AM (#648940)
ALTER TABLE twits MODIFY COLUMN twitid BIGINT UNSIGNED NOT NULL

There, fixed, send check for $20 to [email protected]

Call me back when they reach 18446744073709551615

Re:Twitter's UID associated to each tweet will very soon exc... by sanj on Tuesday June 09, 2009 @10:16AM (#649102)
Assuming that Rails is still interacting with the DB, and the ID in question is the primary key (ie, the 'id' on the 'twits' table) isn't quite that easy.

I had to do this a couple of years back when I realized I had a table that'd grow quickly:

http://snippets.dzone.com/posts/show/4422

Feel free to send the donation to the same place.

Re:Twitter's UID associated to each tweet will very soon exc... by lpgauth on Tuesday June 09, 2009 @05:48PM (#650060)
Is there no easier way then this? Can't believe that wasn't integrated into rails...
Re:Twitter's UID associated to each tweet will very soon exc... by jhancock on Tuesday June 09, 2009 @09:47PM (#650383)
"Can't believe that wasn't integrated into rails..."

You just made my day. I haven't laughed so hard in ages ;)

Re:Twitter's UID associated to each tweet will very soon exc... by joezydeco on Tuesday June 09, 2009 @10:47AM (#649161)
Hmm, didn't fix my iPhone client. Try again!
Re:Twitter's UID associated to each tweet will very soon exc... by fleaflicker on Tuesday June 09, 2009 @10:24AM (#649118)
That command will take several hours to complete and write lock the table.
Re:Twitter's UID associated to each tweet will very soon exc... by Andys on Tuesday June 09, 2009 @09:31PM (#650368)
Unless you use PostgreSQL.

But then you would have had 64 bit ID by default anyway.

Re:Twitter's UID associated to each tweet will very soon exc... by jsonscripter on Tuesday June 09, 2009 @11:34AM (#649245)
Why can't you just redirect the writes to a new temporary table and join the original and temporary on reads? (I'm not a database admin)
Re:Twitter's UID associated to each tweet will very soon exc... by karanbhangui on Tuesday June 09, 2009 @01:28PM (#649445)
I'm actually interested in this answer as well. I know of a company using a proprietary db and now that their growth has literally exploded, they're stuck with it.
Re:Twitter's UID associated to each tweet will very soon exc... by jsonscripter on Tuesday June 09, 2009 @02:47PM (#649632)
I wonder what literally exploding growth looks like...
Re:Twitter's UID associated to each tweet will very soon exc... by blhack on Tuesday June 09, 2009 @03:11PM (#649706)
Re:Twitter's UID associated to each tweet will very soon exc... by TweedHeads on Tuesday June 09, 2009 @11:00AM (#649189)
Nothing the fail whale can't fix ;-)
Re:Twitter's UID associated to each tweet will very soon exc... by sho on Tuesday June 09, 2009 @05:42AM (#648798)
That is quite an amazingly high number. Surely 100m people have not posted 2,000 "tweets" each.

I wonder what proportion of the total is human-typed, and how much is machine generated?

Re:Twitter's UID associated to each tweet will very soon exc... by cperciva on Tuesday June 09, 2009 @06:11AM (#648826)
Surely 100m people have not posted 2,000 "tweets" each.

Two billion tweets would be 1M people posting 2k tweets each, or 100M people posting 20 tweets each.

Re:Twitter's UID associated to each tweet will very soon exc... by sho on Tuesday June 09, 2009 @01:54PM (#649499)
Oh. Oops. Of course you are right. That was pretty dumb of me : /
Re:Twitter's UID associated to each tweet will very soon exc... by axod on Tuesday June 09, 2009 @05:59AM (#648814)
Amazingly high? considering how long twitter has been going? It's extremely small IMHO.

http://news.ycombinator.com/item?id=648725

"Just 10% of Twitter users generate more than 90% of the content, a Harvard study of 300,000 users found."

I'd expect most of it is machine generated.

Re:Twitter's UID associated to each tweet will very soon exc... by hugothefrog on Tuesday June 09, 2009 @06:01AM (#648820)
I'd also expect most of those numbers not to be used. Their db writes are probably not contending on one monotonic counter. They're probably using some numbering scheme which guarantees uniqueness across multiple writable databases, but not global serialisation based on id.
Re:Twitter's UID associated to each tweet will very soon exc... by bretthoerner on Tuesday June 09, 2009 @08:34AM (#648933)
I'm not sure how they'd be doing that and still giving each tweet a number so close to all the others. I have no idea where you see a "public timeline" as people mentioned above, so I just did a search for "i"... the following tweets were about a second apart.

2088840357 2088840362 (difference of 5, and that's without a real public timeline where you could easily prove each tweet takes the next number in a sequence)

So it isn't as if they're using UUIDs or something unique... they very much seem to be relying on a single counter (currently).

Re:Twitter's UID associated to each tweet will very soon exc... by hugothefrog on Tuesday June 09, 2009 @08:47AM (#648953)
For a first guess, if there were 5 writing dbs, each could write every 5th id, starting at a different index (0,1,2,3,4).

You'd end up with globally unique ids, and as long as your writes were (approx.) evenly distributed, all 5 sequences would be (approx.) close to each other.

Who knows, they might have 100 writable dbs, with only 70% up at any point in time, meaning only 70% of all integers are actually used.. anyway, I don't know, I can just imagine them doing something like this if they wanted certain things from the system.

Re:Twitter's UID associated to each tweet will very soon exc... by hboon on Tuesday June 09, 2009 @04:33PM (#649927)
No. Because their API exposes an explicit ordering of the IDs - there is a sinceID: parameter.
Re:Twitter's UID associated to each tweet will very soon exc... by Timothee on Tuesday June 09, 2009 @10:48AM (#649163)
You can get the public timeline from here: http://twitter.com/public_timeline

Note that one can delete a message, which could explain some gaps.

Re:Twitter's UID associated to each tweet will very soon exc... by jackowayed on Tuesday June 09, 2009 @12:29PM (#649328)
I think the gaps are mostly people posting protected tweets, which thus don't get into the public timeline.
Re:Twitter's UID associated to each tweet will very soon exc... by jonknee on Tuesday June 09, 2009 @11:20AM (#649219)
They do something like that with userid's. Used to be incrementing and now they have gaps.