News for hackers. Stuff that matters.
Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by axod on Friday December 19, 2008 @02:38PM (#403745)
I don't believe that's true. Google sure, but twitter is just another message routing system, like any one of the IM networks.

The average irc server probably handles more than twitter - certainly email software does.

There's a ton of software that needs to scale, and be able to handle a large volume of data.

Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by gnaritas on Friday December 19, 2008 @03:53PM (#403830)
There's a ton of software that needs to scale

Yes, but there's vastly more that doesn't, you need to stop thinking that most software runs on the Internet and has tons of users, it doesn't. The vast majority of software is written for businesses and runs either on an intranet or is available on the internet to the users of that business.

For every public app you know about that needs to scale there are a hundred you don't know about that don't need to scale. Most programmers work on systems that will never need to scale, EVER. It is a tiny tiny minority of programmers that actually work on systems that become publicly popular on the Internet and actually face scaling issues.

Secondly, we're talking about programmers here, not the software, most email systems use existing mail software written by that tiny tiny minority of programmers who write such systems. The vast majority of programmers will NEVER write a mail system, a chat system, or anything that handles large volumes of data or users. Most of them of doing one thing, biz apps for in house projects at big and small companies exposing relational data to in house users.

How many IM networks are there? Now how many programmers are there? I'll say it again, most software does not need to scale. Because a lot of software does need to scale does not make that statement in any way false because a shit load more doesn't need to scale.

Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by gaius on Friday December 19, 2008 @06:08PM (#403976)
The vast majority of software is written for businesses and runs either on an intranet or is available on the internet to the users of that business.

That doesn't mean a thing in terms of scalability. One user can max out a fast machine with OLAP cubes or other DSS. Or any sort of simulation or modeling. Think in terms of datasets, not in terms of users (all many users are is a large dataset with a requirement for a quick turnaround of each computation).

Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by gnaritas on Friday December 19, 2008 @06:43PM (#404002)
If an app only has one user, it doesn't need to scale, it needs to perform. Scalability is all about users and only loosely related to performance. Datasets are irrelevant.
Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by sadfsa on Saturday December 20, 2008 @03:52AM (#404422)
I don't know what dictionary you're using, but here are a few definitions from mine:

scalable - How well a hardware or software system can adapt to increased demands.

scalable - the ability of a product or network to accommodate growth

The ability to scale hardware and software to support larger or smaller volumes of data and more or less users.

the ability of a computer system to shrink or grow as requirements change

Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by gaius on Saturday December 20, 2008 @06:27AM (#404487)
That's not true. It needs to scale from 1M of data to 1G to 1T and beyond. You seem to think that scalability only refers to concurrency. It doesn't. It means that you're as close as possible to O(n) or better yet O(1) as your workload increases, whatever that workload is.
Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by mechanical_fish on Friday December 19, 2008 @03:06PM (#403776)
Twitter is not "just a messaging system". See:

http://www.hueniverse.com/hueniverse/2008/03/on-scaling-a-mi...

(Many thanks to Eran Hammer-Lahav for writing this so that I don't have to.)

Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by axod on Friday December 19, 2008 @04:49PM (#403896)
I respectfully disagree. It's a simple message passing system, of which there are a ton of examples. This isn't rocket science.

The problem is that where once it would have been written in c as a server, it's now written with big databases and 'popular' languages - so it's harder to scale.

Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by kragen on Friday December 19, 2008 @05:15PM (#403933)
It would have taken longer to get working in C, which would have dramatically increased the probability that the project would have gotten killed before it saw users (it sounded like a pretty stupid site to me at first!), and it would be harder to rev the UI and features as often as they did before they hit the scaling problems.

But a message server in Ruby with Eventmachine wouldn't have been that hard, would it?

Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by gaius on Friday December 19, 2008 @05:25PM (#403948)
True. IBM have basically solved messaging with MQseries running on OS/390. All that remains is do that, only cheaper.
Re:Coding Horror: Hardware is Cheap, Programmers are Expensive by seiji on Saturday December 20, 2008 @12:26AM (#404311)
Proprietary message queues are a huge problem (and a huge business).

There are better ways: http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protoc...

For a good introduction check out http://google-ukdev.blogspot.com/2008/09/rabbitmq-tech-talk-...