News for hackers. Stuff that matters.
Re:Tell HN: Server Status by cincinnatus on Thursday January 16, 2014 @08:06AM (#7069131)
I'm sure it has been asked many times before, but I'd love to hear the latest thinking... Why in 2013 is HN still running on bespoke hardware and software? If a startup came to you with this sort of legacy thinking you'd laugh them out of the room.
Re:Tell HN: Server Status by matthewmacleod on Thursday January 16, 2014 @08:15AM (#7069175)
this sort of legacy thinking

That's the kind of facile statement that makes people riotously mock the entire startup community, like "MongoDB is webscale" but even less valid.

Cloud services are not a panacea, and there are myriad situations in which running one's own infrastructure can be a good idea. What matters is that the issues and benefits are taken into account; if one can show research demonstrating that a custom infrastructure is cheaper, or more reliable, or less prone to legal issues, for example, then there's nothing to laugh at.

And remember that PaaS in particular can cost a buttload of money - I'm certain it's contributed to the downfall of more than one otherwise promising startup.

Re:Tell HN: Server Status by eloff on Thursday January 16, 2014 @08:30AM (#7069234)
The advantage of cloud servers is if one experiences corruption or goes down you just kill it and start a new one. If your cloudy EBS equivalent experiences corruption, you restore from snapshot and off you go again. Either way it involves less downtime than HN seems to have. The downside is it costs more (usually, depends on how high your server management and data center overheads are.) I'd like to point out that despite several high profile down time incidents in the aws, never was there a case that I'm aware of where you couldn't just restore from your last snapshot to another availability zone or region.
Re:Tell HN: Server Status by shiftpgdn on Thursday January 16, 2014 @08:57AM (#7069363)
You realize cloud services are vulnerable to data loss as well?[1] The cloud isn't some magic machine off in a datacenter somewhere. Its a bunch of servers and SANs just like what you or I would roll out if we needed bare metal infrastructure. The only difference is the extraordinary markup that you're paying amazon to use their servers.

[1]http://blogs.computerworld.com/18198/oops_amazon_web_service...

Re:Tell HN: Server Status by eloff on Thursday January 16, 2014 @09:56AM (#7069660)
Yes, obviously. That's why I said you should restore from backups in that event. If you lose your backups on S3, congratulations, you had better odds of winning in your state lottery. The big difference is not just the price as you say, but the flexibility.
Re:Tell HN: Server Status by throwaway5752 on Thursday January 16, 2014 @09:46AM (#7069619)
Before the cloud we just called those backups.
Re:Tell HN: Server Status by KaiserPro on Thursday January 16, 2014 @11:05AM (#7070104)
And how do you guarantee the validity of your snap shot? has your process been writing corrupt records for the last n hours and not notice?

Do you even have access to that snapshot if the system is down?

the cloud simply exchanges know levels of failure with unknown level of failure. You pay someone else to think about it for you.

If there is a massive data loss at amazon, and people have been backing up to glacier, the recovery times will jump from hours to days even weeks, simply because there aren't enough drives to recover from.

The cloud may be more reliable, but when it goes down, it'll gown down hard, leaving you high and dry if you don't have real backups.

Re:Tell HN: Server Status by eloff on Thursday January 16, 2014 @11:21AM (#7070237)
The same with any other incremental backup. You have a backup retention policy and go back to the last good one. Being in the cloud doesn't change that.
Re:Tell HN: Server Status by perlpimp on Thursday January 16, 2014 @09:03AM (#7069389)
Yeah you can do private cloud as well. Docker lets you do that. It can cost a bit of money in the effort spent but payoff in the long run usually worth it - at scale.
Re:Tell HN: Server Status by throwaway5752 on Thursday January 16, 2014 @09:51AM (#7069639)
Yes, it's great if you want a lot of multi-tenant instances on one box. Why would HN benefit from that, though?
Re:Tell HN: Server Status by joeblau on Thursday January 16, 2014 @12:29PM (#7070861)
HN is about hosting a forum for intelligent people to communicate various intellectual topics, not about being a dev/ops challenge. There is no good reason that they can't run this on some services/hardware that are more stable than what they've got. People behind the YC community (VCs investing in the HN companies and many of the successful founders) are millionaires+ so the "price" argument really dons't make sense. YC should be setting the example of how to do it right.
Re:Tell HN: Server Status by uccrost on Thursday January 16, 2014 @08:20AM (#7069196)
it's definitintely not more reliable in this case...
Re:Tell HN: Server Status by gizmo on Thursday January 16, 2014 @08:18AM (#7069186)
Bespoke hardware is much cheaper, has much more consistent performance, and a beefy dedicated machine is much faster than a large AWS slice. If you can run your entire web app on a single server you have much less infrastructure to worry about, which is a big plus. All this vertical vs horizontal scaling nonsense makes your software so much more complex, and with a tiny bit of optimization it's often not needed.

Besides, HN also uses cloudflare as CDN and DDOS protection, so it's not like they're stuck in 1999.

Re:Tell HN: Server Status by nullspace on Thursday January 16, 2014 @08:30AM (#7069238)
If you can plan your hardware capacity, which is not that hard for a simple website like HN, it becomes far more cost effective to run on real hardware, even with a large overhead.

This is quite hard for startups to do, because their core function is to keep adding and removing features, experimenting and scaling, and the initial cost of buying hardware to support these functions is too large. This plus all the services provided by AWS et al. saves a lot of time and effort.

Nothing to do with "cloud" vs "non-cloud".

Re:Tell HN: Server Status by acomjean on Thursday January 16, 2014 @08:21AM (#7069197)
Periodic HN downtime helps software businesses be more productive.

Repeated Disk corruption maybe indicates a hardware problem unless the file system isn't up to snuff. Generally those file systems are pretty reliable.

Re:Tell HN: Server Status by jrmenon on Thursday January 16, 2014 @09:56AM (#7069662)
FWIW, our start-up began on AWS during early days, but finally due to cost constraints, migrated most of our servers to our own h/w setup. We had substantial number of servers of varying types on AWS (live servers, full hadoop cluseters etc..), on east and west regions, lots of EBS storage etc.., so not some trivial setup.

We really liked AWS, and the support we got for most of the times was good, but as we had 24x7 live traffic, even with reserved instances, the cost finally caught up to justify the move to our own hosting solution.

One thing that the made the migration easier for us was that from day 1, we decided to treat AWS as a co-location, hence we set things up with the usual open-source s/w stack (i.e. avoided proprietary Amazon solutions like dynamo, messaging solutions etc.. maybe just used S3 for offline archiving of logs. It was tempting to build on their components, but ended up building it ourselves.), including own own hadoop setup. When the time came, we could easily migrate things out.

Hope this gives some perspective. Still a fan of AWS, and if I were do another start-up, would follow the same script all over again.

Re:Tell HN: Server Status by TelmoMenezes on Thursday January 16, 2014 @08:14AM (#7069171)
Maybe because it's fun? What's the point of getting rich if not to do what you really want?

Also, I suspect that if a startup came to them with a loyal following like HN enjoys, they wouldn't be laughed out of the room at all...

Re:Tell HN: Server Status by trout on Thursday January 16, 2014 @09:04AM (#7069399)
If HN was on AWS, where would we go to discuss AWS outages?
Re:Tell HN: Server Status by pconf on Sunday January 19, 2014 @02:32PM (#7085605)
There are downside to AWS other than reliability (a la Netflix). One of those is privacy. If you host with AWS Amazon has access to all of your user data, and with Amazon likely the NSA. If you want to maintain a free and open forum for people to express support for people like Snowden and Assange and question the NSA or military / industrial complex' percent of the federal budget (>50) then you have to run your own servers in a skilled, ethical, small to mid-sized ISP's facility.

This doesn't explain the penny-wise (IMO) allocation of hardware or dev/sysadmin resources of course.

Re:Tell HN: Server Status by vacri on Thursday January 16, 2014 @05:50PM (#7073057)
Just don't put it in AWS US East, and you can talk about AWS outages just fine.
Re:Tell HN: Server Status by logicallee on Thursday January 16, 2014 @08:22AM (#7069205)
if a startup came to YC with on the order of 10k uniques within minutes of posting a top link, and the kind of user accounts and engagement that HN has, they wouldn't care if it was written in Perl 2.0 running on a repurposed old 747's partial fly-by-wire computer. Really, Nobody Cares.

Dropbox was an rsync executed every two minutes.[1] AngelList was done via email.

[1] can't find this reference atm, could be apocryphal

Re:Tell HN: Server Status by oneeyedpigeon on Thursday January 16, 2014 @08:10AM (#7069145)
HN isn't a mission-critical business. It's a fun, experimental distraction (albeit one that we all rate very highly, of course).
Re:Tell HN: Server Status by lazyjones on Thursday January 16, 2014 @08:15AM (#7069174)
> It's a fun, experimental distraction

It seems to incur maintenance costs and reputation hits that are avoidable though, isn't that reason enough to invest some time in building a reliable solution?

Re:Tell HN: Server Status by user24 on Thursday January 16, 2014 @08:34AM (#7069251)
No-one's going to say "Yeah I was going to apply to YC but HN keeps going down so I don't think I will after all".
Re:Tell HN: Server Status by lmm on Thursday January 16, 2014 @08:38AM (#7069273)
Speak for yourself
Re:Tell HN: Server Status by randallsquared on Thursday January 16, 2014 @09:21AM (#7069492)
Could you elaborate on why that's a factor at all? Given that YC's core business has nothing to do with HN, I wonder if the idiosyncrasies of HN aren't actually contributing something like intermittent reinforcement training to community members.
Re:Tell HN: Server Status by lmm on Thursday January 16, 2014 @09:31AM (#7069554)
Why's it a factor at all? Because I wouldn't want to work with an organization that half-asses things. I don't understand the second half of your comment.
Re:Tell HN: Server Status by sp332 on Thursday January 16, 2014 @09:43AM (#7069605)
"Positive reinforcement" would be a bar which gives some food to a rat who pushes it three times. If a rat is hungry, he pushes the bar enough times to get as much food as he needs. "Intermittent reinforcement" is a bar that gives some food after a random number of presses each time. A rat will push the bar frantically until he's exhausted, even if he's not very hungry. The technique works pretty well on humans too. https://en.wikipedia.org/wiki/Reinforcement#Intermittent_rei...
Re:Tell HN: Server Status by pitchups on Thursday January 16, 2014 @08:21AM (#7069199)
And also because that is the hallmark of a true hacker - roll your own - rather than rely on a ready made solution!
Re:Tell HN: Server Status by huhtenberg on Thursday January 16, 2014 @08:13AM (#7069157)
Kids these days. Next thing you say you can't whistle at 2400 baud. Sheesh.
Re:Tell HN: Server Status by wlesieutre on Thursday January 16, 2014 @08:33AM (#7069249)
2400 Hz, if I'm not mistaken. Baud indicates how many signals you can send in a second. Hz is the wave frequency, which you have to mimic for the signaling tones.
Re:Tell HN: Server Status by gvb on Thursday January 16, 2014 @08:51AM (#7069332)
No, 2400 baud - phase shifting is used encode the bits, so the actual frequencies used are complex. 300 baud used two tones for Tx and two tones for Rx so it is easy to whistle (albeit the characters coming out are garbage). Higher baud rates are probably impossible to whistle because the encoding is much more complex. I never tried, acoustic couplers didn't work above 300 baud. ;-)

Ref: http://en.wikipedia.org/wiki/Modem#1200_and_2400_bps

Re:Tell HN: Server Status by yebyen on Thursday January 16, 2014 @09:17AM (#7069468)
I am pretty sure he meant 2600Hz anyway

Besides everyone knows, nobody whistles 2600Hz, they just get the toy out of the Cap'n Crunch box to do it for them.

http://en.wikipedia.org/wiki/John_Draper#Phreaking

Re:Tell HN: Server Status by sp332 on Thursday January 16, 2014 @09:38AM (#7069582)
Or rehabilitate an old payphone with an in-house PBX, and then phreak it with a modded Radio Shack tone dialer. http://www.citizenengineer.com/
Re:Tell HN: Server Status by huhtenberg on Thursday January 16, 2014 @10:58AM (#7070054)
Well, no. It was a joke, so it was 2600 baud, not Hz.
Re:Tell HN: Server Status by yebyen on Friday January 17, 2014 @09:13AM (#7075813)
Yes, there were a chain of people who didn't get the joke, I was just pointing out that 2400Hz wasn't a thing either, though now I see it was a thing, maybe still is, thanks Wikipedia[]

: http://en.wikipedia.org/wiki/Blue_box

Re:Tell HN: Server Status by IgorPartola on Thursday January 16, 2014 @10:11AM (#7069734)
Because it is actually 2014 :)

Kidding aside, running a bare metal server you own or rent is always cheaper, assuming you cannot save money by turning things on and off and know you capacity needs. Sure HN grows but not nearly as fast as FB, Twitter, etc. the service is big enough where it would require expensive virtual servers. This is the best case scenario for a physical hardware box run by people who are familiar with such things.

Re:Tell HN: Server Status by rdl on Thursday January 16, 2014 @08:37AM (#7069268)
hn is mostly a single process application, so dedicated hardware is really the only way they can scale performance (and adding stuff in front like a proxy, cloudflare, etc.)

They even went out of their way to get the fastest single core xeon possible, which is a fairly mid-range CPU, singe they care about process performance vs. overall.

Re:Tell HN: Server Status by KaiserPro on Thursday January 16, 2014 @08:44AM (#7069302)
because it's cheap and works my dear fellow.

Yes it may seem archaic, but it a real server has real disk IO, something which on amazon and the like doesn't come cheap

Re:Tell HN: Server Status by raimue on Thursday January 16, 2014 @08:09AM (#7069140)
It's 2014 already!
Re:Tell HN: Server Status by corresation on Thursday January 16, 2014 @10:56AM (#7070046)
If a startup came to you with this sort of legacy thinking you'd laugh them out of the room.

In many circles, you would be laughed out of the room for cargo culting like this, and for trying to draw a bias against a legitimate deployment choice by declaring it "legacy".

There are many scenarios where hosting on your own hardware is a superior option for a variety of reasons: Financially, security, performance, flexibility.

In the case of HN it seems like it's on some pretty meager hardware, making compromises like software RAID. If this were a critical system for YC, they would have it on redundant machines with redundant, flash-based, hardware-RAID equipped platforms, clustered with redundant 10Gb cross connects, etc. Criticizing a deployment strategy because of the peculiar issues they have faced is like writing off AWS because someone's unbacked up small instance got killed and they had no strategy for it.