News for hackers. Stuff that matters.
Re:Ask HN: Is S3 down? by alexleclair on Tuesday February 28, 2017 @12:48PM (#13755710)
Yup, same here. It has been a few minutes already. Wanna bet the green checkmark[1] will stay green until the incident is resolved?

[1] https://status.aws.amazon.com/

Re:Ask HN: Is S3 down? by nostromo on Tuesday February 28, 2017 @01:30PM (#13756307)
The red check mark is hosted on S3...
Re:Ask HN: Is S3 down? by ak2196 on Tuesday February 28, 2017 @02:20PM (#13756895)
Re:Ask HN: Is S3 down? by crowbahr on Tuesday February 28, 2017 @04:43PM (#13758250)
"Care to share the code as an anti pattern?" brilliant.
Re:Ask HN: Is S3 down? by AtheistOfFail on Tuesday February 28, 2017 @01:34PM (#13756344)
Comment of the year.
Re:Ask HN: Is S3 down? by Kiro on Tuesday February 28, 2017 @03:11PM (#13757398)
Fact of the year.
Re:Ask HN: Is S3 down? by emrekzd on Tuesday February 28, 2017 @01:54PM (#13756571)
In December 2015 I received an e-mail with the following subject line from AWS, around 4 am in the morning:

"Amazon EC2 Instance scheduled for retirement"

When I checked the logs it was clear the hardware failed 30 mins before they scheduled it for retirement. EC2 and root device data was gone. The e-mail also said "you may have already lost data".

So I know that Amazon schedules servers for retirement after they already failed, green check doesn't surprise me.

Re:Ask HN: Is S3 down? by smoodles on Tuesday February 28, 2017 @02:43PM (#13757132)
So just as a FYI the reason that probably happened to you is that the underlying host was failing. I am assuming they wanted to give you a window to deal with it but the host croaked before then. I've been dealing w/ AWS for a long long time and I've never seen a maintenance event go early unless the physical hardware actually died...
Re:Ask HN: Is S3 down? by amaks on Tuesday February 28, 2017 @04:48PM (#13758318)
That what happens when cloud provider doesn't support live migration for VMs.
Re:Ask HN: Is S3 down? by problems on Tuesday February 28, 2017 @03:25PM (#13757502)
That's completely ridiculous, get some fucking RAID Amazon.

I order drives off newegg directly to my DC and I'm yet to lose data with the cheapest drives available in RAID10.

Re:Ask HN: Is S3 down? by prdonahue on Tuesday February 28, 2017 @03:56PM (#13757727)
Yes, solving problems at your scale and AWS' are quite comparable.
Re:Ask HN: Is S3 down? by LoSboccacc on Tuesday February 28, 2017 @04:36PM (#13758169)
but I never lost data off an usb stick how hard could it be!
Re:Ask HN: Is S3 down? by aurelianito on Tuesday February 28, 2017 @08:14PM (#13760046)
Really?!?! Several times USB sticks (and USB HDs) failed on me and other people I work with.
Re:Ask HN: Is S3 down? by problems on Tuesday February 28, 2017 @03:58PM (#13757751)
Not saying my scale is the the same at all - but the fact they can't do something so simple that I can do it as a single individual is embarrassing at best.

Simple solutions to this do scale - Linode and DigitalOcean don't have such issues for example - and while they're not Amazon scale, they are quite large and I'd say they prove the concept.

Re:Ask HN: Is S3 down? by phonon on Tuesday February 28, 2017 @04:07PM (#13757828)
EBS data is backed up in multiple redundant ways (using erasure encoding I think).

Local storage is not intended for permanent storage, and is more use at your own risk. That's also why most of the new EC2 instances don't even support local storage.

Availability =/= durability of course

Re:Ask HN: Is S3 down? by problems on Tuesday February 28, 2017 @04:12PM (#13757881)
EBS is incredibly expensive and slow, not really a good solution. It'd be nice if they offered a better local storage option.
Re:Ask HN: Is S3 down? by phonon on Tuesday February 28, 2017 @04:45PM (#13758274)
Incredibly expensive and slow compared to what? A 500 GB SSD (gp2) costs $50/m, and has 1500 - 3000 IOPS. It's okay for most loads.

For higher performance, you can use

1. EBS Provisioned IOPS (kind of expensive)

2. Aurora (for DB use)

3. The new I3 instances (super fast local storage at a reasonable price.)

Re:Ask HN: Is S3 down? by tomc1985 on Tuesday February 28, 2017 @05:01PM (#13758506)
That's the cost of a new 500GB SSD per month! For the cost of three months' EBS storage and a couple of hours you could setup your own RAID array with a spare for backups and possibly get better uptimes than Amazon :P
Re:Ask HN: Is S3 down? by problems on Tuesday February 28, 2017 @05:36PM (#13758881)
Not to mention you'll get 3-5x better IOPS off a dedicated SSD.

I guess this just boils down again to Amazon not being cost effective enough for my use case in yet another way.

Re:Ask HN: Is S3 down? by phonon on Tuesday February 28, 2017 @08:09PM (#13760026)
Actually, you can get way more than 3-5x better IOPS on your own SSD! Different types of storage have different types of trade-offs. EBS is great for some things, slow and overpriced for others...
Re:Ask HN: Is S3 down? by phonon on Tuesday February 28, 2017 @08:05PM (#13759995)
Huh? What kind of 500 GB SSD costs $50? And again--500 GB on EBS is not stored on 500 GB of flash...they use erasure coding and distribute it over ~3x as much, roughly.

Oh, and good luck creating snapshots of your home RAID!

Re:Ask HN: Is S3 down? by problems on Tuesday February 28, 2017 @08:09PM (#13760021)
> Huh? What kind of 500 GB SSD costs $50?

Definitely not $50 to my knowledge but for ~$170 you can get a Samsung 850 EVO which is rated for 98k IOPS. They're fairly reliable drives and much, much faster than anything you'll get on EBS. You could be running that full 3x replication in less than a year of paying for EBS.

> Oh, and good luck creating snapshots of your home RAID!

LVM, ZFS and Btrfs all do snapshotting quite nicely. FreeNAS - commonly used for consumer grade NASes will automatically manage ZFS snapshots for you too. Amazon will sell you extra space to store snapshots, sure, but increasing the size of your devices usually solves that problem. And quite cost effectively as you can probably tell by now...

Re:Ask HN: Is S3 down? by phonon on Tuesday February 28, 2017 @10:08PM (#13760646)
...and Dropbox can be replaced with SFTP and rsync. Can you roll your own? Sure. Will it work as reliably and effectively as EBS? Can I scale it easily? How many people are comfortable using snapshots on their own?
Re:Ask HN: Is S3 down? by problems on Tuesday February 28, 2017 @10:13PM (#13760683)
It's tried and true tech that any competent ops person can use quite easily. Been around much longer than EBS quite frankly.

Dropbox targets end users who don't have the knowledge required to use the alternative, if you're smart enough to use EBS you're probably smart enough to use ZFS snapshotting just as easily. Or could within a day or two. It's really not that hard.

Like I said, there are systems that pretty much manage the whole thing for you and just warn you when something is about to blow up like FreeNAS.

Re:Ask HN: Is S3 down? by tomc1985 on Wednesday March 01, 2017 @02:33AM (#13762022)
It could be replaced with Nextcloud

Shadow volume replication is entirely possible with several filesystems or Hot Copy kernel mod. Also LVM does snapshotting fairly easily

I may have got my prices a bit mixed up (I saw 120GB at Fry's for $60 last month) but my point stands.

Also why is discomfort such a big problem for folks? Learn stuff.

Re:Ask HN: Is S3 down? by Sanddancer on Tuesday February 28, 2017 @08:42PM (#13760186)
Not really that hard.

zfs snap tank/data@$(date '+Y%m%d')

zfs send tank/data@$(date '+Y%m%d') | zfs recv backup/data

advanced magic for off-system backup

zfs send tank/data@$(date '+Y%m%d') | ssh cheapdiskserver zfs recv tank/data

Re:Ask HN: Is S3 down? by Dylan16807 on Tuesday February 28, 2017 @11:21PM (#13761081)
> 1500 - 3000 IOPS

So about as many as this SD card, and nothing compared to a real SSD.

Re:Ask HN: Is S3 down? by phonon on Wednesday March 01, 2017 @02:48AM (#13762085)
In practice, it's actually fine for most purposes. It's equivalent to multiple striped 15K RPM magnetic disks, which used to be high-end enterprise storage a few years ago.

SD cards have much worse write IOPS.

Re:Ask HN: Is S3 down? by Dylan16807 on Wednesday March 01, 2017 @04:45PM (#13768747)
> In practice, it's actually fine for most purposes.

It is, yes, but I wouldn't refer to it as comparable to an SSD.

> SD cards have much worse write IOPS.

Surprisingly not! Testing in ATTO I got read and write speeds that were almost identical, and a peak of 2000 IOps.

Re:Ask HN: Is S3 down? by phonon on Wednesday March 01, 2017 @07:30PM (#13770212)
>It is, yes, but I wouldn't refer to it as comparable to an SSD.

EBS (gp2) is flash based, has far better performance than high end magnetic disks, with excellent latency and consistent performance. So, it's more comparable to SSD than anything else.

>Surprisingly not! Testing in ATTO I got read and write speeds that were almost identical, and a peak of 2000 IOps.

Really? Were you looking at 4K write? Typically that would be under 1 MB/s for an SD card.

Re:Ask HN: Is S3 down? by Dylan16807 on Wednesday March 01, 2017 @08:50PM (#13770701)
https://www.amazon.com/gp/product/B013CP3MMM

http://i.imgur.com/PRoxVPM.png

It's a relatively high-quality SD card, unfortunately hampered by my reader's inability to use bus speed over 25MB/s.

Re:Ask HN: Is S3 down? by phonon on Thursday March 02, 2017 @01:04PM (#13776490)
Nice card! So about 1500 write IOPS at 4K. Performance might be worse (or better) at other queue depths though.
Re:Ask HN: Is S3 down? by foxylion on Tuesday February 28, 2017 @04:36PM (#13758152)
I think most people rely on EBS and are happy with it. Sure it depends on the use case, but I think it works for most use cases.
Re:Ask HN: Is S3 down? by foxylion on Tuesday February 28, 2017 @04:25PM (#13758010)
It's not just a RAID that can fail. And everyone who uses AWS should expect failures. You should build your infrastructure to handle such failures well.
Re:Ask HN: Is S3 down? by problems on Tuesday February 28, 2017 @04:30PM (#13758073)
They offer no RAID on local storage and only the expensive, IO restricted EBS as an alternative.
Re:Ask HN: Is S3 down? by vacri on Tuesday February 28, 2017 @04:11PM (#13757875)
Yes, the only way a server can die is from non-raided disks.
Re:Ask HN: Is S3 down? by problems on Tuesday February 28, 2017 @04:13PM (#13757891)
Otherwise they should at least be providing customers their data back.
Re:Ask HN: Is S3 down? by foxylion on Tuesday February 28, 2017 @04:37PM (#13758183)
I think you misunderstood the local storage. It is not intended to permanently store data. It's a volatile storage like RAM.
Re:Ask HN: Is S3 down? by tuna-piano on Tuesday February 28, 2017 @01:58PM (#13756632)
It's crazy how much better the communication (including updates and status pages) is of the companies that rely on AWS than AWS' communication itself.

https://status.heroku.com/incidents/1059

Re:Ask HN: Is S3 down? by tcsf on Tuesday February 28, 2017 @02:44PM (#13757148)
Blake Gentry gave a full accounting of Heroku's response process here - http://www.heavybit.com/library/video/every-minute-counts-co...

Amazon should take notes.

Re:Ask HN: Is S3 down? by all_usernames on Tuesday February 28, 2017 @04:08PM (#13757841)
I feel for them. Imagine, 40 or 50 different engineering teams all responsible for updating their statuses. At this moment on the AWS status page I see random usage of the red, yellow, and green icons, even though all the status updates are "Increased error rates." What that tells me is that there's no unified communication protocol across the teams, or they're not following it. And just imagine what it's like being on the S3 team right now.

I notice even Cloudflare is starting to have problems serving up pages now.

Re:Ask HN: Is S3 down? by SnowingXIV on Tuesday February 28, 2017 @04:58PM (#13758467)
Font Awesome went out for me for a bit, but they did a great job getting back up and keeping their users in the loop.

https://status.fortawesome.com/

Re:Ask HN: Is S3 down? by busterarm on Tuesday February 28, 2017 @02:05PM (#13756715)
Re:Ask HN: Is S3 down? by tlogan on Tuesday February 28, 2017 @01:22PM (#13756191)
These service health boards are more like advertisement page then actual status of the service.
Re:Ask HN: Is S3 down? by mwfj on Tuesday February 28, 2017 @01:27PM (#13756267)
I guess their bizarre thinking is something along the lines of: "unless we have proof that noone can access the service, we won't change the indicator from green to yellow.

Seriously: I don't understand why you guys stay with AWS.

Re:Ask HN: Is S3 down? by cagataygurturk on Tuesday February 28, 2017 @04:17PM (#13757932)
Because you perceive public clouds only as virtual machine providers, that you can replace with other provider in two days. A detailed cloud migration consists of replacing some parts of your software to use managed services provided by a specific cloud provider, and AWS is still has the best service offerings IMHO. When you use these services carefully also you will see that AWS is very cheap and reliable enough. Outages like today's are happening in every platform and it is possible to mitigate them.

You can use Adwords as a self-service user. Without knowing so much of details you can run your ads but also you can bery easily ruin your budget. But many enterprise customers use it very differently than those users and they are extremely optimizing the cost. Cloud is the same. If you don't know how big customers use AWS, it is normal that you are surprised because AWS is still leading the market.

You say GCP is better than AWS. Which part is better? GCP does not have many services of AWS we benefit from. How can you compare totally different providers? You can only say AWS EC2 is worse than GCP. But you cannot compare whole platforms in one sentence.

Re:Ask HN: Is S3 down? by johansch on Wednesday March 01, 2017 @12:19PM (#13765786)
(Sorry, I'm late to reply, but since you addended your comment you might still be listening...)

After spending a year evaluating both AWS and GCP (with an emphasis on their managed database services; both SQL and no-SQL) my general feeling is this:

"Microsoft Windows is to Unix as AWS is to GCP".

(Or perhaps closer to the truth: "VMS is to Unix as AWS is to GCP".)

Baically AWS services seem like they are badly designed by buerocratic mediocre engineers following some bureocratic template for "a service".

GCP feels a lot saner (both API- and UI/console-wise). I often got the feeling it's designed by people who:

a) are smart and well-rounded in terms of experiences. It does take cleverness and experience to design something elegant that is also useful.

b) take pride in their work (it does show)

(And then, as a bonus: It's cheaper!)

Re:Ask HN: Is S3 down? by cagataygurturk on Wednesday March 01, 2017 @12:57PM (#13766313)
You talk about SQL and No-SQL as managed services and it shows that your experience is limited to a classical application consisting of virtual machines and some data storage. However these are not the only services offered by both platforms and currently AWS has a richer feature set. For example Lambda and its deep integration with whole AWS platform is the biggest game changer from my point of view. If we are talking about virtual machines and databases, I can accept this comparison. However we are talking about 30+ services, some of them are even not available somewhere else and solving serious business problems in production and at scale. It is very wrong to put everything into basket and compare. Maybe GCP has better pub/sub service and AWS has better object storage. These should be compared seperately. Answering to your question, why do we still stay at AWS, because it is solving our problems in the most cost effective way and with reduced complexity, we are happy with it.
Re:Ask HN: Is S3 down? by johansch on Wednesday March 01, 2017 @01:12PM (#13766502)
You're probably assuming too much again :)

I specifically spent a lot of time on Lambda and found it quite annoying compared to GCP AppEngine. So much bureaucracy. Just this thing that you have to specifically register every single Lambda API call and its parameters using an interface built by non-thinking people.. Sheesh.

For on-demand processing I just want a single HTTP-ish entry point, like AppEngine provides. (That way I can I move my service between different providers, if I wanted to move away from e.g. AWS.)

Re:Ask HN: Is S3 down? by johansch on Wednesday March 01, 2017 @01:59PM (#13767013)
Anyway, I just updated my HN profile with more details about my experience. Please visit it to judge if I might know what I'm talking about.
Re:Ask HN: Is S3 down? by cagataygurturk on Wednesday March 01, 2017 @05:15AM (#13762658)
Sorry for endless number of typos and mistakes. Obviously I was sleepy while I was writing this.
Re:Ask HN: Is S3 down? by gtsteve on Tuesday February 28, 2017 @02:28PM (#13756971)
> Seriously: I don't understand why you guys stay with AWS.

Personally I've been using it for ages and I know most services inside and out. They do suffer downtime in some regions occasionally, but it'd be too expensive at this point to move.

And who doesn't suffer downtime? You can't avoid it; you just need a plan to deal with it. For example, having a backup replica bucket in another region and the ability to quickly switch your CDN over would probably be a good idea here; that's what I did.

If you want to go further you can replicate your data to another cloud provider entirely and use low TTLs to switch to a backup CDN if your system is that mission-critical (in the event of a worldwide AWS failure doomsday scenario).

All systems will fail you and it's our responsibility as IT professionals to have a plan to mitigate this.

Re:Ask HN: Is S3 down? by nicolaslem on Tuesday February 28, 2017 @04:26PM (#13758020)
Low TTL on DNS entries might do more harm than good: if your DNS provider gets seriously DDoS, being able to rely on caches can save the day.

Anyway, I agree with your conclusion.

Re:Ask HN: Is S3 down? by johansch on Tuesday February 28, 2017 @02:43PM (#13757127)
Sunk cost fallacy.

I do agree that we should all plan for failures.

However, I also think it's a sign of failure in planning and architecture foresight if it's too expensive to move away from a particular cloud provider.

Re:Ask HN: Is S3 down? by froogle on Tuesday February 28, 2017 @03:03PM (#13757318)
The sunk cost fallacy is when you (irrationally) decide to stick with what you're doing purely because you've already spent a lot of resources on it. It doesn't apply when you've done an economic analysis and found out it doesn't make sense to swap.

There are plenty of cases where it just wouldn't make sense to switch after looking at the costs, opportunity costs, etc. For example, if his site makes him $10 a month, outages cost him $1 a month that could be mitigated by moving, and it would cost $1000 of labor to swap providers. (Depends on interest rates.)

Perhaps it was originally a failure to not have a plan to easily move from a provider, but it doesn't seem unreasonable to me that right now it may cost too many hours of work to justify the move.

Re:Ask HN: Is S3 down? by gtsteve on Tuesday February 28, 2017 @04:33PM (#13758122)
It's not as though it would be impossible; our integration with AWS isn't that deep, it's not as though we use DynamoDB for our core data store or anything like that. But even migrating from one traditional datacenter to another isn't easy from an operational point of view.

There needs to be a clear financial win. Even taking into account the failures we've seen so far, I don't see a compelling reason to leave AWS.

Re:Ask HN: Is S3 down? by johansch on Tuesday February 28, 2017 @03:13PM (#13757415)
(You're right, I used that term incorrectly.)

Still stand behind the other two points I made in that post though.

Re:Ask HN: Is S3 down? by rattray on Tuesday February 28, 2017 @01:35PM (#13756355)
> I don't understand why you guys stay with AWS.

Who do you recommend instead (assuming in-house or Hetzner-equiv is out of reach)? Google Cloud? Azure? Rackspace?

Re:Ask HN: Is S3 down? by johansch on Tuesday February 28, 2017 @01:37PM (#13756370)
Google Cloud if you're looking for something similar. It's just so much better and cheaper. I think a lot of the resistance here towards that kind of move is just because people are inherently lazy and they aren't paying the bill themselves.

(I'm guessing a relatively large part is also selfish attachment to the market leader because of employment reasons. I hate wasting money, both for myself and for my employer, so I don't really understand this kind of thinking - but I do understand how it could flourish in a venture capital-rich time/locale.)

I also recommend reading:

https://thehftguy.com/2016/06/15/gce-vs-aws-in-2016-why-you-...

Re:Ask HN: Is S3 down? by debaserab2 on Tuesday February 28, 2017 @01:53PM (#13756558)
Google Cloud doesn't exactly have the greatest reliability/uptime either.
Re:Ask HN: Is S3 down? by eicnix on Tuesday February 28, 2017 @02:04PM (#13756697)
https://status.cloud.google.com/summary tells a different story or do you have other information?

I have used GCP for some time without being affected from any incident.

Re:Ask HN: Is S3 down? by debaserab2 on Tuesday February 28, 2017 @03:03PM (#13757317)
I'm not sure what you mean. If anything that link underscores my point. GCE has absolutely had it's own catastrophic errors. Remember last April when ALL instances in ALL regions went down?

https://status.cloud.google.com/incident/compute/16007?post-...

Re:Ask HN: Is S3 down? by miken123 on Tuesday February 28, 2017 @06:06PM (#13759139)
At least Google has a post-mortem, at AWS everything would still show up green with some random note about 'increased error rates'.
Re:Ask HN: Is S3 down? by boulos on Tuesday February 28, 2017 @04:16PM (#13757927)
While I agree that was a horrific outage for us, there's a big difference between no external connectivity for a few minutes (note: internal IPs still worked fine, as did access to APIs through that mechanism) and "ALL instances in ALL regions went down".

Disclosure: I work on Google Cloud (and wouldn't want to be an incident responder at AWS today...)

Re:Ask HN: Is S3 down? by eicnix on Tuesday February 28, 2017 @03:14PM (#13757418)
The GCP services are usually within their SLA target so I don't see the problem with the incidents. So you know what you buy and can take actions if you need a higher SLA for your application.
Re:Ask HN: Is S3 down? by debaserab2 on Tuesday February 28, 2017 @03:26PM (#13757508)
> so I don't see the problem with the incidents

All instances going down in all regions is an order of magnitude worse than a single service going down in a single region. You're deluding yourself if you think GCE is any more reliable than any other reputable cloud hosting platform.

Re:Ask HN: Is S3 down? by briffle on Tuesday February 28, 2017 @02:19PM (#13756876)
That is a pretty awesome page.. way better than a page full of green icons, during an obvious outage... I like that they have writeups a few days after the incidents....
Re:Ask HN: Is S3 down? by joatmon-snoo on Tuesday February 28, 2017 @01:44PM (#13756451)
Google also doesn't have the best record for developer tools.
[dead]
Re:Ask HN: Is S3 down? by ecliptik on Tuesday February 28, 2017 @02:37PM (#13757056)
too soon
Re:Ask HN: Is S3 down? by johansch on Tuesday February 28, 2017 @02:56PM (#13757250)
;)
Re:Ask HN: Is S3 down? by chebum on Tuesday February 28, 2017 @03:11PM (#13757401)
GC's CDN doesn't cache files bigger than 4Mb. No Windows VMs. Bound to AWS for these 2 reasons.
Re:Ask HN: Is S3 down? by dragonwriter on Tuesday February 28, 2017 @03:18PM (#13757454)
Re:Ask HN: Is S3 down? by mrmaximus on Tuesday February 28, 2017 @06:58PM (#13759533)
As already mentioned, they do have Windows VM's but there are some caveats that indicate it's not fully baked yet. 1.) They require that each VM MUST have a public IP address so that Windows can talk to an activation server every 30 days. 2.) You cannot yet bring your own license.
Re:Ask HN: Is S3 down? by vgt on Tuesday February 28, 2017 @03:26PM (#13757510)
Someone else already mentioned Windows VMs.

Looks like CDN has a 10MB limit:

https://cloud.google.com/cdn/docs/caching

(work at Google Cloud)

Re:Ask HN: Is S3 down? by kohuma on Tuesday February 28, 2017 @02:07PM (#13756749)
What about something like B2 from https://www.backblaze.com/ ?
Re:Ask HN: Is S3 down? by Twirrim on Wednesday March 01, 2017 @11:57AM (#13765506)
S3 in a single region is based out of multiple data centres / availability zone, with data distributed so that the loss of a single availability zone won't impact either data availability or durability, even to the point of being comfortable with complete physical destruction of an AZ. The same applies for Azure, GCP etc.

B2 is based out of a single DC (or at least, was at launch and I don't see anything that suggests that has changed?) You've got to decide what's most important to you. Data persistence or $$$.

Re:Ask HN: Is S3 down? by pmalynin on Tuesday February 28, 2017 @01:40PM (#13756400)
OVH
Re:Ask HN: Is S3 down? by LeoHaggins on Tuesday February 28, 2017 @01:57PM (#13756606)
Bad idea there, support is horrible.
Re:Ask HN: Is S3 down? by rspeer on Tuesday February 28, 2017 @02:14PM (#13756835)
OVH doesn't even want to take my money to keep my server running. Their auto-billing process is busted and when it goes wrong they just delete your server.
Re:Ask HN: Is S3 down? by fishnchips on Tuesday February 28, 2017 @02:29PM (#13756975)
That's not what I've seen. I misconfigured my auto-billing and got paged in the middle of the night by nodes mysteriously disappearing, but they released those machines minutes after my CC went through. Not that I'm a big fan of OVH but if you design your system to allow for failure you can't match their value for money.
Re:Ask HN: Is S3 down? by jacquesm on Tuesday February 28, 2017 @02:08PM (#13756759)
What is your last datapoint on that?

The last year or two has seen a remarkable improvement according to those customers of mine that host there.

Re:Ask HN: Is S3 down? by derefr on Tuesday February 28, 2017 @05:12PM (#13758633)
I think it's more, "if the service can't do what people need it to do, that's a problem; if the service cluster gets wedged hard enough to stop responding to the requests of our monitoring system, that's a failure."

Which would make sense (and is sorta-kinda a best-practice) if Amazon wrote services such that they "crashed early"—but instead they're seemingly written so the backend lock up and be rendered completely useless at "doing its job" but will continue to run just fine.

Either of those two design decisions is potentially a good thing on its own, but they need to be considered in light of one-another if you want your status page to make any sense. If you want to report cluster failures, code your clusters to actually fail. If you want to keep your clusters up, write your monitoring checks as whole-stack acceptance tests.

Re:Ask HN: Is S3 down? by notyourwork on Tuesday February 28, 2017 @09:35PM (#13760463)
> Seriously: I don't understand why you guys stay with AWS.

You don't seem to have enough experience to comment on the issue.

Re:Ask HN: Is S3 down? by johansch on Wednesday March 01, 2017 @03:15PM (#13767860)
Please visit this comment sub-tree:

https://news.ycombinator.com/item?id=13765786

Re:Ask HN: Is S3 down? by notyourwork on Friday March 03, 2017 @02:21PM (#13785437)
That is a regurgitation of your opinion without any facts.

Comparing technology and saying "it seems" or "i feel" isn't really a good argument to convince me one way or the other.

Re:Ask HN: Is S3 down? by tlogan on Tuesday February 28, 2017 @02:09PM (#13756766)
> Seriously: I don't understand why you guys stay with AWS.

I tried them all and Amazon is still the best.

Re:Ask HN: Is S3 down? by cgag on Tuesday February 28, 2017 @04:30PM (#13758078)
Postgres on RDS
Re:Ask HN: Is S3 down? by boulos on Tuesday February 28, 2017 @04:41PM (#13758231)
Come to NEXT in a week! :).
Re:Ask HN: Is S3 down? by espeed on Tuesday February 28, 2017 @08:46PM (#13760205)
Any chance UDF iterators for Cloud Bigtable are in the works?

Being able to run distributed D4M/GraphBLAS queries in Cloud Bigtable would be killer.

"From NoSQL Accumulo to NewSQL Graphulo: Design and Utility of Graph Algorithms inside a BigTable Database" https://arxiv.org/pdf/1606.07085.pdf

Re:Ask HN: Is S3 down? by hartleybrody on Tuesday February 28, 2017 @01:03PM (#13755938)
I'm seeing green checkmarks across the board, but they just added a notice to the top of the page:

> Increased Error Rates

> We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

Re:Ask HN: Is S3 down? by syntheticcdo on Tuesday February 28, 2017 @01:07PM (#13755994)
I guess sub-1% to 100% failure rate is technically an "increase".
Re:Ask HN: Is S3 down? by Artemis2 on Tuesday February 28, 2017 @01:19PM (#13756149)
I guess file uploads and downloads are technically "API calls".
Re:Ask HN: Is S3 down? by samstave on Tuesday February 28, 2017 @01:48PM (#13756498)
the worst thing is when your system cant handle these "increased error rates" as your control plane cascades failure due to something like this....

The worst "increased error rate" problem I had was when the API was failing and my autoscale system couldnt deal and launched thousands of instances because it couldnt tell when instances were launched (lack of API access) and the instances pummelled the fuck out of all other parts of the system and we basically had to reboot the entire platform....

Luckily, amazon is REALLY forgiving with respect to costs in these (and actually most) circumstance....

Re:Ask HN: Is S3 down? by salvor on Tuesday February 28, 2017 @04:39PM (#13758202)
recalls numerous times

Yes. Yes they are. Thankfully.

Re:Ask HN: Is S3 down? by matwood on Tuesday February 28, 2017 @01:23PM (#13756210)
I always joke that if one of those statuses ever went to red, it means the zombie apocalypse has begun.
Re:Ask HN: Is S3 down? by paulddraper on Tuesday February 28, 2017 @01:42PM (#13756428)
The number of non-green marks is the number of ICBMs currently in flight towards an AWS data center.
Re:Ask HN: Is S3 down? by cperciva on Tuesday February 28, 2017 @01:35PM (#13756353)
The good news is, if Amazon's services are marked as offline, you're allowed to use Amazon Lumberyard to control nuclear power plants.
Re:Ask HN: Is S3 down? by chrisan on Tuesday February 28, 2017 @01:48PM (#13756509)
In case anyone wants to see what mysterious the red icon looks like: https://status.aws.amazon.com/images/status3.gif

At best when there are problems (not like now I guess) I will see the "note" green icon https://status.aws.amazon.com/images/status1.gif

Re:Ask HN: Is S3 down? by krylon on Tuesday February 28, 2017 @02:27PM (#13756954)
I've heard (on the Fnord new show on the most recent CCC congress, so take it with a grain of salt and a bucket of humor) that Amazon's TOS are more or less void when a Zombie Apocalypse breaks out.

They had some convoluted but fairly specific wording in their TOS, whoever wrote must have had a lot of fun.

Re:Ask HN: Is S3 down? by jfim on Tuesday February 28, 2017 @06:37PM (#13759387)
From https://aws.amazon.com/service-terms/

> 57.10 Acceptable Use; Safety-Critical Systems. Your use of the Lumberyard Materials must comply with the AWS Acceptable Use Policy. The Lumberyard Materials are not intended for use with life-critical or safety-critical systems, such as use in operation of medical equipment, automated transportation systems, autonomous vehicles, aircraft or air traffic control, nuclear facilities, manned spacecraft, or military use in connection with live combat. However, this restriction will not apply in the event of the occurrence (certified by the United States Centers for Disease Control or successor body) of a widespread viral infection transmitted via bites or contact with bodily fluids that causes human corpses to reanimate and seek to consume living human flesh, blood, brain or nerve tissue and is likely to result in the fall of organized civilization.

Re:Ask HN: Is S3 down? by grogenaut on Tuesday February 28, 2017 @09:19PM (#13760378)
First the fall of human civilization has to be a real threat per the TOS so not sure they'll care.

Second, I know the lawyer and yes he had fun.

Re:Ask HN: Is S3 down? by obsurveyor on Tuesday February 28, 2017 @02:44PM (#13757136)
Then I guess it has begun, the page is now showing red. I'd put a picture on imgur but it's not loading.
Re:Ask HN: Is S3 down? by jonstaab on Tuesday February 28, 2017 @12:58PM (#13755860)
http://downdetector.com/status/aws-amazon-web-services looks like a reasonable alternative place to check/report downtime.
Re:Ask HN: Is S3 down? by zedpm on Tuesday February 28, 2017 @01:02PM (#13755925)
I just check Twitter, since Amazon's status is always a lie. My personal dashboard is still showing no problems. It's bad enough that the main public status is always green even when there's clearly a problem, but you'd think they could at least make the private status accurate.
Re:Ask HN: Is S3 down? by eicnix on Tuesday February 28, 2017 @12:59PM (#13755881)
Which is coincidently down.
Re:Ask HN: Is S3 down? by talawahdotnet on Tuesday February 28, 2017 @01:04PM (#13755955)
Maybe they are hosted on S3 facepalm or maybe they just got a surge in traffic
Re:Ask HN: Is S3 down? by booleanbetrayal on Tuesday February 28, 2017 @01:03PM (#13755939)
downdetectordown.com ?
Re:Ask HN: Is S3 down? by pure_ambition on Tuesday February 28, 2017 @01:05PM (#13755968)
yep, page won't load.
Re:Ask HN: Is S3 down? by jonstaab on Tuesday February 28, 2017 @01:03PM (#13755944)
Gah. It was up 3 minutes ago. Anyone have any suspicion this is another ddos episode? I saw that SO was down last night too: https://twitter.com/StackStatus/status/836450836322516992
Re:Ask HN: Is S3 down? by cperciva on Tuesday February 28, 2017 @01:09PM (#13756024)
Pretty confident that isn't it. S3 was returning InternalErrors for 22 seconds before it started timing out and/or returning 503s to all my requests.

I'd bet that something broke (causing InternalError responses) and then nodes started marking themselves as failed (causing the timeouts and 503s soon after).

Re:Ask HN: Is S3 down? by chx on Tuesday February 28, 2017 @01:52PM (#13756545)
I want to see the botnet capable DDoSing S3. That would be something.
Re:Ask HN: Is S3 down? by vjdhama on Tuesday February 28, 2017 @01:57PM (#13756623)
Apparently, that's down too. Sigh.
Re:Ask HN: Is S3 down? by Fiahil on Tuesday February 28, 2017 @01:57PM (#13756615)
So, global S3 outage for more than an hour now. Still green, still talking about "US East issue". I'm amazed.
Re:Ask HN: Is S3 down? by gtsteve on Tuesday February 28, 2017 @02:19PM (#13756879)
It doesn't appear to be global; my app in eu-west-1 appears unaffected.

It's possible that the console won't work however as I believe that's served from us-east-1.

Re:Ask HN: Is S3 down? by chebum on Tuesday February 28, 2017 @03:13PM (#13757416)
My site hosted on S3 is also running.
Re:Ask HN: Is S3 down? by gordon_freeman on Tuesday February 28, 2017 @05:03PM (#13758529)
Looks like they have fixed the issue with their health dashboard now.

From https://status.aws.amazon.com/ : Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

Re:Ask HN: Is S3 down? by talawahdotnet on Tuesday February 28, 2017 @01:01PM (#13755906)
There was an alert on the personal health dashboard[1] a second ago, it said S3 Operational issue in us-east-1 but when I tried to view the details it showed an error.

Then I refreshed and the event disappeared altogether.

[1] https://phd.aws.amazon.com/phd/home?region=us-east-1#/dashbo...

Re:Ask HN: Is S3 down? by socialentp on Tuesday February 28, 2017 @01:23PM (#13756206)
Same here. But it is in the general status dashboard: http://status.aws.amazon.com/
Re:Ask HN: Is S3 down? by tuna-piano on Tuesday February 28, 2017 @12:55PM (#13755818)
Still green now, 8 minutes in.
Re:Ask HN: Is S3 down? by bpicolo on Tuesday February 28, 2017 @12:56PM (#13755830)
I've had a few non-Amazon providers tell me AWS things are not working in the last 5 minutes, no note from Amazon though.

Nice.

Re:Ask HN: Is S3 down? by leesalminen on Tuesday February 28, 2017 @01:00PM (#13755886)
Just sent out a notice to our customers via our status page. I really wanted to be able to add a link back to AWS detailing the issue but that's a pipe dream I suppose.
Re:Ask HN: Is S3 down? by fudged71 on Tuesday February 28, 2017 @01:55PM (#13756583)
... still green
Re:Ask HN: Is S3 down? by clamprecht on Tuesday February 28, 2017 @01:25PM (#13756247)
Re:Ask HN: Is S3 down? by adrenalinelol on Tuesday February 28, 2017 @01:28PM (#13756281)
Looks like his personal site isn't loading... :)
Re:Ask HN: Is S3 down? by johansch on Tuesday February 28, 2017 @02:53PM (#13757228)
Yup, it is indeed hosted with Amazon.
Re:Ask HN: Is S3 down? by ceejayoz on Tuesday February 28, 2017 @01:40PM (#13756412)
Re:Ask HN: Is S3 down? by ak2196 on Tuesday February 28, 2017 @01:54PM (#13756573)
We have a slack emoji for it called greenish. It's the classic AWS green checkmark with an info icon in the bottom. Apparently it's NOT an outage if you don't acknowledge it. It's called alt-uptime.
Re:Ask HN: Is S3 down? by foxylion on Tuesday February 28, 2017 @04:44PM (#13758259)
I really liked it. But when trying to add it to my HipChat group it failed to upload. Why? S3 outage, what an irony.
Re:Ask HN: Is S3 down? by nhumrich on Tuesday February 28, 2017 @07:14PM (#13759650)
AWS internal lingo calls this the "green-i"
Re:Ask HN: Is S3 down? by cheeze on Tuesday February 28, 2017 @12:57PM (#13755845)
Just went yellow

Edit: nevermind

Re:Ask HN: Is S3 down? by schneidmaster on Tuesday February 28, 2017 @01:00PM (#13755892)
Did it? Still fields of green for me.
Re:Ask HN: Is S3 down? by malloci on Tuesday February 28, 2017 @01:06PM (#13755986)
While keeping the status green for s3, they have at least put up a notice at the top:

Increased Error Rates

We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

Re:Ask HN: Is S3 down? by schneidmaster on Tuesday February 28, 2017 @01:09PM (#13756029)
Yeah I just now saw that. Probably regional cache clearing or something.
Re:Ask HN: Is S3 down? by matthuggins on Tuesday February 28, 2017 @01:00PM (#13755883)
Still green for me
Re:Ask HN: Is S3 down? by leesalminen on Tuesday February 28, 2017 @01:03PM (#13755947)
Just went yellow

Increased Error Rates

We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

https://status.aws.amazon.com/

Re:Ask HN: Is S3 down? by nkozyra on Tuesday February 28, 2017 @01:09PM (#13756021)
Check individual services ...

Amazon Simple Storage Service (US Standard) Service is operating normally