News for hackers. Stuff that matters.
Re:Prime Down: Amazon’s sale day turns into fail day by throwaway289355 on Monday July 16, 2018 @06:35PM (#17545760)
AWS employee and bar raiser here.

>Something no one has mentioned yet, could it be that the engineering force at Amazon is no longer what it used to be?

In many regards, yes. The bar had to be lowered to meet the demands of growth. We've also taken in a lot of hires from companies that have brought their culture and friends with them. The culture at Amazon is not what is was even 2 years ago. It is in many places day 2.

No one also seems to notice that Amazon retail often suffers widespread issues like this. We can count on SEV1's happening during peak as things blow up badly. This has happened several years in a row, and sadly the themes are pretty much the same across all: forgot to scale (yes...really) or some stupid system bottleneck. It doesn't help that Amazon retail has a good amount of its workforce based in India and seemingly disconnected from the Seattle based leadership.

Re:Prime Down: Amazon’s sale day turns into fail day by ergothus on Monday July 16, 2018 @07:12PM (#17545969)
> It is in many places day 2.

For those unfamiliar with internal Amazon lingo, a big deal is made of it always being "day 1"

https://www.fool.com/investing/2017/04/13/jeff-bezos-says-it...

Re:Prime Down: Amazon’s sale day turns into fail day by dabockster on Monday July 16, 2018 @07:12PM (#17545968)
> The bar had to be lowered to meet the demands of growth.

Not only that, but I've noticed that Amazon has started using way more contractors (not actual firms, but more Mechanical Turk/UpWork like contractors, if not just straight up misclassification) and agencies within the past year or so. I can't say that I'm surprised that Amazon is having technical issues now that they've grew in numbers but not in technical chops.

On a side note, it is a great time to become a recruiter in Seattle with all these agencies popping up. /s

Re:Prime Down: Amazon’s sale day turns into fail day by smt88 on Tuesday July 17, 2018 @12:12AM (#17547248)
This makes me terrified that my users' security relies on AWS...
Re:Prime Down: Amazon’s sale day turns into fail day by abrookewood on Tuesday July 17, 2018 @01:53AM (#17547586)
Honestly don't think that is an issue. AWS is THE most certified company I have ever worked with. Seriously, go look at their compliance page - it is ridiculous. https://aws.amazon.com/compliance/programs/
Re:Prime Down: Amazon’s sale day turns into fail day by usr1106 on Tuesday July 17, 2018 @02:25AM (#17547714)
Certification is a theater for managers and so called decision makers. It might have a positive impact on code quality as long as people are motivated. But if insiders are telling me that the workplace culture is deteriorating that is a serious problem. Certification will not prevent bad code hitting production.
Re:Prime Down: Amazon’s sale day turns into fail day by reeteshv on Tuesday July 17, 2018 @06:47AM (#17548685)
Reminds me of several CMM Level 5 certified IT services firms consistently grounding projects into dust!
Re:Prime Down: Amazon’s sale day turns into fail day by abrookewood on Tuesday July 17, 2018 @11:49PM (#17555455)
I was mainly referring to the security of their systems, but I take your point.
Re:Prime Down: Amazon’s sale day turns into fail day by yayadarsh on Wednesday July 18, 2018 @12:57AM (#17555664)
Not certain on this, but I think Azure is known to have more compliance certifications. This is probably largely due to the long relationships Microsoft has with government organizations etc.
Re:Prime Down: Amazon’s sale day turns into fail day by duxup on Tuesday July 17, 2018 @12:02AM (#17547212)
I got the impression from some former Amazon employees that they were burnt out and put their time in and got Amazon on their resume and were just done with life at Amazon so they moved on.

Do you sense any attrition like that?

Re:Prime Down: Amazon’s sale day turns into fail day by weka on Monday July 16, 2018 @08:11PM (#17546261)
> It doesn't help that Amazon retail has a good amount of its workforce based in India and seemingly disconnected from the Seattle based leadership.

Could you expand on this a bit, please?

Re:Prime Down: Amazon’s sale day turns into fail day by codeonfire on Monday July 16, 2018 @10:49PM (#17546916)
An Amazon manager, full of shit, once tried to tell a joke at a social gathering. He claimed that he and friends removed the engine from someone's car over lunch as a prank. This is the education level that some of them have. He literally didn't think people would know he was full of shit.
Re:Prime Down: Amazon’s sale day turns into fail day by toomanybeersies on Tuesday July 17, 2018 @12:28AM (#17547323)
It would be technically possible to remove a car engine during lunch. However, that's not so much a prank as malicious damage. You'd also have to be a very skilled mechanic, familiar with that specific car, to do it that fast.
Re:Prime Down: Amazon’s sale day turns into fail day by joncrane on Tuesday July 17, 2018 @07:45AM (#17548980)
Not to mention you'd drop a lot of polluting fluids on the ground if you did it in a parking lot.
Re:Prime Down: Amazon’s sale day turns into fail day by exikyut on Tuesday July 17, 2018 @01:04AM (#17547441)
Honest question: education level or mental developmental level?

Mental health is enough of an epidemic that $company_with_exponential_scale is most definitely going to pick up a fair chunk of people with various issues, cognitive included.

Chances are this is even part of why the bar is a bit lower.

(Said as someone with mild high-functioning autism)

Re:Prime Down: Amazon’s sale day turns into fail day by codeonfire on Tuesday July 17, 2018 @03:10AM (#17547904)
My point is Amazon has hired some managers that have really lowered the bar due to the environment they are from and people they are used to dealing with. Imagine someone from a place where you could tell a preposterous lie and 50% of people would believe you.
Re:Prime Down: Amazon’s sale day turns into fail day by hueving on Monday July 16, 2018 @11:22PM (#17547057)
Gullibility has little to do with 'education level'.
Re:Prime Down: Amazon’s sale day turns into fail day by phendrenad2 on Tuesday July 17, 2018 @12:07AM (#17547229)
You misread it - the manager wasn’t gullible, he was the one telling the fib.
Re:Prime Down: Amazon’s sale day turns into fail day by rrjanbiah on Tuesday July 17, 2018 @04:46AM (#17548259)
Re:Prime Down: Amazon’s sale day turns into fail day by ainiriand on Tuesday July 17, 2018 @05:22AM (#17548398)
I can see what you mean. Take for example the email I received to promote Prime. A mix of spanish and english with some very bad translation problems in spanish. It was a bit embarrasing to read it.
Re:Prime Down: Amazon’s sale day turns into fail day by yeukhon on Monday July 16, 2018 @09:05PM (#17546508)
No load testing ever?
Re:Prime Down: Amazon’s sale day turns into fail day by hossbeast on Monday July 16, 2018 @09:10PM (#17546533)
Good luck generating Prime Day load artificially.
Re:Prime Down: Amazon’s sale day turns into fail day by greenleafjacob on Monday July 16, 2018 @11:43PM (#17547138)
At Uber we invested in synthetic load tools that create fake riders and drivers, match them, etc and test our entire dispatch system end to end to arbitrary amounts of online driver and riders. I don’t see why they couldn’t do the same with carts, adding products to carts, etc
Re:Prime Down: Amazon’s sale day turns into fail day by wiredone on Tuesday July 17, 2018 @12:30AM (#17547334)
They do. But I'm guessing you don't scale up 1M+ servers for your Uber canary traffic tests - these are some of the scales Amazon undergoes in these events. The scale is unlike almost any other web property around.
Re:Prime Down: Amazon’s sale day turns into fail day by BinaryIdiot on Tuesday July 17, 2018 @01:50AM (#17547573)
> But I'm guessing you don't scale up 1M+ servers

Do they really need 1 million servers? Many of my friends who work at other tech companies need such few servers in comparison even with significantly high traffic that just screams massive inefficiencies...which seems wrong.

But I've never worked at Amazon so I wouldn't know.

Re:Prime Down: Amazon’s sale day turns into fail day by convivialdingo on Wednesday July 18, 2018 @12:30AM (#17555590)
At one of my previous companies we managed about 100 servers for peaks of about 30 million users.

But we didn’t handle free-form text search like Amazon. I can imagine that would necessitate a huge scaling up of compute and data.

Re:Prime Down: Amazon’s sale day turns into fail day by deegles on Tuesday July 17, 2018 @01:28PM (#17551412)
Judging by all the error pages after launch, they needed more!
Re:Prime Down: Amazon’s sale day turns into fail day by greenleafjacob on Tuesday July 17, 2018 @01:35AM (#17547532)
I’m not sure how it’s relevant. If you have the infrastructure to send 1000 concurrent users you can probably send 1M concurrent users. We only test small integer multiples of our peak traffic, and if your absolute number of servers to service that is in the millions then it would make absolute sense to be routinely running that capacity test. If that means “scale up 1M+ servers” then that is what you have to do, otherwise how can you be sure?
Re:Prime Down: Amazon’s sale day turns into fail day by HankB99 on Monday July 16, 2018 @10:17PM (#17546809)
Could you do something like that from another cloud service? I suppose the difference would be whether 10K requests from one IP address would be the same as 10K requests coming from 10K IP addresses. The further you move from the actual production load, the higher risk that the test doesn't test everything. For example if the network could only handle connections from 5k hosts then the former could pass while the latter failed miserably.
Re:Prime Down: Amazon’s sale day turns into fail day by pageald on Monday July 16, 2018 @10:34PM (#17546863)
Simulating millions of users should be well within the capabilities of a company as large as Amazon. Off the shelf load testing tools like Locust can create thousands of fake users with one worker.
Re:Prime Down: Amazon’s sale day turns into fail day by sitkack on Monday July 16, 2018 @10:14PM (#17546799)
Please, generating load is always easier than servicing it.
Re:Prime Down: Amazon’s sale day turns into fail day by yeukhon on Tuesday July 17, 2018 @07:51PM (#17554537)
Probably can find a hotspot and load test on the hospot.
Re:Prime Down: Amazon’s sale day turns into fail day by jjirsa on Monday July 16, 2018 @11:14PM (#17547030)
(◔_◔)
Re:Prime Down: Amazon’s sale day turns into fail day by jtchang on Tuesday July 17, 2018 @01:10AM (#17547461)
It's actually pretty trivial. Getting the right parameters for the load will be hard and making sure you are loading it properly. Think about DDOS attacks. Generating the load is rarely the issue.
Re:Prime Down: Amazon’s sale day turns into fail day by nikofeyn on Monday July 16, 2018 @07:10PM (#17545960)
what is a bar raiser?

and when has amazon ever been an engineering force? i have always felt the website and service experience is a relic of the 2000s. more often than not, i get the answer “our system can’t do that” from customer service.

Re:Prime Down: Amazon’s sale day turns into fail day by throwaway289355 on Monday July 16, 2018 @07:17PM (#17545996)
>https://www.socialtalent.com/blog/recruitment/raising-the-ba...

I think Amazon has taken on an outsized image to many people that just isn't true. We have good engineers in many organizations, but we don't pay enough, have the right strategy, or take care of individuals well enough to lure the kind of great folks you find at other big tech companies. In many ways, Amazon is a retailer that does technology because it found a way to make money from it. The DNA is still MBAs/finance and retail.

Re:Prime Down: Amazon’s sale day turns into fail day by vadym909 on Tuesday July 17, 2018 @02:08AM (#17547645)
A bar raiser is typically someone in the interview loop for a candidate that is not from the same team and therefore more objective about whether the candidate is truly 'better' than most people on the team (raises the bar). Although good in theory- its not actually practical to find every single new person better than the current employees but it helps keep things in perspective and is a deterrent to bringing in weak buddies.

Its true Amazon has some great engineers but is not a very engineering centric. I remember a senior engineer in Retail once comparing it to a plumbing system kept together with bandages.

Re:Prime Down: Amazon’s sale day turns into fail day by deskamess on Monday July 16, 2018 @07:52PM (#17546167)
How much do you think the on-call contributes to engineers leaving? You would think support tools and support personnel could help to retain engineers.
Re:Prime Down: Amazon’s sale day turns into fail day by jhall1468 on Monday July 16, 2018 @11:31PM (#17547084)
No, bad design and refusal to manage technical debt is the issue. Oncall only matters in some orgs and even then only matters where the tech debt is totally out of control.

Bottom line is Amazon is a product culture not an engineering culture and that makes it really easy to leave for Google or unicorns that really appreciate tech debt tradeoffs.

Re:Prime Down: Amazon’s sale day turns into fail day by chronid on Tuesday July 17, 2018 @04:25AM (#17548177)
Who cares about retaining those support/ops engineers you shit on because you don't want to own the crap you write, right? :)
Re:Prime Down: Amazon’s sale day turns into fail day by kk_cz on Tuesday July 17, 2018 @04:51AM (#17548276)
From the site:

In simple terms, bar raisers are current Amazon employees that come in during the interview process to analyse candidates. They do this alongside their own full-time job, assessing as many as 10 candidates a week and spending 2-3 hours on each one.

In other words 20-30 hours per week on top of the full-time job? That doesn't sound quite right.

Re:Prime Down: Amazon’s sale day turns into fail day by patch_cable on Tuesday July 17, 2018 @10:27AM (#17549935)
Another bar raiser here. That site is wrong. The expectation is 2 candidates a week. And it would be more proper to say that it is part of your full-time job, not on top of it.

10 candidates in a week may happen at some kind of event, but then it isn’t 2-3 hours per candidate, and in that case you’d effectively be taking a couple of days off from your normal job.

Re:Prime Down: Amazon’s sale day turns into fail day by xigency on Monday July 16, 2018 @11:54PM (#17547180)
This bar raiser thing sounds like normal hiring practice in my work experience.
Re:Prime Down: Amazon’s sale day turns into fail day by nojvek on Tuesday July 17, 2018 @12:28AM (#17547322)
The thing that baffles me is the industry perception is that Amazon has subpar engineering but Amazon right now is 2nd after Apple for Marketcap, so they must be doing something right.

I love the Amazon customer service. They’ve managed to crack a difficult problem and execute enough that other Giants haven’t come close to it yet.

GCP and Azure tail AWS by quite a bit. Amazon online retail is a Google search engine level monopoly now.

So Amazon can do a lot of things wrong, but I’d have to say they get the important parts right.

Re:Prime Down: Amazon’s sale day turns into fail day by Godel_unicode on Tuesday July 17, 2018 @01:08AM (#17547458)
> Amazon online retail is a Google search engine level monopoly now.

Not even particularly close. Amazon doesn't even have a majority of online sales, although it's getting close. They seem like a much bigger force than they are because of growth.

https://techcrunch.com/2018/07/13/amazons-share-of-the-us-e-...

Re:Prime Down: Amazon’s sale day turns into fail day by pas on Tuesday July 17, 2018 @04:38AM (#17548224)
They are big, because they invested every penny they made into getting big. And those pennies are converted through human misery in "fulfillment centers". (And AWS on-call is not that fancy either for engineers, though the two are almost incomparable.)

They got some parts right, some parts awfully wrong, and some are just irrelevant now. They make money, they are cheap + convenient, and that usually what people focus on. They are not sophisticated, they are not great designers, etc.

Re:Prime Down: Amazon’s sale day turns into fail day by BryantD on Monday July 16, 2018 @08:12PM (#17546265)
This doesn't completely answer the question, but I would distinguish UI and UX from the ability to build systems that run successfully at Amazon scale.
Re:Prime Down: Amazon’s sale day turns into fail day by cma on Monday July 16, 2018 @11:55PM (#17547181)
Corporate meme thing like six sigma. Each new employee is supposed to raise the bar, or the whole organization gets worse on average. Jeff Bezos as #1 is supposedly the least qualified employee in the company as each new hire raised the bar.
Re:Prime Down: Amazon’s sale day turns into fail day by booleandilemma on Tuesday July 17, 2018 @12:39AM (#17547380)
what is a bar raiser?

A PM who has never opened an IDE in his or her life and who is only familiar with “coding” concepts through Wikipedia. They read books by Malcolm Gladwell, Daniel Kahneman, and Nassim Taleb and majored in one of the humanities. When they’re shown the webpage that the geeks created which loads in 2 seconds, they tell the lead developer that they want the loading time cut down to 1 second and the font to be changed.

Re:Prime Down: Amazon’s sale day turns into fail day by overkalix on Tuesday July 17, 2018 @01:19AM (#17547490)
Hey hey hey. What's wrong with Kahneman?