Re:Update on Christmas Issues
by calpaterson on Wednesday December 30, 2015 @04:42PM
(#10814549)
This is a famous quotation but in my experience it is not true. I have implemented web caching a few times and while some RPC-style forms/APIs have been difficult I've never put a cache in and had it go seriously wrong. Naming things is sometimes tricky but never profoundly difficult.
The hard problems are probably things like doing something concurrently or maybe implementing the right feature...
Re:Update on Christmas Issues
by whizzkid on Wednesday December 30, 2015 @04:58PM
(#10814626)
"... it is not true. I have implemented web caching ..."
As the quote says, it is not the implementing which is hard, it is the invalidating part that is little bit tricky.
If you try to invalidate the cache in a strict manner, you will not enjoy the benefits of the caching much, since it will invalidate most of the time. If you have more relaxed implementation of it, there is a risk of this exact problem happening.
Re:Update on Christmas Issues
by kevinbowman on Wednesday December 30, 2015 @04:50PM
(#10814593)
It's possible when you don't realise that something will be cached (for whatever reason - incorrect verb, missing etag or cache-control headers, some cookie confusion), and suddenly everyone starts seeing one person's experience (hopefully only a slightly tweaked page, but at worst their account screen).
Turning the caching off is easy, but it can be really difficult to track down the cause after that because you tend to need a specifically-timed race condition, and even when you're sure you've fixed it everyone is really nervous to turn the cache back on again...
Re:Update on Christmas Issues
by mst on Wednesday December 30, 2015 @06:50PM
(#10815200)
I, and many other ops-ish people I know, cringed at the obvious caching error that seemed like the most likely culprit.
The "in the middle of a DDoS" makes it make a lot more sense, and while this is still a cringeworthy situation, the cringing at least on my part is now more sympathetic than horrified.
Re:Update on Christmas Issues
by pnt12 on Wednesday December 30, 2015 @05:31PM
(#10814845)
Valve is (was?) known for being silent too often - at least concerning Dota 2 (the diretide event 'scandal'). Maybe they're trying to correct that, after so many complaints about their poor communication, which is nice.
Re:Update on Christmas Issues
by CaptSpify on Wednesday December 30, 2015 @09:56PM
(#10815845)
I think this was big enough that they had to comment. Publicly leaked personal information might be enough to start a lawsuit (IANAL). But I agree, props to them for going against their SOP, and actually communicating.
Re:Update on Christmas Issues
by Zikes on Wednesday December 30, 2015 @03:47PM
(#10814270)
I really hope this catalyzes Valve to create a proper customer support team, one that can publicly address these issues in a timely manner as well as handle day-to-day customer support problems. It's ludicrous that a company as big as Valve has got away with it for as long as they have, and that's due in large part to an extremely devout fanbase that's only now starting to get fed up with them.
Re:Update on Christmas Issues
by Zikes on Thursday December 31, 2015 @11:10AM
(#10818163)
gog.com is becoming really popular, and I actually had a really good experience dealing with EA Origin's support earlier this week. Sure they're not as big as Steam yet, but they're getting big enough to be competitive.
Re:Update on Christmas Issues
by danso on Wednesday December 30, 2015 @03:37PM
(#10814201)
Nice to see the update and I'm very interested in how many people they determine deserve compensation, and why them (if there is any distinction between severely affected and no-big-deal effects). The error seems "basic"...in the sense that everyone who knows devops knows how to prevent it, but according to their account, the circumstances were extreme (plus, it was the holidays) and so mistakes were made. Just like how only a seemingly moron of a surgeon could operate on the wrong limb of the wrong patient...but such catastrophic mistakes often happen when the ER is in a state of chaos.
Re:Update on Christmas Issues
by vive-la-liberte on Wednesday December 30, 2015 @04:47PM
(#10814581)
>Valve is currently working with our web caching partner to identify users whose information was served to other users, and will be contacting those affected once they have been identified.
Re:Update on Christmas Issues
by grogenaut on Thursday December 31, 2015 @12:03PM
(#10818442)
They're glossing over over this a bit. Many sites use the last 4 digits of your cc number as validation so the last 2 digits + email + phone suffix setup is valuable information and is definitely an addition to any cc database for liveliness of credit cards or accounts to go after on steam.
Re:Update on Christmas Issues
by thezilch on Thursday December 31, 2015 @02:48AM
(#10816714)
They didn't make a mistake; they hopefully learned a huge lesson. It's hard finding people already having learned this lesson, so you shouldn't fire over this incident alone.
Re:Update on Christmas Issues
by minimaxir on Wednesday December 30, 2015 @03:38PM
(#10814203)
While an expected response, it's hard to forgive Valve for taking 5 days for any official acknowledgement of the issue (not even a Tweet on the Steam Support Twitter accounts, which you would expect to be the primary source of information), especially since personal information leaked.
To quote a NeoGAF moderator:
> To be honest, I think people would have been willing to give them 5 days (irrespective of what the professionally appropriate timeframe was) if their initial response wasn't so tone-deaf.
Re:Update on Christmas Issues
by Zikes on Wednesday December 30, 2015 @04:01PM
(#10814349)
Call it petty, but hearing their official statement secondhand makes it feel disingenuous to me. They have more appropriate channels in place for that sort of thing, including email, their two official Twitter accounts, and their very own web site, which would communicate directly to the customers affected.
The hard problems are probably things like doing something concurrently or maybe implementing the right feature...
As the quote says, it is not the implementing which is hard, it is the invalidating part that is little bit tricky.
If you try to invalidate the cache in a strict manner, you will not enjoy the benefits of the caching much, since it will invalidate most of the time. If you have more relaxed implementation of it, there is a risk of this exact problem happening.
Turning the caching off is easy, but it can be really difficult to track down the cause after that because you tend to need a specifically-timed race condition, and even when you're sure you've fixed it everyone is really nervous to turn the cache back on again...
The "in the middle of a DDoS" makes it make a lot more sense, and while this is still a cringeworthy situation, the cringing at least on my part is now more sympathetic than horrified.
How will they be able to do that?
Firing people for a single mistake is for fast food restaurants, not for billion dollar tech companies.
To quote a NeoGAF moderator:
> To be honest, I think people would have been willing to give them 5 days (irrespective of what the professionally appropriate timeframe was) if their initial response wasn't so tone-deaf.
[0] http://www.pcgamer.com/warning-steam-is-revealing-private-ac...