News for hackers. Stuff that matters.

andywarfield

Here's a really quick story on how i accidentally worked out this strategy by getting it wrong first. When I started at Amazon and was trying to convince the team that we should do certain things, I did what I'd always been trained to do: I wrote down the problem and then sketched a solution to it. Then I'd start floating the doc around to try to get folks excited about it. And invariably, they'd do what they were trained to do, which was to have a critical response to the proposed solution. They'd argue that I was solving it the wrong way, and I'd be in a spot where we'd have a conversation where I was defending a position. But this was the last thing I wanted — I was trying to get everyone excited about fixing a problem, but I slowly realized that when I approached it this way, I was just getting feedback on my proposed solution.

So I started doing an experiment where I'd write that same doc, including the ideas i had on the shape of the work we should do, but then I'd delete my solution before sharing it. To your question: I'd still totally write my solution ideas down. Partially because I can't help myself and honestly it was a helpful way to think things through. But when I deleted it and shared a doc with just a problem statement, I'd get feedback on the problem statement. It's pretty obvious, but it was also a pretty surprising result: all of a sudden i was in conversations where we were all on the same side of the table. Feedback was either refining the problem (which was awesome) or proposing solutions. And when the person reading your problem statement starts trying to solve it, it's really cool... because they totally start getting invested and the conversations are great.

Like everything, none of this is actually either/or. There are points in between, like including a sketch of the shape of a solution, or properties that a solution would have to have. But the overall thing of separating the problem and the end state of where you want to get to, from the solution and the plan on how to get there is a pretty effective tool from a sharing ownership perspective.

oh shoot. good catch, thanks!
Re:The 80/20 rule for storage systems on Monday December 15, 2014 @12:24PM (#8753080)
Getting things perfect is pretty hard: it basically required being able to see the future. There are still a bunch of examples where imperfect decisions end up being a really significant win. For instance:

1. The counter stacks that we build using the HLLs allow us to draw miss ratio curves. These are a model for how much a given application will benefit from getting additional fast memory (whether that fast memory is flash/ram/etc). A useful aspect of this isn't just trying to give it more memory but also identifying the workloads that you simply can't win on. For workloads with absolutely enormous working sets, you can decide to give them a lot less resources, and then end up speeding up all the other applications in the system. Alternatively, you can decide to give them the large amount of memory that they do need to be successful, possibly by having to buy additional fast memory. In either case, being able to get detailed working set analysis helps make appropriate decisions a lot.

2. 80/20 is completely anecdotal -- the appropriate split is always going to be entirely dependent on workload. That said, as you move out into the tail of the access frequency curve, cost per access increases. So even if you don't move data out to disk -- even with only a single type of storage media -- understanding these curves can allow you to decide to do things like compress the colder half of your storage, trading off some compute work and latency on access for more space. In the case of hybrid systems, strategies like this have the potential to really increase the effectiveness of caches.

Totally agree with your other points. There are many factors to consider and balance in any system design. Note that for raw media, magnetic disks are still way more reliable in terms of archiving your bits without data errors over long periods of time than NAND. In either case, good system design needs redundancy and scrubbing to make sure that device errors are caught and recovered from before data is actually lost.

Re:The 80/20 rule for storage systems on Monday December 15, 2014 @12:00PM (#8752958)
Thanks! We're pretty early in our use of HLLs within the system but are already managing to get some really cool data off of them. I'm excited to see where this all goes as we build out the system over the next year.

Your last-accessed time point is exactly right. Storage systems used to be able to do a lot with file system-level metadata, but with the size and opaqueness of VM image files, those techniques have become a lot less effective. We're currently exploring how we can use HLLs in combination with a couple of other techniques to do things like clustering co-accessed data and then managing operations like prefetching and demotions over much longer time frames than are typically done in OSes and storage systems.