Intel’s first Optane SSD: 375GB that you can also use as RAM

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by sologoub on Sunday March 19, 2017 @01:52PM (#13908581)

At previous employer, we built a system using Druid as the primary store of reporting data. The setup worked amazingly well with the size/cardinality of the data we had, but was constantly bottlenecked at paging segments in and out of RAM. Economically, we just couldn't justify a system with RAM big enough to hold the primary dataset. As the result, we had to prioritize data aggressively, focusing on the more recent transactions and locating them on the few servers with very high RAM that we did have. Historic data segments had to go through a lot of paging in/out of RAM. User experience on YTD (year-to-date) or YOY (year-over-year) reports really suffered as the result.

I don't have access to the original planning calculations anymore, but 375GB at $1520 would definitely have been a game changer in terms of performance/$, and I suspect be good enough to make the end user feel like the entire dataset was in memory.

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by Dylan16807 on Sunday March 19, 2017 @07:39PM (#13910537)

Make sure you're looking at updated prices for ram too. 16x16GB of registered ECC DDR3 is about the same price and enormously faster.

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by sologoub on Sunday March 19, 2017 @09:28PM (#13911008)

Sure, but I believe we were limited by the available chassis to a lot lower than 16 slots.

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by Dylan16807 on Monday March 20, 2017 @02:13AM (#13912131)

Well the first google result for "1u 16 dimms" is a refurbished chassis+motherboard+PSU for a hundred bucks. Brand new costs more but not terribly so; the main cost is the ram whether you go 8 slots or 16.

These SSDs have situational uses but unless you want 10+ TB in one server you can get a system with >50% as much actual RAM for the same price.

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by sologoub on Monday March 20, 2017 @07:00PM (#13918384)

It's not the cost. We ran standardized chassis, so whatever our ops had is what they had...

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by Redsquare on Sunday March 19, 2017 @02:29PM (#13908792)

Would you choose to run with druid again?

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by sologoub on Sunday March 19, 2017 @06:09PM (#13909968)

For that use case, absolutely! We made do with the version that could not even support label appends (limited joins). The current version would allow us a lot fewer workarounds.

The probabilistic hyperloglog data type is also a game changer compared to say redshift, but again it's only viable if you are dealing with counting (estimating) unique entities across billions of rows and super-wide dimension sets.

If you are doing a general purpose analytics store, Redshift is hard to beat because of reliability and ease of implementation.

Druid is a purpose-built race car. Redshift is a good cross-over - far less headache and can do almost any job good enough, but you won't have the tuning or performance (when tuned right) at scale. Although, I'm continuously impressed with what redshift actually can do, dispite the humble feature set.

Druid's main weakness is lack of SQL support, so it's not a great analyst datastore. You pretty much have to wrap it into a reporting app.

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by scapecast on Monday March 20, 2017 @10:10AM (#13914337)

Hi sologoub - can you elaborate a bit on the tuning for Redshift you're referring to? What's the pain there? Asking because we're building a performance management product for Redshift, I'd love your input! lars at intermix dot io

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by otterley on Sunday March 19, 2017 @07:30PM (#13910472)

What do you think of ClickHouse vis-a-vis Druid and Redshift?

[ Parent | on HN ]

Re:Intel’s first Optane SSD: 375GB that you can also use as RAM by sologoub on Sunday March 19, 2017 @10:50PM (#13911349)

Don't have any experience with that tech, but from reading the marketing landing page it sounds more akin to memSQL than Redshift, in that it seems to include options for streaming ingestion.

If I'm going to take on a similar project, I may POC memSQL or Citus DB, and possibly Big Query (if the project is built on Google Cloud as opposed to AWS or raw iron).

[ Parent | on HN ]