News for hackers. Stuff that matters.
Re:Cloud Memorystore: A fully managed in-memory data store s... by dudus on Wednesday May 09, 2018 @01:45PM (#17031971)
There's a fine print that says:

> ¹Basic Tier instances experience a downtime and a full cache flush during scaling. Standard Tier instance experience very minimal downtime and loss of some unreplicated data during scaling operation. ²Applicable for GA release only.

Does that mean that right now there are no persistance options with this Managed Redis cluster? I have never tried to use Redis but I've been eyeing it for a while and the most confusing part is how it handles persistence.

Apparently there are ways to have Redis configured to be pretty durable as outlined here:

https://redis.io/topics/persistence

But it doesn't look like Google is supporting either RDB nor AOF with this managed service, they say RDB is coming though.

Doesn't it heavily limits the use cases for this product and relegate it to a cooler memcache?

This is not criticism, it's just curiosity on good use cases for this product.

Re:Cloud Memorystore: A fully managed in-memory data store s... by antirez on Wednesday May 09, 2018 @05:16PM (#17034171)
Maybe this is just for the initial period. Redis has orchestration commands that allow to stop the clients in one side, migrate, shutdown the first instance, and resume the other, without the possibility of getting writes in the middle of the transition. Other providers do that to ensure safety of operations.
Re:Cloud Memorystore: A fully managed in-memory data store s... by dudus on Wednesday May 09, 2018 @09:14PM (#17035734)
Thanks for chiming in. I'm a big fan of your work. Did Google consult you on this product before launch?
Re:Cloud Memorystore: A fully managed in-memory data store s... by antirez on Thursday May 10, 2018 @06:02AM (#17037542)
Nope, no vendor does (fortunately). AFAIK people get the "Redis way": grab the source and do whatever you want with such code, in the limits of the license / law. Also I get near no contacts from big corps using Redis at scale. This is a sane model otherwise my work would be dealing with communications instead of writing code :-) However when technical people from big companies ping me about bugs / issues / improvements ... then it's great feedback, but is rare.
Re:Cloud Memorystore: A fully managed in-memory data store s... by mrkurt on Wednesday May 09, 2018 @02:05PM (#17032214)
It sounds like they're scaling by adding larger instances, replicating, and then failing over. So they'll lose any data that hasn't been synced to the new instance when the failover occurs. I'd bet it's rare, but they're being explicit about it.

The Basic Tier probably only uses one instance, so they just replace it in place.

Re:Cloud Memorystore: A fully managed in-memory data store s... by trhway on Wednesday May 09, 2018 @02:58PM (#17032777)
What would be the point of downtime they mention other than doing the sync?
Re:Cloud Memorystore: A fully managed in-memory data store s... by hinkley on Wednesday May 09, 2018 @05:55PM (#17034515)
From the docs:

   Basic Tier: Provides a standalone Redis instance. Use this tier for applications that require a simple Redis cache.
If you were only reallocating replicas you'd expect no downtime. But turning on replication isn't seamless.
Re:Cloud Memorystore: A fully managed in-memory data store s... by panarky on Wednesday May 09, 2018 @01:59PM (#17032132)
Their post isn't very clear on persistence, here's how I read it.

1) Yes, they're using Redis persistence (RDB, AOF, or a Google-written persistence layer, it's not specified).

2) Basic Tier scales by tearing down the smaller Redis instance and migrating the data (???) to a larger instance. This causes downtime, but not lost data.

3) Standard Tier migrates using replication during scaling, so there's little downtime but if data is in flight at cutover, it could be lost.

4) And "coming soon" you can migrate data in and out of Cloud Memorystore by importing and exporting RDB files.

Re:Cloud Memorystore: A fully managed in-memory data store s... by gopalashok on Wednesday May 09, 2018 @02:29PM (#17032463)
Full disclosure, I'm the product manager for Cloud Memorystore

1) We are not using persistence at all in beta, this is something we are working towards

2) Scaling of basic tier will result in a full cache flush and will cause downtime during the scaling process.

3) That's correct

4) Yes we will be making Import-Export of RDB very soon

Re:Cloud Memorystore: A fully managed in-memory data store s... by sitkack on Wednesday May 09, 2018 @03:25PM (#17033045)
> 1) We are not using persistence at all in beta, this is something we are working towards

Persistence is a key feature of Redis, it isn't just a fancy memcached. This severely limits the applicability of this offering.

Re:Cloud Memorystore: A fully managed in-memory data store s... by jkaplowitz on Wednesday May 09, 2018 @05:27PM (#17034269)
So does the beta label. :) Both will change in time, from what the PM said.
Re:Cloud Memorystore: A fully managed in-memory data store s... by dannyw on Wednesday May 09, 2018 @09:36PM (#17035829)
It’s like releasing a beta for DNS that doesn’t support IPv6 or CNAMES.
Re:Cloud Memorystore: A fully managed in-memory data store s... by rkangel on Thursday May 10, 2018 @05:00AM (#17037284)
Exactly. It will be still be useful to some people, it allows them to get something working end-to-end, and then iterate on it to provide the features for more people.
Re:Cloud Memorystore: A fully managed in-memory data store s... by sitkack on Thursday May 10, 2018 @09:22PM (#17044324)
Not providing persistence is setting people up to lose data. It is the wrong move.
Re:Cloud Memorystore: A fully managed in-memory data store s... by russell_h on Wednesday May 09, 2018 @04:51PM (#17033965)
This is a great looking product, thanks for following up here.

Can you speak to what kind of durability you expect to achieve once persistence is in place? Is it reasonable to treat this as a durable datastore and simply have a disaster recovery plan in place? Or should applications be prepared to handle occasional minor data loss?

Re:Cloud Memorystore: A fully managed in-memory data store s... by hamandcheese on Wednesday May 09, 2018 @04:37PM (#17033817)
> We are not using persistence at all in beta

Maybe don’t advertise it as a “data store” then.

Re:Cloud Memorystore: A fully managed in-memory data store s... by jkaplowitz on Wednesday May 09, 2018 @05:26PM (#17034261)
Transient caches are still a useful type of data store for better performance... That said, they're noting it as a goal and a beta limitation, so they're not satisfied with pure transience either.
Re:Cloud Memorystore: A fully managed in-memory data store s... by hamandcheese on Wednesday May 09, 2018 @11:09PM (#17036119)
That’s fine, but it should be made way more obvious what this is and what it isn’t. This goes against expectations.
Re:Cloud Memorystore: A fully managed in-memory data store s... by antonvs on Thursday May 10, 2018 @02:42AM (#17036866)
Its name is Cloud MemoryStore, not Cloud Diskstore. That should set expectations right there.
Re:Cloud Memorystore: A fully managed in-memory data store s... by hamandcheese on Thursday May 10, 2018 @03:03AM (#17036946)
They use the term “data store” multiple times, not to mention:

> and features like persistence, replication and pub-sub.

(Customer testimonial) > We have used Redis on everything from storing asynchronous task queues for tens of thousands of CPUs to a centralized persisted key-value pair store for the feature vectors output by our ML models

Additionally, “persistence” isn’t mentioned at all in the “Coming soon to Cloud Memorystore” section.

I’m not trying to be a pedant, I just think it’s dangerous that it’s not made abundantly clear that there is no persistence.

Re:Cloud Memorystore: A fully managed in-memory data store s... by dudus on Wednesday May 09, 2018 @02:04PM (#17032198)
Why do you think they are using persistence at all, and why do you think Basic Tier doesn't have data loss on scaling?

I read it differently. When they say full cache flush doesn't that imply full database flush, I was thinking all data gets wiped on scaling operation.

Re:Cloud Memorystore: A fully managed in-memory data store s... by nodesocket on Wednesday May 09, 2018 @02:12PM (#17032292)
Agree I don't think it is persistent. "a full cache flush" represents all data loss to me.
Re:Cloud Memorystore: A fully managed in-memory data store s... by panarky on Wednesday May 09, 2018 @02:28PM (#17032447)
Could be I read this one wrong.

There's a Redis command called FLUSHALL that deletes all the data.

https://redis.io/commands/flushall

Re:Cloud Memorystore: A fully managed in-memory data store s... by dudus on Wednesday May 09, 2018 @02:18PM (#17032351)
Wouldn't that also imply full data loss on server restarts?
Re:Cloud Memorystore: A fully managed in-memory data store s... by tgtweak on Wednesday May 09, 2018 @10:42PM (#17036031)
Redis persistance is not built or intended to be a write guarantee, but more of a safeguard against cold cache after a restart or failure.

It looks as if free tier has persistence turned off by default and that standard tier uses snapshotting/rdb to replicate.

Re:Cloud Memorystore: A fully managed in-memory data store s... by manigandham on Thursday May 10, 2018 @12:53AM (#17036485)
That's exactly what it's built for, to save data. It's just configurable in terms of how and when its persisted.
Re:Cloud Memorystore: A fully managed in-memory data store s... by zodvik on Thursday May 10, 2018 @12:13AM (#17036335)
Would AOF persistence as 'everysec' not imply a write guarantee of max 1 second data loss for restart?
Re:Cloud Memorystore: A fully managed in-memory data store s... by tgtweak on Friday May 11, 2018 @10:25PM (#17052162)
No, this is not a write guarantee.

You can still lose an acknowledged write.

Re:Cloud Memorystore: A fully managed in-memory data store s... by ihsw2 on Wednesday May 09, 2018 @01:58PM (#17032127)
On the surface it looks like this isn't a true-blue Redis server (or even a fork of Redis) but support for the Redis protocol bolted onto an in-house memory-based caching product.
Re:Cloud Memorystore: A fully managed in-memory data store s... by bradhe on Wednesday May 09, 2018 @06:05PM (#17034589)
> Does that mean that right now there are no persistance options with this Managed Redis cluster?

It's a memory store.

Re:Cloud Memorystore: A fully managed in-memory data store s... by amazingman on Wednesday May 09, 2018 @06:22PM (#17034733)
That's not really an answer to the question. You may not agree with it, but there are many orgs out there using Redis as a persistent datastore. If Google wants their money, they're going to need persistence options.
Re:Cloud Memorystore: A fully managed in-memory data store s... by bradhe on Wednesday May 09, 2018 @06:33PM (#17034837)
It doesn’t guarantee it’s redis. It guarantee’s it’s redis compatible. The use case they’re (rather obviously) optimizing for is an in-memory cache.