Due to our new allegiance to google cloud I’ve been given a little more privileged access to engineers (after the fact) and I can tell that I definitely made the right choice, the people I spoke to favoured having a clean/clear backend with actual quality of life features; but since they don’t go nicely into feature comparison charts people often think that GCP is less mature or featureful.
I’m a real convert, and our company uses all three of the big cloud providers in some fashion- but my team only deals with GCP and we’ve had the least headaches.
What I’m trying to say is: you guys are doing great. I’m really happy with the product for my use-cases.
If it works for you thats good, but it wouldn't be my first choice even if I found it to cost less than other alternatives.
I will agree that GCP has some nice quality of life features, and is certainly favorable for a small project compared to some other providers, but I find it hard to trust google's ability to keep up with the demands of a large organization.
VMs don't have a mess of instance types, just standard but customizable cpu/ram with any disks and local SSDs attached. Live-migrated, billed to the second, and automatically grouped per cpu/ram increments for billing discounts.
Networking is 2gbps/core upto 16gpbs/instance with low latency and high throughput regardless of zone and doesn't need placement groups. VPCs are globally connected and can be peered and shared easily across projects so that it's maintained in one place across multiple teams. Fast global load balancing with a single IP across many protocols and immediate scaling.
Storage is fast with uncapped bandwidth and has strongly consistent listings. BigQuery, BigTable and lots of other services have "no-ops" so there's no overhead other than getting your work done. Support is also flat rate and not a ripoff percentage.
The major downsides to GCP are the lack of managed services, limited and outdated versions for what they do offer, poor documentation and broken SDKs, and dealing with the opinionated approach of their core in-house products. This usually means as a startup that you are more productive on AWS or Azure because you can get going in a few clicks with a vast ecosystem, while GCP is a better home for larger companies that have everything built and just need strong core offerings with operational simplicity.
In my experience, most startups just want managed options they can run themselves rather than engaging partners but if that's what you need then AWS will have more companies to offer, although GCP does have qualified partners. I recommend contacting one of the GCP developer advocates either in this thread or on twitter for help, or email me separately and I'll put you in touch.
Also I havent worked with them but https://shinesolutions.com/ has put out plenty of articles and case studies that seems to show they're pretty capable, that might work for you.
For an early stage startup/Indie developer, GCP support model is not appeasing. $100 Per user for development role is unacceptable. (at least to us, the stage where we are now)
If you're very early then you can also try the older support pricing: https://cloud.google.com/support/premium/
If that's still too expensive then you can probably rely on the free support and forums until you have more spend and revenue.
https://cloud.google.com/solutions/migration-center/ is a reasonable jumping off point. Sorry if that didn't come up more clearly.
Gcp was always exactly the same perf, on the button exactly.
Azure was all over the place, fast then slow then fast then slow.
Aws was in the middle.
If you want guaranteed perf then go with gcp 100%
That is spot on. After a year of research and testing on GCP and AWS, I concluded with the same thing. AWS is much more startup/indie-friendly.
We hate Oracle like the next person, but we're locked in. And there's no way we're going to the "Oracle Cloud"
I can understand why this is a no-go for the GP.
This is the cost of lock-in.
Personally I'd rather have the expertise on staff than pay oracle (and microsoft) for this shitty behaviour.
Edit: I'm being downvoted but it's true, if you need enterprise support you're paying the same percentages you would on any other cloud service.
The role-based support is flat-rate but you're right that the Enterprise tier is $15k or percentage spend. From my experience, most companies are fine with the 1-hour production tier.
I haven't worked with GCP beyond 6 figure scale but I dont think that matters. If you need the 15-min response times and the TAM guidance then that's the fee but at least you can opt-out if you don't.
So, obviously my experience is based on my use-case so things that are important to me are predictability of resources and consistency of data.
From a resource perspective (GCP:Compute) there seems to be a tendency of setting CPU affinity for certain cores, often in the same NUMA zone on the host. This means my tickrate doesn't get hijacked on certain cores randomly. That's quite nice honestly and it's something that on-prem hyperconverged VM solutions don't manage. The NUMA zone affinity / samezone affects our performance quite drastically as certain applications (such as in-memory databases or gameworlds) allocate large chunks of memory and then do little updates millions of times a second. This means memory bandwidth is very important.
GCP also has live migration of instances which is 100% transparent to the application (even the CPU clock seems to be moved over), unlike my colleagues on AWS who get 24h notifications of host machine maintenances I've never had to "deal" with even a single instance outage since starting up on GCP 1y ago. But I would assume if AWS hasn't moved on solving this problem they will soon.
Things like VPC networking being inter-zone by default is something that just makes sense and solves some headaches that I hear other teams having.
From a storage performance perspective, on AWS we were hard-capped to 5GBps per instance to S3, such a hard limit does not exist for GCP instances to cloud storage so we're able to make use of it. (in fact, I can often saturate my 10GBit interface limit even to non-google instances). A n AWS sales rep that this hard-limit can't be removed no matter how much we pay.
I can't attest to the difference in APIs, since I use terraform and that abstracts away anything I would be using.
Regarding storage again; we had reps from AWS talking to us at length and they would not guarantee that fsync() would be honoured on elastic storage. They stated that "it should never be needed" but honestly I'm not comfortable with that answer. I mean, my bare-metal database instances haven't gone down in 3 years but it doesn't mean I'm going to start decreasing the consistency for raw throughput.
There's a lot more but these are the most important things that I can think of right now.
The fixed performance EC2 instances (in contrast to instances with burstable performance) all have dedicated physical CPU allocations, and 1:1 affinity from vCPU to the underlying CPU. Similarly, memory allocations are fixed, pre-allocated, and are aligned to the NUMA properties of the host system. When combined with both our Xen hypervisor and Nitro hypervisor, this provides practically all of the performance of the host system with regard to memory latency and bandwidth.
See: https://twitter.com/_msw_/status/1045032259189760000
With our latest 25 Gbps and 100 Gbps capable instances, EC2 to S3 traffic can make use of all of the bandwith provided to the instance.
Please send me more information about the interaction you had on the EBS side, msw at amazon dot com. All writes to EBS are durability recorded to nonvolatile storage before the write is acknowledged to the operating system running in the EC2 instance. fsync() does not necessarily force unit access, and explicit FUA / flush / barriers are not required for data durability (unlike some storage devices or cloud storage systems). Perhaps there was confusion about the question that was asked.
In hyper converged environments the storage and the compute live together on a service mesh and thus some processing is done on the hypervisor to manage storage, those processes don't have an affinity (in the case of vmware for example) so a guest VM core gets suspended for a couple of hundred clock ticks sometimes.
This is a super challenging problem for general purpose virtualization stacks. EC2 has been working to avoid any guest VM interruptions. With the Nitro hypervisor, there are no management tasks that share CPUs with guest workloads.
See more here, including data from a real customer with real-time requirements running on EC2 under the Nitro hypervisor: https://youtu.be/e8DVmwj3OEs?t=1796
This is a great tool to help uncover if CPU and memory NUMA within a VM aligns to the physical host or not:
https://www.cl.cam.ac.uk/research/srg/netos/projects/ipc-ben...
Paper from the USENIX 2012 conference: http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1....
EC2 instances have been NUMA optimized since we launched our CC1 instances in 2010. I encourage you to try ipc-bench on your cloud provider of choice, if CPU and NUMA affinity is important to your workload.
However it may depend on instance type and other factors. This complexity is a problem with AWS.
Live migrations is amazing.
Now if only Google paid more attention to certain features of their GLB... amazing tech with showstopper bugs means amazing tech I can’t use. (Also means it’s not so amazing)
"We have important news about your account (AWS Account ID: XXXX). EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: i-XXXX) in the us-east-1 region. Due to this degradation, your instance could already be unreachable. After 2018-12-28 19:00 UTC your instance, which has an EBS volume as the root device, will be stopped."
The stop date is 2 weeks after the email is sent, but as the message states, the instance is likely already dead. It was already dead in this case. GCP will live migrate your instance to another host.
Like the parent said; we used to receive these notices quite frequently and the amount of time varies wildly. I should ask the teams who currently/primarily use AWS perhaps the situation has changed?
But it was not uncommon to have 24hrs of notice, especially for instances in US-EAST2 for some reason.
https://aws.amazon.com/blogs/aws/the-floodgates-are-open-inc...
GCP has much more modern APIs (they got to do "right" the first time around, while AWS had to learn from their "mistakes")
GCP is less mature with documentation, but still excellent (AWS is first class). I've found bugs, and others I know have run into straight up wrong docs - a quick message to support yielded the correct information and docs update within the week.
GCP is far more "opinionated" about how you should run a service. AWS is opinionated as well, but less so. What this means is that while GCP will sell you resources in traditional way, to go GCP native or from scratch really requires buying into the GCP "way" of doing things more than AWS does. Basically, K8S or go home.
We use GCP and I enjoy it quite a bit. I have extensive AWS experience as well, but no strong preference.
AWS now has the new arm machines in production which are cheaper. AWS also allows yearly prepaid managed databases - while Google still charges per second billing (with some monthly discount). My AWS TCO comes out to be much, much lower (almost 30%) than GCP if i take into account committed use/prepayment discounts.
For us: on AWS/Azure we had to benchmark each instance when it came up and if it performed poorly it had to be reaped and redeployed constantly. The increased overhead in dealing with things "the AWS way" was enough by itself to offset the pricing difference with the steep AWS discount we were getting.
Understanding administrative overhead is complicated but I'm sure you are taking into account the cost of humans. (or, you've already absorbed the cost of automating it?) :)
As with all things; use what works for you. I'm just very happy with GCP over the others coming from bare metal.
in terms of benchmarking - this is not something we care about (since we dont have so much CPU utilization), so probably its easier for me.
For my usecase, AWS has better administrative tools than google. I use Route 53, SES, S3 - in all those three, no other products come close (transfer acceleration...im looking at you).
The one place that I agree with you is Dataproc. That's far superior to EMR in terms of spin-up/down.
I prefer to think of reserved instances as a purely financial trick.
For example, if I have a new workload that won't fit on existing hardware with AWS, I can either: a) reserve the new instances, or b) not reserve on the assumption I will be able to optimise in the 'near' future.
In practice this is hard to know but AWS forces you to come to a correct decision up front, whereas Google Cloud allows you to not have to answer this up front without being punished later for making the 'wrong' choice.
At the bottom line, commitment impedes change.
I now have a significantly reduced opex that saves me a lot of money. If I grow very fast (and outgrow it)... I really dont care, because I will most likely have a financing event.
If I dont grow fast enough to justify the spend, I have bigger problems.
In addition, I have to commend Google and Azure here - the way they do committed use is very flexible. They price it on number of units (cores, RAM, whatever) - so if you outgrow it, you still get the discount + full cost of additional units. On AWS, if you outgrow, you have to frikking sell off your machines on their auctions and buy new ones.
Only problem is that Google doesnt do this for databases (which are very expensive)
I don't believe this is entirely true. The AWS reserved credits are good within the machine family. So a t2 credit is good for all t2 instance sizes. Not as flexible as CPU/RAM credits, but more so than it was in the past.
its not very nice. that's why they have a reserved instance marketplace - https://aws.amazon.com/ec2/purchasing-options/reserved-insta...
what is new is the convertible reserved instances. I havent used it - and you may be right there. But i still maintain the GCP/Azure way is light years better
I do think gcp and azure do it better though.
Separating capacity from instance type makes it much more natural and easy to use for your actual requirements.
- >500k cores
- >300PB storage
- >12,500 cluster size
- >1T messages per day
And there's also this other talk, "How Twitter Migrated its On-Prem Analytics to Google Cloud" - focused on their migration to BigQuery:
- https://www.youtube.com/watch?v=sitnQxyejUg
- 20 TB/ day of raw log data, >100k events/sec
- Loading ~1TB/hour into BigQuery.
- Serving 5,000+ complex queries / second. p99 ~300ms
Disclosure: I'm Felipe Hoffa and I work for Google Cloud https://twitter.com/felipehoffa.
I’d be interested in their cost/benefit analysis for this one.
How? I mean there are 7B people on earth. Are people that active on Twitter? Or does "messages" mean something else here?
> Disclosure: I work at Google Cloud (and directly with Derek and the Twitter team).
Thanks for making this community awesome. I want to work at Google or with Google one day. One of the few companies I have always admired and always will.
It is possible that while GCP gives a good value based on the writing on the tin, someone else might beat that with a good margin under special terms.
That being said, I have reasons to think very few could beat Google on compute or storage game and definitely much harder to beat the big G on Networking side of things.
Disclosure: I worked for GCP until early 2015, but I haven't worked for Google since then, have no inside info on this deal and am not relying on anything from my time at Google for this comment.
https://getpolarized.io/2019/01/03/building-cloud-sync-on-go...
I like the google storage API the most honestly. Firebase is really nice but the storage API is very polished and basically does exactly what you need it to do without the complexity of S3.
Obviously that's the thing you'd choose to highlight in a public Google-hosted presentation after a strategic partnership that is a bonanza in marketing for Google Cloud. It wouldn't work so well to tell a room full of engineers "We moved to Google cuz they gave us a fat discount, one you can't get because you're not Twitter!"
The fact that you allocated a top level URL for it is embarrassing (https://cloud.google.com/twitter/). Can you imagine if Amazon advertised https://aws.amazon.com/netflix/?
It's a case study. Google didn't create a one-off URL for Twitter, but have the same URL pattern for all their public case studies:
- https://cloud.google.com/chilean-healthcare/
- https://cloud.google.com/now-ims/
- etc.
Amazon has absolutely the same, except that their URL is slightly different, but I didn't think that there is a URL police which determines which URLs are considered embarrassing for case studies and which not:
I’ll try to edit this page tomorrow when I’m at a computer, but there’s much more information in Derek’s talk at NEXT [1]. They (rightfully) didn’t want to get into a detailed “this is what we saw on <X>”, but Derek alludes to their careful benchmarking across providers.
While you should always assume smart people make economically reasonable decisions, Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting). I’m hoping a follow-up talk will let us say more about how the migration is going, while this talk was focused on the decision to move part of (!) their Hadoop environment to GCP.
[1] https://m.youtube.com/watch?v=4FLFcWgZdo4