escardin

> With vitess the sharding logic lives “in the database” so your application doesn’t need to be updated.

That is unfortunately not true in the majority of cases. Queries of medium complexity or higher will often just fail to run on vitess. You also still need a sharding key if you want queries to be efficient. Essentially you don't want the vtgates stitching tables from different shards together.

As you scale, scatter queries become more and more problematic. You need to have carefully considered lookups, and often alter your queries to include sharding keys everywhere.

Cross shard writes and two phase commits are also a recipe for deadlocks.

Finally, you need to keep in mind that every shard you connect to uses up a connection in its pool. If you have relatively long lived sessions (like... request length) that touch a ton of shards in the same query, you can exhaust your connection pool with 'idle' connections.

Vitess _is_ amazing. I am grateful for it every day. It is not a panacea. Neither are the alternatives. Switching to Vitess is a beautiful problem to have, because you have a problem of scale. The switch is harrowing.

It's because CashApp lets you buy and sell bitcoin, that is, CashApp is buying this bitcoin so it can sell it to customers [1]. It's not speculation on the part of Square or Jack Dorsey.

All this speculation about speculation is wildly disappointing.

[1]: https://www.investors.com/news/technology/square-stock-sq-sq...

> Square said revenue jumped 141% to $3.16 billion as more Cash App users bought and sold digital cryptocurrency Bitcoin.

I assume if you are using MSBuild you're building .net or otherwise using visual studio. If you want to build these projects with gradle or maven you still have to invoke MSBuild in my experience. It's not strictly necessary AFAIK, but you'd be giving up a lot of what visual studio offers you for little in return.

In general, if you aren't already using a JVM language, there is not a whole lot of reason to use maven or gradle. Pretty much every language has equivalent native build tooling that you should be using instead.

If you are using a JVM language, even for part of your build process, then both can be used as the primary build tool and invoke the build tooling of the other ecosystems as needed. This is most useful when you have a native component that you are using from the JVM.

I have in the past used gradle to build the native C++ components for apps, but in general it reached out to msbuild on windows, and make files on linux.

I've used gradle it to invoke npm commands to build the frontend component of a WAR java webapp.

My thought was everyones local cache was expired and it's re-syncing everyone. Supposedly they run vitess for mysql sharding, and that could mean they're getting slammed by a lot of scatter or cross shard queries they can normally handle.

My team manages a large scale vitess setup. Nothing is free. Vitess is a much better way of sharding mysql than sharding in app. It is not a silver bullet. No matter how you slice it, sharding a database exposes you to everything you described. For most of our devs, they don't actually have to care all that much about what the DB looks like under the hood. The biggest issue is scatter queries, for which we built a detector so you can find out before your code hits prod.

Cross shard queries are evil.

I count myself lucky that I didn't have to live through the migration and first shard splits, it sounds harrowing. We have a shard split coming up this year, but it's gotten way better now that we have our scale under control.

Vitess is a good solution to 'oh shit we use mysql and our app succeeded beyond our wildest dreams and we now have to migrate without or with minimal downtime ASAP, and can't afford the time to rework everything to use cassandra'. It is not something you should really be thinking about when choosing what storage backend to use for your brand new app. We strongly advise teams to choose other technologies where possible. It's not that it doesn't work, or is bad. It's actually really good at what it does. It's rock solid, and scales really well. It's the tradeoffs that make us not want to use it.

No you still have to send the data over the pcie link, but DPDK should basically offload all the network work to the nic, so that you are just streaming data to it. The kernel won't need to deal with IRQs or timing or anything like that.

I might be making things up, but I believe you can also run code on DPDK nics? i.e. beyond straight networking offload. If that's the case you could try compressing the data before you DMA it to the nic. This would make no sense normally, but if your bottleneck is in fact the pcie x1 link and you want to saturate the network, it would be something worth trying.

I mean really the whole thing is at most a fun exercise as the nic costs more than the pi.

It's probably outside the scope (and possibly cheating) but could a DPDK stack & supported nic[1] push you past the PCIe limit?

[1] https://core.dpdk.org/supported/

I would argue that an ingress is not a barrier to getting started with k8s (even with kubeadm on bare metal). You don't NEED an L7 router to get started. Much more likely is you need a LoadBalancer. An ingress is only going to help route traffic that is already in your cluster.

If you deploying to almost any cloud provider they have an implementation for you. If you're on bare metal, you have to make that choice yourself.

Even if you're hosting yourself you don't technically need a load balancer, you could use host ports, or node ports and it would work for some definition of work. I have in the past set up an ingress to listen on host ports of every node and it works well, if a little janky.

I'm surprised you're complaining about this which is an easy thing to replace and not your choice of network plugin, which can result in having to reprovision the cluster or similar if you want to replace it with something different.

Then you're using something bigger than a Cortex-A55?

They're screwed unless they update. Luckily the iPhone 6 was still getting security updates as of this year. It doesn't have ios 13+ but it does have security updates on 12.

As a 6 user for another month or so, my biggest complaint isn't slowness, it's the lack of ram that causes apps to be closed readily whenever you switch, compounded by the apps having to boot/reload again.

Most of the apps I use aren't IOS apps, they're things like chrome.

This is nothing like using a 3g or a 4 until they stopped getting updates, those were downright painful to use. This is merely frustrating.

I don't know what people expect from their phones, but 5-6 years of updates and the phone still running well is incredible. I hope my next iPhone gets me another 6 years. It seems likely.

I don't know what it's like in Android land, but I expect that the number people using a phone at all for 6 years is even less than those using old iPhones. At least they had way more ram for a same era phone.

If you want an ELK stack, you should look into the operators provided by Elastic [1]. All you need to do is write a very small manifest or two for each thing you want operated. I feel like this is a better solution to 'I want an ELK cluster' than a helm chart because it solves more problems without leaking.

[1] https://www.elastic.co/blog/introducing-elastic-cloud-on-kub...

I also think helm is terrible.

Helm's stated goal is to be something of a package manager for apps in k8s, but this is fundamentally unworkable as shown by... Helm. It's hard to describe just how unworkable this idea is.

Let's start with an example, you want to install an app (let's say Jira) and a DB backend of your choice, postgres or mysql.

The first step where this all falls down is, it may or may not support your preferred DB. Sure, Jira does, but does the chart?

Assuming it does support your preferred backend, maybe it depends on the chart for the db you picked. If it does, it's going to install it for you, hopefully with best practices, almost certainly not according to your corporate security policy. This is also a problem if say, you have a db you want to use already, prefer to use an operator for managing your db, or use a db outside k8s.

You got lucky, and it supports your DB just the way you want it. Next question, do you want an HA Jira? Often, this part is done so differently that HA Jira and Single host Jira are straight up different charts.

Do you want your DB to be HA? Unfortunately, the chart Jira chart author picked to depend on is the non-HA one. Guess you're out of luck.

Maybe you want to add redis caching? Nginx frontend/ingress? Want to terminate TLS at the app host and not ingress? How do you integrate it with your cert management system?

We haven't even looked at the config, where you have to do everything in a variables.yaml file which is never documented as well as the actual thing it's configuring on your behalf, and is not always done in a sensible manner.

Hopefully it's clear from this that as a user, helm isn't going to work for you, because just as there's no such thing as the average person, there's no such thing as an average deployment. Even a basic one is filled with one off variations for every user that a public chart needs to support.

As a developer, helm is unworkable because you're templating yaml inside yaml. This isn't too bad if you're just tweaking a few things on an otherwise plain chart, but a public chart, that naively hopes to support all the possible configurations? Your otherwise simple chart is now 5-10x longer from all the templating options. Have fun supporting it and adding the new features you'll inevitably need to add and support.

As a counterpoint to all this, kustomize gets a lot right. I don't mean that kustomize is perfect, or even good, but I've found like k8s itself, it understands that the problem space is complex and to try and hide that complexity leads to a lesser product that is more complex because of leaky abstractions.

Kustomize acts as a glue layer for your manifests, so instead of some giant morass of charts and dependencies none of which work for you, you're expected to find a suitable chart for each piece yourself and compose them with Kustomize.

Going through the same example again as a user:

Your vendor has provided a couple basic manifests for you to consume, maybe even only one, because they're expecting you to supply your own DB. Since they only need to supply the Jira part, instead of having an HA chart and a Single node manifest, They just give you one manifest with a stateful set. Or maybe they give you two. One as a deployment and one as a Stateful set. The stateful set might also have an upgrade policy configured for you.

Since the vendor punted on the DB, you can do whatever you like here. You'll have to supply a config map with your db config to the Jira deployment, but that's okay, it's easy to override the default one in the manifest with kustomize. You are now free to use a cloud managed DB, an operator managed one, or just pull the stock manifest for your preferred DB.

Want to terminate TLS in your app again? Easy enough, Cert Manager will provide the certs for you, and supply them as secrets, ready to consume in your app, you just need to tell it where to look.

So now you have all the parts of your Jira deployment configured just how you like, but they're all separate. Maybe you just edited the stock manifests to get your changes in the way you like. Dang, you've created a problem when the vendor updates the manifest, as now you need to merge in changes every time you update it. That seems like a huge hassle. What if you could just keep your changes in a patch and apply that to a stock manifest? Then it's easy to diff upgrades of the stock manifest and see what changed, and it's easy to see what you care about in your patch.

All of this seems like it's getting kind of unwieldy, maybe we can make it easier. We'll have a single kustomization.yaml, and it'll be really structured. In it, you can list all the paths to folders of manifests, or individual files, or git repos/branches/subpaths. We'll also specify a set of patches that look like partial manifests to apply to these base manifests to make it clear what is the base and what goes on top. Then finally, for common things like image versions and namespaces, we'll expose them directly so you don't need to patch everything. We can do that because we're using standard manifests that can be parsed and modified.

That is kustomize, and as awkward as it is, it's just trying to make it clear what you're applying, and what your one customization from stock is in a maintainable way. It does a better job of 'package management' by not managing packages. This is pretty similar to how Linux package managers work. If you install wordpress on your server, it's going to install php, but it might not install apache or nginx, as everyone wants something different. It's definitely not going to install a DB for you. You as the admin have to decide what you want and tell each part about the other.

I did an on-prem k8s deployment at my last place. It is definitely challenging compared to EKS and GKE, but the difficultly is not in base k8s.

Following the kubeadm getting started guide on the kubernetes.io site can get you an 'ha', 'production ready' going in a couple hours. Most of it is pretty mechanical, and only needs a couple key decisions, mainly your networking plugin. Generally the most popular ones have instructions as part of the getting started guide, making the process straight forward.

Where it quickly becomes difficult is after this step. You have a cluster ready to serve workloads, but it has no storage, no ingress/external load balancer.

Storage can be as simple as NFS volumes (you don't even need a provider for this, but you should use one anyway). Rook/Ceph will work, but now you've just taken on two complex technologies instead of one.

Without an external load balancer of some sort, you will have trouble getting traffic into your cluster, and it likely won't be actually HA. You can use MetalLB for this, or appliances. If you're just starting out though, you can totally get away with setting up CNAME aliases in DNS to your nodes in a round robin type fashion. It won't be HA, but it will work, and is simple and straight forward.

Ingress is pretty easy to setup for the most part. Usually just applying an available manifest with a tweak or two. If you go the CNAME route, you will need an ingress setup so you can serve http/https on standard ports without too many issues.

If you do all these things, then you have a real deal cluster. Things like ingresses are recommended even if you're running in the cloud, so you may find that you're not all that far off from what you might find there.

Overall, the biggest trouble is all the choices you need to make. If you're starting out, maybe read up on two or three of the most popular choices for each step, and then just pick one. Anything that exists entirely within the cluster can usually be expressed purely as source controlled manifests, and kubeadm deployments can be simple shell scripts if you don't make them do everything (i.e. only support one container driver, not all of them).

One major caveat; If you screw up your network layer, you basically have to start over. This isn't strictly true, but it's the one where you are often better off starting over when you need to make fundamental changes to your network setup (like podCIDR and serviceCIDR or your network plugin). Pretty much everything else can be made to work with multiple setups at once, or you just need to delete and redeploy that component.

Conceptually this is exactly what Terraform is made for. It will deploy things with the parallelism rate you specify, and is designed to neatly scale your infra up and down.

To do a rolling deployment though, it does become awkward, as you essentially have to model both the old version and the new version in your code, and then run the commands several times to scale up the new cluster and scale down the old one in stages. It's basically a modelling of a blue/green deploy.

The tooling for Terraform with bare metal is poor, but it is out there. There are plugins for MAAS, Cobbler, Some giant HPE thing I never wanted to touch.

If you're willing to write code, I found it a good fit for provisioning servers over ILO/Redfish.

Ansible is a good choice for maintaining the k8s infra itself, but IMO is not good at maintaining hardware or managing containers.

The major problem with Ansible compared to container orchestrators is that Ansible has no concept of previously known state. It makes some things that are easy in Terraform (deletions) more difficult, as you have to write a new playbook to delete stuff. So can't say, "I don't want to run this nginx container on this host anymore" by just removing that host from the part of your inventory that gets nginx, or removing nginx from that hosts playbook. You have to explicitly remove it.

This is not a huge problem if you're into immutable infrastructure, where you just wipe any host and redeploy, as it lets you go from known state to known state reliably. This kind of relgates Ansible to yaml shell scripts, but it's generally a lot tidier and gives you a nice framework for knowing where you are in your script.

When trying to apply Ansible to more infrastructure related stuff, again, it can do it, but's not the right tool. You can write a playbook to deploy a host or scale your EC2, but you basically have to do that separate from the provisioning of the hosts you just deployed.

For container orchestration, I found Ansible more of a hinderance than a help. Ansible is very imperative, and most orchestration tools are deliberately declarative. There's no particular reason to use it to deploy, scale, delete basically anything in k8s or swarm, as there is very little value add over using the tools themselves.

Overall I find Ansible best suited to building the containers or vms that you will run on your cluster, as these are necessarily imperative steps that fit it well. This also applies to the hosts that make up your cluster, if you aren't able to prepare an immutable image for them or get them to auto join.

They're not though? The interactive stuff doesn't work on an Apple TV at least.

But that was always the case? At least as late as 2001, they had weekly milk delivery in Moncton, New Brunswick. It cost the same as milk in a store. You put your order in an envelope with some cash for next week. It wasn't just milk, you could get most non-frozen dairy that way.

Is it at all the same as Amazon? No. But it shows that you can be profitable delivering low value items right to peoples doorstep.

> Cool, I thought so, I just wasn't 100% sure! I think it's interesting how different people take away different things here, to me, this demonstrates that you don't have to have namespaces; many of the largest ecosystems do not and have not had them. We did learn a lesson there, just didn't come to the same conclusion that you did. This one is more controversial within the community though, many people do agree with you.

Maven is from 2004 and has namespaces. Maven has it's problems, but the Java package management ecosystem gets many things right that later systems have not.

I don't get this in general? Is it not policy that you need to wear a shirt and shoes while in a commercial building? Do you not have signs on your stores that say 'No Shirt, No Shoes, No Service'?

How are we as a society unable to reconcile the need to wear a shirt in store with the need to wear a mask? I realize that as a society we are dumb panicky animals that are afraid of any change, but this is ridiculous!

What I find most interesting about this is that is has just as much trouble as humans in translating requirements into code. Admittedly it makes some fairly basic mistakes as well, but always in the under specified areas

i.e.:

You didn't say what colour 'welcome to my news letter' should be, so I picked white. The numbers from 1-5 are totally a possible random permutation in the 1-10 range. I find the crying laughing emoji to be really ugly, so it's the worst one.

In some sense, it doesn't. But the point of these measures is to be anti-racist. To make the environment more welcoming to people of colour. Maybe the majority of people of colour don't care, but I am sure that some do.

In my own experiences, being actively welcoming provides a level of safety that silence will not, because the status quo is that it is not safe for me to be myself, even if I might plausibly be accepted among my specific peer group.

Overall, these are small changes (what github is doing notwithstanding), and the amount of time onlookers have spent whinging about them far exceeds the time spent making the changes.

No, this won't address the most pressing systematic abuses by minorities, but it does make one of the most lucrative careers less actively hostile to them.

This is exactly right. Local storage is conceptually the persistent volume equivalent to an emptydir volume mount.

If you want to have a 'storage node' in a simple way, the NFS storage provider is the way to go. You install the nfs client libs on each node, setup an NFS share and configure and run the provisioner[1].

My experience with iSCSI is to stay the heck away from it. It is not what you want. iSCSI is really meant for people who already have iSCSI SANs and not people who have a disk they want to share. The more I learned about it, the more I learned that I should have picked something else for every use. It's not that it's bad, it's that it solves a much different problem than I expected given the networked nature of it.

[1] https://github.com/kubernetes-incubator/external-storage/tre... (I think this is the right one, been a while).

There's this https://github.com/google/zoekt. It's pretty light on features, but dang if it isn't fast and precise.

Yes! Publish a graphql schema that is parseable by apollo graphql please! I tried to use your API on my companies internal source graph setup and had to hand roll my API calls because of these errors.

> In terms of total energy use, it’s almost certain that a ten-year refrigerator should be replaced, for example.

Argh! This is such BS! I'm so sick of hearing this. My parents had a 30+ year old fridge they replaced recently. The energuide sticker on the old one said it's total annual power usage was only marginally less than their brand new one. You'll almost never make up the cost of replacing something based on power usage alone unless you're starting off really inefficient, or power costs a ton. Refrigerators have gotten better in the last 30 years, but they're not computers, they don't double in efficiency/performance every other year (which isn't true of computers anymore either). The math was probably all worked out in the 60's and it's been marginal gains from material science improvements ever since.

Most appliances aren't that complicated. Fridges are insulated boxes with a compressor and a thermocouple. If you read the reliability reports from consumer reports, you'll find that the simple (cheap) fridges all have top marks in reliability. All the fancy ones, it's all their fancy parts that break.

I have never understood why people just buy the idea that a major appliance that is 10 years old is so inefficient that you should replace it without consideration.

My 6 may not have the current OS, but it's still getting security updates, which is all I care about on a phone that is otherwise good enough still.

IMO kustomize handles that case far better than helm does and comes built into kubectl.

My wife replaced her 6 plus with an 8 because it was crashing a lot before this feature was implemented.

That's what I've been finding a lot lately. Kubenetes manifests have very complete schemas, but there's not really any official concise way to express them or relations between them.

I've come to the conclusion that helm/templating is awful. Every helm chart I've ever seen suffers from the kitchen sink problem. Dependencies don't really address the problem either, because that's not really resolving a whole lot in the end.

What I've found so far is that Kustomize strikes a solid balance between an off the shelf working config provided from official sources, and making changes myself to suit my environment. Instead of the developer maintaining a kitchen sink approach, they can have something (or a set of somethings) that are straight forward, and I can maintain a set of tweaks that work for me. Unfortunately Kustomize's documentation is not that great, and it is incredibly awkward to work with.

I'm hoping that solid operators come out for most applications instead of helm charts. I'm not 100% sure that they're going to be substantially better than helm, but I'm hoping between operators and kustomize I can link applications together in a way that I get properly maintained applications for minimal effort (ala AWS), without having to maintain a large code base linking all the bits together.

I do think I need to look into pulumi as it might be in the direction I want to go, but I remain skeptical.

Terraform has some good ideas, but is too rigid for kubernetes, and I feel like I'm giving up the flexibility of kubectl. I like the idea of having a state and resolving differences between desired and actual state, but not enough to rewrite everything in HCL.