Re:Edge Computing at Chick-fil-A
by jacques_chester on Wednesday August 22, 2018 @08:17PM
(#17823446)
Wouldn't Spinnaker be suitable for something like this?
Re:Edge Computing at Chick-fil-A
by danielodio on Wednesday August 22, 2018 @11:28PM
(#17824336)
Caleb, we’re Armory, W17. This does indeed sound like an interesting use case for Spinnaker, as Jacques said. Happy to chat with you about it. [email protected] (we are commercializing Spinnaker — http://www.Armory.io )
Re:Edge Computing at Chick-fil-A
by jacques_chester on Thursday August 23, 2018 @07:17PM
(#17831389)
Your blog post turned up in slack today (congrats). My particular flavour is Concourse but I nodded all the way through.
Re:Edge Computing at Chick-fil-A
by swampthinker on Wednesday August 22, 2018 @05:26PM
(#17822335)
What were the main issues you faced with Helm?
Re:Edge Computing at Chick-fil-A
by coffeesn0b on Wednesday August 22, 2018 @05:50PM
(#17822528)
The primary challenge was just reasoning with the template and using Helm at scale... ie; what exactly did we deploy on those hundreds of varying clusters?
Other issues included; tiller would sometimes become unstable... version mismatch issues between helm local and roller... lack of a clear, outage free canary deployment... we even found cases where helm would not cleanup after itself during a deployment and retain previous config settings within k8s.
For me, Helm caused more problems than it solved. Pulling in packages always seems good, but as soon as you want customization, you're back to merging in the (relatively) straight-forward yaml files from the chart. Also, instead of Helm's templates (which get crazy complex and unintuitive), a simple tool like Kustomize[0] is very straight-forward and allows per-environment configuration. Finally, tiller pods do present (yet another) security risk for Kubernetes.
Each individual restaurant gets it's own cluster (we aren't fully deployed yet). There are too many network latency challenges and too much immaturity around federated clusters to take the route (unfortunately).
We currently use gitops, which houses all the configs per cluster in a single repo per restaurant (CRAZY amount of corgis!)... we call that Atlas (the repos) and we made a little pod called Vessel that polls and applies configs in those repos.
We're almost done building something called Fleet that will generated and manage those repos (Atlas) at scale. Ie; a UI where we can say "send this version to 10% of restaurants" and it will regenerate and deploy those configs to all the appropriate Atlas repos, which will then get pulled down by Vessel.
We tried doing all this with Helm but failed miserably. Maybe it was us? But templating vs gitops... the choice seemed obvious.