uncreativebrian

Hey this is Brian and I wrote the article. Personally I don't love "hand waving" so I'd love to answer your question. Its a personal mission now :)

Can you help me understand what you mean by "more traditional architecture"?

I'd be happy to share more about why we chose k8s vs alternatives.

Hey this is Brian and I wrote the article. Great question.

1) MQTT -- we have a lot of cases where we want to share data between physical or "software" devices in-restaurant, without internet connections. MQTT is our primary channel for sharing messages between things, and for collecting data in general to be exfiltrated to the cloud.

2) "Brain" apps -- these are "smart" applications that collect data from MQTT topics and make decisions about what should be done for a given restaurant process. Today, the reality is we do not automatically cook anything. We do, however, have some restaurants that have more intelligent screens to help them know what to cook at any given moment based off of forecasting models that use the data we collect from many different things. These generally run at the edge to preserve their function in cases where there is high latency or bad WAN connections.

3) In the past we had a POS Server in the restaurant. It was a single point of failure for things like drive thru order taking with iPads and mobile orders. The move to K8s lets us run a microservice that can intake mobile orders on a more resilient infrastructure. To be clear, we have not made this shift just yet, but its something we are working on that will ultimately be transparent to customers but that will make their ordering experience more reliable. We have many cases like this.

Overall, our workloads aren't "heavy" which is why we did an array of three moderately sized commodity compute devices. We wanted a smart, sensible ROI on the hardware. We do have enough business-important apps that need to run in a low-latency high-availability environment to push us out of just cloud (internet dependent) and to the Edge. In the future, maybe this can change and we can use cloud only. It seems Edge is an industry trend though, with many workloads moving towards their consumers (a lot of gaming, but a lot of retail as well).

Great question and I hope this helped -- if not just let me know.

This is Brian and I wrote the article, and I am taking this as a massive compliment ;)

You hit on a great point -- Edge infrastructure buffers outbound internet traffic but does not replace the need for a good circuit.

We use K8s in the cloud as well! Our entire control plane for the IOT activity runs there. The rest of the engineering teams have a mix of solutions, but many are moving to K8s as well.

Hey this is Brian -- I wrote the article. I do want to add that Chick-fil-A isnt looking to replace jobs with "robots". We really want to assign our restaurant team members to tasks that add maximum value to customers, which are generally service oriented (iPad in Drive Thru, personal service in the dining room, etc). Our automation will help our team members be more effecient and enjoy their jobs more rather than replace them.

Hey this is Brian -- I wrote the article. It really is an interesting time, especially when you have the scale that Chick-fil-A has. While traditionally a fairly archaic industry (POS registers, ERP at corp, thats about it), the volumes and challenges of scale have created many challenges that solid engineering can help solve.

Hey, this is Brian and I wrote the article. I have worked at Chick-fil-A for over 17 years. I will volunteer time to take a phone call with anyone that is interested in hearing my first-hand experience with Chick-fil-A and its culture over the years. Just reach out on LinkedIn if you are interested and we will schedule some time.

Hey this is Brian -- I wrote the article. I have heard the term a lot over the years. I think you could call it that -- the idea being its the "foggy" area between the user and the cloud. The reason we like "micro private cloud" is that 1) its a tiny environment with limited cloud properties (no elastic scale) but 2) we do try and provide a platform with cloud-like services for our internal developers, and we like the cloud paradigms being applied to how we think about compute at the edge, in restaurants.

Hi Jay -- this is Brian (I wrote the article). There are constant tradeoffs. I am leery to say we are the most sophisticated bare metal setup. I think all of us that are close to the work would love for many parts to be much more sophisticated. That said, the way we handle distributed data right now is using Mongodb. We have a cluster that spans the three nodes that we have in a restaurant. The primary use case is persistence for our MQTT topics. Most of this data is what I will call "mildly ephemeral". It isn't needed for the long term, but we do `hope` to maintain it long enough to exfiltrate it to the cloud and deliver it to any necessary recipients. If we were to have a global cluster outage, it is not the end of the world. Over time, it probably will be. Back to work! :)

We should have a Medium post about this soon, but we built a set of tools called Fleet, Atlas, and Vessel. The simple premise is that Fleet generates code for 1..n clusters. By code I just mean yaml files. Atlas is a git repo per cluster that holds the declarative state for that cluster. At any time it should be able to be applied to the cluster and result in the ideal state. And then vessel runs in each cluster and does a (I am oversimplifying a bit) git pull and kubectl apply -f. We can use these tools to do canary deployments (generate x percent this way, y percent that way in ingress rules) across a wide range of clusters. The goal here is really to minimize configuration drift and keep a large number of fairly ubiquitous (but not always so) clusters in order.

Most retailers don't have the challenges of scale that we have and I think thats why its weird to see tech innovation in a restaurant. That said, we need it to scale and are grateful to the k8s community for making what we are working to do possible!

This is Brian -- I wrote the article. We are closed on Sunday but its not dogmatic in nature. We want to make sure that everyone at Chick-fil-A has a great work/life balance. Occasionally we need to do some work on a Sunday but we try to architect and engineer in a manner that supports business-hours-friendly deployments, or scheduled deployments over night to protect that. This is truly a first-class design consideration. In my 14+ years at Chick-fil-A, I have only worked a handful of Sundays for any reason. Its a tribute to leadership at all levels and a great company culture that values people (and their need for times to rest).

HA!!!

This was a good learning experience for our engineers at our Innovation Center at Georgia Tech in Atlanta. It may bear fruit in the form of a useful solution in the future, though. What is unique to Chick-fil-A is 'volume'. We do a lot of sales in our restaurants, so anything we can do to try and make our team members lives easier is important to us. We want them to enjoy their jobs and we want to do the best we can to consistently create high quality food experiences for our customers. Our teams in restaurants are the heroes, but we are trying to use technology to help them do what they do. <thumbs up>

This is Brian -- I wrote the article. For what its worth, we considered trying to run our clusters on an array of cell phones since they would natively support connectivity fallback and pack a real effecient resource punch in a small footprint. One of our engineers came up with that idea. With the challenges compiling on ARM and the relative ease of moving forward with x86, we went that way to start. We tried really hard to balance the "ideal" with the short term "MVP", and I think we landed in a decent spot with the NUCs. They give us enough power to run the things we need in the short term, and we have some capacity for horizontal scale in the future as our needs increase. We tried to think "cloud native" at the edge as much as we could without truly being a cloud.

This is Brian -- I wrote the article. Super cool use for a NUC! I'm going to follow the links and check out what you did some more.

This is Brian -- I wrote the article (I suspect I will be typing that a lot tonight). We do clusters with 3 nodes per restaurant, so at full scale (when we have rolled out to every restaurant in the chain) that will be ~6000 nodes, and growing by new stores * 3 going forward. This will support an estimated 100k IoT "things" of various types in the next year and a half to two years (estimate)