In the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand. Then it was reassembled in the morning. This would leave a lot of room for error and possible contamination in the process. And the sight of a shake machine hitting full pressure with a misaligned O-ring in the barrel is a lovely one indeed.
So the newer machines take a different route: they self-pasturize. The machine goes out of service and slowly heats everything up to a bacteria-killing temperature (including the dairy mix inside), then cools everything back down to freezing temperature to serve.
This process takes about four to five hours. But if you set the clock wrong, the machine will go into its cycle at some super-inconvenient time. There are other triggers that can force a self-clean but in general it's not "broken", it's just doing this process at a bad time for you.
I'm curious how this data is being sourced. The last I checked there was an effort to put cloud-based telemetry into every unit and have them report home constantly (both to the equipment maker and to McD), but that was never intended to be public data.
EDIT: Ah, I didn't RTFT. He's using the public ordering site to scrape it (and now the title has changed). But that does mean there's some working connection from the equipment to the McCloud now. That's cool. Or maybe a manager is doing it by hand.
In other cold related story, I was in a "cold country", out in the open, -25 Celsius, my phone rang, my gloves didn't have the electro-thingie so I couldn't "slide to answer" so I too my glove off, answered, talked for 60 seconds. Then I went to a warm pub, and after 10mins I could bend my fingers again.
Yes cold can .... you up badly in seconds.
TL;DR: don’t put a scarf over your nose in very cold temperatures.
I've never had a problem with a scarf freezing stiff like you describe. They build up ice crystals of course, but they don't interfere with air flow and are easily broken.
The biggest problem I expect from wearing a mask in the winter is that they'll get damp quickly because the cold temperatures, and perhaps having a scarf over top, will prevent moisture from evaporating.
In such cold climates I wear an ultra long scarf (2m-2.5m), I wrap my neck/jaw/ears 3-4-5 times (depending how tight) and I tuck it in my jacket. The outside of the scarf freezes, the inside remains warm. I also try to breathe by my nose, less humid.
Weird, as a Canadian who has been through many -35 days, I absolutely use a face covering, whether that's a scarf, neck gaiter, mask...Never had any problem like that.
The machines that I've seen don't have real-time telemetry, and are not networked.
There was an old effort a long time ago where the machines were on a powerline-networking system (LONWorks) but that is dead and buried. The next-gen effort was ramping up a few years ago and I don't really know where it is now. The predictive analytics company they bought was a big part of it.
https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at...
I would have expected some VMWare product, just because it's "enterprisey". Shame on me for judging.
Another project to keep an eye on is AWS's Greengrass which has a bunch of capability overlap, but a bunch of new ideas for edge and fog IoT also. I had a mixed experience with it but at least some ideas bear borrowing.
The 2006 timeframe does fit with the LONWorks effort. The McD implementation used....powerline networking. Man was that a kludge.
- You have a system whereby every instance has a 75%-79% uptime.
- Each instance's 21%-25% daily scheduled maintenance window can (generally) be arbitrarily-scheduled by the ops team.
If you said:
"manage a redundant cluster of at least 2 instances with non-overlapping maintenance windows"
BONUS QUESTIONS:
Q: What's the max number of ice cream machines found in a McDonald's location?
Q: What's the cluster size of the McDonald's ice cream monitoring system?
Q: How much money does McDonald's have?
Q: Why the fuck?
If you said: "1", "at least 3", "metric fuck-tons", and "I don't know", respectively, then you were correct!
Doubling the cost of the ice cream infrastructure to avoid a 25% downtime may not be a profitable venture -- especially if the downtime is scheduled to occur while the store is closed.
https://www.mcdonalds.com/us/en-us/about-us/franchising/real...
In the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand.
Not only were the internals fully disassembled and sanitized, they were left to soak in the solution overnight. All that was left behind was the cylindrical plastic chamber, sanitized and left to air dry overnight.
The messiest part was lubricating the O-rings with a petroleum jelly. Failure to remove all that when cleaning seemed like a good vector for microbes.
The worst job of all was cleaning the fryer vats.
Did you ever have a clever employee that thought they could use the 5-gallon plastic buckets as a quicker way to dispose of the old fryer shortening? I watched that happen one Sunday morning. That was amazing.
I yelped and stepped back. A manager rushed over to me, slammed the lid shut, and told me to not tell anyone.
I quit a couple weeks later.
Currently it's early afternoon and the site says that over 8% are out.
That's about halfway between "all overnight" and "all at completely random times".
Are almost half of the clocks set wrong? Assuming that the actual number of broken machines is relatively small.
I'm not sure what material it's on to keep it heated. But something like coal I assume.
Moreover the $750 you reference must be a commercial product since you can do GPS on a Raspberry Pi for < $50 easily? Which product are you thinking works for this scenario? Think I've seen everything from $300 to $13000 solutions for this over the years.
(...although in fairness some of the other comments here suggest the things might already have some kind of int[er/ra]net connectivity in which case no extra hardware required)
It's actually probably best they never touch the public wifi. I don't know why no one has mentioned ethernet yet, though. These machines are static, they just sit there. There's already tons of wiring going everywhere, so I would assume that it should be relatively easy to get a cable drop to the ice cream machine. Then you don't have to worry about any of this wifi security. You could do port level security, but that's probably overkill for an edge network.
I have actually been wondering lately why there isn't more of a market for some of these things that it seems like you could hack together. There has to be something I'm missing. You'll never be a unicorn, because most of these kinds of sensors have a limited scale, but it seems like a good return on investment for a small team.
WWV might be a better system, but runs you into time zone issues. Do you need a valid SIM to pickup time from 3G?
[1] https://www.etsi.org/deliver/etsi_ts/136300_136399/136331/11...
You used to be able to pick up a time signal almost anywhere in CONUS with an FM receiver. It was encoded in the transmissions of PBS television stations.
This was back in NTSC days. Now that everything is digital, I don't know if it's still true.
A better source today would be the time codes sent out by some FM radio stations with RDS encoders, to allow car radios to set their clocks. Unfortunately not that many radio stations do this and, once again, the time is not always all that reliable since the RDS encoder may not have any synchronization source itself.
The cellular network used to be an excellent source, CDMA cells required GPS time sync for TDMA reasons (well, CDMA reasons, technically speaking...) and broadcast a time code that is directly off of their GPS time source. Unfortunately, while GSM cells (and LTE) do broadcast the time, there is no guarantee made of precise synchronization as they don't broadcast a time code directly from their GPS source (not an expert in this field but I think the GSM/LTE time information comes from the possibly remote controller rather than the local radio hardware). Still, it would probably be good enough for this application.
GPS time sync is actually quite cheap to implement these days but tends not to work in these scenarios since a clear sky view is needed. WWVB is possibly on the way out. SNTP is probably ruled out less by the BOM cost of WiFi hardware and more by the deployment pain of having to get kitchen equipment configured for the corporate WiFi network.
Here's a fun idea: McDonalds presumably centrally controls the in-restaurant audio. Could they encode timestamps into the background music in a way that machines can cheaply recover? You wouldn't need high reliability, just enough for it to work once in a while. The old-ish Nielsen Peoplemeter system would be a model.
https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at...
That’s an awesome way to name their cloud computing stuff that fits the brand really well. Is this an official/semiofficial name?
Jokes are usually frowned upon here, but that's a good one. Have an upvote from me.
#1 hated task was cleaning the broiler.
#2 cleaning the shake machine
It probably sounds insane, but someone from a plastics company spoke at my high school and said this was the case since it was easier to ship and needed no refrigeration. Now, decades later, I'm wondering if this was just complete nonsense or had an element of truth to it. I've been able to find nothing online.
Basically take anything homogeneous, add gum, add air, create a foam. Basically like an Aero bar but with way smaller bubbles.
You just cut a quarter (I'm guessing on this figure) off your ice-cream ingredients by weight and you can now advertise "scoops from the freezer"!
It's genius but just a more complex version of putting all the pizza toppings in the little window, or having the coleslaw tub be much wider at the top, or putting everything in a wide plastic skirt so the box is 20% bigger than it needs to be, ... packaged food sellers are scumbags.
And while I think the whipped foods are stupid, I don't think that packaged food sellers are necessarily scumbags. They're merely people responding to incentives, just like everyone else. The food market has razor-thin margins and consumers are ill-informed. Neither consumers (who are hard to inform) nor regulators (who are captive to the multinational conglomerates) have been able to push back more strongly than the profit incentive, so the so dark patterns like those you describe are inevitable.
So these whipped products seem pretty useful.
My kids eat a lot of creamcheese sandwiches. Block cream cheese just destroys sliced bread.
The products in that category that have been puffed up with air (and don't meet this minimum) get sold as "dairy desserts".
(I don't work on the food industry, so I won't be shocked if I got details wrong.)
https://www.youtube.com/watch?v=oR3Z028prSU
https://www.youtube.com/watch?v=k8t7u5mupS4
I don't think I'd want to eat that
There is no way i would eat something linke that.
These days the trend seems to be to reduce the packaging, at least here in the UK. "Same size - less packaging!". So it seems like reduced packaging and distribution costs, and perhaps fitting more onto store shelves, outweighs any extra sales from customers who are fooled into thinking the product is bigger than it really is?
I'm not a chemist but I've recently come to realize that the meaning of the term in the popular psyche has probably changed over the decades, taking on largely negative connotations, because of its association with a small subset of materials (albeit large by volume), most of which are either toxic, a pollution problem, or both.
AFAIU, in industry "plastics" is just a blanket term for moldable organic materials that chemists are constantly inventing. They're not necessarily sourced from petroleum, nor necessarily toxic. It's a ridiculously huge category of material, and plastics really were and probably always will figure prominently in our future. And I would assume that some traditional, even edible materials have been subsumed into the plastics category given the general scope of the term and the industry itself.
https://www.eater.com/2016/8/8/12403338/subway-yoga-mat-chem...
https://www.npr.org/sections/thesalt/2014/03/06/286886095/al...
Sodium carbonate is sometimes used as an alternative to lye for those who don't want to deal with the safety aspect of handling lye, as lye is very caustic. Of course, lye still provides the best results.
Lye-pickled fish is also a thing (lutefisk)!
Logically though, refrigeration based transport is not a concern for any fast food company. They use ingredients that absolutely must be transported cold, thus the cost of adding another menu item that must be shipped cold to that list is immaterial (even at scale).
Mcflurry ingredients: https://www.mcdonalds.com/us/en-us/product/mcflurry-with-ore... (no mention of plastic)
That's actually incorrect, they list two plastics there. It's just that "plastic" isn't something that is actually necessarily toxic, and the media misuse the terminology all the time.
The two are:
> Cellulose Gum, Guar Gum
Next up, what about redundancy, do stores only have one machine?
But to answer the question: no, the machines aren't net enabled and do not NNTP their clocks. But that's changing.
But I still wouldn't design my architecture to let a central server decide when a machine should go out of service. Think about what else you can fuck up that way.
Why are they off? Good question. Sometimes it's as simple as a store owner setting the heat cycle for 2pm instead of 2am.
Why are they not redundant? They're expensive, take up a lot of valuable space and a lot more valuable energy, they need to be filled with 2x the product, and the customer demand isn't that high at any given moment to warrant the expense in most stores.
From my (old) experience working at one, yes. The annoyance of having to maintain a second machine (especially when cleaning and maintenance involved a physical person cleaning things) was not worth keeping extra 9's of uptime.
/ducks
Don't you mean NTP? We've had this technology for years! If it's connected to the internet, it's a pretty fundamental piece of pretty much any networked OS. You can ensure correct time this way.
If you have a datacenter full of computers, obviously you don't want to run a separate GPS antenna for every system. NTP allows systems to synchronize to GPS over a network.
If a fast food worker at a better paying, better run chain or restaurant said it, it would be still a little mean (as intended), but not kicking down.
Every time McDs changes CEOs I feel bad because that means almost every current project is going to be shitcanned as the new person tries to make their mark. Are you still there now?
It's funny that the machine tells them there's no ice cream. I visited my local library for a tour. They have an automated system that collects returned books, sorts them, scans and processes them, then puts them in boxes to be shipped to other libraries if they need to go there. Or puts them in carts and labels them with instructions about the efficient path to travel to return books...
The humans just take orders from the machine when there is a cart ready with books to be put on the shelf or boxes to be sent to other libraries.
Interesting times.
They would enter their self-pasteurisation cycle, and the display at the front would tell you that is what it was doing, but it wouldn't prevent you from pulling the handle, if you so wished.
The problem, though, was that the barrel of the machine where the mixture was normally frozen was now at some 60 degrees C, it would be much much higher pressure than usual, and since the entire staff of a McDonald's tends to consist of teens who don't exactly take pride in the precision in which they carry out their jobs, at least once a night you'd have some poor kid blast a fine mist of hot sugary milk over themselves and the surroundings. Sometimes twice.
Sometimes I miss working at McDonald's.
Depends on the country, here it is hot 9 months of the year. People queue up for ice cream every day. McDonald's has mini stores only selling ice cream.
Maybe it's the same in hotter parts of the US (which the article is about), for example Florida.
The latter implies continuance of service, the former leaves that open.
When I worked that McDonalds the machine was almost always down simply because the night crew didn't give a shit and it was too much effort to clean. I worked mornings with the old folks because I was too uncool for the night shift.
Even though it's a valid distinction, I'm not sure it's one that's valuable to the interaction being described.
As these kinds of places attract the pettiest and most annoying customers anything else would make the customers try to bargain with you, even for stupid crap like that.
The flip side is that when you have a non-standard request to a business (like a bank) or a government agency, it used to be that a low-level clerk would have some workarounds, or you could escalate to a manager. Now, they all have a convenient excuse: "the system doesn't let me do that". And it's true, because it's all SaaS software with inflexible business rules, controlled by some central team that neither you nor the clerk will ever be able to reach.
People have said "the phone lines are down" for years, meaning landline phone service was temporarily unavailable (often when no actual physical cables were knocked down off of poles, etc, though that might be where the use of the term originated in that case).
I'm sure there are plenty of other examples of using 'down' to mean the same kind of thing (all of which predated computer use being common in a regular person's life).
Like hang up, when was the last time I hung up a phone, that style of phone went out of fashion here 30 or more years ago.
Server looked confused - but which burger did I want? It became apparent that while I could ask for a preset item and ask for removals, I could not actually go fully off menu. So I had to ask for "the cheese burger, without the cheese" (as somehow distinct to say, the bacon burger, without the bacon). I laughed at the ludicrousness of that, but now I see they were channeling Sartre! In retrospect it would have been a truly beautiful absurdity if see had come back and said "sorry, we have no cheese, do you want no bacon?"
Some people would look down on a franchise or a restaurant or a business that had super high failure rates. Like if the comedy section of Netflix went down every 3 times you tried to visit that section. As opposed to Netflix having some system level limitation requiring them to perform some magical dance and perform some spell every few days to keep the server room from exploding.
You would perhaps give Netflix more leeway and not feel as much disdain and switch to a competitor if you knew that the comedy section being down every once in a while was some legitimate limit the engineers had found and not just a shoddy organization. But if you just got some nginx 500 error page randomly you would be like 'wow this site is hot garbage'.
I'd be willing to bet that "broken" is just less likely to start a fight with an enraged customer whose day was very slightly inconvenienced by their inability to consume extra junk food.
Former Blue Bell Creamery CEO was recently charged with seven counts of conspiracy and fraud for not taking proper actions in regards to ice cream based listeria in 2015.
I know very little about food science so apologies in advance but curious as why they don’t add preservatives themselves to the Ice cream and why other pH neutral foods McDonalds do don’t require as much care as the ice cream?
As for duration, that's a function of temperature and desired safety margin. The higher temp you go to sanitize, the faster you can sanitize, but if they're heating the ice cream mix as well, there's likely a limit to how high they can get without damaging/altering the mix. So the other variable is time, and that's probably what leads to the multi-hour sanitize cycle.
How come the the high amount of sugar in ice cream doesn't help?
Would it be any different if it was a non-dairy milk like soy?
You need for there to be little enough water in the solution that cells with permeable membranes will be desiccated through osmosis. This can be seen in a common household science experiment, where you remove the outer shell of an egg by leaving it in vinegar for a while, then submerge it in corn syrup to watch its further transformation.