klaas-

> This investigation is a useful reminder: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well-tested and reliable. Most people use SQLite in a standard configuration and never face this sort of issue. Everything we were doing was a public, documented, supported configuration—but by taking manual control of the checkpointing process and running at our own aggressive pace, we stepped off the well-trodden operational path.

I feel like they missed a key takeaway from their own argument here, they should not be running a non-standard configuration :)

Seems there are more people interested in this: https://learn.microsoft.com/en-us/answers/questions/5962465/... https://www.reddit.com/r/sysadmin/comments/1ve63cg/azure_vm_...

Tracking ID in MSFT is DZ_H-21Z

I would be interested in a severity score or something similar -- otherwise I can't judge how important it is to fix -- is this a "friday right now" problem or a monday is fine problem ;)

from a company (RHEL users) POV: EntraID support -- we're using https://learn.microsoft.com/en-us/entra/identity/devices/how... for SSH at the moment (not really happy with it but I don't have a better alternate) -- and that does not work for cockpit. So another way to authenticate against entra would be nice for corporate users

maybe I missed that, but why not just combine ptp and ntp within chrony, it does support that.

I would guess nextcloud

I think a very short lived cert (like 7 days) could be a problem on renewal errors/failures that don't self correct but need manual intervention.

What will let's encrypt be like with 7day certs? Will it renew them every day(6 day reaction time), or every 3 days (4 days reaction time). Not every org is suited with 24/7 staffing, some people go on holidays, some public holidays extend to long weekends etc :). I would argue that it would be a good idea to give people a full week to react to renewal problems. That seems impossible for short lived certs.

I'd say the primary problem of framework is security updates for bios etc

yeah, that was my suspicion as well, seems that AI generated content is mixed with seo-spam or malware. I even tried to report feedback.azure.com as a deceptive site to the major browsers, but they don't share my concerns ;)

I think Microsoft has a general problem with getting rid of unwanted things within their eco-system. I keep complaining that their feedback.azure.com portal is filled with spam/malware comments and links, but even internally their teams can't reach anyone to get it fixed. Example https://feedback.azure.com/d365community/idea/9d0b22d8-c025-...

any of those switches in the right price range for home? I've been looking for a cheap SONiC switch for a long time but those on the supported hardware list are all rack/datacenter sized I would say -- and none do PoE which I really need for home usage (cameras, wifi APs)

I have a Framework 13 (12th Gen). I have no complaints about the hardware quality. I love the replace everything concept. I rarely do that though, my ports are always the same - 2x usb-c, 1x usb-a, nic (bulky I remove that one for traveling and replace it with another usb-c).

Having said all that, I would not buy another one right now. The software side of things is just not up to what I expect. They still have not figured out lvfs for this notebook; the last bios update I could install was a manual EFI installer (which is beta at this point it in time I guess, only really "production" ready update is windows based installer). Also the bios I have is now a year old, and there only has been one non-beta bios upgrade in the whole lifetime of this model (~2.5 years). So the software side of things is just miserable... At this point in time I guess the next machine will be something with coreboot+lvfs and regular, timely updates (still taking suggestions ...). I guess I would be willing to give a framework+coreboot situation another try in a couple of years. The main reason for buying this is reusablilty once I decide to move on to the next hardware generation.

yesterday they were also serving a update catalog index that did not match it's signature https://downloads.dell.com/catalog/CatalogIndex.gz // https://downloads.dell.com/catalog/CatalogIndex.gz -- but that was fixed after I complained

and their idrac based firmware updater downloads http(s)://downloads.dell.com/Catalog/Catalog.xml.gz without checking the signature -- and by default without verifying https certificates when using https :D

there is something similar for WD drives as well.

https://github.com/not-a-feature/wd_ssd_firmware_update/ https://github.com/not-a-feature/wd_fw_update

I really hope some day there will be a big disk manufacturer that natively supports lvfs - I would pay a premium for that.

they do talk about actual usage in the talk, but only vcpu wise, not instance wise. For vcpus it's also more than 60% linux

the actual talk (video in article) says more than 60% of VM vcpus are from linux VMs, it does not talk about instances though. But I would expect Linux to be a lot of very small and very large VMs :)

so the "real" azure nodes are some kind of special Windows+HyperV but for example their accelerator cards are linuxes: https://cfp.all-systems-go.io/media/all-systems-go-2023/subm...

you can use azure files nfs, that should be posix compliant, but it comes without auth or encryption ... :)

And that windows is backed by Linux if performance is key: https://cfp.all-systems-go.io/media/all-systems-go-2023/subm...

I also have a follow up to this topic. I would say it looks like in Azure nobody is verifying TCP checksums. A bitflip in tcp payload should have been noticed at the NIC/tcp level but it reaches my application with corrupted data.

after a full day of searching for a possible cause, someone from microsoft suggested that they try to move the express route gateways (which are seemingly VMs inside of Azure) to a different hypervisor. For now it seems like the first gateway was flipping bits and on new hardware it is running normal again... That was a very interesting day troubleshooting wise

so my guess would be there are more than a few devices between the edge and the express route gateway (which is more or less a VM I was told). So I am guessing right now someone is trying to figure out what hardware is involved in between the two points we captured and looking at that stuff :) For me as a customer that's just a big black box (cloud)

yeah, that's what we did. We captured it at the last place in our network (a router) and then compared the tcp payloads to the packets that we received in azure in our firewall. Next step was to get the dumps from as much inbetween as possible, it seems that is only the edge router of Azure and our express route gateway. After looking at the pcap of the azure edge myself I verified that the package is still okay there, the MS support verified it's broken in the express route gateway, so it has to be flipped somewhere inside their Azure Germany network stack. They are still searching where ... :)

So it seems this is their software defined network, the packets still look good when entering via the microsoft edge routers and they get bitflips on their way into our express route

you can also send mails with curl via cli https://everything.curl.dev/usingcurl/smtp