How We Found 7 TiB of Memory Just Sitting Around

Re:How We Found 7 TiB of Memory Just Sitting Around by stackskipton on Friday October 31, 2025 @11:28PM (#45779035)

You mentioned in the blog article that it's doing listwatch. List Watch registers with Kubernetes API that get a list of all objects AND get a notification when anything in object you have registered with changes. A bunch of Vector Pods saying "Hey, send me a notification when anything with namespaces changes" and poof goes your Memory keeping track of who needs to know what.

At this point, I wonder if instead of relying on daemonsets, you just gave every namespace a vector instance that was responsible for that namespace and pods within. ElasticSearch or whatever you pipe logging data to might not be happy with all those TCP connections.

Just my SRE brain thoughts.

[ Parent | on HN ]

Re:How We Found 7 TiB of Memory Just Sitting Around by fells on Saturday November 01, 2025 @12:14AM (#45779195)

>you just gave every namespace a vector instance that was responsible for that namespace and pods within.

Vector is a daemonset, because it needs to tail the log files on each node. A single vector per namespace might not reside on the nodes that each pod is on.

[ Parent | on HN ]

Re:How We Found 7 TiB of Memory Just Sitting Around by stackskipton on Saturday November 01, 2025 @10:54AM (#45782091)

I think DaemonSet is to reduce network load so Vector is not pulling logs files over the network.

We run Vector as Daemonset as well but we don't have a ton of namespaces. Render sounds like they have a ton of namespaces running maybe one or two pods since their customers are much smaller. This is probably much more niche setup then many users of Kubernetes.

[ Parent | on HN ]

Re:How We Found 7 TiB of Memory Just Sitting Around by ahoka on Saturday November 01, 2025 @12:04PM (#45782741)

That's where the design is wrong.

[ Parent | on HN ]