dkowalski

I needed realistic OTEL data generator for metrics, logs and traces. I also needed to speak native OTEL to a proto-compliant collector. But my issue was I need different permutations and I don't want to stand up many different containers or slurp up gigabytes of RAM. So while some the existing tools are genuinely good, they also seem to be very purpose based. So I couldn't find the flexibility I needed and decided to write a new tool.

The tool is called Snowglobe. The analogy is on an isolated system that you can "shake". It's a metaphor of the various knobs you can tweak: amount of data, amount of chaos, emit or omit AI spans. So it doesn't just do happy paths.

It simulates about 20 to 28 services because on the complexity the user chooses. I'm also trying to keep up with the always moving GenAI semantic conventions. One big part was I wanted it small so I chose Go and made it a single executable. There's a container too, that's 6.4MB.

Because I hope this is useful for people writing or dealing with APMs (consultants, developers, SEs) I opened as Apache 2.0. Disclosure: I work for Immersive Fusion, the original purpose was to light up our observability tool but I see it being useful for everyone in the industry.

One last thing, I originally called it tracegen but that name was being confused with the OTel native tool. So we renamed it for snowglobe and it's more fun to call it that.

So I'm looking for feedback from consultants, engineers and anyone dealing with OTel data ingestion and processing and visualization. I'd also like to know if this is total overlap and shouldn't exist at all.

I know plenty of senior engineers who couldn't delegate to save their life ;) There is always someone less experienced to make the metaphor work: "Imagine you are doing a group project at school with someone who's sometimes wrong but also diligent"

> Top-tier developers might still outperform AI

That's never been true. It's like saying they are outrunning cars.

Where your expertise is key is when you call out the mistakes it AI and ensuring you are not enabling slop.

In the process, you get to review your stuff you memorized and possibly learning something. Don't focus on "losing competitive edge", but instead bringing your taste to the project and your years of expertise so that the stuff written is coherent and lasting.

I agree with you that encoding yourself in a system is kind of disconcerting. But i disagree with you encoding your judgement. You have your judgement from years of work and when you see something stupid I'm sure you stop it. So your judgement is the critical part and AI will not and should not have it outright as it will almost always make the wrong/non-deterministic decision and you can't keep it accountable. I wrote a blog on this if you are interested.

I would shy away from anchoring on "taking your job". Maybe it is, i don't know you. But generally, you own the outcome. Not looking at AI output or understanding what it does is what outmodes you. Go down the path of making good use of it with owning the outcome, challenging it, rather than just trusting it.

I love writing code most of the time. If that's your fun mode, totally go for it. But otherwise, if it's saving you time or able to help you with stuff you don't actually know, I think it's very useful. I wouldn't try to race the machine for how fast code gets cranked though.

I feel the same sometimes. Then I look at it as handing it off to an intern to do the preliminary work. As long as I actually look at it sanity sake then I'm still on the loop. That opens doors for more tasks. If I want to research I can and it doesn't mean I'm dumb because I offloaded. It just means I delegated and had time for stuff I cared to do.

Hey man, you are not "pushing yourself out of a job market". You can give AI to different people and get different quality work. Think of AI as a good tool not as a replacement. You are right, a ton of startups and even grown up companies are driving/drinking hype and their stuff is just wrappers of crap under a nice wrapper that falls of the moment you use their stuff.

Honestly, the work is still yours, that's the beauty of it. Your understanding of what AI did with you, the shape it takes, the direction and the alterations you made along the way. That's what make is special stil.

I was where you are and managed to get out, reply back if you want to chat more.

Shit breaks man. But i've been there too: worn down by the hype and it all seems related to you or somewhat more relevant because it happened in close succession. I am waiting for my Dell R-whatever 1U server to kick it any day now.

Minimalist is good, reducing footprint, doing more with less is good optimization. look for a different angle because it's really the thing you can do. Reply back if you want to chat more.

this is exactly the problem we keep running into. the cost isn't just "how many tokens did this call use," its "how many tokens did this entire user action consume across all the agent loops, retries, tool calls, and embeddings."

most observability tools show you the LLM call as one flat span. you can see it cost X tokens but you cant correlate it with the API request that triggered it, or see that the agent looped 4 times because the first 3 outputs failed validation. so you end up building custom logging and hoping the numbers add up.

we've been building an APM (immersivefusion.com) where cost is a first-class dimension on every trace. so you can see one request flow from the UI through your backend through the agent workflow, and each span carries its token cost. the idea is you should be able to answer "what does a checkout cost when the recommendation agent is in the loop" without stitching together 3 different tools.

for the forecasting question specifically, i think the answer is you need a few weeks of production data with good instrumentation and then you can build a distribution. the variance is real but its not random, its usually a few specific flows that blow up (retries on bad structured output like @hkonte mentioned, or RAG queries that hit the wrong chunk size). once you can see which flows are expensive the guardrails become obvious.

also wrote a longer piece on this if anyone's interested: immersivefusion.com/blog/end-to-end-observability-from-ui-to-ai-agent-to-invoice