Re:The four pillars of data observability: metrics, metadata...
by kzh_ on Sunday August 14, 2022 @01:56PM
(#32461350)
+1 on existing log viewers being particularly well suited for text over non-textual assets. My experience here is limited but I believe Grafana has a dynamic image plugin if you store a link to an asset in blob storage or Base64 encode it.
I've also heard of people storing those links in a database like Snowflake then creating displays on top using Tableau or Looker, to avoid having to build a web app from scratch.
Re:The four pillars of data observability: metrics, metadata...
by nerdponx on Sunday August 14, 2022 @01:44PM
(#32461252)
It might be interesting to have something like "statistical logging", which saves the intermediate image files 1% of the time and discards them after 30 days.
Re:The four pillars of data observability: metrics, metadata...
by pm90 on Sunday August 14, 2022 @04:04PM
(#32462440)
Maybe wild idea: generate a unique text identifier for the image + an s3 url, log that identifier rather than the image. I guess its logging metadata rather than the actual data.
Re:The four pillars of data observability: metrics, metadata...
by SnowHill9902 on Sunday August 14, 2022 @04:07PM
(#32462471)
- unused memory is wasted, you may be able to store the raw image.
- if your process is deterministic store a hash.
- store a low resolution image.
For example an image processing pipeline. You don't always want to log (it'd never scale) but as part of a trace you might want to keep the intermediate files so you can track down where the problem is. You've already got text logging for each step, recording metrics like duration and which filters were involved. I have saved files and referenced them in the logfile, but no log viewers I've seen understand anything beyond text. So I then have to build my own UI or open the images in turn.
Is there a pattern to handle this?