Re:Debugging random slow writes in PostgreSQL
by sa46 on Saturday May 15, 2021 @04:33AM
(#27163470)
This seems like an instance where BPF could shine. If I had a large magic wand, I’d love to have distributed traces of database query execution with spans or events for significant events like the GIN pending list cleanup described in the article.
[ Parent
| on HN ]
Re:Debugging random slow writes in PostgreSQL
by mulander on Saturday May 15, 2021 @08:32AM
(#27164617)
Check the slides/talk from my comment here elsewhere in this thread: https://news.ycombinator.com/item?id=27163455
We did use DTrace to diagnose what was the root cause of a hanging insert (it was the same pending list cleanup as on this blog post). The whole talk we did was about DTracing PostgreSQL :)
Re:Debugging random slow writes in PostgreSQL
by mettamage on Saturday May 15, 2021 @04:45AM
(#27163526)
After reading a couple of chapters in Brendan Gregg’s book I tend to agree.
Re:Debugging random slow writes in PostgreSQL
by bonobocop on Saturday May 15, 2021 @06:36AM
(#27164037)
I used BPF (using _both_ of Brendan Gregg's recent books!) on our Jenkins builds recently to figure out why `TRUNCATE` statements were taking longer than I expected.
The underlying cause was that `ext4` filesystem was journalled and the `fsync`s were waiting on `jbd2_log_wait_commit`, which offcpu sampling let me pick up.
I don't think I would have been able to trace the kernel call and link it all the way back to the application call without BPF (at least I wouldn't have been able to).
Re:Debugging random slow writes in PostgreSQL
by mettamage on Saturday May 15, 2021 @07:52AM
(#27164399)
Thanks for this story! I haven’t seen many practical examples yet.
Fun fun :)
Re:Debugging random slow writes in PostgreSQL
by avinassh on Sunday May 16, 2021 @03:38AM
(#27171940)
which book?