gregthelaw

If you'll excuse the shameless self promotion, I gave a talk at C++Now last year on how time travel debuggers work: https://www.youtube.com/watch?v=NiGzdv84iDE

(Warning: contains me trying to play Doom :)

Undo founder here. I just got a slack from one of our marketing team who is thrilled you appreciate their work. :-)

Our customers are top tier tech companies and our users are among the smartest engineers on the planet. I'm proud of our marketing, but no-one is going to spend a bunch of money with us just because of that.

So there must be something else in it I guess ;-)

Undo co-founder here. rr is indeed awesome. If it works for your use-case, you should use it!

Undo is mostly used by companies whose world is complex enough that rr doesn't work for them, and they understand how powerful time travel debugging is.

There has now been a LOT of engineering invested by a lot of very smart people into Undo, so it does also have a lot of polish and nice features.

But honestly, if rr is working for you, that's great. I'm just glad you're not doing printf debugging the whole time :)

Undo founder here. We've been at this for getting on 20 years now. Originally it cost $295 for a perpetual license. Eventually we understood that the majority of developers (actually employers of developers) will pay $0. But some are happy to pay for tooling, as long as they're confident they'll get a many multiples return-on-investment. Hence our pricing. Happily, enough do that we can run a modest but profitable business (40+ people). Customer churn is practically zero.

Why do people pay for Undo when they can get rr -- which is also really good -- for free? Those whose code or environment is big enough and complex enough that rr doesn't work for them, and they understand how powerful time travel debugging is. If rr works for you, you should use it. This includes most independent developers.

If rr can work for you and you're still not using any kind of time travel debugging, you have effective tied one hand behind your own back! If you're independent (incl student or academic) and rr doesn't work for you, get in touch -- we give free licenses for academic and certain other use cases.

There is a wider thing here about software companies paying for dev tooling. So many companies over the years who made really cool things who couldn't make their business work.

If it's been a long time I recommend taking another look. TBF you can tell it hasn't had the millions of dollars of investment that the Microsoft debuggers have, but still it's come a long way over the last 5-10 years.

e.g. it now has decent multi-process support.

I agree MI is kinda horrid, but no need for it these days, you can do everything via the Python API, and the modern equivalent is Debug Adapter Protocol which GDB claims to support now (although I haven't tried).

There a million frontends, including both Visual Studio (via ssh) and VSCode, if you like those.

The perfect developer tool does not exist, but I believe that if you're debugging native code on Linux more than a few times per year then you should really know how to drive GDB. It won't always be the best tool for the job, but it often will be.

Exactly right. Undo has "thread fuzzing" which is similar concept to chaos mode, but more targeted.

Co-founder of Undo here. This is a common misunderstanding, and just not true -- neither for Undo nor rr. Most races will reproduce at least as easily in Undo, especially if you use our "thread fuzzing" feature (rr has something similar, called chaos mode).

Sure, there will always be some races/timing issues that just won't repro under recording (Heisenberg principal and all that), but in fact most races are _more likely_ to occur under recording. Part of this is because you slow down the process being recorded, which is equivalent to speeding up the outside world.

And of course, when you do have your gnarly timing issue captured in a recording, it's usually trivial to root-cause exactly what happened. Our customers tell us that races and timing issues are a major use-case.

Congrats on this work -- writing a debugger from scratch is a big job. I have cloned the repo, will take a proper look this w/e.

I haven't tried UScope yet (I shall), but I don't agree with you about GDB. I don't find it especially buggy unless doing niche things like non-stop debugging -- I guess you may well have a different experience though.

I think the UI is much maligned unfairly. It has a few quirks, but ever used git? For the most part it's fairly consistent and well thought through.

By terrible API you mean the Python? I admit it can be a bit verbose, but gives me what I find I need.

What features do you most miss? My experience is that most people use like 1% of the features, so not sure adding more will make a big difference to most people.

I would say yes, your CI should accumulate all of those regression tests. Where I work we now have many, many thousands of regression test cases. There's a subset to be run prior to merge which runs in reasonable time, but the full CI just cycles through.

For this to work all the regression tests must be fast, and 100% reliable. It's worth it though. If the mistake was made once, unless there's a regression test to catch it, it'll be made again at some point.

But if you don't squash, doesn't this render git bisect almost useless?

I think every commit that gets merged to main should be an atomic believed-to-work thing. Not only does this make bisect way more effective, but it's a much better narrative for others to read. You should write code to be as readable by others as possible, and your git history likewise.

I've spent the past two decades working on a time travel debugger so obviously I'm massively biassed, but IMO most programmers are not nearly as proficient in the available debug tooling as they should be. Consider how long it takes to pick up a tool so that you at least have a vague understanding of what it can do, and compare to how much time a programmer spends debugging. Too many just spend hour after hour hammering out printf's.

Related: write down what you're seeing (or rather, what you _think_ you're seeing), and so with pen and paper, not the keyboard. You can type way faster than you can write, and the slowness of writing makes you think harder about what you think you know. Often you do the know the answer, you just have to tease it out. Or there are gaps in your knowledge that you hadn't clocked. After all, an assumption is something you don't realise you've made.

This also works well in conjunction with debug tooling -- the tooling gives you the raw information, writing down that information helps join the dots.

I love the "if you didn't fix it, it ain't fixed". It's too easy to convince yourself something is fixed when you haven't fully root-caused it. If you don't understand exactly how the thing your seeing manifested, papering over the cracks will only cause more pain later on.

As someone who has been working on a debugging tool (https://undo.io) for close to two decades now, I totally agree that it's just weird how little attention debugging as a whole gets. I'm somewhat encouraged to see this topic staying near the top of hacker news for as long as it has.

It's a great talk! I have stolen your "if you smell smoke, find the source" advice and put it in some of my own talks on the subject.

I have a bunch of (36 if you're counting :) short videos and blog posts introducing the advanced features of GDB: https://undo.io/resources/gdb-watchpoint/

Just yesterday I gave a talk at MeetingC++ in Berlin on debugging multithreaded code. It's amazing how few developers know anything beyond the very basic of their debugger. If all you know is print, break, continue, next and then you dismiss the debugger as "not very useful" then you've not made a judgement based on information but on initial reaction.

You need a time travelling debugger!

https://undo.io/

https://rr-project.org/

https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

Sometimes printf debugging is all you have available. But printf can be lousy for debugging race conditions because it's pretty slow and worse, libc's printf takes a lock and so introduces a synchronisation point. So I made a simple library that is both fast (~5ns on a half-decent machine) and lock-free.

It doesn't completely defeat the Heisenberg principle of course, but you can in practice add a decent amount of logging without affecting timing in a meaningful way.

Enjoy!

> Indeed, that fixes the part about it only marching forward. But still, it wouldn't be sufficient in cases where the ultimate cause is near neither the start nor the obvious error, but (e.g.) in the 50th of 100 entries.

Not true. rr (and time travel debugging in general) really excels at exactly that problem. You just put a watchpoint on the offending data at the 'obvious error', and reverse-continue. This usually takes you straight to the root-cause. It's incredibly powerful.

(Disclaimer, I'm not exactly unbiased as I work on another time-travel debugger, https://undo.io)

This is super cool.

Great to see. I'm biassed of course, but reversible debugging is the future (or should that be the past? Sorry.) If you'll excuse the shameless plug, http://undo.io/ for the same with C/C++. I'd like to think we were at least a little influential in this! :) https://vimeo.com/160863576

Greg

I'm a co-founder of the company. Lots of stability improvements over the past year or so: I'd urge you to try it again. Particularly v4 beta. Send email to [email protected] to get a beta license.

I'd never claim it's perfect: it's software after all, and the fact most software has bugs is why we have a business in the first place ;) But we're used by some pretty hardcore folk these days, including Cadence, NASA, and some of the biggest banks in the world. And the PyPy guys seem pretty happy :) Oh, and of course, if you do find issues, please do report them to [email protected] -- then we can fix them.