If something is so hard to implement that everyone who tries gets it wrong (to a first approximation), then maybe the concept is bad. Or, at least, the concept isn't fully baked and is missing something critical.

Most implementations of toasts-with-actions that I've seen don't have the problem OP described. I more often find myself manually dismissing them than wishing they'd have stuck around longer.

So because you are a fast reader and have a fast reaction time with your mouse, toasts are good for everyone? No.