News for hackers. Stuff that matters.
Re:Don't trust default timeouts by asdfasgasdgasdg on Saturday August 29, 2020 @09:27AM (#24315916)
I think the opposite. Use infinite timeouts for outbound calls. If you're an interactive application, display progress/activity to the user. Allow the user to manually cancel. If you're a server application or system, maintain an operation wide timeout and, if you do time out, propagate cancellation.

Systems I work with that have a default timeout are a pita. You end up having to make pointless retries when you'd have been happy to wait.

There are exceptions. If cancelling and retrying is has a decent chance of routing around the original problem. In that case, a timeout makes total sense. The other case is if you have a workload where operations tie up a mixed set of resources (e.g. threads + blocked backend calls) and only some of your incoming ops are dependent on the blocked resource. In that case, timeouts make sense in that they at least allow you to make forward progress on the unblocked requests. Although tbh separate queues and thread pools is the safer way to handle this. Because your caller with the timed out calls is gonna keep retrying and eventually these retries will crowd out the requests that can make progress in your incoming request mix.

Re:Don't trust default timeouts by jasonhansel on Saturday August 29, 2020 @12:25PM (#24317211)
I think the problem is partly that many programming languages make it difficult to propagate cancellation correctly and idiomatically. So people end up adding timeouts to individual network requests instead of to the operation as a whole.
Re:Don't trust default timeouts by networkimprov on Saturday August 29, 2020 @01:03PM (#24317537)
Go, Rust, and Zig make it impossible to interrupt most blocking syscalls, because they automatically retry on EINTR. [1]

Go supports timeouts on network & file read/write ops, which can be used to interrupt them.

[1] https://github.com/golang/go/issues/41054

Re:Don't trust default timeouts by Arnavion on Saturday August 29, 2020 @02:52PM (#24318512)
>Go, Rust, and Zig make it impossible to interrupt most blocking syscalls, because they automatically retry on EINTR.

This is not true of Rust. Some of the convenience wrappers (std::io::Read::read_exact, etc) on top of the basic primitives (std::io::Read::read, etc) do retry for you (and explicitly document it), but not "Rust" as a whole. The primitives map one-to-one to calls of read/write/sendto/recvfrom and bubble up ErrorKind::Interrupted to the caller just fine.

Re:Don't trust default timeouts by networkimprov on Saturday August 29, 2020 @03:33PM (#24318810)
The Rust filesystem API works (or once did) as I described. This can render an application unusable when trying a network filesystem or storage device that's unavailable.

https://github.com/rust-lang/rust/issues/11214

To support user intervention when a task takes longer than expected, all blocking syscalls should be interruptible.

Re:Don't trust default timeouts by Arnavion on Saturday August 29, 2020 @03:40PM (#24318860)
Linking to discussions from pre-1.0 does not help your assertion.

std::fs::File's impl of std::io::Read:

https://github.com/rust-lang/rust/blob/7fc048f0712ba515ca11f...

-> https://github.com/rust-lang/rust/blob/7fc048f0712ba515ca11f...

-> https://github.com/rust-lang/rust/blob/7fc048f0712ba515ca11f...

Again, as I said, it corresponds one-to-one with a call to the underlying read API. The retries for ErrorKind::Interrupted are done by higher abstractions like std::io::Read::read_exact, and they explicitly document that they do this.

https://github.com/rust-lang/rust/blob/7fc048f0712ba515ca11f...

Re:Don't trust default timeouts by networkimprov on Saturday August 29, 2020 @04:04PM (#24319022)
Apologies if I got this wrong, but you've linked read(), and I'm referring to mkdir() et al.

Have they taken out the EINTR retries which were added for the issue I linked?

Re:Don't trust default timeouts by cyphar on Saturday August 29, 2020 @10:14PM (#24320807)
Go only recently started handling -EINTR correctly (retrying the operation) though I do agree that this should've only been done in the higher-level wrappers. The "os" package shouldn't be doing retries IMHO.

Pre-1.14 -EINTRs were quite rare in "normal" Go programs so the stdlib basically ignored them, but 1.14 introduced preemption which resulted in many more -EINTRs and quite a few Go programs were broken as a result. So in many ways this behaviour was necessary to un-break backwards compatibility. If Go had made the interruption semantics -- which had existed for at least a decade before Go came about -- clearer from the outset then maybe this whole business could've been avoided.

This is symptomatic of the reasons why container runtimes (at least, those written in Go) have historically been vary wary of Go updates. Several years ago, each Go release would change some minor semantics of the Go runtime and cause breakages...

Re:Don't trust default timeouts by kijin on Saturday August 29, 2020 @09:58AM (#24316137)
I prefer the opposite, because of the reality that people are lazy. Increasing the timeout often leads to a programming habit where you just turn a blind eye to potential bottlenecks and/or cross your fingers and hope that the network will always be available.

Enforce short timeouts, preferably less than 3 seconds, and definitively no longer than you expect the user to have patience for, if the operation is part of an interactive workflow. Any task that has a legitimate reason to take longer gets pushed to a background process or cron job. This makes timeouts on the frontend a regular occurrence, so you'll be forced to handle them just like any other error condition. Result: a more robust program.

But this probably depends on what kind of program you're building. Most of the stuff I build and support are consumer-facing, so anything that isn't instantaneous is cause for concern. Other types of applications, though, might have more patient users.

Re:Don't trust default timeouts by cortesoft on Saturday August 29, 2020 @02:15PM (#24318196)
The problem is this will make your site unusable for people on slow connections. Some people have internet connections that take seconds to connect, and nothing the site does can make that faster.
Re:Don't trust default timeouts by Roboprog on Saturday August 29, 2020 @04:59PM (#24319395)
Those users don’t get past our single sign on app in the first place. The “ticket granting ticket” used to give out sessions in individual apps expires after a few seconds, presumably to discourage various kinds of attacks.

Not every app is forced to support terrible connections. We definitely had issues with hung operations until we had a timeout, though there was some disagreement about failing “fast” (10 seconds is “fast”???) vs hanging inexplicably for much longer, but not forever, periods of time.

Edit: when REST calls occasionally failed after 10 seconds, rather than hanging the UI, the support calls stopped. 2 or 3 people a day had to hit submit a second time. They got over it, as opposed to reloading the app/page. This was the most cost effective way for us to handle this.

Re:Don't trust default timeouts by hderms on Saturday August 29, 2020 @10:25AM (#24316317)
Most services of some kind have some form of resource contention at play. Even the simplest CRUD app likely has a DB connection pool to deal with. Imagine some external service isnt responding for some non-tricial amount of time and your requests are holding open DB connections as a result, you could very easily cause a total availability loss for that service.

Timing out would at least let you, for instance, flip a circuit breaker off or fail fast and have the resulting monitoring very specifically tied to the actual problem in the system, not to mention avoiding resource contention issues like I mentioned.

Re:Don't trust default timeouts by magicalhippo on Saturday August 29, 2020 @09:38AM (#24315985)
> Use infinite timeouts for outbound calls.

Don't use infinite timeouts if you don't have a another way to cancel the operation.

Re:Don't trust default timeouts by asdfasgasdgasdg on Saturday August 29, 2020 @09:43AM (#24316031)
Counterpoint: always have another way to cancel the operation.
Re:Don't trust default timeouts by RhysU on Saturday August 29, 2020 @10:01AM (#24316155)
Not retrying is implicitly a way to cancel. Also, it is a codepath that's trivially tested vs the effort to test an extra cancel path.
Re:Don't trust default timeouts by asdfasgasdgasdg on Saturday August 29, 2020 @10:17AM (#24316263)
I'm not sure how testing timeouts is trivial compared to cancellation. They both take about the same amount of code to write a test for, IME. (Not much.)

Not retrying+timeouts has similar effects to cancellation. The operation ceases to go forward. But it is not the same. It's a lot more expensive than imperative cancellation (need to rebuild, resend, reparse the request) and it has a lot of production risks that waiting with cancellation doesn't. For example, naive retries can expose backends to thundering herds, and less naive retries can have strange issues caused by exponential backoff where you'll have requests sitting around doing nothing for half their own timeout, before giving up because the next retry did not hit before the end of the parent request's timeout.

Re:Don't trust default timeouts by RhysU on Saturday August 29, 2020 @02:35PM (#24318362)
All good points. By trivial I meant 2 tests (works/fails) vs 3+ (works/fails/cancel with the latter possibility having its own works/fails cases). A timeout is just a status code on failure.
Re:Don't trust default timeouts by marcosdumay on Saturday August 29, 2020 @10:16AM (#24316257)
Yet, if you are making an interactive application, that easy to test codepath is a great way to put bugs into your requirements.
Re:Don't trust default timeouts by orisho on Saturday August 29, 2020 @10:40AM (#24316416)
In Go, it is a single code path. Contexts can be canceled and they also come with propagating timeouts. The timeouts simply trigger a cancellation, so the only code path is handling cancellation.

There's nothing complicated about it, so there's no reason your code can't implement timeouts and cancellation the same way: timeouts are a cancellation triggered autonomously after some time passes.

Re:Don't trust default timeouts by RhysU on Saturday August 29, 2020 @02:37PM (#24318389)
Not disagreeing, but having trouble seeing it. Could you elaborate?
Re:Don't trust default timeouts by marcosdumay on Saturday August 29, 2020 @03:21PM (#24318730)
By adding that timeout you just created a user-visible behavior that nobody asked for and people will only notice in production while dealing with the most complicated use-cases.
Re:Don't trust default timeouts by RhysU on Saturday August 29, 2020 @03:46PM (#24318896)
Nobody asks for it but some choice must be made. As a user, I have often cursed things that hang indefinitely. And I don't trust application state after touching a cancel button. That stuff is seldom tested well.
Re:Don't trust default timeouts by megous on Saturday August 29, 2020 @02:45PM (#24318451)
Well, not always possible. For example the latest systemd has a bug where it sometimes deadlocks in a PAM module, so it blocks all remote access to a machine over ssh (openssh uses PAM, optionally). If openssh had a timeout on the PAM child process, it would simply retry after timeout, instead the whole machine is lost and needs to be restarted with physical access.

There's no way to cancel the operation remotely, because you're not authenticated yet. And you may not have any other access.

Timeouts are also a good defense strategy against bugs.

Re:Don't trust default timeouts by magicalhippo on Saturday August 29, 2020 @10:30AM (#24316348)
This requires the API you're using to support this. If the API doesn't, then using infinite timeout is a bad idea.
Re:Don't trust default timeouts by asdfasgasdgasdg on Saturday August 29, 2020 @10:37AM (#24316387)
Of course. I did not mean to convey that timeouts should be avoided in all cases. In fact I listed several such cases where they should be used. An API that has no way to cancel would be another example. Although I would argue that such an API is fundamentally flawed.
Re:Don't trust default timeouts by magicalhippo on Saturday August 29, 2020 @10:44AM (#24316432)
Yeah I just wanted to highlight it, as I've see far too much code passing INFINITE to WaitForSingleObject or similar.

And yeah, not having another way of cancelling is not nice, but sadly not entirely uncommon.

Re:Don't trust default timeouts by asdfasgasdgasdg on Saturday August 29, 2020 @10:49AM (#24316468)
Right I think the suggestion in that case would be to upgrade to an API that does support cancellation wherever possible. E.g. wait for multiple objects with the original argument and an additional cancel event.
Re:Don't trust default timeouts by tlb on Saturday August 29, 2020 @01:08PM (#24317587)
As a user, I prefer short timeouts that pop up an error message with a 'retry' button.

You need the retry button anyway, in case the server is throwing errors. And there's often no good place for a cancel button, without putting up a big 'Loading' animation.

Re:Don't trust default timeouts by cortesoft on Saturday August 29, 2020 @02:17PM (#24318201)
Retrying doesn’t help on a slow connection... if the best case connections speed of your user is slower than your timeout, you can retry infinite times and it wont help.

Maybe increase the timeout on retry?

Re:Don't trust default timeouts by nitrogen on Saturday August 29, 2020 @02:35PM (#24318372)
Exponential backoff but for the timeout instead of the interval, and/or a Wait Longer button, sound useful.
Re:Don't trust default timeouts by iainmerrick on Saturday August 29, 2020 @02:51PM (#24318505)
As a user, I hate it when a program tells me I need to press “retry”! You’re the computer, you retry, and keep trying until I ask you to stop.
Re:Don't trust default timeouts by pixl97 on Saturday August 29, 2020 @03:04PM (#24318617)
Until it sits there and retires till hell freezes over and you don't realize anything is wrong.

And that's the problem with timeouts, everyone has different expectations.

Re:Don't trust default timeouts by asdfasgasdgasdg on Saturday August 29, 2020 @05:47PM (#24319646)
It's hard to go wrong by telling people what's happening and why. "it's taking longer than usual to X" is a fine thing to expose in a UI.
Re:Don't trust default timeouts by iainmerrick on Saturday August 29, 2020 @06:01PM (#24319731)
I should have stated my implicit assumption, that the program will tell me that it’s retrying. If I decide it’s not worthwhile I can stop it, sure.

For any user-initiated action, automatic retry seems strictly better than failing on a single timeout.