News for hackers. Stuff that matters.
Re:Opnieuw: A simple and intuitive retrying library for Python by miguendes on Wednesday February 05, 2020 @08:09AM (#22246399)
What's the difference between this one and tenacity?

[1] https://github.com/jd/tenacity

Re:Opnieuw: A simple and intuitive retrying library for Python by BiteCode_dev on Wednesday February 05, 2020 @08:18AM (#22246450)
At least a good readme, which is always a good sign.
Re:Opnieuw: A simple and intuitive retrying library for Python by mixmastamyk on Wednesday February 05, 2020 @06:41PM (#22252058)
From the article, they claim that the parameter names are easier to understand and therefore easier to enter correctly. As I've not used either lib directly, I'll have to take their word for it.
Re:Opnieuw: A simple and intuitive retrying library for Python by oefrha on Wednesday February 05, 2020 @08:01AM (#22246352)
Personally I would avoid this kind of npm-esque micro-dependency in my Python projects. I’ve written essentially the same decorator for many of my projects; it takes ten minutes and a few dozen lines of code.
Re:Opnieuw: A simple and intuitive retrying library for Python by whateveracct on Wednesday February 05, 2020 @12:38PM (#22249136)
Would you really want to roll your own exponential backoff w/jitter for every project though? It's not the easiest to get right, and since you're looping in a somewhat complicated way, it's definitely prone to accidental infinite loops in my experience.

Retrying seems perfect for a library to me. For instance, Haskell has a popular retry library https://hackage.haskell.org/package/retry-0.8.1.0/docs/Contr... and I wouldn't consider it bad to just bring it in.

Is this more symptom of Python deps being "riskier" than Haskell deps due to types or versioning or something?

Re:Opnieuw: A simple and intuitive retrying library for Python by oefrha on Wednesday February 05, 2020 @01:09PM (#22249463)
This library posted here is only ~150 loc without docstrings, and it’s more complicated than my typical equivalent decorator factory with basically the same features, sans logging.

I just don’t find this tricky. If only every problem I need to solve is as simple as this one.

> Is this more symptom of Python deps being "riskier" than Haskell deps due to types or versioning or something?

No, it’s a distaste for tons of small third-party packages that could easily be done in place. If anything I found Haskell deps more prone to breakage with upgrades compared to Python deps (given my limited experience with Haskell), at least with the set of popular packages I tend to use. Part of that is Python packages tend to not depend on a ton of other packages.

Re:Opnieuw: A simple and intuitive retrying library for Python by whateveracct on Wednesday February 05, 2020 @01:44PM (#22249822)
By "breakage with upgrades" do you mean bugs introduced due to compiler/library bumps or just PVP constraint violations?
Re:Opnieuw: A simple and intuitive retrying library for Python by oefrha on Wednesday February 05, 2020 @02:41PM (#22250315)
The latter — not necessarily PVP constraint violations though, but rather dependency hell situations. One example that pops to mind: packaging git-annex for Homebrew was always “great fun” until we settled on resolving with LTS Haskell (IIRC).
Re:Opnieuw: A simple and intuitive retrying library for Python by cjfd on Wednesday February 05, 2020 @01:07PM (#22249440)
Any dependency is also a risk. What if the next version does not just contain a nice feature that you want but also a nasty and subtle bug that you notice only later? Also, maintenance of a small library is by no means guaranteed to remain at the same level of quality. Generally, one is much safer with larger and more serious libraries. I would say one should just have dependencies that provide some rather serious set of features that would cost a lot of time to implement oneself.
Re:Opnieuw: A simple and intuitive retrying library for Python by guitarbill on Wednesday February 05, 2020 @01:07PM (#22249441)
Not at all, I think they just have some notion that npm = bad and npm = (small) deps, so (small) deps = bad.

I agree, it's harder to get right that it seems from first glance. The number of implementations I've seen with broken or not-honoured timeouts...

Python also had a retrying library; apparently it's no longer maintained. I've used the tenacity "fork", which has also evolved/introduced some great features like combinable and chainable wait/backoff.

Re:Opnieuw: A simple and intuitive retrying library for Python by oefrha on Wednesday February 05, 2020 @01:13PM (#22249499)
> Not at all, I think they just have some notion that npm = bad and npm = (small) deps, so (small) deps = bad.

No, I don’t have that notion. Apparently not every package on npm is small or trivial, but the abuse of trivial packages is most well-known and disastrous in the npm ecosystem.

Edit: My god, iOS autowrong is all the rage today.

Re:Opnieuw: A simple and intuitive retrying library for Python by ekimekim on Wednesday February 05, 2020 @11:38AM (#22248416)
The thing that always gets me with retries is error handling.

What errors should be considered retryable, vs raise immediately?

Do you log a retryable error? I've had systems mysteriously "hung" before which on closer inspection were infinitely retrying with a retry library that didn't log anything...

And if you do exceed your max retries, which error do you raise? The first one? The last one? There's no guarentee they'll be the same. A completely different "TooManyRetries" error?

These are the concerns I'd like to see addressed in a high level overview of a library like this one.

Re:Opnieuw: A simple and intuitive retrying library for Python by inetknght on Wednesday February 05, 2020 @12:29PM (#22249029)
Also: what if you get different retriable errors every time you re-try: should you have different retry count for each distinct retriable error? When you finally give up, should you report _all_ of the distinct errors or just the earliest or latest retriable error, or ...
Re:Opnieuw: A simple and intuitive retrying library for Python by karlicoss on Wednesday February 05, 2020 @07:44AM (#22246256)
Would be interesting to read more specific comparisons with existing mature libraries, e.g. backoff [0]. Feeling I get from reading the motivation behind a new library is that most of it can be achieved by using backoff and perhaps a custom delegating decorator that remaps parameter names and does extra checks if necessary.

[0] https://github.com/litl/backoff

Re:Opnieuw: A simple and intuitive retrying library for Python by tiku on Wednesday February 05, 2020 @07:18AM (#22246094)
The title is a dutch word, in English translated to "again" or "repeat".
Re:Opnieuw: A simple and intuitive retrying library for Python by jillesvangurp on Wednesday February 05, 2020 @08:21AM (#22246470)
The vowel combinations always trip up foreigners. Nieuw is Dutch for new and it is pronounced essentially the same. The ew sound basically transliterates to ieuw. Op (on) combined with Nieuw, basically means "on new", or 'again' in proper English.
Re:Opnieuw: A simple and intuitive retrying library for Python by mercer on Wednesday February 05, 2020 @10:09AM (#22247288)
I'll be the pedant and argue that if you want to figure out how to sound proper Dutch, there's a difference.

"new" sounds like "neeuuuuu" in US English, and a diphtong "nuuu" in British English.

"nieuw" sounds like "niii" followed by a very short "oooo" as in "do" followed by a slightly more pronounced "w" at the end.

I'm aware that there are proper ways to codify pronunciation, but it's more fun this way and I know nothing of it.

Re:Opnieuw: A simple and intuitive retrying library for Python by Kim_Bruning on Wednesday February 05, 2020 @08:18AM (#22246446)
Yes, but it's one of those words, where pitch and intonation (and -in this case- past memories of dread) are as important as the actual syllables.
Re:Opnieuw: A simple and intuitive retrying library for Python by timwaagh on Wednesday February 05, 2020 @12:24PM (#22248973)
Still seems a decent company. Nice that you take the time to properly deal with technical problems instead of just doing the whole 'plough forward even though you're just ploughing air' thing that happens just about everywhere.
Re:Opnieuw: A simple and intuitive retrying library for Python by leetrout on Wednesday February 05, 2020 @07:46AM (#22246264)
I don’t have a better solution but I was taken back by the length of the very verbose / descriptive param

retry_window_after_first_call_in_seconds

Am I the only one that thinks that is a bit excessive?

Re:Opnieuw: A simple and intuitive retrying library for Python by m12k on Wednesday February 05, 2020 @07:51AM (#22246294)
If you're using something like this, then the details of when, what and how you retry is probably something you care about, so having the library be explicit about it and making it possible to know what something means without having to look at the docs seems sensible to me.
Re:Opnieuw: A simple and intuitive retrying library for Python by anton_gogolev on Wednesday February 05, 2020 @08:06AM (#22246380)
This is some Java-level explicitness. Maybe, drop the `_seconds` suffix and replace it with something like `retry_window_after_first_call = '60s'`?
Re:Opnieuw: A simple and intuitive retrying library for Python by tijmendj on Wednesday February 05, 2020 @08:20AM (#22246463)
I agree it might look a bit excessive, but it greatly cuts down on complexity and possible unexpected behavior.

If you accept '60s' as 60 seconds, then the next questions are:

- Will it also accept '60000ms'?

- What does it do if I pass an int or float anyway? Will it implicitly take it to be seconds?

- Will it throw an exception if I use an unexpected value? If so, which exceptions can I expect? Will it just chug along and use a default?

Re:Opnieuw: A simple and intuitive retrying library for Python by VBprogrammer on Wednesday February 05, 2020 @08:46AM (#22246629)
Something I picked up from another developer is to use timedelta for these type of parameters. It avoids hoping that you get the right granularity for everyone.

   retry_window_after_first_call=timedelta(minutes=3.5)
   retry_window_after_first_call=timedelta(days=2)
   retry_window_after_first_call=timedelta(minutes=50)
Re:Opnieuw: A simple and intuitive retrying library for Python by leetrout on Thursday February 06, 2020 @03:02PM (#22259732)
Nice! Yes this is the better option. IMO, should just be

  retry_window=timedelta(...)
Re:Opnieuw: A simple and intuitive retrying library for Python by mercer on Wednesday February 05, 2020 @10:04AM (#22247241)
While the idea of '60s' is interesting, in lieu of that I always err on the side of explicitly specifying the time units when the argument provided is just a number. Not doing so has bitten me too many times.
Re:Opnieuw: A simple and intuitive retrying library for Python by bluebasket on Wednesday February 05, 2020 @08:49AM (#22246649)
60s. was wondering how we could add physical units to python? highjack the number literals and variables like this: 1.s, pi.radians? or maybe the way golang does it: 1 * time.seconds?

would be nice to have

Re:Opnieuw: A simple and intuitive retrying library for Python by bobbiechen on Wednesday February 05, 2020 @12:55PM (#22249315)
I also found this package: https://github.com/sbyrnes321/numericalunits

Which uses the interesting approach of setting all "units" to random floats: A complete set of independent base units (meters, kilograms, seconds, coulombs, kelvins) are defined as randomly-chosen positive floating-point numbers. All other units and constants are defined in terms of those. In a dimensionally-correct calculation, the units all cancel out, so the final answer is deterministic, not random. In a dimensionally-incorrect calculations, there will be random factors causing a randomly-varying final answer.

Which presumably is done to avoid overhead from carrying around units with each number, but forces you to run calculations multiple times (with different random seeds) to verify the result is correct.

Re:Opnieuw: A simple and intuitive retrying library for Python by magnusmundus on Wednesday February 05, 2020 @12:03PM (#22248729)
Re:Opnieuw: A simple and intuitive retrying library for Python by ekimekim on Wednesday February 05, 2020 @11:34AM (#22248377)
Sounds like a fun hack, I encourage you to give it a try and then never EVER use it in real code :P
Re:Opnieuw: A simple and intuitive retrying library for Python by bluebasket on Wednesday February 05, 2020 @08:39AM (#22246587)
yes it is excessive!
Re:Opnieuw: A simple and intuitive retrying library for Python by tignaj on Wednesday February 05, 2020 @07:56AM (#22246323)
If you are not familiar with how retrying is used and why exponential back off, jitter, etc are useful here is a shameless plug to a blog post I wrote recently https://blog.najaryan.net/posts/autonomous-robustness/ (the first half pretty much is about all these things).
Re:Opnieuw: A simple and intuitive retrying library for Python by sgt on Wednesday February 05, 2020 @07:12AM (#22246063)
I like the name. Maybe this should be merged into requests at some point?
Re:Opnieuw: A simple and intuitive retrying library for Python by sammorrowdrums on Wednesday February 05, 2020 @07:24AM (#22246134)
Do you speak Dutch? It's like the Python Zeep (SOAP - API protocol) client is Dutch for soap, Opnieuw is Dutch for again.

I always wandered if that was a nod to Guido Van Rossum that people like naming python packages in Dutch?

Re:Opnieuw: A simple and intuitive retrying library for Python by tijmendj on Wednesday February 05, 2020 @07:35AM (#22246204)
Channable is a Dutch company, and a lot of people who work here speak Dutch. I don't remember where we got it from but it is possibly a nod to Zeep.

I joked that we should name our library such that you would have to do

  from driemaal import scheepsrecht
Re:Opnieuw: A simple and intuitive retrying library for Python by Kim_Bruning on Wednesday February 05, 2020 @08:15AM (#22246428)
"There should be one-- and preferably only one --obvious way to do it.

Although that way may not be obvious at first unless you're Dutch."

-- Tim Peters, The Zen of Python

O:-)

Re:Opnieuw: A simple and intuitive retrying library for Python by sgt on Wednesday February 05, 2020 @10:08AM (#22247282)
No, but I speak Afrikaans, which is closely related to Dutch. So "opnieuw" is easily understood. In Afrikaans it would be "weer" (or "weer oor").
Re:Opnieuw: A simple and intuitive retrying library for Python by mixmastamyk on Wednesday February 05, 2020 @06:57PM (#22252149)
Most English names are already taken on PyPI. Using another language is one way to get around it.
Re:Opnieuw: A simple and intuitive retrying library for Python by lawlorino on Wednesday February 05, 2020 @10:08AM (#22247279)
I believe at least one of the contributors to Opnieuw is Dutch.
Re:Opnieuw: A simple and intuitive retrying library for Python by rkrzr on Wednesday February 05, 2020 @07:20AM (#22246103)
We did an internal naming competition for the library and one of the suggestions was "retreitz - Retries for humans".
Re:Opnieuw: A simple and intuitive retrying library for Python by sammorrowdrums on Wednesday February 05, 2020 @07:31AM (#22246172)
En jullie hebben een kantoor in Utrecht! Gezellig.

(And you have an office in Utrecht! Cozy.)

So that's why you guys speak Dutch anyway!

Re:Opnieuw: A simple and intuitive retrying library for Python by sdan on Wednesday February 05, 2020 @08:15AM (#22246429)
I think I’ll try this, but for devs:

a simple while True + try/catch + time.sleep(n*2) will do exponential backoff pretty easily.

Re:Opnieuw: A simple and intuitive retrying library for Python by x3ro on Wednesday February 05, 2020 @08:21AM (#22246476)
The post clearly states the problems they were trying to solve, none of which are taken into account here (consistency, ease of parameterization,..) It seems like people are reading too many "X in Y lines of code" posts :D
Re:Opnieuw: A simple and intuitive retrying library for Python by BiteCode_dev on Wednesday February 05, 2020 @08:20AM (#22246464)
But no jitter, and no timeout.
Re:Opnieuw: A simple and intuitive retrying library for Python by bluebasket on Wednesday February 05, 2020 @08:55AM (#22246695)
https://github.com/invl/retry works well for me. only problem is that logging issue that's still open i believe.

seems to me that the authors wrote their own library instead of trying to make the ones that didn't meet their need better. well, they never mentioned they tried...

anyways, some of their choices like that long variable name are odd to me. i can understand the unit problem though. golang solves that so nicely!