You must be fun on PRs

I hope you’ll be stuck debugging “error happened” without stuck trace and “x, y or z” are missing forever.

My forever favourite error message is iverilog encountering a problem and simply printing: I give up

>> long running batch job

'process exited with status 1'

You must be eternally frustrating junior.

thoughtful programming tips? on my HN??

Glad I'm not working with you if you consider that kind of PR comment to be worthy of a negative comment.

I would have just replied with "Oops, something went wrong."

If it hadn't already been flagged, that is.

Which is unfortunately very commonplace.

Funny how statements like these get upvoted and not flagged, but my "Lmao" did. I should be more toxic it seems. Which I thought is worse than an "Lmao" but hey, HN knows best.

("Lmao" is useless, but definitely not worse than some other responses.)

Why not add a boolean flag and use a single exit(1) at the end? That way the user can see all the environment variables that have not been set.

This is nitpicking, but this is a good usecase for the := operator:

  if not (API_KEY := os.getenv("API_KEY")):
      ...

For internal tools I just let os.environ["API_KEY"] raise a KeyError. It's descriptive enough.

I write a decent amount of Python, but find the walrus operator unintuitive. It's a little funky that API_KEY is available outside of the `if`, perhaps because I had first seen the walrus operator in golang, which restricts the scope to the block.

This isn't really unique to the walrus operator, it's just a general python quirk (albeit one I find incredibly annoying). `for i in range(5): ...` will leave `i` bound to 4 after the loop.

Oddly enough, "except" variables don't remain bound!

    try:
        x = int('cat')
    except Exception as e:
        pass
    print(e)  # <- NameError: name 'e' is not defined

So, it appears Python actually has three variable scopes (global, local, exception block)?

Also true of JavaScript pre-ES5, another language that on first glance seems to only have function scope: it actually does have block scope, but only for variables introduced in `catch` blocks. AFAIU that was the standard way for a dumb transpiler to emulate `let`.

I wonder if that was ever popular, considering the deoptimization effects of try/catch, and given that block scope can also be managed by renaming variables.

exceptions being the exception is funny somehow

very recursive

you can say that again.

> Oddly enough

It's not that odd, since it's the only situation where you cannot keep it bounded, unless you enjoy having variables that may or may not be defined (Heisenberg variable?), depending on whether the exception has been raised or not?

Compare with the if statement, where the variable in the expression being tested will necessarily be defined.

> Compare with the if statement, where the variable in the expression being tested will necessarily be defined.

    if False:
        x = 7
    print(x)

    print(x)
          ^
    NameError: name 'x' is not defined

Ruby does this sort of stuff, where a variable is defined more or less lexically (nil by default). Python doesn't do this. You can have local variables that only maybe exist in Python.

While somewhat true, what would this be bound to?

    for i in range(0):
        pass

Well, after writing my comment, I realized that a python interpreter could define the variable and set it to None between the guarded block and the except block, and implicitly assign it to the raised exception right before evaluating the except block, when the exception as been raised. So technically, it would be possible to define the variable e in GP example and have it scoped to "whatever is after the guarded block", just like what is done with for blocks.

Is there any chance this would cause trouble though? Furthermore, what would be the need of having this variable accessible after the except block? In the case of a for block, it could be interesting to know at which point the for block was "passed".

So, maybe "None" answers your question?

The answer is: it is unbound. Intellisense will most likely tell you it is `Unbound | <type>` when you try to use the value from a for loop. Would it be possible that it could be default initialized to `None`? Sure, but `None` is a destinctivly different value than a still unbound variable and may result in different handling.

Heisenbug is the word you are looking but may not find.

Are you saying that this should result in a name error?

   if <true comparison here>:
       x = 5
   print(x)  # <- should give name error?

I stand corrected, the exception case is definitely an oddity, both as being an outlier and as a strange behaviour wrt Python's semantics. Or is it a strange behaviour?

In the case of an if like in your example, no provision is made about the existence of x. It could have been defined earlier, and this line would simply update its value.

Your example:

   if True:
       x = 5
   print(x)  # 5

Same with x defined prior:

   x = 1
   if False:
       x = 5
   print(x)  # 1

What about this one?

   if False:
       x = 5
   print(x)  # ???

On the other hand, the notation "<exception value> as <name>" looks like it introduces a new name; what if that name already existed before? Should it just replace the content of the variable? Why the "as" keyword then? Why not something like "except <name> = <exception value>" or the walrus operator?

While investigating this question, I tried the following:

    x = 3
    try:
        raise Exception()
    except Exception as x:
        pass
    print(x)  # <- what should that print?

In any sane language it would, yes.

Nope, it's more complicated than that:

    e = 'before'
    try:
        x = int('cat')
    except Exception as e:
        e2 = e
        print(e)
    print(e2) # <- This works!
    print(e)  # <- NameError: name 'e' is not defined

It's not a scoping thing, the bound exception variable is actually deleted after the exception block, even if it was already bound before!

lol - raku maybe weird, but at least it has sane variable scoping

I may be rusty, but wasn't there a "finally" scope for those situations?

edit: writing from phone on couch and the laptop... looks far, far away...

Exception blocks don't create a different scope. Instead, the name is explicitly (well, implicitly but deliberately) deleted from the scope after the try/except block runs. This happens because it would otherwise produce a reference cycle and delay garbage collection.

https://stackoverflow.com/questions/24271752

https://docs.python.org/3/reference/compound_stmts.html#exce...

This is the type of things that make me roll my eyes at all the wtf JavaScript posts[0], yes there are a lot of random things that happen with type conversions and quite a few idiosyncrasies (my favourite is that document.all is a non empty collection that is != from false but convert to false in an if)

But the language makes sense at a lower level, scopes, values, bindings have their mostly reasonable rules that are not hard to follow.

In comparison python seems like an infinite tower of ad-hoc exceptions over ad-hoc rules, sure it looks simpler but anywhere you look you discover an infinite depth of complexity [1]

[0] and how half of the complaints are a conjugation of "I don't like that NaNs exist

[1] my favourite example is how dunder methods are a "synchronized view" of the actual object behaviour, that is in a + b a.__add__ is never inspected, instead at creation time a's add behaviour is defined as its __add__ method but the association is purely a convention, eg any c extension type need to reimplement all these syncs to expose the correct behaviour and could for funzies decide that a type will use __add__ for repr and __repr__ for add

> yes there are a lot of random things that happen with type conversions and quite a few idiosyncrasies... the language makes sense at a lower level, scopes, values, bindings have their mostly reasonable rules

The "random things" make it practically impossible to figure out what will happen without learning a whole bunch of seemingly arbitrary, corner-case-specific rules (consider the jsdate.wtf test currently making the rounds). And no, nobody is IMX actually simply complaining about NaNs existing (although the lack of a separate integer type does complicate things).

Notice that tests showcasing JavaScript WTFery can work just by passing user data to a builtin type constructor. Tests of Python WTFery generally rely on much more advanced functionality (see e.g. https://discuss.python.org/t/quiz-how-well-do-you-know-pytho...). The only builtin type constructor in Python that I'd consider even slightly surprising is the one for `bytes`/`bytearray`.

Python's scoping is simple and makes perfect sense, it just isn't what you're used to. (It also, unlike JavaScript, limits scope by default, so your code isn't littered with `var` for hygiene.) Variables are names for objects with reference semantics, which are passed by value - exactly like `class` types in C# (except you don't have to worry about `ref`/`in`/`out` keywords) or non-primitives in Java (notwithstanding the weird hybrid behaviour of arrays). Bindings are late in most places, except notably default arguments to functions.

I have no idea what point you're trying to make about __add__; in particular I can't guess what you think it should mean to "inspect" the method. Of course things work differently when you use the C API than when you actually write Python code; you're interacting with C data structures that aren't directly visible from Python.

When you work at the Python level, __add__/__iadd__/__radd__ implement addition, following a well-defined protocol. Nothing happens "at creation time"; methods are just attributes that are looked up at runtime. It is true that the implementation of addition will overlook any `__add__` attribute attached directly to the object, and directly check the class (unlike code that explicitly looks for an attribute). But there's no reason to do that anyway. And on the flip side, you can replace the `__add__` attribute of the class and have it used automatically; it was not set in stone when the class was created.

I'll grant you that the `match` construct is definitely not my favourite piece of language design.

> methods are just attributes that are looked up at runtime

At runtime when evaluating a + b no dunder method is looked up and there is no guarantee that a + b === a.__anydunder__(b) https://youtu.be/qCGofLIzX6g

What i mean with weird scoping is

    def foo():
      e = 'defined'
      try:
        raise ValueError
      except Exception as e:
        print(e)
      print(e) # this will error out

    foo()

I also dislike how local/global scopes work in python but that is more of a personal preference.

I agree that that Javascripts standard library is horrible the jsdate.wtf is an extreme but apt example, IMO most of these are solved with some "defensive programming" but I respect other opinions here.

> And no, nobody is IMX actually simply complaining about NaNs existing

I watched many Javascript WTF! videos on youtube and NaNs and [2] == "2" were usually 90% of the content.

Anyway actually my biggest gripe with python is that i find the module/impor/export system counterintuitive

Oh yeah, that's a good point.

Python really is a bit of a mess haha.

Maybe Python will get a let one day

I find it incredibly intuitive and useful that it does that. sometimes it drives me nuts that it doesn't do it for comprehensions but I can see why.

But if something fails in a loop running in the repl or jupyter I already have access to the variables.

If I want to do something with a loop of data that is roughly the same shape, I already have access to one of the the items at the end.

Short circuiting/breaking out of a loop early doesn't require an extra assignment.

I really can't see the downside.

Python 2 actually did let comprehension variables leak out into the surrounding scope. They changed it for Python 3, presumably because it was too surprising to overwrite an existing variable with a comprehension variable.

That almost sounds like having the "variables" eax, ebx, ecx, and edx.

Oh wow, maybe that's why I expect it to work that way! I can't believe it's been long enough since I used 2 that I'm forgetting it's quirks.

It would be utterly nuts otherwise. For loops over all elements in a sequence. If the sequence is a list of str, as an example, what would the «item after the last item» be?

the issue isn't the value of i, the issue is that i is still available after the loop ends. in most other languages, if it was instantiated by the for-each loop, it'd die with the for-each loop

I cannot tell you how many times I've hit issues debugging and it was something like this. "You should know better" -- I know, I know, but I still snag on this occasionally.

> `for i in range(5): ...` will leave `i` bound to 4 after the loop. reply

This "feature" was responsible for one of the worst security issues I've seen in my career. I love Python, but the scope leakage is a mess. (And yes, I know it's common in other languages, but that shouldn't excuse it.)

I would love to hear about the security issue if you're able to talk about it

I don't remember the exact details, but it basically involved something along the lines of:

1) Loop through a list of permissions in a for list

2) After the loop block, check if the user had a certain permission. The line of code performing the check was improperly indented and should have failed, but instead succeeded because the last permission from the previous loop was still in scope.

Fortunately there was no real impact because it only affected users within the same company, but it was still pretty bad.

Oof that's a near miss. That's the sort of hard-to-find issue that keeps me up at night. Although maybe these days some ai tool would be able to pick them up

This is just Pythons scoping, which is not restricted by block, but function. You have the same effect with every other element.

There's no block scope in Python. The smallest scope is function. Comprehension variables don't leak out, though, which causes some weird situations:

    >>> s = "abc"
    >>> [x:=y for y in s]
    ['a', 'b', 'c']
    
    >>> x
    'c'
    
    >>> y
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'y' is not defined

Comprehensions have their own local scope for their local variables, but the walrus operator reaches up to the innermost "assignable" scope.

Wow. I had been writing Python for 15 years and I didn't even know that operator exists

It's only existed for 6 of those years so perhaps you can be forgiven :)

The last time I wrote Python in a job interview, one of the interviewers said "wait, I don't know Python very well but isn't this kinda an old style?" Yes, guilty. My Python dates me.

I always forget this syntax exists. When was this introduced?

In Python 3.8 - https://peps.python.org/pep-0572/

The Walrus operator! Introduced in Python 3.8 according to the PEP 572.

Really nice to combine 1) checking if something exists and 2) act on it

You forgot to end with "goo goo g'joob" or the less correct Simon and Garfunkel version "coo coo ca choo".

And for lambdas

October 2019 in Python 3.8

https://docs.python.org/3/whatsnew/3.8.html

Why would you use it though? I always thought it a bad thing that legacy code (such as in C or C++) had these side effects inside of checks happening

no one uses walrus

It seems that way! I briefly experimented with it when it first came out, but never used it in any production code. I've never seen it used by anyone else, either.

I use it very often. Avoids duplication of expensive operations in comprehensions.

I find it interesting that none of the voluminous python code I've had AI tools generate has ever had a walrus operator in it. Reflects the code they're trained on, I guess.

I use it fairly often

Why are people writing Python like it's Go?

Letting the KeyError go is usually fine, but if you want to log it or can recover somehow then:

    try:
      API_KEY = os.environ["API_KEY"]
    except KeyError:
      logger.exception("...")
      ...

Even better is to try to fetch all the env variables and then report on all of the missing ones.

It would be better to do both: print out the detailed string or strings, then exit if either are printed.

Even better, check all the conditions and report all the errors. Don't make me add the first variable, run it again and get another error.

Sometimes that's inevitable, bit noisy of the time it isn't.

I'm surprised there isn't an argparse like thing for documenting expected environment variables.

That's basically pydantic settings: https://docs.pydantic.dev/latest/concepts/pydantic_settings/

Taking ephemeral arguments like channel ID from the environment is more offensive to observability and user comfort

I feel like use case and audience matters when making these decisions. In this case, the user is probably someone interacting with a python script they're running in a console (I assume by print), then I really don't think it matters - the user will check that both things are set. Should you also give them some documentation about setting env vars? Should you customize that documentation to the OS they're running? etc.

If the user is a typical consumer using a typical consumer interface, then yes you want to handhold them a bit more.

Neophytes take notice. Attention to details like this is what separates truly great programmers from merely good ones. That said, for scripts reusable by others you should use command line arguments . Environment variables in lieu of command line arguments is a huge code smell.

You don't generally want API keys accidentally recorded into someone's bash history.

What exactly is your threat model, where the attacker can read ~/.bash_history but can't execute (or capture output from) /usr/bin/env?

Threat model: shell scripts

You do NOT save passwords in shell history, it is insanely insecure. Lets begin with the passwords being readable by everything that can list tasks.

You can protect passwords in a password manager. You do not need to keep the passwords in env and I do not.

> Lets begin with the passwords being readable by everything that can list tasks.

Why are processes running that can do this, that I don't already fully trust?

> You can protect passwords in a password manager.

What's your plan for supplying the password to the program, given that people will want to automate use of the program?

How about not accidentally showing tokens and credentials when live screen sharing?

CI and other build systems. Where tokens have been stolen in the past, by users not caring, and a VM not being properly cleaned.

The threat model is that history is persistent while the environment isn't. That said, whenever possible you should handle secrets using file descriptors as opposed to environment variables.

For this example, don't just command line arguments. There's an API key there, you don't want an API key visible in your cmdline.

Then how would you SET the API key in the first place? :) The argument doesn't make any sense at all.

In some .profile or .envrc or what you'd call such a file, I suppose.

And you expect someone will be able to read your bash_history, but not your .profile?

Fair, I wasn't thinking about the details of how someone would lift out the bash history.

Using read or an equivalent, presumably. Just because you don't know why a practice is recommended against doesn't mean that there isn't a good alternative.

Typer has a great feature that lets you optionally accept argument and flag values from environment variables by providing the environment variable name:

https://typer.tiangolo.com/tutorial/arguments/envvar/

It's especially nice for secrets. Best of both worlds :)

No, that's an anti-feature. :) Sibling comments here claim that command line arguments "leak" whereas environment variables does not. It's plain wrong. An attacker with access to arbitrary processes' cmdline surely also has access to their environ. Store secrets in files, not in the environment. Now you can easily change secret by pointing the --secret-file parameter to a different file. The only reason people use BLABLA_API_KEY variables is because Heroku or something did it back in the day and everyone cargo-culted this terrible pattern.

One could write a huge treatise on everything that is wrong with environment variables. Avoid them like the plague. They are a huge usability PITA.

This is bad advice. Please don't make claims about security if you're making it up as you go.

Environment variables are substantially more secure than plain text files because they are not persistent. There are utilities for entering secrets into them without leaking them into your shell history.

That said, you generally should not use an environment variable either. You should use a secure temporary file created by your shell and pass the associated file descriptor. Most shells make such functionality available but the details differ (ie there is no fully portable approach AFAIK).

The other situation that sometimes comes up is that you are okay having the secret on disk in plain text but you don't want it inadvertently commited to a repository. In those cases it makes sense to either do as you suggested and have a dedicated file, or alternatively to set an environment variable from ~/.bashrc or similar.

Good luck storing your private keys in environment variables!

    exit("Missing ...")

This prints the message and exits with code 1.

Even slightly better is to first check both

    if not API_KEY and not CHANNEL_ID:
        print("Missing both YOUTUBE_API_KEY and YOUTUBE_CHANNEL_ID.")
        exit(1)
    if not API_KEY:
        print("Missing YOUTUBE_API_KEY.")
        exit(1)
    if not CHANNEL_ID:
        print("Missing YOUTUBE_CHANNEL_ID.")
        exit(1)

That way you don't end up fixing one just come back and be told you're also missing another requirement

Even better would be to only check each once and buffer the decision:

    valid = True
    if not API_KEY:
        print("Missing YOUTUBE_API_KEY.")
        valid = False
    if not CHANNEL_ID:
        print("Missing YOUTUBE_CHANNEL_ID.")
        valid = False
    if not valid:
        exit(1)

This way you only check each value once (because your logic might be more complicated than just checking it's not set, maybe it can be wrongly formatted) and you still get to do whatever logic you want. It also removed the combinatorial problems.

This is a pretty general principle of separating decision from action.

why print then exit(1) instead of raising an exception?

Oh, I know this one:

  $ python3 -c "print('clear messaging'); exit(1)"
  clear messaging

  $ python3 -c "raise ValueError('text that matters')"
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ValueError: text that matters

and that story gets _a lot_ worse when some programs raise from within a "helper" module and you end up with 8 lines of Python junk to the one line of actual signal

Or

  API_KEY = os.environ.get("YOUTUBE_API_KEY")
  CHANNEL_ID = os.environ.get("YOUTUBE_CHANNEL_ID")

  assert(API_KEY, "Missing YOUTUBE_API_KEY")
  assert(CHANNEL_ID, "Missing CHANNEL_ID")

Do you use parens with assert? In Python, the assert statement is not a function, right?

That bit me before... it created a tuple which evaluated as true.

Assertions are disabled via `python -O` so they probably shouldn't be used like this.

On the other hand, presumably the program will just crash anyway with some terse permission/request error later on if these values aren't set

Doesn't matter. Assertions should be used for things that are always true (barring bugs in the program). That isn't the case here.

I usually go through an array of settings that must be defined and list any missing ones, to avoid repeated attempts from the user/deployer.

Thank you. I see something like this all the time on one of the sites I use for work. If you fail the 2-factor, it'll tell you your password was wrong and reset the whole thing instead of telling you the 2-factor code was wrong or expired.

Just use

  if not API_KEY:
    sys.exit("Missing YOUTUBE_API_KEY.")

It will print message to sys.stderr and do system exit with code 1.

P.S. exit() is just an alias to sys.exit(), I prefer longer form.