News for hackers. Stuff that matters.
Re:Box: Python dictionaries with recursive dot notation access by jerf on Friday May 05, 2017 @01:54PM (#14275401)
I think every Python programmer implements this at some point.

I think the real reason not to do this is one that as of this writing still hasn't been mentioned yet, which is that when using dot notation, your key names have to fit the Python grammar for identifiers, so you can't have a key "the thing" with a space, or anything else that isn't a Python identifier. So you can't just say "I can access this dictionary with dots", it has to be "I can access some keys of this dictionary with dots, but others I have to access another way". This greatly, greatly reduces its utility. There's a variety of ways of trying to address this; this package appears to try to normalize key names, which is very prone to surprising behaviors and makes it difficult to reason about what will go into what bucket, and also seriously mitigates the virtue of this entire approach because you, the programmer, must also run the normalization algorithm yourself in order to use the dot notation, which rapidly eats away the gains of typing a dot instead of two brackets and two quote characters. Rewriting keys like that is really icky.

The other problem is that the "dot" namespace, as it were, is as others have noted here used for method and property resolution, so you also end up with "dictionaries with random values they can't really contain because they're being used as method names" or "dictionaries that if you put the wrong key in them override a method" or something else like that. Also, consider the ability to subclass these things, making many of the things you might think to do to hack around this break in subclasses.

It's very superficially tempting but it has a looooot of issues that become evident over time. This isn't even a complete list.

Re:Box: Python dictionaries with recursive dot notation access by cjhanks on Friday May 05, 2017 @04:31PM (#14276619)
There are quite a few nay-votes here, so I'll give a yay vote.

In the past I have had to deal with very complex configuration systems; several thousands of lines of XML which were supposed to direct many applications.

And writing:

    value['this']['that']['the_other']
vs

    value.this.that.the_other
Really starts to matter for code legibility. Of course if you change your XML configs your Python will break - but it would likely break in any case.

The other case where I have found this applicable is when I have complicated JSON structures in the form of user input. Typically I augment this style by using a JSONSchema that ensures attributes do exist and have defaults (or None).

Re:Box: Python dictionaries with recursive dot notation access by smoe on Friday May 05, 2017 @05:01PM (#14276825)
I see the value in the latter for code legibility. My gripe with it is, when I'm new to a project that uses a dict implementation like that: how do I know what happens if for example `that` is missing? Does it raise AttributeError, KeyError or just return None?

Personally I'd prefer a helper functions like

    deep_get(dict_, dotted_path[, default]) -> value
You stil have to check the docs/source what exactly happens, but it is just one simple function instead of magic methods. And i can keep using plain dicts everywhere.

EDIT: To clarify a bit. An experienced Python developer should immediately recognize a call like `deep_get(value,'this.that.the_other')` as something project/framework specific and not built-in, while `value.this.that.the_other` is ambiguous.

Re:Box: Python dictionaries with recursive dot notation access by CloudYeller on Friday May 05, 2017 @06:47PM (#14277378)
You can get part of the way there with operator.attrgetter:

NT = collections.namedtuple('NT', ['x', 'y'])

nt1 = NT(1, 2)

nt2 = NT(nt1, 'asdf')

assert nt2.x.y == 2

get_two = operator.attrgetter('x.y')

assert get_two(nt2) == 2

If you wrap the operator.attrgetter in a try/catch, you can force a default too: https://pastebin.com/bFxGr22E

Re:Box: Python dictionaries with recursive dot notation access by smoe on Saturday May 06, 2017 @02:06PM (#14281512)
I never thought much about best implementation because I only used it i think once in ten years. But afaik I used something like this, that does the job and is easy to read

  _empty = object()
  def deep_get(dct, dotted_path, default=_empty):
      for key in dotted_path.split('.'):
        try:
          dct = dct[key]
        except KeyError:
          if default is _empty:
            raise
          return default
      return dct
Re:Box: Python dictionaries with recursive dot notation access by nerdponx on Friday May 05, 2017 @09:35PM (#14278171)
You can use `get` from the `toolz` package:

    toolz.get(value, ['this', 'that', 'the_other']
There's also a Cython implementation called `cytoolz`.
Re:Box: Python dictionaries with recursive dot notation access by cjhanks on Friday May 05, 2017 @07:31PM (#14277605)
Personally I try to implement it as an AttributeError, it seems the most intuitive. By the way, I do understand your point and it's pretty valid in a lot of cases.
Re:Box: Python dictionaries with recursive dot notation access by lobster_johnson on Friday May 05, 2017 @04:37PM (#14276646)
One option would be:

    value['this.that.the_other']
Bonus: Make it support JSONPath and you can use it to extract lots of things at arbitrarily nested levels.
Re:Box: Python dictionaries with recursive dot notation access by Varriount on Friday May 05, 2017 @05:05PM (#14276848)
What do you typically use for enforcing a schema? I've been using Good[0], however since it is meant for general data structures, users don't get line/column printouts for invalid data.

[0] https://pypi.python.org/pypi/good/0.0.7-0

Re:Box: Python dictionaries with recursive dot notation access by cjhanks on Friday May 05, 2017 @07:35PM (#14277621)
I like jsonschema [1][2], primarily because it's a cross-language standard. Though, I have never used `Good` - it seems to be pretty nice if you're Python focused.

[1] https://pypi.python.org/pypi/jsonschema

[2] http://json-schema.org/

Re:Box: Python dictionaries with recursive dot notation access by msellout on Friday May 05, 2017 @08:04PM (#14277779)
Give `destructure` a try https://github.com/selik/destructure
Re:Box: Python dictionaries with recursive dot notation access by xapata on Friday May 05, 2017 @07:04PM (#14277468)
Better would be to code depth-first-search and breadth-first-search functions and use those on plain nested dictionaries. That's what XML did via XPATH.
Re:Box: Python dictionaries with recursive dot notation access by nas on Friday May 05, 2017 @02:16PM (#14275580)
If you use something like this, future maintainers are going to hate you.
Re:Box: Python dictionaries with recursive dot notation access by aidos on Friday May 05, 2017 @02:53PM (#14275900)
Agreed. This is not the sort of code you want to be dealing with in a few years time.

Also, I seldom find that string based dicts are what I need. One of the best things about Python is being able to use virtually anything as a key.

Re:Box: Python dictionaries with recursive dot notation access by user5994461 on Friday May 05, 2017 @04:08PM (#14276457)
> One of the best things about Python is being able to use virtually anything as a key.

I find that to be the thing one of the worst thing ever made into a programming language.

You are facing a program/API/library and you can have absolutely no clue of what it will give or expect without extensive reverse engineering.

Re:Box: Python dictionaries with recursive dot notation access by dragonwriter on Friday May 05, 2017 @07:10PM (#14277503)
> You are facing a program/API/library and you can have absolutely no clue of what it will give or expect without extensive reverse engineering.

Or reading the docs.

Re:Box: Python dictionaries with recursive dot notation access by xapata on Friday May 05, 2017 @07:07PM (#14277492)
I find that's more a problem of bad design and bad names, rather than flexible types.
Re:Box: Python dictionaries with recursive dot notation access by aidos on Friday May 05, 2017 @09:13PM (#14278076)
Wait, you choose what to put in the dict. I like being able to map from, say, db objects to internal representations. Why wouldn't you want that?!
Re:Box: Python dictionaries with recursive dot notation access by dllthomas on Friday May 05, 2017 @10:54PM (#14278428)
The concern is facing something that expects a map, and not knowing how to correctly populate it.
Re:Box: Python dictionaries with recursive dot notation access by aidos on Saturday May 06, 2017 @06:00AM (#14279556)
Time has taught me to prefer lists / flat structures as arguments to functions (and in most other storage and transfer contexts). Besides, either you're passing in entities that conform to the api, or you're not. Nothing about arbitrary keys in dicts changes that.
Re:Box: Python dictionaries with recursive dot notation access by makecheck on Friday May 05, 2017 @02:27PM (#14275672)
Yep. Or keys with dots in their names, or keys actually named something like "__eq__" (strange but not impossible).

The worst thing about abstractions is when they break. I can always figure out how to fix or extend ugly but straightforward code. I may not even know where the manual is when a fancy wrapper dumps a trace.

Re:Box: Python dictionaries with recursive dot notation access by mannykannot on Friday May 05, 2017 @03:47PM (#14276284)
>I may not even know where the manual is when a fancy wrapper dumps a trace.

I read somewhere that if it is written in self-documenting code, the problem and its solution will be obvious.

Re:Box: Python dictionaries with recursive dot notation access by xapata on Friday May 05, 2017 @07:09PM (#14277496)
Was that sarcasm?

What counts as "self-documenting"?

Re:Box: Python dictionaries with recursive dot notation access by Retra on Friday May 05, 2017 @08:39PM (#14277954)
Undocumented code is self-documenting. Because it only ever does exactly what is written, it must be intended to do exactly what it does.
Re:Box: Python dictionaries with recursive dot notation access by xapata on Friday May 05, 2017 @11:37PM (#14278528)
Ah, Dr. Pangloss.
Re:Box: Python dictionaries with recursive dot notation access by zubat on Friday May 05, 2017 @08:14PM (#14277823)
I ended up making a custom data structure and query language around this kind of notation. Not in Python(am working in Haxe) but easily generalized. The structure is either array or string map or ordered string map with a dynamic data field and optional string tags. So it maps very well onto JSON or XML style data, but comes with the benefit of a very unambiguous way to do queries, where I can mix together addressing styles(take this numbered index OR take the named key) and pull elements into an array without much difficulty. The lookup syntax is "@<key>#<index>..." and each key or index entry is parsed into an ADT. I can pass along partially constructed queries as the relevant data to be processed. It's not quite as flexible as SQL since the data has to come in tree form, but it creates a major design affordance: the same query can be passed to a custom algorithm or custom data structure, with less ambiguity than a simple string lookup.

To some extent this retreads ground covered by e.g. XPath, but the specific structure puts it closer to my day-to-day needs.

Re:Box: Python dictionaries with recursive dot notation access by strunz on Friday May 05, 2017 @02:22PM (#14275628)
Yep. Last time I tried one of these I had a dict that had keys that conflicted with the dictionary keywords, like "items" or "keys" or "values", so I wound up functions back instead of data.
Re:Box: Python dictionaries with recursive dot notation access by chris_griffith on Friday May 05, 2017 @02:17PM (#14275588)
Author here: I totally agree with the fact it has shortcomings and definitely isn't aimed to replace everyone's dict everywhere.

I actually created the first iteration of this around 3 years ago, aimed primarily for sysadmins and others who use the console more than actual developing. As I find it extremely convenient to tab complete keys and a lot less typing.

From there it has obviously evolved to try and fit more peoples needs, and I use it regularly in my projects.

Re:Box: Python dictionaries with recursive dot notation access by lkirk on Friday May 05, 2017 @03:54PM (#14276339)
Just an FYI: the newest version of ipython allows you to tab complete keys
Re:Box: Python dictionaries with recursive dot notation access by lesingerouge on Friday May 05, 2017 @03:00PM (#14275965)
Also dot means attribute means less speed.
Re:Box: Python dictionaries with recursive dot notation access by dom0 on Friday May 05, 2017 @03:13PM (#14276053)
If you have to worry about that, you've taken many wrong turns in the past.
Re:Box: Python dictionaries with recursive dot notation access by deathanatos on Friday May 05, 2017 @08:52PM (#14277997)
That's a huge micro-optimization. For a trivial example of a class having the attributes "a", "b" and "c", and a dict of the same,

    In [4]: %timeit d['b']  # dict
    10000000 loops, best of 3: 42.7 ns per loop

    In [6]: %timeit f.b  # class using __slots__
    The slowest run took 30.10 times longer than the fastest. This could mean that an intermediate result is being cached.
    10000000 loops, best of 3: 44 ns per loop

    In [9]: %timeit b.b  # class using __dict__
    10000000 loops, best of 3: 48.1 ns per loop
You're talking <6ns per access; except in the very exceptional case where you know you need this, in Python of all languages, it's an over optimization. The maintainability of having the stupid-simple class vastly outweighs the speed.