News for hackers. Stuff that matters.
Re:Why databases use ordered indexes but programming uses ha... by d_burfoot on Monday December 09, 2019 @09:08PM (#21748956)
As a programmer, I always use an ordered map (Java TreeMap) in preference to hash tables. My reasons:

- It is much easier to write a transparently correct comparison method for a new object that I want to use as a key. With hash tables you need to worry that your hash function might not be properly distributed.

- With hash tables you have to worry that your language's under-the-hood resize/reallocate algorithm works well, or you need to make sure you choose the capacity and load factor numbers correctly. Even if things work as expected, you might end up paying a penalty because, for example, the array backing the hashtable was auto-resized to 128 million elements when you only need 70 million.

- Java TreeMap has lots of cool utility methods like pollFirstEntry (pull off the first K/V pair), higherMap (view of map that only includes pairs with higher key), etc.

- TreeMap key traversal is deterministic!! The worst bug I encountered in years of programming work was caused by non-determinism in HashMap traversal (you could blame Java's implementation here, I suppose).

- The O(1) vs O(log N) performance difference is minor to the point of being undetectable in every case I've ever encountered.

- When you do something with the data in the map, you often want to sort by key anyway: might as well have the sort taken care of by the data structure.

Re:Why databases use ordered indexes but programming uses ha... by kodablah on Monday December 09, 2019 @10:26PM (#21749317)
You should reach for the best tool for the job instead of giving job-specific reasons for your general choice. Without going into detail explaining how your bullets either don't apply to most uses, prematurely code for a future case that may not exist, dismiss performance, reference other languages as if language implementation is not a factor in the choice, etc... suffice to say they both have their place and in general you probably shouldn't reach for an ordered map over a unordered one without reason. I.e. it shouldn't be your default choice in most languages and it'd be bad advice to give other programmers.
Re:Why databases use ordered indexes but programming uses ha... by ummonk on Tuesday December 10, 2019 @12:30AM (#21749861)
An ordered map is much better as a default simply because it can do everything that an unordered map can. Choosing to use an unordered map instead is a premature optimization that is nearly always unnecessary.
Re:Why databases use ordered indexes but programming uses ha... by mantap on Tuesday December 10, 2019 @01:09AM (#21750085)
IMO, premature optimization doesn't apply to cases where you are making general decisions about policy or defaults, otherwise you can get into a "death by a thousand cuts" situation where none of your functions are slow individually but the whole program is slow (i.e. big overheads). To me, premature optimization means don't optimize a specific function unless you know it is slow. I'd also say that premature optimization doesn't apply in cases where both alternatives are equally easy to write.
Re:Why databases use ordered indexes but programming uses ha... by mlyle on Tuesday December 10, 2019 @01:28AM (#21750183)
On the other hand, if you pick a TreeMap--- odds are it will be a non-issue either way.

If it does show up on your profiles, it's easy to reconsider and say "I don't think I'll ever access this in order, after all..." and change it and see if you see a benefit.

Re:Why databases use ordered indexes but programming uses ha... by JadeNB on Tuesday December 10, 2019 @09:01AM (#21752458)
> I'd also say that premature optimization doesn't apply in cases where both alternatives are equally easy to write.

I think that premature optimisation applies not only to ease of writing, but even more to ease of debugging.

Re:Why databases use ordered indexes but programming uses ha... by unethical_ban on Wednesday December 11, 2019 @05:56PM (#21767675)
Prematurely optimizing for death by a thousand cuts is what you're proposing.
Re:Why databases use ordered indexes but programming uses ha... by GeneralMayhem on Tuesday December 10, 2019 @12:37AM (#21749910)
That argument works both ways, though.

An unordered map has the narrower interface and generally better performance, so it's much better as the default. Choosing to use an ordered map instead is YAGNI interface bloat that is nearly always unnecessary.

Re:Why databases use ordered indexes but programming uses ha... by kodablah on Tuesday December 10, 2019 @03:37AM (#21750741)
Premature optimization concerns effort. We do a lot of things when writing code that could be considered premature optimization by your vague definition. By this definition too, staying in Java land, I should probably not instantiate my ArrayList or HashMap with a capacity even if I know it...technically premature optimization.

All else equal wrt simplicity, choose the highest performing option that contains the features you need. Dogmatic premature optimization rules are unhelpful. We can apply reasonable thinking, and we should.

Re:Why databases use ordered indexes but programming uses ha... by naasking on Tuesday December 10, 2019 @10:32AM (#21753354)
> Premature optimization concerns effort

And the OP is correct that writing a correct comparison function needed for an ordered map is easier than writing a correct hash function that is properly distributed. So the ordered map is lower effort for that reason alone, although the other properties the OP lists also produce valid reductions of effort.

Re:Why databases use ordered indexes but programming uses ha... by naniwaduni on Monday December 09, 2019 @11:01PM (#21749488)
The problem with choosing "the best tool for the job" for every job is that now you have a lot more unknown unknowns and they don't generalize when you find them.
Re:Why databases use ordered indexes but programming uses ha... by loeg on Tuesday December 10, 2019 @12:42AM (#21749939)
Especially when an ordered map is backed by a binary tree (TreeMap), which is the worst data structure. Btree ordered maps good, hash tables good (if point queries are sufficient); binary trees have terrible cache locality.
Re:Why databases use ordered indexes but programming uses ha... by kccqzy on Tuesday December 10, 2019 @02:53AM (#21750539)
Cache locality issues can be fixed by using a good allocator (or a good compacting GC).
Re:Why databases use ordered indexes but programming uses ha... by AstralStorm on Tuesday December 10, 2019 @07:17AM (#21751751)
How can an allocator optimize the layout of a dynamically sized structure that's being changed without compacting?

Now, compacting is an O(n) memory move plus additional calculation to figure out optimal-ish layout.

(Can be amortized, but it's same as amortizing hash table resizing. Optimal layout cannot be decided without profiling, and that's expensive.)

Re:Why databases use ordered indexes but programming uses ha... by dajohnson89 on Monday December 09, 2019 @11:16PM (#21749538)
doesn't this force you to define a comparator for everything though?
Re:Why databases use ordered indexes but programming uses ha... by _greim_ on Tuesday December 10, 2019 @01:38AM (#21750234)
As opposed to defining a hash code for everything, yes. Parent asserts that writing obviously-correct comparators is easier.
Re:Why databases use ordered indexes but programming uses ha... by LgWoodenBadger on Tuesday December 10, 2019 @11:22AM (#21753866)
IDEs have had good equals/hashcode auto-generation mechanisms for years.

Java itself introduced Objects.hash(Object...) in 1.7 https://docs.oracle.com/javase/7/docs/api/java/util/Objects....

It's never really been an issue at all to implement equals/hashcode, which you're required to do for effectively all Collections-based usage anyway.

Re:Why databases use ordered indexes but programming uses ha... by 91aintprime on Tuesday December 10, 2019 @10:43AM (#21753467)
funny enough, a hash code defines an order on a set
Re:Why databases use ordered indexes but programming uses ha... by tempguy9999 on Tuesday December 10, 2019 @01:08PM (#21754990)
Is it in any sense a useful order though? You can assign any order to any set of objects, but you'd want it to be useful. A hash - the more hash-ey it is, the less the order means AFAICS.
Re:Why databases use ordered indexes but programming uses ha... by n3k5 on Wednesday December 11, 2019 @09:03AM (#21762369)
For the purpose of making a tree data structure, all linear orderings would be equally useful. Hashing only provides a partial order, so it's important to choose a hash function that won't assign the same hash to large batches of items.

But it only needs to have this property for hashes of one particular size, whereas a hash map requires a function that behaves well for a variety of hash sizes. So there's an opportunity for using a cheaper hash function.