News for hackers. Stuff that matters.
Re:Removing Array Duplicates by hellllllllooo on Friday April 12, 2019 @02:49AM (#19642565)
My first thought was using a map to store what's been seen and run through the list keeping copying the values that haven't been seen to a new list in order or just marking them as dupes in place with the empty string.

This seems to be the "alt" case and is dismissed by the author but would like to hear a fuller explaination of why this is a problem?

Re:Removing Array Duplicates by roel_v on Friday April 12, 2019 @03:02AM (#19642634)
Too slow, too much hashing, too many copies.
Re:Removing Array Duplicates by bjoli on Friday April 12, 2019 @03:09AM (#19642675)
It will probably be faster than the code in the article.
Re:Removing Array Duplicates by roel_v on Friday April 12, 2019 @03:46AM (#19642857)
How?
Re:Removing Array Duplicates by bjoli on Friday April 12, 2019 @11:44AM (#19645826)
It requires a single pass through the array, whereas the article suggests different approaches that all go through the array multiple times (or even O(n^2)). For small lists this is probably ok, but for big lists it is a very slow approach.

Edit: I would probably use a bloom filter for this.

Re:Removing Array Duplicates by saagarjha on Friday April 12, 2019 @04:13AM (#19642987)
It requires a single linear pass instead of a convoluted sort.
Re:Removing Array Duplicates by roel_v on Friday April 12, 2019 @04:38AM (#19643107)
... what sort? Are talking about the same thing here?
Re:Removing Array Duplicates by saagarjha on Friday April 12, 2019 @04:49AM (#19643159)
The code in the article has a sort in it.
Re:Removing Array Duplicates by KirinDave on Friday April 12, 2019 @05:19PM (#19649112)
Why would that be "more copies" than building up a hash table is what I don't get.
Re:Removing Array Duplicates by bjoli on Wednesday April 17, 2019 @05:12PM (#19686526)
It requires many passes through the list, and even O(LogN) is probably enough to make it slower than a hash based O(n) approach depending on list size. Hashing is fast. If space is an issue, just use an appropriately sized bloom filter.
Re:Removing Array Duplicates by fluffycat on Friday April 12, 2019 @05:59AM (#19643481)
No I think it would be actually faster and simpler.
Re:Removing Array Duplicates by hellllllllooo on Friday April 12, 2019 @04:01AM (#19642944)
That's not really a fuller explaination.
Re:Removing Array Duplicates by roel_v on Friday April 12, 2019 @04:39AM (#19643110)
What more is there to explain? How is it not immediately obvious that the GP's approach is inferior the the OP's?
Re:Removing Array Duplicates by ww520 on Friday April 12, 2019 @05:35AM (#19643385)
It really is not obvious how OP's approach is faster than GP's approach.

OP's approach is by sorting which has complexity of O(m * N * logN) where m is the average key length and N is the size of the array. GP's approach with hash lookup has a complexity of O(m * N) where m is the average key length and N is the size of the array. The extra logN term makes OP's approach slower.

Re:Removing Array Duplicates by magicalhippo on Friday April 12, 2019 @09:14AM (#19644479)
Of course, for low values of N the constants dropped from the big-O notation start to matter. For very small values of N and a random-access input, even an O(N^2) approach may be optimal (for each input element, check for equality against previous elements of the input, output if no hit).
Re:Removing Array Duplicates by ww520 on Friday April 12, 2019 @08:41PM (#19650298)
Well, when N is small, any complexity analysis is moot.
Re:Removing Array Duplicates by magicalhippo on Sunday April 14, 2019 @05:35AM (#19658390)
Yes, but it's easy to forget and still apply it blindly in regimes where it's not really applicable anymore.
Re:Removing Array Duplicates by KirinDave on Friday April 12, 2019 @11:07AM (#19645427)
Many hash tables use tree structures rather than actual key hashing because it turns out that this is usually faster on modern machines. So many hash tables are technically O(n log n) for some high log factor like 32.

I've posted what I think is the optimal solution and the research behind it above, if you're curious how to hit O(n) time without brutal constants.

Re:Removing Array Duplicates by ww520 on Friday April 12, 2019 @08:47PM (#19650326)
Since the array size is known ahead of time, it's easy to initialize a hash table with an optimal capacity. No need to use fancy tree structure.
Re:Removing Array Duplicates by KirinDave on Saturday April 13, 2019 @01:35AM (#19651274)
The size of the table isn't the trick. It's the perfect hash function that you want. If you have that then you've already solved the problem at hand.