News for hackers. Stuff that matters.
Re:Perl 11 by nopacience on Tuesday October 23, 2018 @08:18PM (#18288626)
Perl5 is a great language. For internet and system related tasks it is great! It was created long time ago and evolved over the years.

One thing is true, it had a lot of advantage over other languages for a lot of years. However, now in 2018, other languages have also evolved and gained a lot of traction. One example is javascript/node which is now being used more and more. For a lot of time, a lot of frontend focused developers didnt really want to try backend development... specially because backend development was usually done in another language which was not javascript. So some never really attempted to do backend development. However when node was released, that opened a lot of new opportunities for the front end developers.

There are a lot of languages these days and many of them are capable enough to do many types of projects.

The important thing is, some languages have a bigger concentration of more capable engineers. One of these languages is perl. The perl programmers are excelent. They have real close experience with the kernel and systems and usually have great experience in complex backend tasks. Many are not as great with front end development. Some are.

Some projects can be done and deployed by 1 single perl developer. The same project if done in java for example might require more than 1 java developer.

The microsoft language creates a bubble and their developers sometimes dont know universal technical terms, they only know the microsoftish terms. Ie. they might not know what a "web server" is, but they know what "IIS" is. They might now know what a "hash" is, but they know what a "dictionary" is. Perl developers use universal terms.

Amazingly, javascript, especially ES6 is looking more and more like perl. Those that know both will agree sometimes it reminds of perl.

There are many projects people dont know about that use perl behind the scenes. But of course there are not as many perl programmers as there are in other languages and thats one reason you dont hear much about these projects.

Companies embrace more and more the modern perl development. Java, microsoft, node, python, ruby, etc are more popular of course. Specially because a lot of banks use java and they push java via universities. In the past I have seen labs with SUN machines on universities that teach Java. Microsoft also gives students free software licenses via universities. Thats another reason Microsoft and Java are very popular.

Re:Perl 11 by singingfish on Wednesday October 24, 2018 @04:11AM (#18290529)
The thing about perl which people really don't appreciate is that it scales. From a thow-away oneliner to a few hundred thousand lines of code [1]. But achieving that coherently takes a depth of knowledge of the language[2] and a reluctance to reinvent the wheel.

My main problems with perl is that I haven't found any normal programming activity I can't do with it, and that it's sucked enough of my time up over the past decade that people pay me and give me good working conditions in which to use it.

[1] But if you get to "a few hundred thousand lines of code" it's likely that there's some significant architectural mistake in your code. Architectural mistakes are not a perl-specific thing. Perl's biggest strength is also its biggest weakness - it's flexible.

[2] Depth of knowledge of a language is something that seems to be quite unfashionable in the industry at the moment, and some of perl's design decisions seem to be optimised for a style of knowledge that's also unfashionable.

Re:Perl 11 by totalperspectiv on Wednesday October 24, 2018 @08:58AM (#18291972)
Could you elaborate on point 2 there? I think that's an interesting point. I think it fits with a theme I've been mulling over regarding more people being tool users than tool builders.
Re:Perl 11 by DougWebb on Wednesday October 24, 2018 @10:37AM (#18292752)
One example I can think of is this: if you write perl using 'canonical' expressions, your code will often get compiled to optimized function calls. But if you use non-canonical expressions to perform the same operation, the compiled code won't be optimized and will run a lot slower.

An example would be using a map expression instead of a for loop. The map expression will typically run faster, even though its body is pretty much the same as the for loop's body.

Re:Perl 11 by rurban on Wednesday October 24, 2018 @07:13PM (#18297279)
No. map should be much slower than a for loop, because map has to cons the result list, while the for loop only does it for side effects, but ignores all the return values. If map is faster, then there's some thing wrong with the loop optimizer or loop run-time overhead.

In cperl there are many loop optimizations which are not in perl5. I'm pretty sure "for" is faster than "map" there.

Re:Perl 11 by lizmat on Wednesday October 24, 2018 @04:46PM (#18296200)
FWIW, in Perl 6 there is no real difference between a `map` and a `for` loop: in the case of the `for` loop, the final value of each iteration is simply discarded.
Re:Perl 11 by singingfish on Thursday November 01, 2018 @07:39AM (#18352967)
Most decent computer languages are heavily inspired by theoretical linguistics. So education tends towards that. Perl is inspired by practical linguistics. It's a total mess like English and amazingly expressive like English. This doesn't fit well with standard computing paradigms, but it does make perl a decent language for people from non-standard backgrounds.
Re:Perl 11 by lizmat on Thursday November 01, 2018 @08:35AM (#18353303)
> perl a decent language for people from non-standard backgrounds

I would argue "for any background". Why would you have to think like a computer to be able to program? Don't we have computers to do that?

Re:Perl 11 by singingfish on Friday November 02, 2018 @05:22AM (#18361638)
A lot of people from "standard backgrounds" have the theoretical linguistics approach baked in to their literacy. Perl on the other hand has a much easier time adapting to varying thinking styles. Here's a comment I made about a former colleague's code earlier today:

> one thing people like about perl is it can shoehorn into pretty much any mental model. [ colleague ]'s model is quite solid but also pretty mental

There's something to be said for enforcing a certain kind of mental model. Java and C# do a terrible job of this. Python does a reasonable job. Ruby is just perl done differently - seems to make the "mental" part a bit easier. Personally aside from perl the programming I've most enjoyed is arduino flavoured C, but that's just for fun. I do perl for fun and profit.

Re:Perl 11 by aasasd on Wednesday October 24, 2018 @03:40AM (#18290408)
Ironically, the universal terms for ‘hashes’ are ‘map,’ ‘associative array’ or, at the worst, ‘table.’ Even ‘dictionary’ makes sense and is definitely used outside of MS.

Hashes are completely different things, the use of them is an implementation detail and not necessary.

Re:Perl 11 by CydeWeys on Wednesday October 24, 2018 @04:18AM (#18290556)
OK, I'll bite. How do you get O(1) lookup without using a hashtable?
Re:Perl 11 by rurban on Wednesday October 24, 2018 @04:53AM (#18290746)
For a small amount of short keys hash tables perform worse than a simple linear scan. 127 is a good number but it depends on the design.

O(1) is completely missing the point of hidden constant factors. For many applications, like the one used in javascript, ruby or perl - map as objects with 3-7 short keys - hash tables are totally overblown.

For compile-time known keys you better create a perfect hash table, where the fastest could be a switch of pre-compiled word-aligned memcmp statements. http://blogs.perl.org/users/rurban/2014/08/perfect-hashes-an... Haven't done that yet with SIMD instructions, which should be much faster.

For mostly read maps (like e.g. for mysql applications) it's mostly faster to run-time compile a shared lib with a perfect hash table. Compilers are very fast with data-only.

For many other use cases a judy tree, compressed patricia tree, ctrie, HAMT, CHAMP or even a btree may be faster than a typical hash table. Esp. considering cache oblivious architectures. The modern swiss table doesn't look like an old-style hash table anymore, more like a patricia tree.

E.g. almost everything is faster than the default C++ hash table.

Re:Perl 11 by outadoc on Wednesday October 24, 2018 @04:24AM (#18290580)
That's not the issue. If you have a Dictionary interface that says it provides you with O(1) lookup, you don't care if it uses a hashtable or magic dust to do it.
Re:Perl 11 by CydeWeys on Wednesday October 24, 2018 @04:32AM (#18290626)
My point is that, since a hashtable is the only way to provide the desired performance characteristics, it's not unreasonable at all to say "I want a hashtable", instead of "I want a Dictionary implementation with O(1) lookup". The former is more succinct.
Re:Perl 11 by outadoc on Wednesday October 24, 2018 @04:36AM (#18290647)
I get ya, but I think you just described abstraction. Sure, if we're speaking low-level, we might as well use "hashtable".
Re:Perl 11 by CydeWeys on Wednesday October 24, 2018 @04:49AM (#18290724)
When someone says "hashtable" in the context of writing software, they're most often specifying desired performance characteristics, not the specific implementation details. This is how Perl does it, for example.

Also, there are Map implementations that don't have these performance characteristics, e.g. a TreeMap in Java which internally uses a red/black tree and thus provides logarithmic lookup and insertion. If I'm writing something that needs to do a lot of efficient lookups but only write "Map" in the spec, there's danger that the wrong thing will be used, and performance will suffer. It really does need a hashtable, not just a generic lookup interface.

Re:Perl 11 by bloak on Wednesday October 24, 2018 @06:10AM (#18291052)
In theory, I don't think hash tables are O(1): even if you ignore the atrocious worst-case performance and just look at average performance you still have to make the size of the hash depend on the number of entries, which brings in a factor of log(n), which is exactly what tree-based algorithms have.

In practice, the performance of a hash table definitely does depend on the number of entries because of the cache hierarchy and stuff like that.

I could easily be wrong about the theory: there are, of course, lots of different kinds of hash table and different ways you might formalise the meaning of "O(1)".

Re:Perl 11 by CydeWeys on Wednesday October 24, 2018 @07:52AM (#18291565)
Hashtables are typically considered amortized O(1) for insert and retrieve. The atrocious worst-case performance doesn't happen in common implementations because the hashtable gets enlarged before or when it occurs (and these resizes still allow for amortized O(1) performance).

And logarithmic growth in the key size is definitely not as bad as logarithmic growth in the number of nodes in a tree to traverse, because going from calculating e.g. a key size of 28 to 29 bits is not a big deal at all, but going from 28 to 29 levels in your tree means an entirely new memory lookup to grab that one node. The hash calculation is always cheap, whereas traversing a tree thrashes caches -- you only need one memory lookup in a hashtable, whereas you need logN in a tree. Those lookups are orders of magnitude more expensive time-wise than the simple integer math involved in calculating the hash.

Re:Perl 11 by palant on Wednesday October 24, 2018 @07:34AM (#18291465)
You are definitely right for most hash tables. Determining the right bucket is O(1) but there will be collisions. Open hashing solves it by creating a linked list within buckets which you will need to traverse. Closed hashing solves it by probing, you might test several buckets before finding the right entry. In both cases, you get O(log(n)) as the average lookup performance.

You actually get O(1) with perfect hash functions, but doing so requires choosing a hash function by the data set. This is normally only feasible for precalculated hash tables, not those that are being filled dynamically.

Re:Perl 11 by CydeWeys on Wednesday October 24, 2018 @07:59AM (#18291618)
The average lookup performance on a hashtable is definitely not O(logN) -- it's amortized O(1). Just Google for "hash table performance" and observe that all the results are saying O(1), not O(logN). I'd like to request some reputable sources from you showing O(logN).

The modal number of buckets you have to look into (or the modal size of the linked list) is 1. If it's more than 1, then your hash table is too full, and you need to enlarge it. This is admittedly an expensive operation, but it still amortizes out to O(1). And if you know in advance how many elements will be in a hashtable (this is common), then you never even need to resize.

Re:Perl 11 by bloak on Wednesday October 24, 2018 @10:30AM (#18292697)
It's important to distinguish between graphs of performance as measured on real hardware and theoretical results from computational complexity theory. It might be clearer to say "near-constant time over the region of interest" if you're talking about the former because "O(1)" has a precise meaning and refers only to what happens in the limit. Real programs usually only work for "small" n: they stop working long before the size of the input reaches 2^(2^(2^1000)), for example. But whether the algorithm is O(1) or not depends only on what happens when the input is bigger than that.

Another way of putting it: if you modify your practical program so that it runs on a theoretical machine that handles any size of input then you will probably find that another term appears in the execution time, one with a "log" in it, and that term eventually dominates when n gets big enough.

Yes, I am being a bit pedantic. But "O(1)" is mathematical notation and you have to be pedantic to do maths properly.

Re:Perl 11 by palant on Wednesday October 24, 2018 @09:41AM (#18292303)
It should be obvoius that hash table lookups are above O(1) as long as collisions exist. Tweaking the load factor can reduce the probability of collisions, but that's merely a constant factor to the lookup complexity. As to whether O(log(n)) is a good approximation for the average, we can certainly discuss that. It depends a lot on the algorithm, but https://probablydance.com/2017/02/26/i-wrote-the-fastest-has... shows (first graph) that the reality is typically even above logarithmic for some reason.
Re:Perl 11 by CydeWeys on Wednesday October 24, 2018 @06:47PM (#18297138)
> It should be obvoius that hash table lookups are above O(1) as long as collisions exist.

This doesn't follow at all. If the average number of lookups is 2 instead of 1, then it's O(2), which is O(1) because it's just a constant factor. The number of collisions needs to grow proportionally to the total number of elements being stored, which does not happen because the hashtable itself is also grown to keep the average number of collisions constant and low.

A hashtable that is 80% full has the same small number of collisions regardless of whether the hashtable has a thousand elements in it or a billion. Meanwhile, a BST would have to make 10 lookups in the first case and 30 in the second case -- that's logarithmic for you.

Your link shows lots of real performance figures which are dominated by CPU cache misses and other non-intrinsic factors. That's what's responsible for the reduction in performance on larger data sets, not anything intrinsic to hashtables themselves. If you look at the aforementioned first graph, the best hashmap performs at around at worst around 10ns up through ~200K elements, whereupon it blows out the fastest CPU cache and then takes around a max of 25 ns for the max size tested (~700M?). Factoring out the cache issue, it definitely does not look super-logarithmic to me. It might not even be logarithmic; I'd need to see raw data to know for sure (it's hard to tell by estimating from the graph).

Re:Perl 11 by palant on Wednesday October 24, 2018 @07:25AM (#18291412)
There are some options. For example, if the indexes are known to be within a certain range, a plain array will do better. For tables below a certain size, an array of key/value tuples might also perform better. An implementation could even switch between the underlying approaches depending on the data at hand, choosing between hash tables and something else depending on what should perform better.

Yes, the "hash" part is an implementation detail. The performance is also an implementation detail by the way. Often, tables will optimize for a different criterion such as memory efficiency. Lookup will be less efficient then. And, as another person noted, you cannot really expect O(1) from a hash table anyway.

Re:Perl 11 by Arkanosis on Wednesday October 24, 2018 @06:10AM (#18291051)
Lookup in a trie is O(L) where L is the length of the key; not that different if you take into account that to lookup into a hashtable, you also have to compute the hash of the key.
Re:Perl 11 by malkia on Tuesday October 23, 2018 @08:33PM (#18288718)
Actually Perl seems to be used quite a lot at Microsoft.

Look at page 3 in this publication: https://www.microsoft.com/en-us/research/wp-content/uploads/...

Re:Perl 11 by ty_a on Tuesday October 23, 2018 @10:05PM (#18289118)
Perl was considered as the defacto scripting language in Windows until they settled on PowerShell.
Re:Perl 11 by smacktoward on Tuesday October 23, 2018 @11:50PM (#18289572)
cough VBScript cough
Re:Perl 11 by malkia on Wednesday October 24, 2018 @11:48AM (#18293433)
I also remember quite a lot of Perl script in the WDK (Windows Driver Kit) back then. Was surprised by it that there were so many perl hackers @ Microsoft.
Re:Perl 11 by jug on Wednesday October 24, 2018 @02:54AM (#18290232)
How come? It didn’t ship with WSH but (only) JScript and VBScript did.
Re:Perl 11 by gaius on Wednesday October 24, 2018 @03:24AM (#18290346)
De facto, not de jure.

Perl’s COM module was excellent and made Windows very scriptable. This was back in the 90s.

Re:Perl 11 by wink on Wednesday October 24, 2018 @02:56AM (#18290239)
I saw the light[tm] too late.

My only real dealing was ~2001 and fixing shitty cgi-bin-webscripts in Perl. PHP4 was a huge improvement over that :)

As the years went on and I did more admin work I've come to really enjoy having perl5 available basically anywhere - the power of python with the availability of shell scripts.. but without the hassle. I'm also kinda sad they didn't write puppet in Perl but chose Ruby. I bet it would have prevented all my "too slow" and "needs too much RAM" problems with it...

Re:Perl 11 by donavanm on Wednesday October 24, 2018 @03:05AM (#18290270)
My memories over a decade old here but ... as I recall there was an early perl based protopuppet that luke wrote and used in his consulting roles in the early 2000s. This was after splitting from cfengine and moving towards a declarative model.

For better or worse Ruby really does seem a good fit for the puppet use case. A flexible DSL for expressing config intent, coupled with a very extensible/inheritance based parse & apply model.

Re:Perl 11 by ricardobeat on Wednesday October 24, 2018 @03:40AM (#18290409)
There are no “universal” terms. You used hash as an example, that’s already less universal than calling it a dictionary - it’s a hash map/table, and I’ve only ever heard this term from Perl developers.
Re:Perl 11 by rurban on Wednesday October 24, 2018 @04:59AM (#18290781)
That's because perl developers knew lisp and functional languages before and therefore avoided the misleading term "map", which is an operator to apply a function on a vector (=array) or list since the late 50ies. a map vs to map is misleading.

"table" is shorter than "dictionary" and also describes the structure better. A dictionary is mostly a book, or a list of keys (/usr/share/dict). prolog also used the term "table" or "tabling" for its caching of backtrackings, what we would call "memoize" or memoization via a hash table.

Re:Perl 11 by kazinator on Wednesday October 24, 2018 @11:22AM (#18293184)
> perl developers knew lisp and functional languages

It's a practical certainty that the vast majority do not, and its design (up to Perl 5, at least) smacks of total Lisp ignorance.

"Map" is used in mathematics as a kind of synonym for "function", especially from some arbitrary set to another one. If you assign the domain < 1, 2, 3, 4 > to the range < 3, 2, 4, 1 >, that's a map. We can draw it as two bubbles containing labeled dots, and draw arrows between the bubbles.

This sense of "map" is widely used in computing. For instance "keyboard scan codes get mapped to characters" or "a coordinate in the abstract canvas get mapped to the viewport, which is mapped to the screen" or "the temperature sensor is mapped to GPIO-s 13 to 17."

The "map" in Lisp's mapcar is exactly this sense. The map is the function that is being applied: mapcar is using the verb sense: take the car of every cell through the function: i.e. map each domain value to the range. This is called "projection" or "mapping"; "map" is just the verb describing what happens to each individual element.

Re:Perl 11 by kod on Wednesday October 24, 2018 @12:34PM (#18293942)
Re:Perl 11 by rurban on Thursday October 25, 2018 @11:58AM (#18301915)
Yes. If Larry knew how a LISP is implemented, many internals would have looked better. the symtab, the optree, the stack and esp. the lexicals. Just compare to the ruby internals.

But at least he knew the terminology, and got the concepts right. In opposition to php and python which failed miserably.

Re:Perl 11 by kazinator on Thursday October 25, 2018 @10:54AM (#18301334)
"I did dabble in LISP more than I did FORTRAN and it looks like oatmeal with toenail clippings."
Re:Perl 11 by collyw on Wednesday October 24, 2018 @04:11AM (#18290526)
Java always had hash maps as one of their map implementations. (I did use both languages years ago, so maybe that's why I noticed)
Re:Perl 11 by CydeWeys on Wednesday October 24, 2018 @04:24AM (#18290578)
For what it's worth, the three main Java map implementations are HashMap, which does what it says on the time, LinkedHashMap, which has some additional pointers to maintain insertion order (and thus has performance that suffers by a constant factor), and TreeMap, which is really a red/black tree which thus offers logarithmic lookup time instead of constant lookup time (i.e. it's not a hashtable at all).