News for hackers. Stuff that matters.
Re:Homogenization of scientific computing – Python is eating... by Xcelerate on Thursday January 09, 2014 @01:06PM (#7032047)
I do scientific computing, and Python is one language I never actually got around to learning for some reason. However, as a long-time hobby, I do have an interest in programming languages so I like exploring things like Haskell, Clojure, Lisp, etc.

One language I'm really excited about for scientific computing though is Julia. From a language-design perspective, it's beautiful. It was actually thought out rather than kludged together. I've been trying to gradually use it more and more for my research, but the only problem I've found so far is the large mental context-switch I make going from my usual languages to Julia. It's hard to tell what Julia code will be the most performant because there's many ways of doing the same task. I saw someone in the comments on this page mention that you can hand-tune the LLVM generated output within the REPL itself. I imagine this would be very useful if I can get around to learning it (anyone know a good tutorial?)

Re:Homogenization of scientific computing – Python is eating... by xfax on Thursday January 09, 2014 @01:33PM (#7032222)
The big question for me is whether Julia will be able to maintain its "purity" as it gains adoption.

R probably started out "beautiful" and "thought out" but has lost that edge with years of community driven development. It's also what make it so damn useful -- you can pretty much find anything on CRAN, often multiple implementations of it.

Re:Homogenization of scientific computing – Python is eating... by mbq on Thursday January 09, 2014 @02:50PM (#7032710)
R is actually one of the most pure languages out there; it basically says "I have vectors; they can have missing values, be nested, and can have other vectors as attributes. And I have functions with lexical scoping. Now go and build the rest as you like." So people did this, one better, one worse -- but the core and beautiful stuff here is that all those approaches will work together and just do the job.
Re:Homogenization of scientific computing – Python is eating... by michaelhoffman on Thursday January 09, 2014 @01:54PM (#7032357)
Sorry, no. Even the core S language would not be described by very many users as "beautiful" according to modern standards.
Re:Homogenization of scientific computing – Python is eating... by epaladin on Thursday January 09, 2014 @07:30PM (#7034253)
Once all of Bioconductor is ported to Julia, I will probably shed some joyous tears.

R is really nice for some things, because of how easy it is to work with table-like data, and operations like applying a function over a vector or a matrix, etc are cake. However, it's also slow, leaky (and sometime seems to use more memory after gc()), and often difficult to debug because so many things fail silently. If it wasn't for so many standard bioinformatics packages being available only for R, I'd probably be using Python. Once we stop running microarrays, I'll probably be using R much less.

Re:Homogenization of scientific computing – Python is eating... by tokipin on Thursday January 09, 2014 @02:15PM (#7032506)
python is a deliberately-straightforward language. i don't see how anyone couldn't become highly proficient in it after writing one or two scripts
Re:Homogenization of scientific computing – Python is eating... by benhoyt on Thursday January 09, 2014 @02:50PM (#7032712)
Deliberately-straightforward -- agreed. But "highly proficient" after writing one or two scripts? That's quite a stretch.

For instance, one of the questions I give in phone screens is for the candidate to write a program to count the number of occurrences of unique words in a text file. The "after writing one or two Python scripts" approach is something like this:

    counts = {}
    f = open('test.txt')
    lines = f.read().split('\n')
    for line in lines:
        for word in line.split(' '):
            if word:
                word = word.lower()
                if word in counts.keys():
                    counts[word] += 1
                else:
                    counts[word] = 1
    f.close()
    count_items = [(count, word) for word, count in counts.items()]
    count_items.sort()
    for count, word in reversed(count_items):
        print word, count
Whereas the "highly proficient" (and much simpler and more Pythonic) approach might look something like this:

    import collections
    counts = collections.Counter()
    with open('test.txt') as f:
        for line in f:
            for word in line.lower().split():
                counts[word] += 1
    for word, count in counts.most_common():
        print word, count
Re:Homogenization of scientific computing – Python is eating... by wyager on Thursday January 09, 2014 @05:22PM (#7033703)
My solution:

    lines = [line for line in open("bible.txt")]
    words = [word for line in lines for word in line.split()]
    counts = {word:0 for word in words}
    for word in words:
        counts[word] += 1
No imports needed. Linear time. A bit inefficient in the dictionary comprehension, but easy to read.

The "lines=" and "words=" can be compressed into one line, but I figure this is a bit easier to read for people who aren't familiar with nested list comprehensions.

Re:Homogenization of scientific computing – Python is eating... by ajayjain on Saturday January 11, 2014 @03:21AM (#7042031)
I'm not too familiar with Python, but this is case sensitive unlike benhoyt's examples. Would this be case insensitive?

    lines = [line for line in open("bible.txt")]
    words = [word.lower() for line in lines for word in line.split()]
    counts = {word:0 for word in words}
    for word in words:
        counts[word] += 1
Re:Homogenization of scientific computing – Python is eating... by gejjaxxita on Tuesday January 14, 2014 @10:19PM (#7061452)
yes!
Re:Homogenization of scientific computing – Python is eating... by burntsushi on Thursday January 09, 2014 @06:25PM (#7033984)
> A bit inefficient in the dictionary comprehension, but easy to read.

If you allow one import to sneak in, I think a `defaultdict` would do nicely. :-)

Re:Homogenization of scientific computing – Python is eating... by drkevorkian on Thursday January 09, 2014 @03:24PM (#7032934)

  for word, count in Counter(open('test.txt').read().lower().split()).most_common():
      print word, count
Re:Homogenization of scientific computing – Python is eating... by Goladus on Thursday January 09, 2014 @08:05PM (#7034392)
I like this one. Though I would do it like this to keep the lines under 80 characters:

    words=open('test.txt').read().lower().split()
    for word, count in Counter(words).most_common():
        print word, count
(edited per child comment)
Re:Homogenization of scientific computing – Python is eating... by benhoyt on Thursday January 09, 2014 @09:14PM (#7034663)
Yeah, those are nice -- and may actually be more efficient on smaller files, as you're only doing the lower() once on a big string. However, for big files you don't necessarily want to read the whole thing in at once.

One nitpick: it's Pythonic (I think) to just name the list of words "words" rather than "word_list".

Re:Homogenization of scientific computing – Python is eating... by Goladus on Friday January 10, 2014 @09:29AM (#7036694)
Yes that's a classic tradeoff, a proficient programmer will have to pick one.

Personally I always read entire files into memory first unless I have reason to believe memory will be an issue or need to program defensively against malicious/careless input. The code is always much cleaner and easier to read and if you need to do a second pass on the data you don't need to re-read it from disk.

Re:Homogenization of scientific computing – Python is eating... by krapp on Friday January 10, 2014 @06:25PM (#7040383)
Here's mine for what it's worth, since this is one of the Google python assignments. Admittedly, i'm not very proficient at all.

   def get_file_words(filename):
     file = open(filename, 'rU')
     words = {}
     for line in file:
       for word in [word.lower() for word in line.split()]:
         if not word in words.keys():
           words[word] = 1
         else: words[word]+=1
     return words


   def print_words(filename):
     wordcount = get_file_words(filename)
     for word in wordcount:
       print word, wordcount[word]
Re:Homogenization of scientific computing – Python is eating... by eeshanC on Friday January 10, 2014 @02:12AM (#7035484)
I'd like to jump in with a little R here- it not all that difficult in "that" either!

open(con <- file('text.txt'))

text = readLines(con, n= -1L) # n is number of lines to read, -1L means read all of it

words = strsplit(text,split = " ")

counts = table(unlist(words))

I put this in because the good thing about R is that it provides functions for many such mathematical operations. And along with this, I'll say something any self-respecting pythoner will know- Less is better than more.

Re:Homogenization of scientific computing – Python is eating... by yohanatan on Thursday January 09, 2014 @03:04PM (#7032796)
Actually that's not 'highly proficient'. This is:

    def read_words(words_file):
        return [word for line in open(words_file, 'r') for word in line.split()]
    len(set(read_words('test.txt')))
Re:Homogenization of scientific computing – Python is eating... by bnolsen on Thursday January 09, 2014 @03:26PM (#7032946)
smashing tons of crap together isn't necessarily "highly proficient". In some cases it makes things harder to read and/or harder to maintain and many times certainly harder to edit.
Re:Homogenization of scientific computing – Python is eating... by eeshanC on Friday January 10, 2014 @02:40AM (#7035567)
True that. But there is also another side to this story, where such nested calls translates naturally to what we are thinking, i.e.

length(unique(text)) comes from the thought "How many(length()) unique (unique()) words are there in this text(text)?"

Re:Homogenization of scientific computing – Python is eating... by mbq on Thursday January 09, 2014 @03:09PM (#7032839)
R version for comparison (;

    length(unique(scan('test.txt',character(),sep=" ")))
Re:Homogenization of scientific computing – Python is eating... by eeshanC on Friday January 10, 2014 @02:17AM (#7035496)
Very clean. 1 small change to make it count of frequencies-

table(scan('test.txt',character(),sep=" "))

Re:Homogenization of scientific computing – Python is eating... by agumonkey on Thursday January 09, 2014 @03:28PM (#7032962)
Don't know when set comprehensions and resource handlers were introduced so that might not run everywhere

    def read_words(words_file):
        with open(words_file, 'r') as f:
            return {word for line in f for word in line.split()}
Re:Homogenization of scientific computing – Python is eating... by pwang on Thursday January 09, 2014 @03:54PM (#7033147)
len(set(open("test.txt").read().split()))
Re:Homogenization of scientific computing – Python is eating... by wyager on Thursday January 09, 2014 @05:37PM (#7033784)
Oh, I like this one. Not quite what OP asked, but very clean.
Re:Homogenization of scientific computing – Python is eating... by jessaustin on Thursday January 09, 2014 @04:43PM (#7033470)
GP asked for counts, not uniques.
Re:Homogenization of scientific computing – Python is eating... by yohanatan on Thursday January 09, 2014 @05:17PM (#7033675)
Oops, you're right.

    words = open("test.txt").read().split()
    [(k, len(list(g))) for k, g in groupby(words, lambda x: x)]
Re:Homogenization of scientific computing – Python is eating... by leephillips on Thursday January 09, 2014 @05:58PM (#7033873)
Doesn't that require an "import itertools"?
Re:Homogenization of scientific computing – Python is eating... by yohanatan on Wednesday January 15, 2014 @02:22PM (#7065506)
Yup. Sorry.
Re:Homogenization of scientific computing – Python is eating... by yohanatan on Thursday January 09, 2014 @05:20PM (#7033692)
Oops, should have been this:

    words = open("test.txt").read().split()
    [(k, len(list(g))) for k, g in groupby(words, lambda x: x)]
Re:Homogenization of scientific computing – Python is eating... by hnriot on Thursday January 09, 2014 @08:57PM (#7034599)
except that's wrong, all you're doing is counting the number of unique words, and you didn't even consider Foo and foo as the same in your example. Part of proficiency is understanding the problem.
Re:Homogenization of scientific computing – Python is eating... by chocolate_ on Thursday January 09, 2014 @10:13PM (#7034884)
How does this type of question go during a phone screen? Does the candidate send the code to you afterward?
Re:Homogenization of scientific computing – Python is eating... by tokipin on Thursday January 09, 2014 @10:38PM (#7034961)
that first version is hilarious. i'd consider it hideous at any proficiency level in any language :D
Re:Homogenization of scientific computing – Python is eating... by leephillips on Thursday January 09, 2014 @02:47PM (#7032697)
The straightforward design makes it fast to go from zero to having a working proficiency. But I'm not sure about "highly proficient": even experienced Pythoners get tripped up by things such as mutable function arguments, and it's often not clear why some simple-looking code is running slowly, and how to speed it up.
Re:Homogenization of scientific computing – Python is eating... by michaelochurch on Thursday January 09, 2014 @01:29PM (#7032196)
What are your thoughts on Clojure's suitability as a go-to language for scientific computing (except in lower-level, high-performance scenarios that might recommend Julia)? I think it has serious potential in this field. It's not there yet, but it's getting there, and the core.matrix standardization helps a lot.
Re:Homogenization of scientific computing – Python is eating... by jamesjporter on Thursday January 09, 2014 @04:20PM (#7033344)
I agree with collyw; Lisp is too much for most scientists, who just care about getting their research done and can't be bother to learn all this weird FP/paren stuff [1]. Supplying an obvious, familiar syntax that looks like math written on paper for, e.g., matrix operations, is critical. This is actually one of the core tensions in Julia development imho: balancing having a sane, well designed language (from a programmer's perspective) vs. having a tool that allows scientists to quickly and easily crank out results.

[1]: Note; I am not insinuating that Lisp is bad, I like it personally. Just relaying the response you will get from most practicing scientists who are not trained as programmers.

Re:Homogenization of scientific computing – Python is eating... by michaelochurch on Thursday January 09, 2014 @08:46PM (#7034558)
Just relaying the response you will get from most practicing scientists who are not trained as programmers.

Is it that hard of a gap to cross? I was a math major who took a couple CS courses, and wouldn't say I was "trained as a programmer". I found Scheme pretty easy to get from the go.

I tend to think that anyone who can get a PhD in the physical sciences can become a half-decent programmer-- if the desire is there, the intelligence and work ethic being established (one hopes, at least) by the degree.

Re:Homogenization of scientific computing – Python is eating... by jamesjporter on Sunday January 12, 2014 @05:42PM (#7047801)
It's less a matter of ability and more a matter of having the time/motivation. If you took a bunch of practicing natural scientists and/or engineers and forced them somehow to enroll in a course teaching Scheme/Clojure/etc., would many of them do well? Almost certainly. If you gave them the choice between a Lisp and something like Julia or MATLAB to use in their everyday work when they need to do some computation? They'll probably choose the later because it's easy and familiar and doesn't have a huge up-front time investment; scientists tend to be very busy people.
Re:Homogenization of scientific computing – Python is eating... by collyw on Thursday January 09, 2014 @02:04PM (#7032438)
Its functional. Looking at the way bioinformaticians code, that will make it too complicated for many of them. SQL seems too much for many of them.