Re:How to Make Common Lisp go Faster Than C
by gibsonf1 on Wednesday May 21, 2008 @02:08PM
(#196337)
My co-founder tried out some of the techniques in the paper on our system: "I was skeptical, probably because I've never done much performance tuning like this. But I changed (triples) to create a 2-d array of fixnums instead of the default list of lists from the select. I added the declarations. It's about 100x faster. 100!" Very nice :)
Re:How to Make Common Lisp go Faster Than C
by xirium on Wednesday May 21, 2008 @03:11PM
(#196428)
When I mentioned that it may be possible for a web startup in beta to improve the speed of their code by a factor of 1024 over no particular timescale, I was downmodded by five points ( http://news.ycombinator.com/item?id=187692 ).
I can only advise that you shouldn't quit while you're ahead. Now that you've eliminated the most obvious performance bottlenecks, you may find that less obvious bottlenecks have emerged. Removing these could give you an additional factor of 10 improvement.
Re:How to Make Common Lisp go Faster Than C
by gibsonf1 on Wednesday May 21, 2008 @03:18PM
(#196443)
We're using the SBCL profiler on a regular basis to speed up the system whenever we find an area that feels a bit sluggish. We've made very large speed gains with only minor coding modifications.
Re:How to Make Common Lisp go Faster Than C
by xirium on Wednesday May 21, 2008 @03:36PM
(#196466)
I'm very impressed with your quantative approach. I'm also very curious about the type of modifications you're making. What proportion of changes are local changes which affect common cases, what proportion of changes are local changes which affect corner cases and what proportion of changes are architectural? Is it 70:20:10?
Re:How to Make Common Lisp go Faster Than C
by gibsonf1 on Wednesday May 21, 2008 @03:53PM
(#196478)
Thats a good question. Our workflow is to look at the profiler results, and work on the functions that are taking the most time. Sometimes we'll find a function being called far too many times for the page being viewed, so we'll look into the program logic to solve that. Our code is highly integrated, so most pages are using functions or methods common to many others, so a fix (or a bug) effects a large part of the system. The example 100x triples improvement sped up the entire system as triples are used at the core of our security and relations. This is a long winded way of saying I'm not sure what the proportions would be, a guess would be 60:10:30
Re:How to Make Common Lisp go Faster Than C
by xirium on Wednesday May 21, 2008 @03:29PM
(#196460)
I'm aware of Amdahl's law. I also know of a case where adding an index to a production database gave a factor of 1000 improvement. If your design started as a proof-of-concept, you made a rash assumption, you had a tight deadline or you're just plain incompetent then there's almost no lower bound to initial system efficiency.
Re:How to Make Common Lisp go Faster Than C
by scott_s on Wednesday May 21, 2008 @04:00PM
(#196491)
My comment was aimed at your down-modded comment. You can make 10% performance improvements in dozens of modules throughout your system, but if they only account for, say, 5% of the total execution time, it won't make much difference.
There is an easy work around for this. Submit the pdf link. Then comment with the pdf link. That adds one level of indirection, until pg gets around to fixing it.
This is hacker news, So I'm assuming people can use their browser and copy and paste.
1) copy link location.
2) paste link location in address bar.
2) remove scribd.com crap, leaving the original pdf link.
3) load original pdf link.
Re:How to Make Common Lisp go Faster Than C
by Hexstream on Tuesday May 20, 2008 @02:15PM
(#195100)
"While Lisp is mainly known for its basement on list processing, the Common Lisp standard features very efficient data types such as specialized arrays, structs or hash tables, making lists almost completely obsolete."
Re:How to Make Common Lisp go Faster Than C
by sah on Tuesday May 20, 2008 @03:07PM
(#195156)
In languages that can mutate data structures using side-effects, linked lists are rarely the best choice of data structure. Note, for example, that Python and Ruby have built-in array-based lists, but no linked lists. Similar structures are the default for lists in C++, Java, and C#.
That said, I'm pretty sure people use linked lists a lot in Common Lisp.
Re:How to Make Common Lisp go Faster Than C
by astine on Tuesday May 20, 2008 @04:34PM
(#195276)
Memory usage and random access times are far better in arrays, but for dynamic resizing, deletion, insertion, appending, and splitting, that is, operations that deal with data structure rather than the content, lists are far superior to arrays.
I used to hand-code linked lists back when I used C/C++ heavily. They have some very significant advantages for modeling certain kinds of data.
Re:How to Make Common Lisp go Faster Than C
by sah on Tuesday May 20, 2008 @07:23PM
(#195441)
I didn't mean to imply that linked lists aren't a useful data structure, just that arrays and array-based lists are well suited to an apparently wider range of problems.
Re:How to Make Common Lisp go Faster Than C
by Zak on Tuesday May 20, 2008 @09:10PM
(#195528)
Many of the interesting macros I've seen use list operations heavily to slice and dice their arguments and body code. Most experienced Lispers use macros quite a bit.
Re:How to Make Common Lisp go Faster Than C
by ExplodingBob on Tuesday May 20, 2008 @04:20PM
(#195258)
The Scribd file was degraded in quality to absolute illegibility, I had to grab the original link out of the URL just so I could read it.
Opt-out? It's annoying. They should just stop doing it for everyone perhaps, or just use it as a backup in case the site is knocked off line by the influx of social news traffic.
Re:How to Make Common Lisp go Faster Than C
by jcl on Tuesday May 20, 2008 @02:27PM
(#195116)
They do check. The sequence of events here seems to be:
- Person submits PDF article.
- HN rewrites article to scribd link.
- People complain about scribd link.
- Admin manually corrects link to point directly to PDF.
- A month later, another person submits same PDF article.
- HN rewrites article to scribd link, but because the original article was manually corrected, the scribd link doesn't match the original article, so it doesn't register as a dupe.
Re:How to Make Common Lisp go Faster Than C
by jcl on Tuesday May 20, 2008 @02:59PM
(#195148)
To be fair, the purpose of HN is to serve YC -- that's why we don't see ads on the site. People have suggested not scribd-ing everything before; I can only assume that the reason this has not yet happened is because the feedback we are providing is more valuable to YC than a few upset readers.
And this may well be the case: When most people have problems accessing a web page one way, they won't tell you directly; instead, they'll try to find another way to access it or give up. By providing only one way to read PDFs on a discussion board, you get access to this otherwise invisible negative feedback.
Re:How to Make Common Lisp go Faster Than C
by dreish on Tuesday May 20, 2008 @03:12PM
(#195164)
Yes, but not every idea turns out to be a good one.
Based on the unanimous jeers every time a PDF submission gets autoconverted, scribd appears not to be one of the good ones. Maybe it's time to try something else.
Re:How to Make Common Lisp go Faster Than C
by dpapathanasiou on Tuesday May 20, 2008 @02:30PM
(#195119)
But wouldn't either the original pdf link or the resulting scribd link show up as a duplicate in HN's dupe checker, since both were in the submitted list at one time or another?
Re:How to Make Common Lisp go Faster Than C
by jcl on Tuesday May 20, 2008 @02:37PM
(#195123)
Look at the link in http://news.ycombinator.com/item?id=172432 ... It's been changed from the scribd link to the original pdf file. I'm guessing they only check for dupes against a list of the current article links, and that they don't maintain a separate list of submitted links (which, except for cases like this one, would be the same list).
Re:How to Make Common Lisp go Faster Than C
by cheponis on Wednesday May 21, 2008 @05:47AM
(#195864)
We can hack around the scribd censorship. Everybody who aubmits a .pdf knows it will be converted into the useless scribd format. So, the original submitter needs to immediately post a comment that is the original .pdf link.
If HN scribd damage occurs on that link, then we will find other ways to route around that damage, for example, by using tinyurl.
I can only advise that you shouldn't quit while you're ahead. Now that you've eliminated the most obvious performance bottlenecks, you may find that less obvious bottlenecks have emerged. Removing these could give you an additional factor of 10 improvement.
Glad to hear your results were so dramatic.
Interestingly, the "comments" link for it points to http://news.ycombinator.com/item?id=196342 -- the original post without any comments shown!
There is an easy work around for this. Submit the pdf link. Then comment with the pdf link. That adds one level of indirection, until pg gets around to fixing it.
This is hacker news, So I'm assuming people can use their browser and copy and paste.
Uh?!?
That said, I'm pretty sure people use linked lists a lot in Common Lisp.
I used to hand-code linked lists back when I used C/C++ heavily. They have some very significant advantages for modeling certain kinds of data.
http://www.lrde.epita.fr/~didier/research/verna.06.imecs.pdf
Opt-out? It's annoying. They should just stop doing it for everyone perhaps, or just use it as a backup in case the site is knocked off line by the influx of social news traffic.
- Person submits PDF article.
- HN rewrites article to scribd link.
- People complain about scribd link.
- Admin manually corrects link to point directly to PDF.
- A month later, another person submits same PDF article.
- HN rewrites article to scribd link, but because the original article was manually corrected, the scribd link doesn't match the original article, so it doesn't register as a dupe.
If they're trying to send more traffic to that site, then we're S.O.L.
OTOH, if they're just trying to increase the number of documents in scribd's repository, then a parallel submission process would be the solution.
I.e., keep HN posts pointing to the original pdf, but also auto-submit to scribe, behind the scenes in a separate thread.
And this may well be the case: When most people have problems accessing a web page one way, they won't tell you directly; instead, they'll try to find another way to access it or give up. By providing only one way to read PDFs on a discussion board, you get access to this otherwise invisible negative feedback.
Based on the unanimous jeers every time a PDF submission gets autoconverted, scribd appears not to be one of the good ones. Maybe it's time to try something else.
We don't see ads because the site is run by millionaires and hardly takes any computing resources or bandwidth.
Ah, I didn't see that (I thought once it's been trans-scribd, it stays that way).
But that raises another question: if http://news.ycombinator.com/item?id=172432 points to the original pdf (i.e., http://www.lrde.epita.fr/~didier/research/verna.06.imecs.pdf) and the guy who submitted this also posted the same original pdf link, why wasn't it detected as a dupe right then?
If HN scribd damage occurs on that link, then we will find other ways to route around that damage, for example, by using tinyurl.
Down with scribd!!!
Hacker News != Digg