News for hackers. Stuff that matters.

Ask HN: Good source code search engines?

I recently saw an article on HN about how to use GitHub's search as a starting point when writing code. That indeed is a great trick and it has already helped me save a lot of time.

Unfortunately, GitHub's search is very lacking and most of the time instead of showing relevant search results it just shows lots and lots of duplicates for the main library.

For example, I was searching for this[1], but the search results show the Main SDK files and not the files created with the SDK (which is what would've helped me). I couldn't find a way to suppress the files or duplicate searches.

So I tried looking for a good source code search engine and I've find none so far. I've tried almost everything I could find on Google and Quora but most of these services are either discontinued (Google code, etc) or very bad.

So, I was wondering what you guys use to search source code? And are there any other tricks like GitHub's search that can increase coding productivity (I think searching SO is also very useful).

[1] https://github.com/search?l=php&q=createEnvironment+ElasticBeanstalkClient&ref=searchresults&type=Code&utf8=%E2%9C%93

( 13 comments | 20 points | on HN )
Re:Ask HN: Good source code search engines? by RmzH on Monday August 29, 2016 @09:27AM (#12381673)
Re:Ask HN: Good source code search engines? by supersan on Monday August 29, 2016 @09:32AM (#12381694)
Thanks, I've already tried both of them and they're not returning a lot of results, esp. not many relevant results.

But so far I think searchcode is okay because at least it does show something.

Re:Ask HN: Good source code search engines? by boyter on Monday August 29, 2016 @08:04PM (#12386420)
Hi, I'm the guy behind searchcode.com Could you possible let me know what results aren't working for you? Im well aware that its due for some improvements in the search area and feedback is always helpful.
Re:Ask HN: Good source code search engines? by supersan on Monday August 29, 2016 @08:16PM (#12386462)
First of all great work! Of all the search engines I tried (and I've tried at least 10), I found yours to be the next best thing after GitHub search.

Anyway, here are some links you can try for yourself:

No php results:

1. https://searchcode.com/?q=CORSConfiguration+CORSRules 1. https://github.com/search?l=php&q=CORSConfiguration+CORSRule...

Only SDK libraries, what I think would be useful if the search engine somehow prioritized code built on SDK/API over libraries (though I don't have the first clue how you would distinguish the two but that would be a killer feature because that is super useful)

2. https://searchcode.com/?q=createEnvironment+ElasticBeanstalk... 2. https://github.com/search?l=php&q=createEnvironment+ElasticB...

I can provide you with more examples, if that helps. Thanks!

Re:Ask HN: Good source code search engines? by boyter on Monday August 29, 2016 @08:34PM (#12386527)
Thank you.

That's one that comes up a lot. The trick is knowing the difference between the two which I am still thinking about how to implement.

If you could email me at [email protected] that would be very helpful. It looks like most of these are down to just not having enough code indexed.

For the record I tend to crawl bitbucket/gitlab more often then github simply because they do not have a search and I want that code to be available. I have started a full refresh of everything though so hopefully your results should improve in a few weeks.

Re:Ask HN: Good source code search engines? by supersan on Monday August 29, 2016 @09:10PM (#12386685)
That is really great news. One way I can think off to distinguish a library from other code is how many times the file shows up in your search results, i.e. the more the duplicates of a file you find (in various repos), the higher the chance it is a library.
Re:Ask HN: Good source code search engines? by boyter on Monday August 29, 2016 @09:23PM (#12386741)
Thats a pretty good point I suppose. I do track that information so it wouldn't be too difficult to implement I guess.
Re:Ask HN: Good source code search engines? by prodicus on Monday August 29, 2016 @02:38PM (#12384110)
Off topic but how would you build something like this from scratch. Just for the learning part of it. Any examples?
Re:Ask HN: Good source code search engines? by boyter on Monday August 29, 2016 @08:32PM (#12386515)
Re:Ask HN: Good source code search engines? by supersan on Monday August 29, 2016 @07:14PM (#12386184)
Yes, that is very interesting to me as well. I think Amazon's ElasticSearch can be used for searching. GitHub would be the obvious choice for crawling the source code.

Of course, there are a lot of roadblocks, like how much GitHub allows you to crawl the source code (sometimes you can find huge dumps on legit torrent sites or I don't remember the name of the site but it provides you with S3 buckets with crawled data where the requester has to pay for bandwidth but the data is free).

Then of course you will have to index the source code by language, project, author, date, etc. It's definitely not easy, which could be the reason why there is this huge gap.

Re:Ask HN: Good source code search engines? by ekr on Monday August 29, 2016 @03:09PM (#12384373)
I've recently ran into debian code search again, https://codesearch.debian.net/. Built on top of Russ Cox's Code search (https://github.com/google/codesearch), and quite well documented.

Can't say I've used it much, but it gives a few pointers into how you could build a better one yourself.

Re:Ask HN: Good source code search engines? by blkhp19 on Monday August 29, 2016 @05:12PM (#12385389)
There's a startup solving this! https://sourcegraph.com
Re:Ask HN: Good source code search engines? by supersan on Monday August 29, 2016 @07:17PM (#12386202)
Tried it, didn't return anything. Also for the life of me, I can't figure out a way for it to search php code.