News for hackers. Stuff that matters.
Re:How Google Code Search Worked (2012) by burntsushi on Thursday March 12, 2020 @07:27AM (#22555660)
I am planning on adding functionality like this to ripgrep. If folks have opinions on how it should work, I'd love to hear from you! https://github.com/BurntSushi/ripgrep/issues/1497
Re:How Google Code Search Worked (2012) by hangtwenty on Thursday March 12, 2020 @11:21AM (#22557758)
Thank you for ripgrep! I will mention it in that thread too but I'm AFK so before I forget...

I'm imagining a "drill down" TUI with rg and fzf. fzf can be good for both filenames and other filter-downs. Thinking of breadcrumbs and easily stepping forward or backward, ability to easily bookmark/"pin" parts of search paths as presets for easy reuse later, etc.

EDIT: I recognize this would be outside of the scope of rg itself, I'm voicing it in case it sparks ideas about the functionality you're thinking of adding. I'll think more about it and see if I can explain better

Re:How Google Code Search Worked (2012) by rkhassen9 on Thursday March 12, 2020 @02:32AM (#22554341)
Code search was “too good to be true, gotta pinch myself” awesome. I miss it to this day.

Here is how I used it - I’d type in some code I was working on and the search result would show similar code and how it was used. Great for debugging and thinking by looking at similar solutions. Sigh.

Re:How Google Code Search Worked (2012) by tehlike on Thursday March 12, 2020 @03:05AM (#22554487)
For me it was f:cc$ f:contentads some keyword I wanted to learn more about. Then a bunch of cross refs.
Re:How Google Code Search Worked (2012) by londons_explore on Thursday March 12, 2020 @06:54AM (#22555478)
You can still play with it here: https://cs.chromium.org/
Re:How Google Code Search Worked (2012) by fzzfff on Thursday March 12, 2020 @07:44AM (#22555754)
wow this site is so laggy, even on chromium
Re:How Google Code Search Worked (2012) by londons_explore on Thursday March 12, 2020 @05:32PM (#22562003)
Google engineers have superfast high end desktops and laptops, and 10 gbit internet connections. They don't tend to optimise their internal tools for low spec machines or internet connections.
Re:How Google Code Search Worked (2012) by hangtwenty on Thursday March 12, 2020 @11:09AM (#22557633)
Re:How Google Code Search Worked (2012) by j_knowles on Thursday March 12, 2020 @07:47AM (#22555780)
This gives me so much nostalgia!
Re:How Google Code Search Worked (2012) by greenyouse on Wednesday March 11, 2020 @11:24PM (#22553394)
Weird, I was just looking into google code search this weekend so I could use something like it on my work computer. It's a little surprising that big co git storage companies don't have a proper code search tool as part of their package. I use Bitbucket right now but the search is built over Elasticsearch and special characters aren't handled so regular expressions won't work.

A couple open source projects that I've seen are Hound and Zoekt. Hound actually uses this code search backend with a nice frontend in React. Zoekt is what I was going to use since it scales really well, is faster, and has good search operators for filtering by repo name, language, etc. Google was using Zoekt until recently for code search across all their open source repos.[0]

[0]https://cs.chromium.org/

Re:How Google Code Search Worked (2012) by Cyph0n on Wednesday March 11, 2020 @11:56PM (#22553587)
We use Opengrok at Cisco. It’s a pretty barebones interface, but it works well.
Re:How Google Code Search Worked (2012) by sebastos on Thursday March 12, 2020 @01:39AM (#22554092)
Interesting, because the search function that Cisco's intranet provides for documents and such is perhaps the single most useless piece of technology I've ever encountered. You could search something like "401k plan" and you'd get marketing materials written in Japanese. Utter trash.
Re:How Google Code Search Worked (2012) by dvirsky on Thursday March 12, 2020 @01:21AM (#22554019)
The current internal CodeSearch is one of the best tools available for Google engineers. It's really a marvel.
Re:How Google Code Search Worked (2012) by tehlike on Thursday March 12, 2020 @03:06AM (#22554495)
Code search, critique, Piper/citc/cider are amazing for developing

Power drill is fantastic for drilling down data. So much money was made thanks to this one.

Re:How Google Code Search Worked (2012) by dvirsky on Thursday March 12, 2020 @12:01PM (#22558175)
Piper was actually a big source of frustration for me. Yeah it's dead simple, but once you have a CL chain, you're entering a world of pain. I've switched to Fig a while ago and haven't looked back. Beyond a tiny fix I'll start editing from CS or a throwaway citc client, it's just simpler to use fig. I've been able to juggle 4-5 CL chains easily and it makes my workflow much easier. Also splitting CLs before review is much simpler with Fig.
Re:How Google Code Search Worked (2012) by ignoramous on Thursday March 12, 2020 @04:40AM (#22554891)
Are these code tools built using https://github.com/kythe/kythe ? Any other OSS projects by Google that back these?
Re:How Google Code Search Worked (2012) by cameronbrown on Thursday March 12, 2020 @06:40AM (#22555418)
I believe so. Kythe seems to have spawned out of the internal CodeSearch.
Re:How Google Code Search Worked (2012) by hangtwenty on Thursday March 12, 2020 @11:12AM (#22557674)
Something that is not by Google but which you would probably like - SourceGraph. Has commercial options but it is OSS.

https://about.sourcegraph.com/

Re:How Google Code Search Worked (2012) by jules on Thursday March 12, 2020 @03:43AM (#22554639)
Do you have a list of all major Google tools and why they're better than what's available elsewhere? (if so)
Re:How Google Code Search Worked (2012) by antoinealb on Thursday March 12, 2020 @06:42AM (#22555423)
In my opinion, what makes them really great is the tight integration that they have. For example, since the whole company uses one build system and one single repository, you can build a truly awesome IDE that knows about every library in the company and can autocomplete for it. Same for code search, where cross references are accurate and work cross languages (for example a class generated from Protobuf).
Re:How Google Code Search Worked (2012) by londons_explore on Thursday March 12, 2020 @06:57AM (#22555494)
Outside Google, the percentage of my coding time I spend hunting some dependencies source tree for the relevant header files or documentation or "Where on earth is this constant defined" is huge.

With codesearch, answering those kind of questions is near instant.

Re:How Google Code Search Worked (2012) by cameronbrown on Thursday March 12, 2020 @06:41AM (#22555420)
Re:How Google Code Search Worked (2012) by dvirsky on Thursday March 12, 2020 @12:10PM (#22558262)
So CodeSearch, Critique, Borg, Sherlog, Cider come to mind as top notch tools that are not available outside. As far as libraries go, the C++ Fibers thing is incredible and I don't think it's open.

Blaze is amazing (albeit a bit slow) but Bazel should be more or less the same, haven't used it. Dremel, Spanner, Tensorflow, Proto, grpc are all available outside. Abseil (https://abseil.io/) is a great library available to everyone.

Re:How Google Code Search Worked (2012) by kannmig on Wednesday March 11, 2020 @10:40PM (#22553109)
Re:How Google Code Search Worked (2012) by ninkendo on Thursday March 12, 2020 @06:28AM (#22555354)
I’ve always wondered why regular expressions and full text indexes are the best thing we expect out of a code search engine.

I mean, we’re talking about code here. Text meant to be interpreted and understood by a compiler. Why can’t we do better?

Why can’t I say “show me everyone that’s calling this function”, like an IDE lets me do? Or “show me functions that accept <type> as one of their arguments and return <type>”, in a way that integrates with the real grammar/AST of the language(s) in question, without resorting to clunky regular expressions?

I should be able to write structured queries against a codebase, with regexes being just one part of that query language.

Re:How Google Code Search Worked (2012) by michaelt on Thursday March 12, 2020 @07:12AM (#22555568)
Only some languages can readily support such features.

For example C has a preprocessor and linking step driven by a build system. And C has a bunch of different build systems available, some of which are procedural rather than declarative.

Maybe you'll need to support package management - if a function signature calls for a CopyOnWriteArrayList do you need to know what the subclasses and superclasses of that type are? Do you need to resolve all the dependencies to be able to do that?

If you're thinking "No problem, everyone compiles their programs in CI anyway" - are you happy to skip indexing unused code and uncompileable code?

And of course you'll be chasing after language and build tool changes - not only to one language, but every language.

On the other hand, a nice simple grep? Sounds much simpler to me.

Re:How Google Code Search Worked (2012) by londons_explore on Thursday March 12, 2020 @06:51AM (#22555457)
Modern Google Codesearch does exactly this.

You can try it out here: https://cs.chromium.org/

In typical google style, the documentation is all google-internal, but by clicking bits of source code you should figure out most of the commands.

Doesn't work well on mobile unless you have a beefy CPU - sorry!

Re:How Google Code Search Worked (2012) by ovi256 on Thursday March 12, 2020 @05:14AM (#22555041)
It's worth taking a look at livegrep (Try it here: https://livegrep.com/search/linux) as an alternative to your git provider's code search.

Quoting patio11: "I intend to boot up a livegrep instance on the first day of every startup for the rest of my life. It borders on miraculous."

It is indeed very good.

Re:How Google Code Search Worked (2012) by hnaccy on Thursday March 12, 2020 @03:20AM (#22554551)
I'm baffled at how bad github code search even for enterprise github deployments. Is there some third party solutions that are popular or standard?
Re:How Google Code Search Worked (2012) by newusertoday on Thursday March 12, 2020 @04:27AM (#22554825)
try sourcegraph https://github.com/sourcegraph/sourcegraph . It is backed by company so you can buy enterprise support as well if required.
Re:How Google Code Search Worked (2012) by generatorguy on Wednesday March 11, 2020 @10:56PM (#22553224)
for some reason I always thought russ cox was 60 years old (even 15 years ago) with a big grey neck beard. boy was I wrong!
Re:How Google Code Search Worked (2012) by kick on Wednesday March 11, 2020 @11:54PM (#22553572)
He's not that old, but he's not that young, either. He worked on Plan 9 for like a decade before this happened. A few years before he put out this blog post, he finished his PhD thesis. While a bit silly to hire someone with that much experience as an intern, it is how most PhDs are treated.
Re:How Google Code Search Worked (2012) by generatorguy on Thursday March 12, 2020 @01:03AM (#22553946)
I was actually thinking of Alan Cox!
Re:How Google Code Search Worked (2012) by kick on Thursday March 12, 2020 @01:14AM (#22553985)
Alan Cox is in his fifties!
Re:How Google Code Search Worked (2012) by enneff on Thursday March 12, 2020 @01:11AM (#22553974)
Try to imagine my shock and dismay when, after working with Russ for a few years, I discovered that we were the same age.
Re:How Google Code Search Worked (2012) by bminor13 on Thursday March 12, 2020 @11:12AM (#22557673)
A minor note in the article reads:

> To minimize I/O and take advantage of operating system caching, csearch uses mmap to map the index into memory and in doing so read directly from the operating system's file cache. This makes csearch run quickly on repeated runs without using a server process.

Does anyone know some resources where I can read more about this technique? (how-to, pros/cons, caveats, etc.) I'm interested in figuring out the best way to have a commandline tool persist state that it can quickly access across multiple runs, but so far a background server process is the only technique I'm familiar with.

Re:How Google Code Search Worked (2012) by akkartik on Thursday March 12, 2020 @12:20PM (#22558391)
Yeah, you have to run a server. When a process exits, all its mmap'd pages are reclaimed. Just like any other memory.
Re:How Google Code Search Worked (2012) by bminor13 on Thursday March 12, 2020 @01:53PM (#22559639)
But this excerpt says that this technique obviates the need for a server process? Are they saving the contents of memory into files using mmap, and then using this state on every run?
Re:How Google Code Search Worked (2012) by akkartik on Thursday March 12, 2020 @01:58PM (#22559703)
I just reread your quoted passage and noticed the reference to the OS's file cache. Ok, this is outside my knowledge :)
Re:How Google Code Search Worked (2012) by chubot on Thursday March 12, 2020 @12:09AM (#22553685)
Re:How Google Code Search Worked (2012) by YesThatTom2 on Thursday March 12, 2020 @06:37AM (#22555403)
Wait... /usr/include on Mac OS Lion included constants for DATAKIT?

Was that a joke? Does someone have a Lion system around that can verify?

Re:How Google Code Search Worked (2012) by mjlee on Thursday March 12, 2020 @10:40AM (#22557326)
https://github.com/apple/darwin-xnu/blame/a449c6a3b8014d9406...

It's been there for 11 years in this repo. Along with DECnet. Suppose there's no real drive to remove it.

Re:How Google Code Search Worked (2012) by ausjke on Thursday March 12, 2020 @12:02PM (#22558179)
opengrok is the best I have used so far, java-based, can search _huge_ code base (e.g. android source code, linux kernel, whatever you throw at it)

https://oracle.github.io/opengrok/