News for hackers. Stuff that matters.
Re:Grep.app – Search across a half million Git repos by IncRnd on Tuesday February 07, 2023 @07:50AM (#34692043)
Please don't check your keys or passwords into github.

This was my first search to show AWS API Keys. [1] and my second to show google API keys. [2] Unfortunately, there are many more of these types of searches - and far too many results.

[1] https://grep.app/search?q=AKIA%5B0-9A-Z%5D%7B16%7D&regexp=tr...

[2] https://grep.app/search?q=AIza%5B0-9A-Za-z%5C%5C-_%5D%7B35%7...

Re:Grep.app – Search across a half million Git repos by capableweb on Tuesday February 07, 2023 @08:27AM (#34692417)
As far as I can tell, all the git repos come from GitHub, and GitHub automatically sends you a warning if you commit access keys of various kinds (and I seem to remember they even de-activate some, maybe AWS keys, automatically? Not 100% sure about this). Most of the keys shown are unlikely to be working, especially the ones older than just a couple of days (either owner notice, or someone else was faster than you, got access and regenerated new keys for themselves)
Re:Grep.app – Search across a half million Git repos by IncRnd on Tuesday February 07, 2023 @08:46AM (#34692631)
> or someone else was faster than you, got access and regenerated new keys for themselves

There are many people who do these types of searches - regularly. If there is any key material checked into gh, consider it immediately compromised, even if the repo is removed.

Re:Grep.app – Search across a half million Git repos by capableweb on Tuesday February 07, 2023 @08:48AM (#34692644)
Tons of automated scraping for these sort of things as well, with the whole flow of "check if key is working, check permissions, if authorized, generate new key and deactivate old one".
Re:Grep.app – Search across a half million Git repos by IncRnd on Tuesday February 07, 2023 @09:13AM (#34692960)
Yes. There are indeed several mitigations in place. However, my point was that within 5 seconds of looking at this tool I was able to uncover hundreds of potential issues. There are many other regexs I have on my list. Please don't fall into the trap of "we have anti-virus, so we can't get attacked."
Re:Grep.app – Search across a half million Git repos by capableweb on Tuesday February 07, 2023 @09:25AM (#34693136)
Sorry if I was unclear, my previous comment was about attackers doing that in order to compromise infrastructure, not that companies are doing it in order to defend themselves. But companies probably do it to defend themselves as well, but I'm not as familiar with that as with the other side of things.
Re:Grep.app – Search across a half million Git repos by greysteil on Tuesday February 07, 2023 @09:01AM (#34692809)
GitHub PM here, just confirming that that's correct (re: us sending detected secrets to the relevant service provider, who take action automatically). There's more detail here: https://docs.github.com/en/code-security/secret-scanning/abo....
Re:Grep.app – Search across a half million Git repos by iambateman on Tuesday February 07, 2023 @01:08PM (#34696499)
This is probably one of the single most important security measures on the whole internet.

Thanks for keeping an eye out.

Re:Grep.app – Search across a half million Git repos by abraham on Tuesday February 07, 2023 @09:16AM (#34693004)
Most of the Google API keys look like they are client side keys and should be configured to only work on specific origins or apps.
Re:Grep.app – Search across a half million Git repos by TYMorningCoffee on Tuesday February 07, 2023 @10:17AM (#34693891)
Your search has a lot of false positives like AKIAIOSFODNN7EXAMPLE
Re:Grep.app – Search across a half million Git repos by madeofpalk on Tuesday February 07, 2023 @11:55AM (#34695263)
Most of these actually look like fake keys to mock data in unit tests.
Re:Grep.app – Search across a half million Git repos by teaearlgraycold on Tuesday February 07, 2023 @01:08PM (#34696485)
Hacker noises - "I'm in"
Re:Grep.app – Search across a half million Git repos by kristopolous on Tuesday February 07, 2023 @01:50PM (#34697213)
----- BEGIN RSA NOT SO PRIVATE KEY -----
Re:Grep.app – Search across a half million Git repos by arp242 on Tuesday February 07, 2023 @01:57PM (#34697332)
Many of these are tests, false positives (e.g. base64 images or "Migration1599570560FixSlovakiaDisplayedAsSloveniaTest"), or examples from documentation.

Can't find a single match for an actual API key.

Re:Grep.app – Search across a half million Git repos by codalan on Tuesday February 07, 2023 @04:10PM (#34699701)
These are probably invalid by now.

https://grep.app/search?current=2&q=api_key%20%3D%20%22%5Ba-...

As others have posted, Github implemented an alerting system for keys/passwords/etc. that have been checked into the codebase. My guess is most of these will be invalidated by now, either by the developer requesting a new key (and not publishing it), or the provider invalidating it for them.

Re:Grep.app – Search across a half million Git repos by zikohh on Tuesday February 07, 2023 @05:18PM (#34700616)
https://sourcegraph.com/search?q=context:global+AKIA%5B0-9A-...

Sourcegraph is much better than this site.

Re:Grep.app – Search across a half million Git repos by asicsp on Tuesday February 07, 2023 @08:18AM (#34692328)
Previous discussion: "Show HN: Search code in GitHub repos using regular expressions"

https://news.ycombinator.com/item?id=22396824 (614 points | Feb 23, 2020 | 155 comments)

Re:Grep.app – Search across a half million Git repos by barbinbrad on Tuesday February 07, 2023 @08:27AM (#34692423)
At first glance, this is great!
Re:Grep.app – Search across a half million Git repos by khaledh on Tuesday February 07, 2023 @08:38AM (#34692546)
Previous discussions:

- https://news.ycombinator.com/item?id=28424845 (September 5, 2021 — 54 points, 6 comments)

- https://news.ycombinator.com/item?id=22396824 (February 23, 2020 — 614 points, 155 comments)

Re:Grep.app – Search across a half million Git repos by NoZebra120vClip on Tuesday February 07, 2023 @09:09AM (#34692900)
This is a great idea and I commend it and I hope others find it useful. However, I have a rather narrow use-case for GitHub searches, and in my initial experiments, this tool comes up 100% empty on common searches I use every day for work.

Most of my GitHub searches don't actually revolve around actual code-code. I search for text strings in prose that turn up in code searches. So the file types may be text or MarkDown instead.

Perhaps grep.app only indexes and presents actual code-code and filters out all the prose I'd be interested in. Sad, because there are actually a few instances where my searches involve code with many intricate special characters, and GitHub's current code search ignores all those.

Re:Grep.app – Search across a half million Git repos by coldpie on Tuesday February 07, 2023 @09:15AM (#34692993)
Put ".ai" at the end of the URL and you can legally use anything you find here regardless of license!
Re:Grep.app – Search across a half million Git repos by voytec on Tuesday February 07, 2023 @09:48AM (#34693449)
GitHub can be searched directly without any JS bloat.
Re:Grep.app – Search across a half million Git repos by bcjordan on Tuesday February 07, 2023 @10:30AM (#34694051)
I'd be curious to hear how the ingestion/search pipeline works — the searches are incredibly fast!
Re:Grep.app – Search across a half million Git repos by Scaevolus on Tuesday February 07, 2023 @12:18PM (#34695625)
Re:Grep.app – Search across a half million Git repos by birriel on Tuesday February 07, 2023 @11:06AM (#34694514)
Missed opportunity to call it "Grit."
Re:Grep.app – Search across a half million Git repos by throwanem on Tuesday February 07, 2023 @11:08AM (#34694538)
Not sure if anyone associated with grep.app is here, but if so, some feedback: it'd be nicer for your infra and my back button alike to coalesce search query changes and push a single history state once the value settles, rather than pushing state each time it changes :)
Re:Grep.app – Search across a half million Git repos by dang on Tuesday February 07, 2023 @11:20AM (#34694724)
Recent and probably related:

The technology behind GitHub’s new code search - https://news.ycombinator.com/item?id=34680903 - Feb 2023 (168 comments)

Re:Grep.app – Search across a half million Git repos by nikolahristov on Tuesday February 07, 2023 @12:25PM (#34695729)
There's Sourcegraph - https://sourcegraph.com/search, which is far more powerful and feature wide.
Re:Grep.app – Search across a half million Git repos by ptannor on Tuesday February 07, 2023 @01:34PM (#34696964)
Haha I can share my first use case - searching for repos I'm involved in, kind of like searching for myself on Google Scholar
Re:Grep.app – Search across a half million Git repos by moneywoes on Tuesday February 07, 2023 @02:16PM (#34697704)
How Is this different than GitHubs search?
Re:Grep.app – Search across a half million Git repos by 100k on Tuesday February 07, 2023 @02:50PM (#34698368)
Hi, I work on GitHub code search. Grep.app is based on Solr and indexes about 500,000 public repositories. As far as I know, the author has not shared their tokenization strategy, but based on the the results I expect they are indexing trigrams to perform regular expression matching.

The current GitHub code search is based on Elasticsearch and indexes more than 100M repositories. Its tokenization is based on whitespace, case changes (like CamelCase) and punctuation (like kebab-case) and strips out non-letter characters like < or { which is why it can't do exact match search.

Our new code search, currently in beta and indexing about 45 million repositories, uses an search engine we built in house that indexes content using a technique we call sparse ngrams. This allows us to execute searches faster than a trigrams index, while also being smaller than a positional trigram index. My teammate discussed some of the technology behind it in our blog post that was published yesterday: https://github.blog/2023-02-06-the-technology-behind-githubs...

Re:Grep.app – Search across a half million Git repos by xigoi on Tuesday February 07, 2023 @03:54PM (#34699439)
Search across a half million GitHub repos. It doesn't support any other git hosts than the proprietary one.
Re:Grep.app – Search across a half million Git repos by sebastianconcpt on Tuesday February 07, 2023 @05:38PM (#34700862)
Yet it doesn't have a clue about the existance of one of my 17 star GitHub public repos