Wikimedia narrows down the app sendin 90M requests to a pic of flower

Re:Wikimedia narrows down the app sendin 90M requests to a p... by oefrha on Thursday February 11, 2021 @05:31AM (#26100071)

> To narrow down the app, we decided to observe connections to the image from clients (phones) to our servers. We did this by opening the popular apps one-by-one and noting down the time. After doing this for all the apps, we then ran this query in Hive: SELECT * FROM wmf.webrequest WHERE year=2021 AND month=2 AND day=9 AND parse_media_file_url(uri_path).base_name='/wikipedia/commons/1/16/AsterNovi-belgii-flower-1mb.jpg' AND webrequest_source='upload' AND uri_host = 'upload.wikimedia.org' AND user_agent='-' AND ip=<IP>;

> We then found the specific app that was making the request by matching the time when it was opened and the time image was requested from our servers, restricting the results to the User-Agent '-' and from the IP we tested.

Unless I missed something, running mitmproxy/Charles etc. in front of the phone would have been way easier than querying the entirety of Wikimedia server logs and trying to match IP & timing windows.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by jrochkind1 on Thursday February 11, 2021 @12:09PM (#26104161)

"way easier" depends on what you are familiar with.

If you are familiar with "querying the entirety of wikimedia server logs" and do it all the time (the word "entirety" makes it seem like a big deal, but they clearly have tools meant for this that they use all the time)... and have never learned to use "mitmproxy/Charles etc" before....

It sounds like the "querying the entirety of the server logs" for this task probably took them tens of minutes at most. It would probably take me at least an hour or two to learn how to use "mitmproxy/Charles etc".

So "way easier"?

If you have to do this sort thing all the time, it might be useful to install and learn how to use "mitmproxy/Charles etc", why not? Certainly worth considering. But if the tools you have are working for you...

I mean, what they did seems like it worked to get them the answer and was pretty efficient, using the toolset they use all the time for dealing with wikimedia ops... Seems like some good detective work to me. I get the desire to point out other tools that would be well-suited for this kind of task, but why the need to point it out as if they did something wrong or not "way easier"? Sounds like what they did was pretty easy for them, and they didn't need to learn new tools to make it "way easier".

I enjoyed hearing about how they tracked this down, and found it useful. Pointing out how they didn't use the "right" tools just makes it less likely people will be willing to share their processes.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by oefrha on Thursday February 11, 2021 @12:54PM (#26104809)

You added way too many assumptions to a short non-accusatory comment.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by detaro on Thursday February 11, 2021 @06:03AM (#26100253)

I mean, the query doesn't look that complicated, and is something they'd obviously be already familiar with how to use - not that mitmproxy etc is particularly hard, but its another thing.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by sdfhbdf on Thursday February 11, 2021 @05:47AM (#26100169)

Judging by the last point they seemed to want to doublecheck findings and confirm before throwing accusations around.

> To further confirm this finding and to ensure that we had the correct app, we decided to log DNS queries from a phone by setting up a local resolver to capture DNS traffic. After pointing the phone towards it and launching the app, we noticed that it was indeed the one looking up upload.wikimedia.org on startup.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by Muley on Thursday February 11, 2021 @05:44AM (#26100153)

Certificate pinning has made this a pain in the arse.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by oefrha on Thursday February 11, 2021 @06:54AM (#26100551)

You can only pin your own certificate, not someone else’s. In this case you probably don’t even need SSL proxying to pin down the culprit, as I dare say not many apps connect to wikimedia on startup. You do need SSL proxying to be sure though.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by zwily on Thursday February 11, 2021 @07:53AM (#26100941)

The app may not load at all with mitmproxy if it has pinned its server cert though.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by oefrha on Thursday February 11, 2021 @09:01AM (#26101565)

No, you can selectively decrypt HTTPS requests for only some domains, and act as passthrough for others.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by ohgodplsno on Thursday February 11, 2021 @10:34AM (#26102574)

Nope. Starting from Android 10, unless an app has explicitly allowed user certificates (and no-one reasonably does, it's all behind a <debug-overrides> flag), you will not be able to MITM it. You may inject your certificates as much as you want. The only option is to have a device on which you have root access, which can push system certificates with adb. This pretty much only means the android emulator these days.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by oefrha on Thursday February 11, 2021 @11:07AM (#26103114)

I don’t use Android so I wasn’t aware of that. But that’s a completely separate concern from cert pinning which does not hinder decrypting third party connections at all.

Edit: after looking into this a bit, this is pretty nuts. How do enterprises inject certificates now?

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by olliej on Friday February 12, 2021 @02:48AM (#26111513)

re enterprise injection:

They don't. It's been made increasingly clear that allowing certs roots to infect unrelated apps is a Bad Thing. MDM profiles etc presumably allow internal certs to be deployed, but those are hopefully limited as countries, let alone companies, have attempted to use those mechanisms to spy on millions of people.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by matheusmoreira on Thursday February 11, 2021 @03:36PM (#26106778)

So it's still possible on rooted devices? Seems good enough.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by simonh on Thursday February 11, 2021 @05:41AM (#26100131)

If they have the logs in Elasticsearch or something similar it wouldn't be all that hard.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by whatatita on Thursday February 11, 2021 @01:23PM (#26105212)

It looks like it wasn't a big deal to perform the lookup, so I guess it comes down to what the engineer was familiar wit.

It does say later in the post that they used a local proxy to confirm their findings, too. Maybe they wanted to check from both sides, just to be sure.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by antimius3000 on Thursday February 11, 2021 @05:57AM (#26100228)

They didn't have 'the phone'. They only had traffic logs and had to find the app that matched a certain traffic pattern. And they did have 'something like ElasticSearch', it's all in the link, and even in this reply.

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by abluecloud on Thursday February 11, 2021 @06:02AM (#26100248)

I'm confused, it says

> We did this by opening the popular apps one-by-one and noting down the time

[ Parent | on HN ]

Re:Wikimedia narrows down the app sendin 90M requests to a p... by Rebelgecko on Thursday February 11, 2021 @06:17PM (#26108831)

It sounds like they did have the phone. Otherwise, Wikipedia is doing some sketchy stuff if they can figure out what apps are on my phone just from my IP address (IMO unlikely)

[ Parent | on HN ]