How practical is it (or not) to intercept TLS traffic for IoT devices on my network where I don't have the option of setting a proxy?

I've been on a crusade the past few years to try and get my self-signed CA onto "smart" devices in my home, but with some it's difficult or I risk damaging expensive hardware.

I've tried various ARP poisoning techniques and such in the past with limited success.

I think that's a classic "it depends". I see two main challenges for IoT monitoring:

1. Getting the IoT traffic to your proxy machine. As you say this is fairly trivial when you can set a proxy, but a bit tricky otherwise as you need to run mitmproxy in transparent mode (https://docs.mitmproxy.org/stable/concepts-modes/). With some luck you can set a custom gateway on your device which simplifies the setup, otherwise you need to have the capability to set iptables rules on the router. I personally have a small $20 OpenWRT router dedicated for that. Alternatively you can run your own DNS server and point everything to mitmproxy. v7 now also supports SNI/host-header based proxying! The good news is that this setup is the same for all your devices and once it works, it works.

2. Getting the device to trust your (personal) mitmproxy CA cert. This one really depends on the particular IoT device and may involve some serious reverse-engineering if you are unlucky. I wish there would be a more of a "I should be able to inspect my own traffic" movement, but it's a tricky problem to solve.

The first part is the easy part, I have routers and run DNS, and have experience with IPTables, I can usually find ways to get hold of the traffic.

The second part is the challenge, and for some devices I've been unable to embed my own CA. I was hoping there was a proxy trick or some magic sauce I was missing. Perhaps where DHCP could automatically configure clients with a (insecure) proxy or something along those lines?

> I was hoping there was a proxy trick or some magic sauce I was missing. Perhaps where DHCP could automatically configure clients with a (insecure) proxy or something along those lines?

Nope, you're not missing anything.

Thanks for confirming, will save me some time messing around! Ill stick to getting my soldering iron out and EPPROM dumping/flashing.

You may be able to intercept a firmware update and load a binary poisoned with your own CA cert. (Lots of factors at play here, of course.)

I'm working on a similar problem (https://github.com/elahd/esp2ino/issues/16) with a project I maintain to sideload IoT device firmware (https://github.com/elahd/esp2ino).

I've been using both mitmproxy and IOXY (https://github.com/NVISOsecurity/IOXY), an intercepting proxy made specifically for MQTT. IOXY is a small, less mature project, but it's definitely worth checking out as a compliment to mitmproxy. Many devices managed via AWS IoT phone home over MQTT and, my limited experience aside, it looks like many don't bother validating certificates when authenticating over this protocol.

...because of enforcement on cheap Chinese etc goods.

Many IoTs have local search and scan, so block the DNS server at your firewall then provide your own DNS server via your own DHCP server.

mitmproxy is a great tool, but it lacks upstream certificate lookup and attributes transfer to a generated host certificate. For example, Charles does that. Some applications perform attribute inspection to detect MITM proxies, namely Spotify. Are you planning to implement this feature? Or maybe you already did it in version 7?

mitmproxy does lookup the upstream certificate by default, but at the moment we only transfer common name, subject alternative names and (new) organization name. The good news is that with version 7 it's very easy to add more here, I'd be more than happy to accept contributions. :)

Good news indeed! I tried to contribute 1–2 years ago and quickly realized that such a feature would require a substantial rewrite after a quick code inspection.

If you (or anyone else reading this) is interested, please do reach out on GitHub or our dev Slack. Happy to help you get started. :)

Is this related to https://github.com/mitmproxy/mitmproxy/issues/4575 ?

Yes, exactly!

Thanks for Mitmproxy! It would be amazing to be able to see non-HTTPS (but still TLS) traffic in the Flows view, and even to intercept it live instead of having to drop to raw TCP mode and write Python. Any idea whether that's on the roadmap?

Starting with this release you do see TCP-over-TLS in the flows view, we have a small screenshot in the blog post. Interception/modification is still something where we need to iron out the UX for, with version 7 you can at least inject messages with the `inject.tcp`/`inject.websocket` commands. This is not very well documented yet, but the technical foundations are there now. :)

If I wanted to use mitmproxy to mitm and observe traffic during automated testing, what mode/settings should I be using on Windows?

The setup is: A client (dev machine or pipeline) running tests that communicate to a Windows VM that hosts the app under test. The client RPCs to call automation APIs on the host/server.

However I couldn't get both communicating and using mitmproxy to observe traffic to work at the same time.

If your server is really just one service on one port, I would just run mitmproxy as a reverse proxy and point your client to that (https://docs.mitmproxy.org/stable/concepts-modes/). If there are multiple services, having it running as a regular proxy sounds best to me.

I know that's not the purpose of mitmproxy, but wondering if it's possible to create automation scripts, like for puppeteer or export requests to curl/httpie/vscode rest client. I tried through web-interface but couldn't find anything like.

You can export to cURL/httpie export in mitmproxy, it'll be exposed in mitmweb with the next release. :) For automation, take a look at https://docs.mitmproxy.org/stable/overview-features/#server-...!

I'm interested in how you generate your docs. I'd like to do something similar using Jekyll.

Our main docs are built with Hugo (https://github.com/mitmproxy/mitmproxy/tree/main/docs). For our API docs we use pdoc (https://pdoc.dev), which integrates well with most static site generators. pdoc is also maintained by us. :)

Thanks. When you began working on the main docs, did you start with a pre-built theme or did you start from scratch?

If I'm not mistaken they are built from scratch, but that part wasn't built by me. IMO the by far hardest part is actually writing the docs. :)

I am still using version 2.0 after the big changes in shortcuts. I used mitmproxy weekly, now I have switched to mostly using Charles, but would love to switch back.

Any plans for a recommended shortcut setup again?

Not writing 30 characters for copy body of a POST request, but fast key bindings.

I don't recall what changed with v3, but sorry for letting you down. Shortcuts are fully configurable since a few releases: https://docs.mitmproxy.org/stable/concepts-commands/#custom-....

If you have specific feedback about where the keyboard UX sucks I'd be happy if you could open an issue on GitHub so that we can take a closer look!

Thanks for mitmproxy!

Suppose I have set an interception filter and I am viewing a live packet. I am about to edit it, etc.

But when a new packet arrives, the text UI moves to the new packet! I have lost the packet that I was looking a few seconds ago.

Is this an issue in mitmproxy or am doing things the wrong way?

Could you clarify what you mean by packet? Are you talking about HTTP requests or about the new non-HTTP TCP streams? The UI/UX for the latter is very much work-in-progress.

I am talking about the HTTP requests.

The currently focused flow should never change when a new one comes in. If that's happening to you, please file a bug over on GitHub or ping me on our dev Slack and we'll get this fixed ASAP. :)

Do you have by any chance enabled focus to follow new requests? Keyboard shortcut Shift + F. Although it's disabled by default (IIRC), you may have enabled it by accident.