> It does tend to get me profiled as a "bot" by CloudFlare

Since Cloudflare employees are known to lurk here, I'd like to know: is this considered a false positive, or working as intended?

Not a CF employee but I am pretty sure it's working as intended as they built and use this:

https://github.com/cloudflare/mitmengine

Here's hoping that, if they do it, they'll start blocking corporate TLS mitm regimes too.

Blocking? Cloudflare provides corporate TLS MITM regimes.

"Corporate MITM" is built into the very idea of PKI. Running your own custom PKI is a completely valid way to operate a network. This is a feature because it allows anyone to establish their own trust tree, completely outside of the public certificate trust network.

"Regimes" sounds pejorative but in truth, companies have a duty and in many cases a legal obligation to protect their networks. Prima facie, I don't see any reason at all why interception of traffic in this circumstance is "bad," except maybe a potential for political misuse like any other written medium.

I actually think the reverse would be substantially worse: if _only_ the public trust chain was valid in major browsers, we would be completely hosed and there would be no distinguishing factor at all between remote attestation and trust.

Thus, corporate TLS interception is, at worst, a necessary byproduct of a very well chosen tradeoff.

And as a personal user, I also should have the right to intercept the traffic on my network.

I have a minor quibble with "my network". You should have the right to intercept the traffic originating or terminating at your devices, but not to intercept any traffic going between other people's devices just because it's on your network.

Then it wouldn't be my network anymore.

Should your ISP be able to MITM all of your traffic just because it's going over their network? If not, then what's the difference?

That's what I'm standing up for in this comment, but you've characterized it as opposition. I don't understand, perhaps you'd like to clarify.

I'm not opposing, just adding more context.

Ever since Google started pushing HTTPS I understood that non-encryption would be a future privacy problem.

Why? This isn't the 90s any more. You shouldn't use your employer's systems for anything other than work.

Hah, remember how Amazon invented "Cyber-Monday"? They claim the Monday after Thanksgiving when people go back to work it'll be their first encounter with Internet-connected computers after the Thanksgiving vacation, and it's the time where they'll be most busy shopping for Christmas presents...

So that's where thta came from...

Amazon was barely a blip when Cyber Monday first came out, let alone the inventors of the term.

> The term "Cyber Monday" was coined by Ellen Davis, and was first used within the ecommerce community during the 2005 holiday season. According to Scott Silverman, the head of Shop.org, the term was coined based on 2004 research showing "one of the biggest online shopping days of the year" was the Monday after Thanksgiving (12th-biggest day historically). Retailers also noted the most significant shopping period was December 5 through 15 of the previous year. In late November 2005, The New York Times reported: "The name Cyber Monday grew out of the observation that millions of otherwise productive working Americans, fresh off a Thanksgiving weekend of window shopping, were returning to high-speed Internet connections at work Monday and buying what they liked." At the time, a lot of people had slow Internet at home. The idea for having such a holiday was created by Tony Valado, in 2003 while working at 1800Flowers.com, and coined "White Wednesday" to be the day before Thanksgiving for online retailers.

https://en.wikipedia.org/wiki/Cyber_Monday

Why should they only allow MItM’ed traffic when a third party is the one defeating the encryption?

Privacy of communications is a protected right in lots of jurisdictions, not by default overridden by employer interests. From a values pov it makes sense as well, privacy is a human right and the fact that you're at work doesn't invalidate that, social interactions at work can be of a personal nature that is good to keep confidential.

You never watch a conference talk on YouTube at work?

I've always created a google account specific for that company though, so it is still technically segregated from my personal anything

Corporate TLS mitm is so often misunderstood. There are, unfortunately, a lot of legit reasons to do so.

Use this to evade it: https://github.com/lwthiker/curl-impersonate

Greetings from a country whose (almost) entire IP space is blacklisted by Cloudflare!

Which country?

From his profile:

> Software engineer from Belgrade, Serbia

Wow thanks, I was looking for this exact piece of software a few weeks ago when testing against some geo restrictions

necessity: still the mother of invention

greetings from the other side of the world, commandline brother. I did not know of this tool, but I do now. Thanks

Wow. Need this as a golang net/http shim.

> We consider an HTTPS connection to be intercepted when there is a mismatch between the expected client request signature corresponding to the browser identified by the User Agent, and the actual client request fingerprint of the request.

Sounds like something that would be "trivial" to defeat, by means of "emulating" other TLS implementations more closely?

which is what https://github.com/lwthiker/curl-impersonate does: it's a special build of curl that can impersonate Chrome & Firefox

Can this be used as a proxy to your real browser? Can you paint the whole picture/setup for a noob?

Presumably so. You'd just need to compile the curl-impersonate for the appropriate browser you want to impersonate, and then link it in to a proxy that used libcurl for its https traffic.

I have no releation to CF but from my perspective, if a human is marked as a bot - that's a false positive.

99.9999% of humans won’t be decrypting protobuf traffic but a large % of bots are out of necessity to interact with the website. That’s what I’d call a good filter.

Since you’re not a Cloudflare customer, you need to think like a customer who is using Cloudflare’s bot protection. Do you want to filter people who are blocking your ads through mitm? I would guess yes.

You should be okay as long as you solve the challenge(s).

I also run a forward proxy and found adding a user-agent header was all that was necessary.

would you mind providing more details about this? Is it any user agent header, or do you need to do some thing like curl-impersonate?

No, actually it just needed to be well-formed. UA headers follow a certain format. Some folks like to try to "blend in" and use known headers seen in the wild but personally I like to make up new UA headers that reveal nothing, not even fake details. On the whole, I very rarely send a UA header. The number of sites that demand one is very small for me. I had just assumed Cloudflare was doing TLS fingerprinting or something more involved than just checking for presence of a UA header. But it worked. The StackExchange family of sites is one example where a UA header is needed. They will block otherwise. Not using Cloudflare to do the blocking though.

s/StackExchange/StackOverflow/

> among other things

what else do you MITM for?

MITM is kind of a silly term for what the user is doing when using a forward proxy. If the computer owner binds the proxy to a localhost address, the unencrypted requests need only go over the loopback. There need be no encrypted requests travelling over the LAN. The computer owner, the "MITM", is on their own computer sitting in between an application and the network interface. That is exactly where they should be, IMHO. Corporations do it. They own the computer and they own their internal networks. It's no different for home users. And WFH is blurring the distinction anyway.

Once the requests leave the computer and travel onto the internet destined for another computer, then of course "MITM" makes sense as a concept.

The computer owner controls the proxy and it's the proxy, not the untrusted application, like a "modern" web browser for example, that handles authentication of the remote peer. Compiling and fully controlling a "modern" browser is a PITA. Almost no one does it, even software developers. Instead people beg for an advertising company or their partner to make changes to a browser. That does not seem to work. Sometimes when people complain it stops the company from making undesired changes. But only temporarily.

Whereas compiling a proxy is easy and the user can fully control it.

Having used many different applications that implement support for TLS, I actually trust the proxy's implementation better than most applications. It's arguably easier to audit one program, the proxy, than it is to check every application to make sure the developer didn't make a mistake when adding TLS support. I recall the incident with socat as one example. That mistake when undetected for a long time. Elinks was another.

For any gamers out there, Reshade's installer is another example where TLS istn't implemented properly. It fails to download effect/shader packages for basic setup on up-to-date Windows machines because it requires that you disable TLS 1.3 globally on Windows so it can use 1.1 or 1.2.

[1] https://reshade.me/forum/troubleshooting/8746-reshade-v-5-8-...

I never heard of reshade before but that is quite crazy. The only explanation I can think of is that the backend they connect to implements TLS 1.3 but in a broken way (because the client crypto lib would otherwise fall back to TLS 1.2 on its own) which really makes you wonder just how broken the crypto impl on their servers is and how many vulnerabilities they’re wide open to.

TLS 1.3 is IMHO still too new to have stabilised and had all the edge-cases worked out. 1.2 was first released in 2008 and only started becoming a requirement more than ~10 years later. I'm not sure how the situation with 1.2 was but I know there were several draft versions of 1.3, which do not interoperate yet had public implementations. That might be what's happening there.

Here are a few sites one (IMHO) would expect to support TLS1.3 but do not

   www.digitalocean.com
   arstechnica.com (Amazon)
   git.kernel.org
   cdn.netbsd.org (Fastly)

> The Proxomitron

Googling this, development ended in 2004? An informed summary of the current state of play would be interesting, since a lot of different "continuations" of it seem to be around. Also, is it Windows-only? (I've been casually looking for a simple proxy that would enable injection of local links into remote content.)

Privacy might be what you're after. Open source and cross platform.

That's impossible to google.

Didn't help that they spelled the name incorrectly, but I found it :D

https://github.com/barre/privaxy

In this case "incorrectly" is what makes it possible to find it, search-engine-autocorrect-stupidity aside. The name is obviously a combination of privacy and proxy. There's another one called Privoxy which works on similar principles.

Apologies, I meant Privoxy. Privaxy looks pretty decent too, but I've never used it.

Auto-correct issue?

We call that Auto co-wrecked or maybe a spill check.

Sorry all, I meant Privoxy. Didn't even realise Privaxy was a thing.

(With thanks to your grandchild) https://github.com/barre/privaxy seems definitely worth a look - thanks!

How does this compare to adguard home???

https://github.com/AdguardTeam/AdguardHome

Adguard seems to offer blocking only, whereas Privaxy "is able ... to inject scripts as well as styles into HTML documents", which was the feature my comment expressed interest in.

The readme goes on to say: "Privaxy is also way more capable than DNS-based blockers as it is able to operate directly on URLs and to inject resources into web pages."

One acts as a proxy while the other one is a DNS server. Different tools

> Windows-only

Can’t Wine help with this?

Presumably this involves installing new CA certs on your devices, no?

Yes, and this also answers the other comment about what else I MITM for: blocking devices that I can't inspect the traffic of is another reason, since the proxy is the only way out of the network.

Pardon the lay question, but is it really the only way out of the network? Can't programs initiate connections while skipping the system-wide configured proxy?

Sure they can.. but they can’t get around your outgoing firewall rule that reroutes alle traffic for certain ports to the proxy.

Hm, I have to see if Mikrotik has rule syntax for this. I can already force every app who thinks they will use their own DNS server to use mine but not sure how I could do the same with a proxy. Maybe just force ports 80 and 443? But what's stopping these apps to communicate on non-standard ports?

There's no reason to allow arbitrary traffic in either direction other than convenience. If you want a more secure network, you block everything by default and narrowly open as needed.

That means I'll stop 99% of all outgoing traffic. Still interested in how to force all traffic to a proxy though.

Thanks, I'll give this a thorough read.

If it's only for certain ports, they can just use non-standard ports.

Not uncommon to have a drop all rule as default on outgoing packets as well.

Regular http gets redirected to proxy, non-standard traffic needs to be explicitly allowed out.

Cooool. Any blog post or something about your network setup?

layperson here: won't, say a banking app complain about the now new cert it deals with instead of the bank-cert?

Banking apps generally hardcode their certs in to prevent this, so this wouldn't work unless you find a way to replace that cert too.

most banks already do certificate pinning, even tho it is not necessary.

clearly it is necessary in some circumstances

Network level ad-blocking like Privoxy and pi-hole have so many downsides like they can't handle inline adverts. My pi4 pi-hole is unplugged right now because it just didn't work well. After hours wasted trying to make it work with all the services, I've thrown in the towel. The time involved is just not worth it for a home network.

What works are browser based ad-blockers and app patchers like ReVanced. As my savings have gone up, I've relied more and more on just paying for ad-free services like YouTube Premium, Hulu, Netfix, and Max for the cases those 2 can't handle.

I run pi-hole strictly for my mobile devices, I let my desktop run through the router normally

I think this is an unfair critique of the PiHole and I believe you’re throwing the baby out with the bath water. The PiHole blocks DNS and that’s all it’s designed to do, and it does it well. There’s a grandiose amount of telemetry, marketing, and general unsavory traffic flying through your network that didn’t originate from a browser. The PiHole is not exactly a turnkey solution and does take some adjusting to your preferences, but once you get it set up, you basically only have to touch it to do system updates.

The problem is that unless you're willing to do surgery on a regular basis as you come across blocked sites, PiHole is going to cause more problems than it solves when you have your entire family using it. Even google stops working correctly because results for products on the front page start leading to blocked links. I like PiHole, but I had to stop using it because it was so aggravating for my wife.

I used a PiHole for a while but after a corrupted SD card I said screw it and started using NextDNS. I have very few subscriptions, in fact nothing else except Spotify, but this one was worth the time saved and hasn't required any maintenance on my end.

Why not AdGuard DNS? It's free.

How heartwarming that they run such a wonderful service purely out of the goodness of their hearts, with no income from it, and no expectation of financial compensation.

It forms the backbone of their paid AdGuard apps on iOS, MacOS and Android. Those apps create a loopback VPN on your system so they can reroute the DNS requests to AdGuard DNS.

I have our PiHole set up with just ad lists, not trackers or privacy. I follow that up on my own devices with uBO with more thorough lists, but this way my whole family gets ad protection rather than nothing.

> google stops working correctly because results for products on the front page start leading to blocked links.

Because they're often ads made to look like search results. Pi-hole working as intended.

That's my experience anyway. My family network is behind a pi-hole and rarely if ever does an intended website visit break as a result.

This is not my experience. Pihole works flawlessly for me and it’s sitting in my closet connected to my cable modem. Haven’t touched it in a year. No broken mainstream sites and I’m blocking telemetry and other crap from apple, Amazon, google, Netflix, Hulu, Wyze, etc. related to advertising and device analytics trying to profile my usage. Pi-hole has been a blessing in the war on privacy and targeted advertising.

I’m a novice, and figured it out and have zero maintenance. Unless you count occasionally updating pihole every now and then. I have a word file with all my exceptions (mostly white lists for Microsoft gaming stuff my kids want) in the event I ever need to reload (ever since I switched to a better SD never have had corruption issues). I understand the spouse issue, mine loves clicking ads on her phone. Easiest workaround is to pop off WiFi click link then go back to WiFi.

What services are you having trouble with? My RPi PiHole has been working great for whole-house blocking for a half-decade now.

Likewise. My biggest tip is set your secondary DNS to something outside your networks so if your pi-hole goes down your entire network doesn’t go down you just get ads.

There’s also nextdns which I was planning on switching to anyways

I’ve been using NextDNS for a few years now. I don’t have time to mess with an ‘on prem’ solution.

It works really well and is simple to manage. It runs on my mobile devices easily.

I turn it off occasionally to shop, but immediately turn it back on as the internet is a whole different place without some sort of robust ad blocking.

It’s nice to see the logs fill up with blocked telemetry and other crap all day.

Is it okay to use cloudflare dns?

Airplane booking sites seem incapable of working with a Pihole. Presumably the entire travel industry is similar

I use a chrome extension or app to switch DNS to something like 1.1.1.1 when a site I need can’t handle blocked DNS calls.

Fair enough, but I've been super happy with the ability to do network level blocking; if you're willing to decrypt your own traffic, you can even do things like strip advertisements from getting embedded in Smart TVs and whatever else you have connected to your insecure network partition. I like it that I can hook up some random Roku and I won't get ads. Or my kid can connect his school laptop to our network and it wont get ads.

Does this just apply to the roku menu or are you saying you are stripping ads from the video streams themselves?

Inline adverts can be blocked, as long as you're decoding layer-7 (which is effectively what this ad is doing).

What kind of problems have you been running into.

I use a layered approach. I’ve been using Pi-hole for 8-10 years, don’t remember, with about 1-2M in the block list.

In addition, I use uBlock as well as pfsense with pfBlocker-NG for blocking countries and other features.

They all work well together. Sometimes I have to bypass them, when I momentarily use the ISP’s Wi-Fi router directly.

It’s been working fine.

This is what I fail to understand. It’s a cool hack but ad-free YouTube is a few dollars a month. Spend that creative time elsewhere!

It’s $18.99 a month for an individual, I agree with your sentiment but really don’t feel like it’s that inexpensive. Then again I don’t consume large amounts of YouTube as most people seem to do.

Most creative projects are not cost effective. If people enjoy the journey, that's what's important.

I haven't thought about Proxomitron in about two decades - are you still using it? I never used it for anything you described, but it was great to use as a proxy with our company firewall. Many programs back in the day couldn't connect with the Internet because our firewall back then required login information to make outbound connections.

I loved Proxomitron! I hadn't thought of using it to MITM YouTube!