SaveTheRbtz

"An In-Depth look at Lerp, Smoothstep, and Shaping Functions"[1] by SimonDev is the best 8min introduction to Lerp for a software engineer that I saw.

[1] https://www.youtube.com/watch?v=YJB1QnEmlTs

We're mostly on the Grafana stack, so we tend to favor their open-source solutions for metric backends. For client libraries, we're currently using Prometheus for metrics and OpenTelemetry (OTEL) for tracing, but we're looking to migrate fully to OTEL once metrics are stable there. As for logging, it depends on the project, but generally we use either Zerolog or Zap, with either Loki Docker plugin or Promtail.

Thanks for creating this proposal! Would love to see it committed upstream. For me it worked just fine -- I'll need to play a bit more with it and maybe put it into a real-world production setup to see how well it preforms on real-world data.

I did PoC experiments with compression, chunking, and IPFS here: https://github.com/SaveTheRbtz/bazel-cache

If you need a mature compression implementation for bazel I would recommend using recent bazel versions w/ gRPC-based bazel-remote: https://github.com/buchgr/bazel-remote

bazel nowadays supports end-to-end compression w/ `--experimental_remote_cache_compression`: https://github.com/bazelbuild/bazel/pull/14041

I was recently playing[0] with the ZSTD seekable format[1]: a *valid* ZSTD stream format (utilizing ZSTD skippable frames, that are ignored by de-compressor) w/ an additional index at the end of the file that allows for random access to the underlying compressed data based on uncompressed offsets. This combined w/ a Content-Defined-Chunking[2] and a cryptographic hash[3] in the index allows for a very efficient content-addressable storage. For example I've successfully applied it to a bazel-cache, which gave me between 10x and 100x wins on repository size w/ negligible CPU usage increase.

[0] https://github.com/SaveTheRbtz/zstd-seekable-format-go

[1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...

[2] e.g FastCDC https://www.usenix.org/system/files/conference/atc16/atc16-p...

[3] https://github.com/facebook/zstd/pull/2737

The analysis itself is quite impressive: a very systematic top-down approach. We need more people doing stuff like this!

But! Be careful applying tunables from the article "as-is"[1]: some of them would destroy TCP performance:

  net.ipv4.tcp_sack=0
  net.ipv4.tcp_dsack=0
  net.ipv4.tcp_timestamps=0
  net.ipv4.tcp_moderate_rcvbuf=0
  net.ipv4.tcp_congestion_control=reno
  net.core.default_qdisc=noqueue

Not to mention that `gro off` that will bump CPU usage by ~10-20% on most real world workload, Security Team would be really against turning off mitigations, and usage of `-march=native` will cause a lot of core dumps in heterogenous production environments.

[1] This is usually the case with single purpose micro-benchmarks: most of the tunables have side effects that may not be captured by a single workflow. Always verify how the "tunings" you found on the internet behave in your environment.

re. syncookies: Linux by default starts issuing syncookies when listening socket's backlog overflows, so it may be accidentally triggered even by a small connection spike. (This, of course, is not an excuse for a service misconfiguration but it is quite common: somaxconn on Linux before 5.4 used to be 128 and many services use the default.)

re: pacing: Awesome!! I would guess it is similar to Linux "internal implementation for pacing"[1]. Looking forward to it eventually graduating form being experimental! As a datapoint: enabling pacing on our Edge hosts (circa 2017) resulted in ~17% reduction in packet loss (w/ CUBIC) and even fully eliminated queue drops on our shallow-buffered routers. There were a couple of roadbumps (e.g. "tcp: do not pace pure ack packets"[2]) but Eric Dumazet fixed all of them very quickly.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

> tcpdump ... linux TCP stack ... ebpf

> Networks are not my specialty

I wish all network non-specialist were like you!

Thanks! Added it back to the blogpost (seems like we accidentally lost the `pktmon` reference during the editing process =)

A couple of questions:

* What are the reasons for disabling TCP timestamps by default? (If you can answer) will they be eventually enabled by default? (The reason I'm asking is that Linux uses TS field as storage for syncookies, and without it will drop WScale and SACK options greatly degrading Windows TCP perf in case of a synflood.[1])

* I've noticed "Pacing Profile : off" in the `netsh interface tcp show global` output. Is that the same as tcp pacing in fq qdisc[2]? (If you can answer) will it be eventually enabled by default?

[1] https://elixir.bootlin.com/linux/v5.13-rc2/source/net/ipv4/s... [2] https://man7.org/linux/man-pages/man8/tc-fq.8.html

Sadly, middleboxes are a real problem, esp. with our Enterprise customers. We had this problem even with HTTP/2 rollout so there is even a special HTTP/1.1-only mode in the Desktop Client for environments where h2 is disabled.

In the future we are planing on having an HTTP/3 support which will give us pretty much the same benefits as SCTP with a better middlebox compatibility.

Oh, Windows 7? That does explain it. Windows TCP stack really improved in 8.1. The main change there is the auto send buffer tuning which allows automatic growing of SNDBUF. Let me see if we can put a dirty hack^W^Wworkaround for Windows < WIN2012R2SERVER that would unconditionally set SO_SNDBUF to something like 1Mb.

We didn't mention RSS/RPS in the post mostly because they are stable. (Albeit, relatively ineffective in terms of L2 cache misses.) FlowDirector, OTOH, breaks that stability and causes a lot of migrations, and hence a lot of re-ordering.

Anyways, nice reference for TAPS! Fo those wanting to dig into it a bit more, consider reading an introductory paper (before a myriad of RFC drafts from the "TAPS Working Group"): https://arxiv.org/pdf/2102.11035.pdf

PS. We went through most of our low-level web-server optimization for the Edge Network in an old blogpost: https://dropbox.tech/infrastructure/optimizing-web-servers-f...

We will be eventually migrating to UDP (HTTP/3) once it is rolled out on Envoys[0] on Dropbox Edge Network[1].

[0] https://dropbox.tech/infrastructure/how-we-migrated-dropbox-...

[1] https://dropbox.tech/infrastructure/dropbox-traffic-infrastr...

Microsoft Devs explain this in their "Algorithmic improvements boost TCP performance on the Internet"[1] article.

TL;DR is that they had RACK (RFC draft) implemented as an MVP but w/o the reordering heuristic.

[1] https://techcommunity.microsoft.com/t5/networking-blog/algor...

Interesting, can you try disabling upload limiter in settings? Also what is your RTT to `nsf-1.dropbox.com`?

PS. One known problem that we have right now is that we use a multiplexed HTTP/2 connection, therefore:

1) We rely on the host's TCP congestion. (We have not yet switched to HTTP/3 w/ BBR.)

2) We currently use a single TCP connection: it is more fair to the other traffic on the link but can become bottleneck on large RTTs.

Alexey from Traffic Team is here. Traffic is definitely a non-negligible part of the budget. We try to reduce it as much as possible for both lower operational expenses and better user experience. Main drivers for that improvement (besides owning our own Edge infrastructure) on the client side are:

1) Brotli (Broccoli) compression.

2) Differential updates through librsync.

3) "LANSync" a P2P sync within a broadcast domain (secured through server issued short lived TLS certs.)

That said, Desktop Client is only 1/3 of the overall Dropbox traffic -- the rest 2/3 are split between Web and API.

We've made a note about how inefficient our solution was and what was the plan to fix it. Sadly, to get proper stats in nginx we needed two things:

* C interface for stats, so we can would have access to from C code.

* Instrument all `ngx_log_error` calls so we would have access not only to per-request stats but also various internal error conditions (w/o parsing logs.)

That said, we could indeed just improve our current stat collection in the short term (e.g. like you suggested with a per-worker collection and periodic lua_shared_dict sync.) But that would not solve the longterm problem of lacking internal stats. We could even go further and pour all the resources that were used for Envoy migration into nginx customizations but that would be a road with no clear destination because we would unlikely to succeed in upstreaming any of that work.

One of the senior engineers once said to me that "Bazel is like a sewer: you get back what you put in."

Bazel requires a lot of upfront effort but the power of (a programmatically accessible/modifiable) dependency graph and a common build/test system across all the languages is very hard to underestimate.

It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config generation, filesystem state, service configuration (e.g. syslog.)

On a more practical note, each nginx `reload` will double the number of workers, almost doubling memory consumption and significantly increasing CPU usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So there is only that many reloads that you can do in an hour.

We've actually started experimenting with converting our static file serving to "proxying to S3 + caching." This is simpler from deployment and development perspectives (for companies that do not have a distributed filesystem, like Google with its GFS):

* for deployment we do not need to maintain a pool of stateful boxes with files on them and keep these files in sync.

* for development, engineers now have a programatic interface for managing your static assets.

open source argument is valid -- most software enterprise vendors do provide source code access (under NDA.) The rest of the arguments stand though: as it is right now, it way more developer/operator friendly to use Envoy in our production.

To tell you the truth, we didn't consider it. From what I can get from the architecture docs[1], it can be a decent platform for apps, but might not be the best choice for a general purpose ingress/egress proxy (at least for now.)

[1] https://caddyserver.com/docs/architecture

Totally get it! The team (@veshji and @euroelessar) struggled a bit in convincing me that the new Envoy way is a simpler one. I do not regret giving in.

Operationally, there are many differences (esp. around Observability) but if I were to distill it down to one thing it is a clean separation between data- and control-plane. This basically means that it was designed to be automated and the automation layer (xDS) itself runs just like any other normal service in production.

It works beautifully. We use driver offload (i40e on the Edge.)

Actually, all the props for that go to Katran's author himeself. When we hired Nikita V. Shirokov (tehnerd), the first thing he did was replacing IPVS with XDP/eBPF-based Katran, which improved our Edge servers throughput by 10x, from ~2MPps to ~20Mpps.

He also contributed a lot to Envoy migration migrating our desktop client to it and adding perf-related thing like TLS session tickets' lifetime to SDS.

The subject of monetizing opensource software is a tricky one. Some companies pursue the Open-Core principle, others monetize through the consulting services or cloud infrastructures.

As for investing into opensource, Dropbox is trying to do that when possible, for example we (along with Automattic) did sponsor HTTP/2 development in Nginx.

We had a large rundown of our Traffic Infrastructure some time ago[1]. TL;DR is:

* First level of loadbalancing is DNS[2]. here we try to map user to a closest PoP based on metrics from our clients.

* User to a PoP path after that mostly depends on our BGP peering with other ISPs (we have an open peering policy[3], please peer with us!)

* Within the PoP we use BGP ECMP and a set of L4 loadbalancers (previously IPVS, now Katran[4]) that encapsulate traffic and DSR it to L7 balancers (previously nginx, now mostly Envoy.)

Overall, we have ~25 PoPs and 4 datacenters.

[1] https://dropbox.tech/infrastructure/dropbox-traffic-infrastr... [2] https://dropbox.tech/infrastructure/intelligent-dns-based-lo...

[3] https://www.dropbox.com/peering [4] https://github.com/facebookincubator/katran

Our technology stack is very gRPC friendly, so developer experience is actually better with it, than without (though this is very subjective.)

As for the middleboxes, using gRPC-WEB[1] allowed us to switch Desktop Client App to gRPC even behind firewalls/IDSes that do not speak HTTP/2 yet.

As for the HAProxy, Dropbox used to use (circa 2013) it specifically for loadbalancing, but we eventually replaced it with our Golang proxy. That said, recent HAProxy improvements (v2.0+) make it quite an awesome dataplane and an excellent loadbalancer!

[1] https://github.com/grpc/grpc-web

Can you describe your use-case?

If you are talking about the ability to select a certificate on the fly via `ssl_certificate_by_lua_block`[1] we are not aware of such functionality. If you are missing something, I would highly encourage you discuss it with the community on a github!

From Oleg Guba, Traffic Team TL, co-author, and person driving the deployment:

* ListenerFilters + NetworkFilters are flexible enough, that some of the custom logic could be just moved to the config.

From Ruslan Nigmatullin, our head Envoy developer:

If you are talking more about a custom verification code there is already couple of ways to do that:

* Client TLS auth Network Filter: https://www.envoyproxy.io/docs/envoy/latest/configuration/li...

* Alternatively, if you are writing C++ extension you can use Network::ReadFilter, Network::ConnectionCallbacks.

[1] https://github.com/openresty/lua-nginx-module#ssl_certificat... [2] https://github.com/openresty/lua-resty-core/blob/master/lib/...