News for hackers. Stuff that matters.

kffl

That's an interesting use case. I am not aware of any tools that would allow for such content-aware proxying of HTTP traffic with artificial latency introduced in the response headers content parts separately.

As of now, speedbump is protocol agnostic - it just proxies the data over TCP without analyzing it (I've used it to proxy HTTP, Postgres, Mongo and AMQP traffic). While the idea of delaying the last byte seems promising at the first glance, especially in the context of retaining protocol agnosticism, the issue is that the proxy doesn't really know which byte is the last one (unless it actually understands the traffic flowing through it - i.e. detecting end of the HTTP response).

EDIT:

You may want to look into echo-nginx-module, as it allows for adding timeouts between output flushed to the client. Perhaps it could be used in conjunction with NGINX's proxy module.

netmem allows for traffic shaping at the kernel-level, whereas speedbump operates on a TCP connection level.

While netmem can be used for adding a fixed delay to network traffic (or a randomly generated one for each packet following a given distribution), speedbump can be used for introducing latency that changes predictably over time (i.e. forming a sawtooth wave or a sine wave), which was particularly desirable in my use case (testing app instrumentation). On the flip side, netmem can simulate packet loss, which speedbump can't do.

Wow, I didn't expect my project to get posted here. Thanks a lot!

I can see that many people are drawing comparisons with tc/netmem, which speedbump is not intended to replace, so I think it's best if I provide some additional context info regarding the use case that prompted me to develop this project.

When setting up application metrics collection and visualization (i.e. via Prometheus + Grafana), I've often found myself trying to introduce artificial latency within the instrumented system for the purpose of generating more interesting timeseries data to test a given monitoring solution. Even when running load tests against an instrumented system, the data plotted on Grafana dashboards was often rather boring, making it difficult to catch bugs in PromQL queries due to lack of immediate visual feedback. I figured that one way of adding predictable variability to the instrumented application’s metrics, would be to introduce variable latency between it and its upstream services (i.e. a databases, message brokers or other services called synchronously).

Let's say that you have instrumented your app using Prometheus client so that it collects latency of DB queries in a histogram and that you are now building a Grafana dashboard to visualize these metrics. If you knew that the DB query latency over time should form a sine wave with a period of 2 mins and amplitude of 10 ms, it would be much easier to validate the correctness of metrics collection and visualization (you would know what to look for on the latency histogram/grah). Speedbump allows me to achieve just that by introducing latency that is variable, yet predictable over time.

Sure, tc can be used for adding a fixed delay (or random delay that follows a given distribution) to network traffic. In the use case that prompted me to develop speedbump (testing application metrics collection and visualization), I wanted to achieve artificial latency being added value of which would change predictably over time (i.e. forming a sawtooth wave or a sine wave so as to give me immediate visual feedback when testing/debugging PromQL queries visualized in Grafana dashboards), which I don't think is possible to achieve with tc alone.

On top of that, I usually work on metrics collection, visualization and load testing in a staging k8s cluster, in which it would be rather tedious to set up tc rules on the worker nodes themselves (or even impossible in the case of managed K8s services with no direct access to the VMs acting as worker nodes).