News for hackers. Stuff that matters.
Re:Tollbooth: HTTP rate limiter middleware in Go by swah on Friday May 22, 2015 @10:42AM (#9588578)
Related: this thread mentions a bunch of things a rate limit package has to consider: https://groups.google.com/forum/#!searchin/golang-nuts/rate$...

(I thought rate limiters were very simple timers)

Re:Tollbooth: HTTP rate limiter middleware in Go by jfolkins on Friday May 22, 2015 @12:32PM (#9589354)
If interested, I wrote a different type of rate limiter in my spare time for my use case.

http://badactor.org

Re:Tollbooth: HTTP rate limiter middleware in Go by Xeoncross on Friday May 22, 2015 @12:55PM (#9589539)
I really like your simple splash demo. Good work, I might even check out your code.
Re:Tollbooth: HTTP rate limiter middleware in Go by jfolkins on Friday May 22, 2015 @04:53PM (#9590860)
The code is solid. I gauge this by the simple fact that when I review the project, I don't absolutely hate it ;-)

I need to circle back around as there are some features I'd like to add. PRs are welcome.

Re:Tollbooth: HTTP rate limiter middleware in Go by Xeoncross on Friday May 22, 2015 @12:48PM (#9589485)
I was writing an netfilter/iptables rate limiter and a nginx rate limiter (both able to handle quite a lot of requests) but all the system admins I talked to shamed me for trying to rate limit. They raised all kinds of points about blocking access to large companies or small countries and it not being effective for DDoS anyway. Any thoughts on this?
Re:Tollbooth: HTTP rate limiter middleware in Go by jerf on Friday May 22, 2015 @02:46PM (#9590196)
A bandwidth rate limiter is indeed of dubious utility. There's a lot of ways that can go wrong, and there's a lot of ways it can fail to do what the user expected anyhow, and there's a lot of ways it can do both.

But there's a lot of things that can be sensibly rate limited, such as logins attempts to a given account. Now that has its own considerations, too, if pushed to the limit... you'd prefer that an attacker can't lock down your service just by spuriously trying to log in to all your accounts 5 times every 5 minutes or something. But loud downtime (which you can then react to) may be preferable to getting your users silently hacked.

I definitely agree that they are less useful than they look at first glance, a great deal more complicated than you'd like, and more subtle than you'd think. But they can still be a useful tool.

Re:Tollbooth: HTTP rate limiter middleware in Go by jusssi on Friday May 22, 2015 @02:32PM (#9590126)
The only effective counter against a proper DDoS is to bring more bandwidth than your attacker has.

A small site is unlikely to attract a proper DDoS, but will get hit by the usual misbehaving web scrapers, broken clients in a silly loop (I'm looking at you, iTunes) and keyboards with super sensitive F5 key. In those cases, serving an occasional 503 might work better than the alternatives.

Re:Tollbooth: HTTP rate limiter middleware in Go by thwd on Friday May 22, 2015 @11:15AM (#9588789)
It's built around github.com/juju/ratelimit
Re:Tollbooth: HTTP rate limiter middleware in Go by mackross on Friday May 22, 2015 @12:54PM (#9589527)
It looks to me like the map of rate buckets will grow unbounded and never be cleaned up?
Re:Tollbooth: HTTP rate limiter middleware in Go by kellros on Friday May 22, 2015 @04:34PM (#9590785)
Looks to be the case based on the config. I'm suprised storage wasn't abstracted to accommodate redis or the like which is essential in a load balanced scenario.

The API doesn't seem quite idiomatic either, I'd expect to create a struct containing options and a function that closes over the http.Handler interface e.g. func(l *Limiter) Limit(next http.Handler) http.Handler or a function that takes options and a next http.Handler that creates a struct implementing http.Handler.

Re:Tollbooth: HTTP rate limiter middleware in Go by heyalexej on Friday May 22, 2015 @05:49PM (#9591033)
https://github.com/etcinit/speedbump is a Redis backed rate limiter that we're currently evaluating for our app.
Re:Tollbooth: HTTP rate limiter middleware in Go by blakesterz on Friday May 22, 2015 @10:20AM (#9588419)
This is neat. What's the best way to do rate limiting in from of Apache for wordpress/drupal sites these days? everytime I go looking for something new I'm dissapointed.
Re:Tollbooth: HTTP rate limiter middleware in Go by giaour on Friday May 22, 2015 @10:33AM (#9588517)
You could use the OP's project and point everything that makes it through the limiter at Apache with a reverse proxy http://golang.org/pkg/net/http/httputil/#ReverseProxy
Re:Tollbooth: HTTP rate limiter middleware in Go by meritt on Friday May 22, 2015 @12:06PM (#9589153)
Use mod_security as most Apache installs already have it enabled and it achieves the same thing.
Re:Tollbooth: HTTP rate limiter middleware in Go by lordlarm on Friday May 22, 2015 @11:40AM (#9588955)
Serious question: why would you not use HAProxy, nginx or similar software which arguably are more battle tested in this domain?
Re:Tollbooth: HTTP rate limiter middleware in Go by jamescun on Friday May 22, 2015 @11:49AM (#9589007)
For general rate limiting, yes HAProxy or Nginx make more sense.

Having different rate limits for different endpoints becomes unmanageable in HAProxy/Nginx; and having different rate limits say across different plans or users is all but impossible.

Re:Tollbooth: HTTP rate limiter middleware in Go by Thaxll on Friday May 22, 2015 @11:44AM (#9588972)
You can do that when the HTTP call is simple, when you need some logic in the rate limitation ( auth, login ect... ) it's impossible.
Re:Tollbooth: HTTP rate limiter middleware in Go by brightball on Friday May 22, 2015 @03:54PM (#9590594)
The best reason I can imagine is so that it can all be wrapped in your portable Go binary. Because of the portability aspect, there are a lot of use cases for Go similar to Java where there is an incentive to do everything "in Go".
Re:Tollbooth: HTTP rate limiter middleware in Go by didip on Friday May 22, 2015 @12:24PM (#9589291)
Four startups ago, I did a simpler one in Nginx using embedded Lua and Redis. It can definitely be done, but writing it in Go is easier, and funner :D