The first two use cases are all about traffic visibility, which has been, and remains a big problem. If we're talking about improving security, the third use case—virtual patching—is the exciting part. The story goes something like this: you discover a vulnerability in your code but you can't fix it immediately, for whatever reason (take your pick). In practice, it may be weeks until the problem is fixed. However, if you have ModSecurity installed, you can write a virtual patch for your problem, often to fix it 100% reliably.

However, it's very important to understand that virtual patching doesn't mean just slapping together some rule that appears to be doing the job. Blacklisting doesn't work. To properly virtual-patch, you have to fully understand the vulnerability and the entry point to the application. You have to carefully whitelist all parameters in that particular location. In other words, you have to almost replicate the work that should have been done by the developer.

Even in this context, it's not all rosy. Virtual patching requires a great deal of expertise to do properly. It's almost always better to spend your time fixing the actual problem. Sometimes, however, that's not possible to do in a reasonable time frame, and that's when a WAF (according to my definition), can be a life saver.

- - - -

There's few people I am more pleased to see commenting here on a topic like Sec more than yourself Ivan, your work's exceptional. And the tools you've provided like Hardenize and SSL Labs have saved me countless hours.

Your books have been an immense source of knowledge for me (and I'm relatively a newcomer by my colleagues standards).

Really pleased to see you here on HN. Though, I'm not entirely surprised. The best of the best seem to haunt this place.

* Your logs should be there to help you find out what happened in the event of a compromise, not to detect it in progress.

What OP should be monitoring is the activity that occurs on the system - logins, processes spawned (e.g. php or mysql suddenly spawning shells for no reason), changes to content, resource volume spikes and as the parent said, configuration changes.

Much of this can be done with a variety of tools but there isn't really a one-size fits all approach. I'd start with resource spikes as it's easiest and work up to an OSSEC configuration reflecting the actual environment.

Also by way of middle ground on the log front, Graylog2 is a happy medium between Splunk and ELK, but it's better to log fewer (but more useful) things than a load of noise.

In my experience, monitoring this takes about as much effort as adding a new apparmor / selinux / firejail / ... rule which would block it in the first place. I mean, with processes spawned we're almost at auditd level already.

I vehemently disagree with this assertion. I also disagree with tptacek's asseration that any admin interface should be inaccessble from the public Internet. That's just not how a lot of small businesses, non-profits, etc. operate. 2FA? Sure. Geo-fencing? Keeps out noise. Basic auth / some front-end to minimize app-specific attack surface? Perhaps.

Saying "you shouldn't look at logs to detect anomalous activity before the fact" is awful. So is "If you have and admin availability publicly, you're screwed".

My saying is "Security is a spectrum". A bank should not have their firewall admin interface publicly available. A community newsletter blog would probably be alright.

There are organizations on AWS that have publicly-routable admin interfaces. They should all be working on plans to migrate away from that, and, in the meantime, they should be extra careful about authentication and monitoring.

If you're too small to have a non-routable admin interface, you are definitely too small to pay for security products.

As for the content of your comment, it's hard to distinguish what's aimed at me and what's aimed at him, so I'll assume everything aside from your first sentence is aimed at him.

You buried the lede on this one.

If you do security for an enterprise with dozens or hundreds of apps that you don't develop yourself, you might not have the luxury of only trying to reduce attack surface, or instrumenting 3rd-party supplied code.

I Also have some mod_rewrite rules in to 403 some URL’s that bots currently hit, mostly because I don’t want to spend 200ms of PHP time for the bot to discover that it’s exploit didn’t work.

I accept that when you’re hosting websites for a load of other people you’re gonna need backups too, that’s the bottom line.

Converting our WordPress sites to static sites greatly simplified and improved our infrastructure.

When I was young I too used to try implement detection and banning but realised it's a waste of time. Key only access to ssh and make sure you at least subscribe to security updates for all packages is enough.

For our own app we went the ELK route due to the number of log messages coming from our own application. But yes as you said it's a pain to manage. Also creating kibana graphs/views/desktop for each type of log is also painful.

A good service I believe would be someone to offer a turnkey/shrink wrapped managed ELK for self hosting. Not only managing the system side of it, but also offering services to help create custom kibana graphs.

Using a short ban time keeps the load of running fail2ban to a minimum. I at one point was banning the kiddies/bots for a week and 40% of the server load was chewed up just running fail2ban.

The solution I came up with to this problem is to use a custom ban action that adds an IP to the blocked ipset list with a given TTL, but set fail2ban to think that the IP has to be only banned for 5 seconds or so.

The result is that ipset purges the IP from the list after the TTL expires, but fail2bans working set is a 5 second rolling windows of blocked IPs.

This works great and it's being used in production for a couple of years.

  > so that my logs don’t fill with a zillion pointless attempts

You don't really need fail2ban at all. Just don't log failed password attempts. Key access only, no one is going to get in.

Of course there are ways to deal with it, but that's just extra work.

I'm pretty happy with sumo. A couple things I wish they'd change, but their core competency is searching logs and metrics and they do that really well. It is paid (there is a free tier but we outgrew it quickly) bit it's well worth it IMHO.

> But really: focus on removing attack surface, rather than continuously monitoring it.

So much this. Logs are great for telling you something bad has already happened and what that was, both security and faults. Reducing the attack service will get you more mileage from a security pov (again, in my opinion). Logs need to he there and some basic trawling of them is also necessary, but is purely reäctive.

Edit: updated to be more clear

Yeah, I should have been more explicit. When I say build secure applications, I imply the full stack including infrastructure. I updated my comment accordingly.

Amen to that. I stopped caring about these sorts of entries in my HTTP logs when I realized these automated attacks target software packages written for language runtimes I don't even have installed. I'm sure not losing any sleep at night that someone is trying to hit phpMyAdmin or wp-admin on my server that doesn't even have a PHP interpreter.

You have railed against network isolation as defense in other threads. Locally-scoped IPs, VPNs, and ACLs are all about network isolation. Care to shed some light on the inconsistency?

Just out of curiosity has anyone with a cryptographically strong ssh password ever been hacked via the password authentication path unless they gave this password to someone else?

Do you know how they were owned? Is there a write up anywhere as this is a super interesting topic.

By having their password brute forced, or some other means?

Remove? Probably not. There are many options for minimizing the attack surface with http.

In a realistically achievable world for virtually every startup, no routable SSH. No security rule allows 22/tcp from a routable address into a VPC. You want to SSH into a host, you VPN to your network first. This is how most organizations with security teams set up SSH.

If you can't get there, at least get to jump box SSH --- 22/tcp is allowed to exactly one isolated host from the outside world (some people filter source addresses to their jump boxes, I don't bother); to get to real hosts, you bounce through that one.

A network where you can randomly connect to the sshd on real servers is one most security professionals will assume isn't being run competently.

"just" that? That's a very bold statement that most of the security community would disagree with.

An example: you don't monitor network traffic on the border because you trust your server not to be compromised. Then one gets compromised and the incident goes undetected due to lack of monitoring.

> If you remove attack surface altogether

That's a big if. Services can have a complex life, with software and configuration changing often. Unless you pull the plug you attack surface is never going to be zero and, on top of that, there so guarantee that it stays small. Hence organizations usually have different people taking care of different layers.

  Scanners:
   - Nikto (https://cirt.net/nikto2)
   - A comparison of tools from 2017 (http://sectoolmarket.com/price-and-feature-comparison-of-web-application-scanners-unified-list.html)

  Web Application Firewalls:
   - ModSecurity (https://geekflare.com/install-modsecurity-on-nginx/)
   - Enable OWASP rules in ModSecurity (https://geekflare.com/modsecurity-owasp-core-rule-set-nginx/)
   - Vulture (https://www.vultureproject.org/)
   - Pay $500 to have SpiderLabs configure ModSecurity for you (https://ssl.trustwave.com/web-application-firewall)
   - Nginx (https://www.nginx.com/products/nginx-waf/)
   - Azure (https://azure.microsoft.com/en-us/blog/azure-web-application-firewall-waf-generally-available/)
   - AWS (https://aws.amazon.com/waf/)
   - Symantec (https://www.symantec.com/products/web-application-firewall-reverse-proxy)
   - Incapsula (https://www.incapsula.com/website-security/web-application-firewall.html)
   - Smoothwall (holy cow, they're still around? https://us.smoothwall.com/utm/)
   - Shadow daemon (probably defunct, https://shadowd.zecure.org/overview/introduction/)
   - IronBee (same, https://github.com/ironbee/ironbee)

  Hardening guides:
   - Apache (https://www.tecmint.com/apache-security-tips/)
   - Nginx (https://geekflare.com/nginx-webserver-security-hardening-guide/)

Don't use any of those scanners, all of which look almost exclusively for vulnerabilities you don't have. Do look into getting a Burp license for every developer on your team.

By all means, harden Nginx. Don't pay SpiderLabs to do it for you.

Thanks !

Your message is being lost/ignored as a result of your delivery.

We want to understand your message, but you aren't giving enough background. I don't trust your message because there is no supporting background. If you're going to tell me to not do something, either:

A) provide a compelling alternative (not doing $it is not a compelling alternative to $it in nearly all cases) B) explain in detail why you shouldn't do $it. Provide more details if asked.

I want to believe what you are saying and be convinced by it, please help me. (Note: I'm not trying to be a dick, I've just known a lot of people who are perpetually ignored because of this communication style and take forever to learn that lesson because no one bothers to say anything)

I'd also like to add that you may want to work on the tone of your comments if you'd like to get enough karma to really participate in the community. Your comment history seems a bit confrontational and argumentative. There are better, friendlier ways to make the same points. I'll defer to the guidelines[0] to explain how you may improve your communication skills.

[0]https://news.ycombinator.com/newsguidelines.html

I hope your day gets better.

Example, same author: Cryptographic Right Answers.

(Disclaimer: I work with tptacek.)

First of all - it takes less than an hour to set up a WAF, and it doesn't worsen your security position. Not only is it free, it's fast and easy and will demonstrably be better than no security at all.

Saying "don't put up a firewall because it isn't perfect" is stupid. You don't have to be a security guru to get that, which obviously several people have, or they wouldn't all be giving you crap about it.

Securing your application long term is a complex, expensive, time consuming process. Suggesting that spending a couple months to be "serious about security" is somehow better than first taking one hour to do the most bare minimum effort is ridiculous.

If I have a computer on the internet, I put a firewall up. Is it defense in depth? No. But that doesn't mean it won't stop the most basic, most common attacks, and it takes no time and no skill to apply it.

I don't know what weird thing you have against WAFs other than apparently they aren't cool or sophisticated enough, but 1) they remove attack surface, 2) they filter the most common drive-bys, 3) they can be both passive and active, 4) they're free, 5) they're easy, 6) they're quick to deploy.

How is it 2018 and I have to explain to you of all people that the web equivalent of iptables might be a good idea if you have zero security?

They do what you configure them to do. Without knowledge of what's allowed and what's not, they're going to have either false positives or false negatives. And if you have that knowledge, you can fix those issues in the web app rather than hide them with the WAF. A lot of preconfigured WAFs will remove specific matches you don't care about anyway rather than attack surface.

> 2) they filter the most common drive-bys

Common as in known proxy or webapp exploits? If that's the case, why not patch them? (There's a reason to use WAF if you can't patch but that costs time/money to get right and test)

> 3) they can be both passive and active

I'm not even sure what passive WAF means... just logging?

> 4) they're free

Only for very small traffic.

> 5) they're easy 6) they're quick to deploy.

See point 1. If you can't put the right restrictions on the app, why do you think you can put them in WAF? If they're not in the app in the first place, why do you think this way is quicker? It's not a magic box will know your requirements. You need a proper setup or the only thing you achieve is removing obviously silly cgi-bin requests from your log files.

I'm not even against WAFs. Just think the gain from them is been none and minimal if you reduce it to "turn it on, it's cheap and easy". It's really similar to iptables - you can turn it on, but why are you doing it? (On a server anyway) If you have a specific reason/route to protect - great. If not, make sure you bind non-public services to local interfaces and you don't need iptables.

> it takes less than an hour to set up a WAF

Not if you want it to do anything useful. Default WAF rules are mostly useless and come with bypasses -- kinda like XSS filters in browsers, and browsers are in a way better position to judge if XSS is going on or not. You can sort-of get useful WAF rules if you encode most of the invariants of your system in your WAF -- but your app is in a strictly better position to do that sort of validation for you instead.

> and it doesn't worsen your security position.

Yes, except for the shiny new TLS MITM, the pathological parser performance cases, the new attack surface in that WAF itself (hands up if you've seen XSS in a WAF interface) and the continued operational cost of having new infrastructure that sees plaintext... Sure! And then I'm being generous in assuming we're not talking about a virtual appliance or as I like to call them "mystery *nix box you can't patch or audit".

> Securing your application long term is a complex, expensive, time consuming process. Suggesting that spending a couple months to be "serious about security" is somehow better than first taking one hour to do the most bare minimum effort is ridiculous.

Also, unlike WAFs, it's effective.

To be clear: I am not conceding your point about cost. It does not take "months" to identify systemic appsec issues. It also does not take an hour to configure a WAF and have it do something vaguely useful for your application.

Anecdotal evidence: Latacora literally does this systemic-discovery-and-remediation-plan thing for a living as a part of our engagement. "Months" is not our timeline for any kind of systemic appsec issue like SQLi or XSS.

> ... the web equivalent of iptables might be a good idea

Are you really sure you want to stand by that equivalence? How often does one come across iptables bypasses?

Yes. Both require some knowledge of how to use the tool and rules, but both are not difficult for even a novice to pick up quickly. Not to mention both can be done by either a dev or a sysadmin in the cloud right now, neither of them require in-depth knowledge of the code, and the scope of the change is simple enough that it can be tested and applied quickly.

> "Months" is not our timeline for any kind of systemic appsec issue like SQLi or XSS.

The discussion was about a dev or sysadmin fixing all this themselves ("taking appsec seriously" etc). Not just running Burp on their website. After they identify the vulns they have to have meetings to discuss fixes, figure out how to patch them and test that it works, go through QA and then roll into production. A fix may involve several systems and most people's deploy process is painful. I've never seen that take less than three weeks, and six is more reasonable. Can you do it in less time? Of course. Does that actually happen? Not without paying a consultancy $300k to hold your hand for two weeks.

It is completely plausible to get a WAF up and running and have it provide real protection immediately. CloudFlare's is probably the best example of this, considering the extensive add-ons they provide based on the intrusions they've seen. I don't like CloudFlare's TLS MITM either, but tens of thousands of their customers don't seem to mind.

Just like iptables, ModSecurity doesn't do anything at first. You have to actually make some decisions. Does that mean you shouldn't spend the extra 15 minutes to read a manual and enable some protection?

Earlier it was suggested that protecting against Drupal or Wordpress vulns is unnecessary. I disagree! There's no reason not to protect against drive-by vulns. Filters for XSS/SQLi, unnecessary HTTP features, malicious bots and scanners, directory traversal, RCE, etc are well worth preventing whenever possible.

As for false positives, they're not as bad as they used to be. There are thorough guides designed to help you identify and remove them from your site. https://www.netnea.com/cms/apache-tutorial-7_including-modse... https://www.netnea.com/cms/apache-tutorial-8_handling-false-... For a sysadmin this may take more time, just like it would if you were using iptables to secure a black box. A dev can change the filters immediately to suit their app.

But even if you don't do any active filtering, the passive filters can at least alert you that something is happening so you can go fix it. This is so standard that Kubernetes includes it as an option in their Nginx Ingress Controller. This guide can get you up and running on your existing K8s install in 10 minutes: https://karlstoney.com/2018/02/23/nginx-ingress-modsecurity-...

This points out how WAF/NIDS is just a speed bump for a serious attacker.

You may conclude that they don't really 1) remove attack surface 2) Unclear if they really do this, and if they do what value there is 3) See the papers for detail on the differences 4) Zero cost to purchase, but what cost to maintain, run, keep up? 5) see #4 6) even if you have a large web farm?

This paper was writen 20 years ago.

Or you go to the doctor and expect him to justify your treatment down to the minor detail? You sound like you talk to your doctor about $overpriced_drug.

It's just reasonable to ask for rational arguments. That's the whole point of the rants here.

Please speak for yourself alone. tptacek's comments in the thread are quite clear.

You are succeeding without trying. You're essentially complaining some free advice does not fit your exact expectations and specifications. Which is fine, lots of free advice doesn't. Trying to pass this complaint off as some kind of favour to the person giving you free advice, though, is... less than great.

Is this advice assuming that it's devs who want the WAF, rather than ops or management?

For devs clearly it's not what they should be focusing on. But saying something isn't worth doing because it shouldn't work is a terrible security principle.

So, I'm just going to go with: no, if you're really building an application and care about application security, no, you shouldn't waste time setting up a WAF tool designed to help 10,000 employee insurance companies make sure their Wordpress marketing site isn't exposing some vulnerability from 3 years ago.

Not wasting time with dumb stuff like this isn't "terrible security"; every dumb thing you do exacts a cost from good things you could be doing instead.

It's like throwing gauze over an open wound. Some companies don't care enough to put thought into healing the wound altogether, whereas some just wrap it in gauze and hope they don't bleed out. And it usually works - at least for a while, until you get big enough/become a lucrative enough target to a persistent hacker.

It isn't a case of "more security tools == better security", it's a case of "why do I need these tools in the first place". If you aren't analyzing attack surfaces and building security into your software development process, I can only hope the code you write isn't accessible externally.

* using 'you' incorrectly here, speaking to those looking to apply these tools to their setup.

What's wrong with that use case though? If you get hired as the new CTO, you're not even going to know all the managers, let alone all the developers.

Clearly auditing all code, doing developer training, etc. is the right longterm approach. But putting a security culture in place and actually getting things right in this kind of environment is a multi-year process.

I'm writing for an audience of people whose businesses own the applications they're deploying. If you control code, you should have at least the security staff required to cut out the extremely low-hanging fruit attack surface that things like WAFs address. Most startups, even with 20+ developers, don't have any full-time security people, and that's fine, because even at the ~20 engineer scale, it's still perfectly feasible to cut out all the WAF-addressed attack surface for your environment.

On rare occasions I'll write advice or give an answer from the perspective of enterprise secops, but unless I say I'm doing that, I'm never really writing from the enterprise perspective. Reddit r/netsec is the right place for that POV.

If your counterexample is just reinforcing the parents erroneous point that WAFs only protect against specific applications with known vulnerabilities, and that not all websites have "doors", then you are missing the point that many WAFs have detectors for fairly generic sql injection and path traversal techniques that could very well find problems with even custom application endpoints....

So, no, not really interested in generic SQL detection rules in WAFs.

Company X, a content publisher, has a wordpress site with a bunch of plugins. They hear that the security record of such a setup is less than ideal. They buy a WAF solution to protect them.

Company Y, also a content publisher in the same space, realizes that they don't need dynamically generated content. They don't need tons of JS. They start publishing static content and what little dynamic functionality they have is well-compartmentalized, with interactions to the outside world carefully audited.

Company Y could also get a WAF, but why? It increases the attack surface, Company Y probably don't have the time/expertise to audit it, it is not clear what benefits it will have.

If you follow the Geekflare link to modsecurity, and then follow the link there to a download page, there's this warning:

>NOTE: Some instabilities in the Nginx add-on have been reported (see the Github issues page for details). Please use the "nginx_refactoring" branch where possible for the most up to date version and stay tuned for the ModSecurity version 4.

That's not something I would suggest throwing in production, given that warning has been there for years.

The post itself describes Shadow daemon as "probably defunct".

Ironbee's last commit was two years ago and I get an nxdomain looking for their website.

AWS WAF is both expensive and and extremely manual. You won't get anything out of it without a major labour investment.

Nginx's built in WAF is only in the pro edition, and outside the price of many people.

This whole discussion however depends on your customer. I have an agency I support where I have to provide a report on every single hack attempt. That means if my Rails application gets hit with said three year old Wordpress exploit, and the web server "404", I have to report on how we "blocked" it. And no, "not applicable" is not an answer.

So we throw money at commercial products. And sometimes I recommend doing so. But like security can involve just being aware of your risks, this use of a WAF is about being aware of the real problem you're solving, which is a paper work based one.