I've gotten a good number of DDoSes sent my way. For the ones I've noticed, there's usually pretty good IP diversity. Volumetric attacks are either like tcp syn floods, spoofed from everywhere, or udp reflection spoofed from you to reflecting hosts, which have pretty good diversity. If you want to survive these, you need to have either a big connection, or packet filtering by someone with a big connection. As of a few years ago, 10Gbps was enough to ignore casual attacks, as long as your IP stack is up for it -- you may need to do a bit of tuning and make sure you've got recent syn handling. On the other hand, if you're running a 10Gbps connection, be sure you're not a reflection target -- be extremely careful about running UDP servers that send significantly larger replies than the requests, if they're exposed on public ips.

Layer 7 attacks are different; you can't spoof those, so you don't get perfect distribution -- but there are lots of ways to distribute simple requests. If the requests are coming from a botnet, there's usually a lot of control about what the requests look like, but if they're coming in through tricking other software (which is unfortunately common), then at least you'll likely have some identifying information; it's dumb to block things by user-agent, but it can be pretty effective. The way to handle these is really to try to make sure the effort your server spends is roughly on par with the effort the client spends; and try to make sure you're running the best optimized TLS handshakes you can (ECC certs are easier on servers than RSA).

I don't know where the line goes between DOS and DDOS, but in my youth I had a virtual server attacked by an acquaintance for a few days. They still won't admit to doing it, but all available evidence at the time pointed to them.

It started as a TCP SYN flood, which I had never seen before -- but since it originated from only two addresses, I could block them. Once I did, more addresses joined in on the attack. I figured out what was happening and how to prevent the connection table from filling up completely under the SYN flood, and then the attack changed shape into a UDP flood, from yet more addresses.

(Many of the addresses corresponded to free shell hosts and managed webhosts with easily exploitable PHP scripts. I tried to inform the people running those hosts that they were being used for an attack, but the vast majority of them seemed to ignore that. A couple of people responded and we could figure out kinda what had happened and what scripts were involved. Remember, kids, that connecting a machine to the internet is a great responsibility.)

They tend to look like that in your Apache logs when they target at the web application level.

Once you're flooded and your process load spikes, you can't see what your web server has run out of workers to handle.

Netstat will give you a better picture as far as IPs/connections once you get the process load down so you can actually run anything.

Anybody know why blocking the offending IP didn’t work?

Most likely because he was trying to block the IP at a point where it was forwarded by some other service. At that point, the offending IP would only exist in the x-http-forwarded-for http header and not as the source address of the request.

Yup! This was it.

I have had this before similar situation to op they found a heavy page that hammered the database until it went down. Banning their ip at the firewall stopped them, they came back a few days latter using a bunch of proxies about 1000. I wrote a script like Fail2Ban that detected the ip and blocked it worked a treat.

I was also attacked by a single ip that sent 10gbit of nonsense to apache, I had to contact my isp to get them to block the ip down stream from me, if they had used a bot net I don't think I would have been able to stop it.

By definition, DDoS is Distributed. From my experience working on a Layer4 DDoS Protection solution, a typical case often range from 1000 to 100k flows.