DDoS attack? Or idiot administrator?

A few weeks ago, my network sustained an external DDoS attack.  Or so I thought.  I had hundreds of connections coming into my network on TCP 80 (web traffic), which caused my already small pipe to become saturated.  When I stopped Apache, or closed TCP 80 at the firewall, they all went away.  I contacted my ISP to see if they would help.  Too keep things simple and sweet, we both decided just to change my IP addresses to an entirely different block.  Problem solved... until today.

That was about three weeks ago.  Today, I had the same set of symptoms, and when I checked into things, I realized the problem was back.  But why would anyone launch a DDoS attack against me?  The problem with DDoS attacks is that they are run by an external force and there's really not much you can do about it.

Software firewalls can help, but it depends on the load imposed on such an attack so there's only so much you can do with software.  A hardware-based firewall is ideal, but not every home-based network administrator has access to the kind of cash and expertise necessary to obtain and use one.  My ISP was somewhat powerless because the law prohibits them (and every other ISP in the US, actually) from analyzing traffic.  All they can tell you is how much bandwidth is being used, but they cannot tell you from where the connections originate, their payload, or any other details.

Hmm, so this was up to me to "fix".  I didn't want to change my IP addresses again, because not only would that require a couple of hours of DNS changes, but it was apparent that such a move was only a bandage, not a cure.  Okay, let's play with `netstat` and see what we can see.

Here's a sample of what I was seeing with netstat (the actual command was `netstat -antw | grep TIME_WAIT | grep :80`).

tcp        0      0 172.16.144.14:37431     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:48184     206.57.28.16:80         TIME_WAIT
tcp        0      0 172.16.144.14:37515     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37465     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37564     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37378     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37561     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37479     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37405     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:58340     116.126.142.87:80       TIME_WAIT
tcp        0      0 172.16.144.14:37426     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37518     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37519     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37546     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37388     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37448     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:54687     66.104.86.10:80         TIME_WAIT
tcp        0      0 172.16.144.14:37526     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:36184     216.255.235.79:80       TIME_WAIT
tcp        0      0 172.16.144.14:48366     206.57.28.16:80         TIME_WAIT
tcp        0      0 172.16.144.14:37559     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37400     76.13.222.11:80         TIME_WAIT
tcp        0      0 172.16.144.14:37536     76.13.222.11:80         TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:55688    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:60988    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:60190    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:57198    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:44688    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:36655    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:40883    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:39652    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:60034    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:49779    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:59507    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:56360    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:47909    TIME_WAIT
tcp6       0      0 172.16.144.14:80        61.139.105.166:4530     TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:33878    TIME_WAIT
tcp6       0      0 172.16.144.14:80        61.139.105.166:3908     TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:32968    TIME_WAIT
tcp6       0      0 172.16.144.14:80        61.139.105.162:2831     TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:55521    TIME_WAIT
tcp6       0      0 172.16.144.14:80        61.139.105.166:1064     TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:36158    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:43896    TIME_WAIT
tcp6       0      0 172.16.144.14:80        89.149.209.100:38941    TIME_WAIT
 

After a few hours of checking things, I realize that the problem was right there, staring me in the face!  If you look at the output above, you'll see that I not only have connections coming in on TCP 80, but I also have a lot of connections going out on TCP 80.  I hadn't noticed that before, mainly because the netstat output was so long.  So what does this mean?

The server undergoing what I thought was a DDoS attack is a linux-based web server running Apache.  Some sites are hosted on this particular server, but other are hosted on another Windows-based server.  So, this server is a reverse-proxy for the Windows-based server as well.  Are you starting to see where I'm going and what my screw-up turned out to be?

Yes, folks, I am an idiot.  On the sites which were reverse-proxied, a few were missing one crucial directive: ProxyRequests Off.  Apache uses mod_proxy (and a few other modules) for the proxy function.  Apache can be a typical forward proxy (like Squid), but it can also be used as a reverse proxy -- brilliant!  However, with mod_proxy enabled, forward proxying is enabled and wide open by default.  Without ProxyRequests Off, my server was happily acting as a proxy server for outside users.

So now I know the whole story.  Apparently, someone (or some automated process) out there on the Internet found one of the virtual hosts on my server and determined that I had a hole -- it was a wide-open forward proxy.  They exploited the hole and began using my server as a proxy for visiting other websites.  I fixed that (temporarily) by changing my IP addresses, but sinde the problem was not fixed, the exploit returned.

I don't know what sites were being proxied nor by whom, but I do know it was consuming all of my bandwidth.   I've since added that directive to the server's main configuration, and the hole has been sealed.  Running a forward proxy without securing it can be dangerous because users can visit unsavory websites, but the connections appear to come from the proxy instead of the original requester!  My intention was NOT to run a forward proxy, but because I didn't pay attention when I configured some of the virtual hosts and failed to add the ProxyRequests Off directive, that's exactly what I had.