I'm looking after a site for a company as a bit of a favour,

The webserver has been running fine for the last year or so, occasionally it stops working and usually restarting apache2 gets it running again for another few months.

Today though it stopped running, I issued the above command and the website appeared instantly, but then 15 minutes later down it went again. I restarted apache2 again, and again 15 mins later it went down again. The next time it was more like 25 mins. Following that I rebooted the whole server - but 20 mins later it went down again.

Any ideas on what I can do to stop this or where to start? I have basic linux abilities but nothing amazing.

It's running Ubuntu.

I've checked the error.log file but can't see anything obvious that stands out to me.

There are loads of the following line:
[Fri Aug 20 22:48:14 2010] [debug] mod_headers.c(665): headers: ap_headers_output_filter()

There are a few errors such as missing files, and an xml2html error every so often but they don't match up with when it stops workng.

No changes have been made to the website code for the last few months.

Any ideas on where to start?

Many thanks!

Whats the memory on the server like when it crashes?

blaa@blaa$ free -m

Oh and,
Is there anything odd in the /var/log/httpd/access.log ?

tail -n 50 /var/log/httpd/access.log

I had top open during the last time it stopped and took a screen grab which had the following info:

load average: 0.02, 0.31, 0.37

Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.0%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2062564k total, 1562964k used, 499600k free, 15704k buffers
Swap: 3196892k total, 0k used, 3196892k free, 708136k cached

The server still seems very repsonsive through SSH, it just that all page requests timeout until I restart apache2

Thanks for the help

That really is strange, Some process must be hanging...

Tbh what id do is... ( bit risky, but id manage to get some sleep tonight )

Set a cron job to restart the httpd server every 10 mins over night.
Then when im ready i would run a system wide update.

apt-get update? in ubuntu i have no idea how to do this..

But id really check the access logs to make sure no one is doing something odd thats crashing it..

Can't see anything odd in the access log all looks very normal, just can't think what this would be. Are there any commands worth running whilst the web server isn't responding?

ps ax | grep httpd

netstat -tunap ( see if you have lots of connections from some random IP )
Maybe block it if you are?

With the netstat command I notice there are 30 or so (server is currently running properly) from 58.187.99.250 which seems to be from viet nam.

Not sure if that's enough to be a problem?

Ubuntu is not the best Linux distro to be running as a production webserver, even the server version is setup more for personal use.
I tried it a few months ago as a dev server (ver 10) and very quickly replaced it with Centos 5.5.

I left it for a few minutes till the next crash - at this point there was 92 connections from the viet nam address all to different ports in the 4xxx range. I've blocked that IP for the minute and restarted apache2 to see if it makes any difference

After blocking the IP Address the server has run flawlessy all night with no restarts needed. I would now like to look in to what was happening and why it was causing apache to hang? What was that IP address doing connecting to ports in the 4000 range?

The ports in the 4000 range might just be the outbound connection ports..

2 things id do is..

Post your IPtables rules here, So i can see if theres something missing..
iptables -L

Also, Go through your access.log for that IP.
cat /var/log/httpd/access.log | grep xx.xx.xx.xx

You may see something odd there..
It may be a simple ddos attack and they are just leaving the connections open, thus killing the server.

Good to know that you found the issue though.


Also as chris said,
If you feel daring or need a little project, Try making a new server with centos and migrate to it.

Iptables -L gives the following output:

Chain INPUT (policy DROP)
target prot opt source destination
DROP all -- 58.187.99.250 anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere icmp any
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh state NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:www state NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:https state NEW
ACCEPT udp -- anywhere anywhere udp dpt:domain state NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:domain state NEW
ACCEPT udp -- anywhere anywhere udp dpt:tftp state NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:69 state NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:smtp state NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:26 state NEW
ACCEPT udp -- anywhere anywhere udp dpt:ntp state NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:ftp-data state NEW
ACCEPT tcp -- anywhere anywhere tcp dpt:ftp state NEW
ACCEPT tcp -- anywhere anywhere tcp dpts:38000:38200 state NEW
DROP all -- anywhere anywhere
DROP all -- 58.187.99.250 anywhere
DROP all -- 58.187.99.250 anywhere
DROP all -- 58.187.99.250 anywhere

Chain FORWARD (policy DROP)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere state NEW,RELATED,ESTABLISHED


As for activity from the IP address, there was about 4600 lines in the activity log file from that IP address - The activity started just before the time of the first crash and continued until blocked. Looks like a regular query no attempted hacks through the query strings etc. Looks like typical bot activity as there was a lot of searches.

We get quite a bit of bot activity though and it never normally trips apache.

The lines in the activity log look something like:

58.187.99.250 "-" [20/Aug/2010:19:47:59 +0100] "GET /##################### HTTP/1.1" 200 5393 "http://www.google.com.vn/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.6; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" www.#####.com - 1 461 "-" "lastSeen=###################"

Just hashed out the bits which are actual addresses on our website.