So... your Tomcat has created a huge catalina.out log file that is now larger than the remaining space on your disk. Logrotate kicks in, tries to make a copy of the file before truncating it, but the copy (as expected) takes up the rest of the disk space, logrotate fails and exits in humiliation and you end up with 100% disk usage.
If you can afford to restart Tomcat, things are going to work out for you.
Stop Tomcat, transfer catalina.out to a different partition/server and start Tomcat again.
If Tomcat is on your production server and you can't afford a restart because its peak-traffic-time-of-the-day, things are more complicated.
You can NOT just move catalina.out to a different mountpoint or server or just plain delete it. The file is actively open by the Tomcat process. 'rm catalina.out' will only delete the reference to the file, but the actual space on the disk will remain occupied by the file (and still actively written to). You'll end up needing to stop Tomcat so that it releases the space.
Make a copy of the file somewhere else, then truncate that thing.
'man truncate' is your friend, but this should be ok:
sudo truncate -s 0K catalina.out
It's not an instant process, so this will probably mean you'll lose a few seconds of logging.
If few minutes of downtime is more expensive than Tomcat logs (it usually is), this is the way to go.
Now, if I only I had set up central logging and/or log rotation properly...
If you can afford to restart Tomcat, things are going to work out for you.
Stop Tomcat, transfer catalina.out to a different partition/server and start Tomcat again.
If Tomcat is on your production server and you can't afford a restart because its peak-traffic-time-of-the-day, things are more complicated.
You can NOT just move catalina.out to a different mountpoint or server or just plain delete it. The file is actively open by the Tomcat process. 'rm catalina.out' will only delete the reference to the file, but the actual space on the disk will remain occupied by the file (and still actively written to). You'll end up needing to stop Tomcat so that it releases the space.
Make a copy of the file somewhere else, then truncate that thing.
'man truncate' is your friend, but this should be ok:
sudo truncate -s 0K catalina.out
It's not an instant process, so this will probably mean you'll lose a few seconds of logging.
If few minutes of downtime is more expensive than Tomcat logs (it usually is), this is the way to go.
Now, if I only I had set up central logging and/or log rotation properly...
3 σχόλια:
Nice trick, didn't know about this one. In such a case, wouldn't "echo > catalina.out" work?
As you said... proper log rotation would help a lot! Why dont you, stop the intance (as you do now) and change a bit the tomcat config?
@Lampros it seems to work, but I prefer using something "dedicated" for this kind of things. Maybe it's paranoia kicking in.
@Paris downtime is not always an option.
Δημοσίευση σχολίου