After I got Concerto running at work, everyone was super happy with it, it was working great and was very reliable. I barely touched it for the first 60 days that it ran. The HR folks made their updates and all was good with the world, until it wasn’t. I came into work one day and Concerto was down. There was no content flowing to the 10 TV’s we had setup and no matter how hard I tried I could not get it running even though the VM was up. I was eventually able to figure out the issue. For some reason the developers of the program didn’t put any log management into the program. This simply means that if left unattended the production log file will continue to grow and eventually fill up the hard drive to the point where Concerto can’t run because the hard disk is out of space. The VM is only a few gigs so that took approximately 65-70 days for me.
What I decided to do was to restore the VM from a recent backup so I would have some breathing room and then did what I should have done initially, setup log management for Concerto. Obviously, system administrators should follow good log management practices and the developers of Concerto apparently expected the same of me LOL! The good news is the Concerto VM is running Debian Linux and Linux makes log management very easy with the “logrotate” command that runs via a system cron job every day. All you need to do is setup an entry for Concerto in the configuration file “logrotate.conf”.
The logrotate configuration file is located in the /etc directory, you can edit it using the following command:
sudo nano /etc/logrotate.conf
You will need the location of the production log file for Concerto for the logrotate entry, it is located in:
/usr/share/concerto/log/
Add the following to the bottom of the logrotate.conf file (“www-data” is the user and group that Concerto runs as and therefore owns the logfile):
# system-specific logs may be configured here
/usr/share/concerto/log/*.log {
su www-data www-data
daily
missingok
rotate 5
compress
notifempty
copytruncate
}Save the file, you can verify this works by using the following command:
sudo logrotate -f /etc/logrotate.conf
This will rotate the logs daily compressing the rotated log and keeping the 5 most recent log files (you can certainly keep more but I haven’t found the need to do so). When you go back to the Concerto log file directory you should see a new log file as well as a compressed older log file, here is what mine looks like currently with 5 days of compressed older logs:
concerto@concerto:/usr/share/concerto/log$ ls -ailh
total 9.3M
145032 drwxrwxr-x 2 www-data www-data 4.0K Oct 15 06:25 .
142615 drwxr-xr-x 16 www-data www-data 4.0K Jun 25 13:31 ..
145033 -rw-rw-rw- 1 www-data www-data 4.0M Oct 15 15:19 production.log
145429 -rw-rw-rw- 1 www-data www-data 1.2M Oct 15 06:24 production.log.1.gz
145410 -rw-rw-rw- 1 www-data www-data 1.1M Oct 14 06:25 production.log.2.gz
145422 -rw-rw-rw- 1 www-data www-data 1.2M Oct 13 06:24 production.log.3.gz
153459 -rw-rw-rw- 1 www-data www-data 1.1M Oct 12 06:24 production.log.4.gz
145426 -rw-rw-rw- 1 www-data www-data 865K Oct 11 06:24 production.log.5.gz
This will eliminate the possibility of the Concerto production log running amok and filling up your hard drive. There is one more thing you need to do to maintain Concerto and avoid space issues which is clearing out cache files, we’ll cover it in another upcoming post!
Thanks,
Rob
