I recently had a problem with a hypervisor where the dom0 was underprovisioned for RAM, and it OOMKiller'd the hypervisor processes for VMs during scheduled rsync backups. After nice-ing the processes appropriately to prevent this I decided to implement a simple cron to email me whenever oomkiller fires. This obviously isn't foolproof, as something could be killed preventing outgoing mails. It is however unlikely, so this will likely be a good enough solution going forward.
#!/bin/bash touch /root/ooms.log FSZ=$(stat --printf "%s" /root/ooms.log) grep -i oom-killer /var/log/messages >> /root/ooms.log echo $(uniq < /root/ooms.log) > /root/ooms.log NEWSZ=$(stat --printf "%s" /root/ooms.log) if [ $FSZ != $NEWSZ ] then echo "New OOM detected, investigate /var/log/messages:" tail -n1 /root/ooms.log fi
Pretty simple altogether, just make sure to run it once before you install it to root's crontab. Don't forget that you can send crons to multiple recipients by making the relevant updates to /etc/aliases for that cron's user and running newaliases.