Dead Simple cron Monitoring With Sentry; Avoiding the “Oh S$%T” Moments

Imagine you, like me, have several important cron jobs running on remote servers. While it’s important to you that these jobs succeed on their scheduled cadence, you don’t have the time to manually check in on them.

With a simple python script and Sentry we can easily send an alert email to ourselves when a problem occurs.

Our python script will read in the previous command’s exit status and alert to sentry if non-zero.

Begin by creating a Sentry account. They give you 5000 free events a month, so this should more than cover your needs for personal work.

Next, take the secret url they give you and write it to a file on your server as json:

{"sentry_url": "your-secret-url"}

Now install Sentry with pip. We are using pip3 and python3 here.

pip3 install --user --U sentry-sdk

Now we will write a python script that reads in an exit code and sends alert if it is non-zero.

Our python script:

#!/usr/bin/env python3
import json
import sys
import sentry_sdk

# write 
sentry_url = json.loads('/path/to/sentry/secret').get('sentry_url')
sentry_sdk.init(sentry_url)


class FailureAlert(Exception):
    pass


def alert_on_failure(sys_arg):
    """Send a sentry alert if exit status of previous command was non-0"""
    exit_code = int(sys_arg.argv[1])

    if exit_code != 0:
        raise FailureAlert(f'script failed with {exit_code}')


if __name__ == '__main__':
    alert_on_failure(sys)

Assuming you put your script in ~/alert_on_failure.py, don’t forget to make it executable with:

$ chmod u+x ~/alert_on_failure.py

Now test the script by running it and passing a non-zero argument to it:

./alert_on_failure.py 1

With that, you should get an alert in your Sentry dashboard.

Now let’s test passing a real command’s status code:

$ asdf; ./alert_on_failure.py $?

Assuming you do not have a command named asdf on your system path, this command will fail and exit with a non-zero status code.

The ; indicates to bash that it should run the second command regardless of the status code of the first. As a result, our alert_on_failure script will now run with $? as an argument.

Conveniently, $? is a bash shortcut for the value of the previous command’s exit code.

Bringing it all together, the asdf command should fail and return a non-zero status code, which alert_on_failure will pick up, sending another event to sentry.

Finally, let’s update our crontab tasks to use our new script:

$ crontab -e

Update your tasks as follows:

your-exiting-cron-schedule existing-task-script; /full/path/to/alert_on_failure.py $?

And there you go! Feel free to create a task that executes immediately to test that it really works.

We just have one more thing to take care of.

Sentry has many great features, including sensible default settings that only alert developers once when a new *type* of error occurs.

However, our exceptions only contain an exit status right now, so if we receive multiple exceptions Sentry will only email us once.

We can change this default though to alert on every error event instead.

Here is a screenshot of how to change it in the Sentry UI. You can find it under Settings -> [Your Team] -> Projects -> [Your Project] -> Alerts -> Rules -> Edit.

Now you’ll get an email straight to your inbox every time one of your cron jobs fails. Finally, peace of mind!