Prometheus ALERT Rules with Arithmetic Expressions

Prometheus does not directly allow math expressions in alert rule labels and annotations (v1.7 at time of this writing).  The prometheus query function supports basic math operations so it is possible indirectly.

Create a label to express a duration in days (24 hours):

ALERT ssl_cert_expiring
  IF probe_ssl_earliest_cert_expiry - time() < 86400 * 7
    FOR 2d
    LABELS {
      days_left = '{{with printf "%f / 86400" $value}}{{query .|first|value|printf "%.2f"}}{{end}}',
      seconds_left = '{{ printf "%.2f" .Value }}'
    }

In the above example, we wanted to call {{query "$value / 86400"}} but inserting $value into the query argument is not supported.  The first step to get around this is to use printf to build the query argument.  For example if $value was 600, the printf would create "600 / 86400" which can be passed to query by using a pipe.  Because the query returns an array, we have to get the single element from the array and then take the value field from the struct.

Build a label from a query that needs to filter on the current instance

ALERT high_load
  IF node_load1 > 1.5 or node_load15 > 0.4
    FOR 5m
    LABELS {
      load_1m  = '{{ printf "node_load1{instance=\'%s\'}" $labels.instance | query | first | value | printf "%.2f"}}',
      load_15m  = '{{ printf "node_load15{instance=\'%s\'}" $labels.instance | query | first | value | printf "%.2f"}}'
    }