How To Set startingDeadlineSeconds For Kubernetes CronJobs

The startingDeadlineSeconds property in Kubernetes CronJobs is an important parameter. It determines the behavior of jobs that miss their scheduled time for any reason.

startingDeadlineSeconds determines the deadline in seconds for starting the job if it misses its scheduled time. If a CronJob misses its scheduled time for any reason and more time than startingDeadlineSeconds has passed, the job will not be started.

A Kubernetes CronJob might not start on time for a few reasons:

• If the pod has trouble coming up for any reason, including resource constraints, image pull backoffs, taints and tolerations, etc.
• If there’s an issue with the cluster, control plane, networking, or scheduler.
• If it misses its start time by startingDeadlineSeconds

The default value is nil, meaning there’s no deadline for the job. This means that if a cron job misses its scheduled time and startingDeadlineSeconds isn’t set, the job can be executed at any time.

At the next scheduled time, the same job will be executed again without regard to the number of missed executions.

I’ve seen incidents caused by this behavior.

For example, we had a job deployed and missed its execution for some reason. This job didn’t have its startingDeadlineSeconds set. Then, a deployment changed the cron’s schedule. Out of nowhere, the job that missed its deadline was scheduled again and its original time.

I’ve also run into issues where jobs without startingDeadlineSeconds set don’t start because of one of the abovementioned hiccups.

Moral of the story? Set startingDeadlineSeconds

Here are some things to consider when setting startingDeadlineSeconds:

• If the job is critical and has to be run even if it’s a little late, set it to a higher value. For example, I work at a FinTech company. It’s critical that some of our jobs are executed even if it’s a little late.
• If your job is frequently run, like every minute, set it to a smaller value so jobs don’t run too far out of their intended schedule.
• If the job takes a lot of time to start or initialize, set a higher value to be safe.

Regardless of what you set your job’s deadline to, always monitor your CronJobs and set up alerts.

---