Resolving Failed Queue Jobs

This article walks through common causes and troubleshooting tips for failed Craft queue jobs.

General Debugging Tips #

If Craft doesn’t see any activity from a job after five minutes, it will be marked as “Failed” in the control panel. At that point you can choose to either Retry the job to restart it, or Release job to cancel it and delete it from the database.

In the rare case that a job is still running after five minutes without providing any status update to Craft, the job will still be marked as “Failed” in the control panel. As soon as the job provides the next status update, the control panel will show that it is back in a running state again.

If the job continues to fail after re-trying, you’re likely running into some environmental limitation. Reasons for this may include (but not be limited to) the following:

• PHP’s low execution time limits cause the job to time out.
  For Apache, that usually means increasing your php.ini file’s max_execution_time. See this article on timeout errors for more info.

• It’s a memory-intensive job and PHP is running out of memory.
  Check your php.ini file’s memory_limit setting.

• The job is causing some other fatal PHP error.

There’s a good chance something is getting logged in Craft’s storage/logs/phperrors.log. Check that file and see if there are any recent logs in it. If it was a fatal error coming from your web server (Apache/nginx/etc.), check their error log files as well.

Debugging Stuck “Generate Pending Transforms” Jobs #

The “Generate Pending Transforms” job is created when a template or GraphQL query references an image transform that hasn’t been generated yet.

This job is particularly sensitive to memory limitations and execution time limits listed in the General Debugging Tips section because processing images in PHP is, by nature, a memory-intensive operation.

Be sure your template is not trying to transform twenty 4000×2000px images in a template on a shared host where PHP is limited to 64MB of memory.

At a 4000×2000px resolution, a single 32-bit CMYK PNG file requires nearly 31MB of memory just to load the image into memory—much less manipulate it—and that’s not taking into account the memory overhead Craft needs to operate.

It’s also important to know the underlying image processing library your server’s using to process these images.

Craft supports both GD and ImageMagick, but we recommend ImageMagick because it supports a wider range of image formats, generally produces better-quality output, and has more image manipulation options available. By default, Craft will automatically use it when it’s available. You can explicitly force Craft to use one or the other with the imageDriver config setting.

Aside from the normal memory and execution time limits, it’s common for the “Generate Pending Transforms” job to get stuck if you are using GD and trying to transform an image GD doesn’t support, like a 32-bit PNG or animated GIF.

It’s also common to run into bugs if you’re running older versions of either GD or Imagick. We recommend running the latest stable version that you can for either library.

• GD releases: https://github.com/libgd/libgd/releases
• ImageMagick releases: https://github.com/ImageMagick/Website/blob/main/ChangeLog.md
  (PHP Imagick wrapper)

Debugging Stuck “Delete Stale Template Caches” Jobs #

The “Delete Stale Template Caches” job is created when you use the native Craft template {% cache %} tag and one of the elements inside that tag changes.

While the General Debugging Tips still apply, there may still be an underlying issue to identify and fix in order to prevent the job from becoming stuck.

A job is a PHP request just like any other request, subject to the same environmental limitations as PHP. The more data you’re caching, the more time and memory will be required therefore increasing the chances for failure.

If you’re unable to narrow the exact cause of the issue, you can take more general steps to get around it:

• Don’t use the native {% cache %} tag. Use Varnish or some other front-end caching solution instead.
• Use the {% cache %} tag but set the cacheElementQueries config setting to false, and come up with your own strategy for deciding when to delete template caches. i.e. a cron job, or every 30 minutes on Tuesdays, etc.
• Use a caching plugin with different caching behavior from the native {% cache %} tag implementation. There are several options available in the Plugin Store.
• It’s probable you’re using the {% cache %} tag as a solution to the N+1 database problem. Use eager loading instead of caching.

Some general rule-of-thumb things to know to reduce the chances of this job failing:

• Craft has to keep track of every single element (Asset, User, Matrix Block, Entry, etc.) inside of a {% cache %} tag to know when to intelligently bust the cache if one of those elements has changed. Each one of those elements basically ends up being another step in the “Delete Stale Template Caches” job. Smaller, more focussed cache tags that just focus on database-heavy parts of the template are much better than, for example, putting a {% cache %} tag in your base layout template and just caching everything.
• Be explicit with your cache keys. By default, Craft will use the current page URI as the cache key (including the query string), which may or may not be the best behavior for your site. If you’re getting pages hammered with tons of query string variations, that creates extra (and probably unnecessary) caches that the job also has to account for.