4

My image transform task always gets stuck halfway through so that I have to "repair" the task as per this guide. Im trying to convert about 150 images (amounts to ~300 steps). Resetting the task within the database does work but it's not a permanent solution.

I've upped the php memory limit to 1024M, execution time is 0 and Apaches timeout is set to a ridiculously high number. Disk space is available and it's happening with imagemagick as well as gd. It also happens both on my local dev VM running Ubuntu 14.04 as well as on my Debian server. PHP and Apache error logs are empty on both machines. I'm running the Personal edition and the latest version of Craft.

I also tried to set ini_set('max_execution_time', 0); in craft/app/services/AssetTransformsService.php but it didn't change anything.

Some more info:

  • Some image transforms are missing after restarting the task until it's complete. That renders the workaround by resetting the task in the db basically useless because not all images are transformed by the time it's complete.
  • It's not a specific image that is preventing the task from completing, as it's always a different image that it's stuck on.
  • There are usually 75-85 images converted until it's stuck.
  • Update Asset Indexes doesn't help.
  • Clearing all caches makes Craft convert all images again. It doesn't delete the old files, it just overwrites them (and gets stuck, as usual).
Sören Kampschroer
  • 143
  • 4
  • Any chances some (or all) of these 150 images you're transforming are animated GIFs? – Brad Bell Aug 15 '16 at 22:47
  • Nope, they are all JPGs. Between 0.4 and 2.5MB in size. – Sören Kampschroer Aug 15 '16 at 23:18
  • Have you bumped https://craftcms.com/docs/config-settings#phpMaxMemoryLimit from its default 256M as well? – Brad Bell Aug 15 '16 at 23:19
  • Hey Brad, just tried that but it didn't help. I've set it to 1024M and it now gets to 110-120 before it stops. I tried to set the max exectution time to 0 in AssetTransformsService.php but that didn't change anything. Logs are still empty as well. Last entry in craft log is [Forced] Starting step 116 of 310 total steps. and then nothing. – Sören Kampschroer Aug 16 '16 at 17:20
  • I just tried using completely different images, in case something was wrong with them, but it still gets stuck at step 120. – Sören Kampschroer Aug 16 '16 at 17:33
  • Made some transform changes in https://craftcms.com/changelog#build2911 (mostly around animated GIFs, but wouldn't hurt to update and test). If not, do you get any related errors in craft/storage/runtime/logs? Any chance you're running older versions of Imagick/GD that could be updated? – Brad Bell Aug 16 '16 at 22:23
  • I updated Craft but it didn't help. I tried using GD again and it did get to 223/310 but then it stopped. GD version is 2.1.1, Freetype 2.5.2. Imagemagick was on 6.7. I installed Imagemagick/imagick from source but that doesn't work at all. Images are not resized and getUrl() gives me the url of the original image. There were no errors when compiling/installing imagemagick though and phperrors.log is still empty. – Sören Kampschroer Aug 17 '16 at 02:25
  • It could be that GD is choking on an image/format is doesn't recognize. Imagick supports a much wider range of images and formats. I think I'd focus on getting Imagick installed and working and try the test with that. – Brad Bell Aug 17 '16 at 02:37
  • Yeah thats why I wanted to use imagick in the first place. The reports from above are all on Imagemagick 6.7. Is there some kind of minimum required version? Have you tried ImageMagick v7.0.2 with imagick v3.4.3RC1? I'm using vagrant, so I could give you my config and you could see for yourself. Should be easily reproducible. – Sören Kampschroer Aug 17 '16 at 02:42
  • Okay I think I found the problem now. Despite setting my php memory limit to 1024M in both my php.ini and Crafts general config, memory usage gets crazy high when converting images. By the time the task reaches step 100, php is using 85% of available memory. Once memory and swap is completely full it returns so 1.5% memory usage and thats when the task gets stuck. We are talking about 212MB worth of images, how can that fill up 2GB+ of memory? I can't be the only one experiencing this. The exact same thing happens in my Ubuntu VM and Debian server with different versions of Imagemagick and GD. – Sören Kampschroer Aug 17 '16 at 12:17
  • @SörenKampschroer, unfortunately you're not alone... I've also some serious issues for images generations that get stuck again and again, despite a server configuration higher than the minimum requirements. Like I'm currently experimenting, perhaps you could look for cloud-based solutions like imgix.com or cloudinary.com to externalize these tasks. – Romain P. Dec 20 '16 at 12:40
  • @Romainpoirier It was an ImageMagick issue actually. Have a look at natebeaty's answer below. ImageMagick's memory limit is independent from PHP's but it looks like PHP is using up all that memory in htop. I'd rather not use external image hosting since I have a server that is able to handle it. – Sören Kampschroer Feb 13 '17 at 18:42

1 Answers1

2

I was having this issue (w/ PHP 5.6, Craft 2.6.x, on WebFaction hosting) because it seemed like PHP had no memory limit and single processes were handling huge image transform queues, bombing when it got over 1.5gb or so.

Making sure Imagick has a memory cap seems to do the trick to avoid the queue getting stuck as PHP runs out of memory, which is discussed here:

Possible memory leak - GeneratePendingTransforms/ImageMagick

Relevant code:

// From http://stackoverflow.com/a/12835966
// Pixel cache max size
$this->setResourceLimit(\Imagick::RESOURCETYPE_MEMORY, 256);
// Maximum amount of memory map to allocate for the pixel cache
$this->setResourceLimit(\Imagick::RESOURCETYPE_MAP, 256);

in craft/app/vendor/pixelandtonic/imagine/lib/Imagine/Imagick/Imagick.php at line 33, at top of smartResize() function.

Update:

If you are not on shared hosting but on a VPS (or otherwise have sudo access on your server, OR are able install ImageMagick + php-imagick from source), you can set memory caps for ImageMagick directly using a policy.xml file, e.g.:

<policy domain="resource" name="memory" value="1GiB"/>
<policy domain="resource" name="map" value="1GiB"/>
<policy domain="resource" name="area" value="1GB"/>
<policy domain="resource" name="file" value="768"/>
Community
  • 1
Nate Beaty
  • 576
  • 5
  • 10
  • Thanks for your answer! I actually managed to fix this in the meantime but I forgot to post the results. Your way does work but I'd recommend changing these values globally for ImageMagick since your changes could be overwritten by a craft update. You can do so in /etc/ImageMagick-6/policy.xml. I've made a pastebin with the needed additions. If you could add this to your answer I will mark it as accepted. – Sören Kampschroer Feb 13 '17 at 18:34
  • Ok, done! I'm assuming a lot of people are running Craft on shared hosting, so this wouldn't help them. But it's obviously a better solution (and a good thing to do in general) if you have access to /etc/ImageMagick/policy.xml or /etc/ImageMagick-6/policy.xml or wherever else your distro might place that file. Thanks for the update. – Nate Beaty Feb 14 '17 at 19:07