0

Problem

I was trying to down scale number of very big images, to reduce computing time required to edit them, so I created a little python script and down scaled them using pygame. The script had taken a input directory and down scaled every image and saved them in output directory, keeping directory structure. This had worked fine, although little slowly. So I decided to run program more times, to make use of more cores. I have modified the original code to skip files, so I can run more of them (for example one program would do every even image, and the other would do every odd image)

Turns out this works only mostly. I have run 3 programs this way at once, and about 5% of images are corrupted (missing end of image, image replaced by parts of others) and I have no idea why. Any ideas what could I do ?

Details

The script being run can be found here http://pastebin.com/Cw8kiDsy. (The script expect two lines with one number each, so it can do every (A*N + B)th file. ). The script was run with python 3.2.3 and pygame 1.9.2

The images resolutions are about 2700 x 2700 or 2700 x 5400, the target size is 600 pixel on the shorter side (600x600 and 600x1200). The input folder (containing only images) has about 40 GB

There may be problems with paths, as directories have a Unicode filenames, although I have taken steps to prevent problems with Unicode chars.

Roukanken
  • 116
  • 1
  • 4
  • The way you do your concurrency seems that it may introduce the errors you describe. How are you picking `A`, `N`, and `B` in your `A * N + B` calculation? You'd probably see clearer if you wrote this program a) with threads, or b) as a worker/queue model (one master program reads the filesystem, dispatches the filenames to a queue, and worker programs read a filename from the queue, convert it, and repeat until the queue is empty. – bitgarden Nov 14 '13 at 19:05
  • For massive image resizing I would choose tools like [ImageMagick](http://www.imagemagick.org/Usage/resize/) – furas Nov 14 '13 at 19:12
  • @bitgarden I convert file if `files_done % A == B` (A and B are read from input), while iterating with `os.walk()`. I have modified script to list all file it wants to convert, and compared outputs - there wasn't any conflicting filename. As for the other solutions, I'm not quite sure how to implement them. – Roukanken Nov 14 '13 at 21:10

0 Answers0