How to change the directory where images are saved, scrapy

Question

I'm scraping images from a webpage. I've several list of images:

list_1,list_2,list_3

each one of those list corresponds to a different category, so far all images are saved in the same directory, the one which is specified in the settings file

IMAGES_STORE = '/home/user/Desktop/folder1/folder2'

I want to save the images from each list in a different forder.

score 2 · Accepted Answer · edited May 23 '17 at 11:52

This is answered here and here.

Basically override the item_completed method and include your logic of where you want to save the images based on which list they come from.

def item_completed(self, results, item, info):

    for result in [x for ok, x in results if ok]:
        path = result['path']
        slug = slugify(item['designer'])


        settings = get_project_settings()
        storage = settings.get('IMAGES_STORE')

        target_path = os.path.join(storage, slug, os.path.basename(path))
        path = os.path.join(storage, path)

        # If path doesn't exist, it will be created
        if not os.path.exists(os.path.join(storage, slug)):
            os.makedirs(os.path.join(storage, slug))

        if not os.rename(path, target_path):
            raise DropItem("Could not move image to target folder")

    if self.IMAGES_RESULT_FIELD in item.fields:
        item[self.IMAGES_RESULT_FIELD] = [x for ok, x in results if ok]
    return item

How to change the directory where images are saved, scrapy

1 Answers1