How to prevent memory consumption during collection enumeration?

Question

In the following script:

    $pageSize = 1000;
    $productCollection = Mage::getModel('catalog/product')
        ->getCollection()
        ->addWebsiteFilter($website->getId())
        ->addAttributeToFilter('status', Mage_Catalog_Model_Product_Status::STATUS_ENABLED);
    for ($currentPage = 1; $currentPage <= $pages; $currentPage++) {
        $productCollection->setCurPage($currentPage);

        foreach ($productCollection as $productItem) {
            $this->writeProductForStores($ioObject, $productItem, $stores, $defaultStore);
        }

        $productCollection->clear();
    }

Let's say I've processed page #1, now I'm on the page #2, processing products 1001-2000, and I don't need products 1-1000 from the first page, but they're still in the memory (I'm monitoring the script execution from top)! And the memory consumption grows as the execution goes, then falling with exception of surpassing the memory limit.

How do I clear memory after finishing processing the previous page?

What $this->writeProductForStores() for? – sv3n Oct 16 '18 at 20:21 — sv3n, Oct 16 '18 at 20:21

score 2 · Accepted Answer · answered Oct 16 '18 at 02:02

2

You need to change how you iterate the collection.

Inchoo did a great write-up on the issue:

https://inchoo.net/magento/working-with-large-magento-collections/

answered Oct 16 '18 at 02:02

ProxiBlue

9,926
3
34
60

I want to use pages. I believe, it's possible to achieve with usage of them, isn't it? I wouldn't like to write my own iterator every time I work with big collection - it seems to be an overwhelming approach. – Nikita Abrashnev Oct 16 '18 at 18:05

score 1 · Answer 2 · answered Oct 16 '18 at 20:21

1

I want to use pages. I believe, it's possible to achieve with usage of them, isn't it? I wouldn't like to write my own iterator every time I work with big collection - it seems to be an overwhelming approach.

I agree with @ProxiBlue ... example code mentioned in Inchoos write-up work also in "standalone"-sripts ... and performs well on large collections ...

Examples:

answered Oct 16 '18 at 20:21

sv3n

11,657
7
40
73

I agree that this approach is good, but... Can we achieve the same thing doing some magic with clearing products from the processed pages? – Nikita Abrashnev Oct 16 '18 at 21:01
@NikitaAbrashnev I'm not sure what you mean with "pages" ? Something like bulk-operations? (I guess this code works also for several thousends of products.) May add some info to ypir question? – sv3n Oct 16 '18 at 21:06
Yep, there're pages in my code in the question:) $pageSize = 1000; $productCollection->setCurPage($currentPage); – Nikita Abrashnev Oct 16 '18 at 21:08
@NikitaAbrashnev I've seen this code. :) But why? Is it really neccessary? – sv3n Oct 16 '18 at 21:09
I just think it's possible. I got an answer on my inital question, actually, and I'll pick the answer, but just for myself I wanna know why paginating doesn't really clear previous pages from the memory and how can I manually clear them (pages)? – Nikita Abrashnev Oct 16 '18 at 21:11
@NikitaAbrashnev reminder for me ... https://magento.stackexchange.com/questions/106962/what-does-getloadedproductcollection-clear-do? Will check it tomorrow. :) – sv3n Oct 16 '18 at 21:18
1

as you can see in my question, this method is already used, but it doesn't clean the memory of the running script – Nikita Abrashnev Oct 16 '18 at 21:20

score -1 · Answer 3 · answered Oct 16 '18 at 03:53

-1

This is what I normally do:

$ids = Mage::getModel('catalog/product')
    ->getCollection()
    ->addWebsiteFilter($website->getId())
    ->addAttributeToFilter('status', Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
    ->getAllIds();
foreach ($ids as $id) {
    $product = Mage::getModel('catalog/product')->load($id);
    $this->writeProductForStores($ioObject, $product, $stores, $defaultStore);
}

getAllIds() fetches all the IDs in one DB read but it requires loading the model n times. This is best when you need all the attributes of the model. The walk() example by Inchoo reads the DB n times but no loading of model. If you have addAtributeToSelect to limit the amount if info, the walk() method is more efficient.

answered Oct 16 '18 at 03:53

kiatng

694
3
16

Please do not recommend to load products in a loop. – sv3n Oct 16 '18 at 07:24
May I know why? – kiatng Oct 16 '18 at 08:24
Its one of the baddest things you can do. load($id) loads the whole product model an may lead to heavy memory consumption and load times. – sv3n Oct 16 '18 at 08:46
Yes, I'm agree with sv3n. I also want to add that there's no reason to reload the product inside of the cycle as you may add to select all the attributes you need on the collection itself before it's loaded. Example: $collection->addFieldToSelect('*') or $collection->addFieldToSelect('sku') – Nikita Abrashnev Oct 16 '18 at 11:34
1

@kiatng some numbers from antother question ... https://magento.stackexchange.com/questions/179273/find-min-max-value-of-product-attribute-in-a-large-category/179275#179275 – sv3n Oct 16 '18 at 20:10
1

It is one of the biggest performance killers – ProxiBlue Oct 16 '18 at 22:50
Depending on the size of your results, and memory setup allocated to PHP, this will just as well have out of memory issues. You simply have not encountered out of memory as your datasets is likely small. – ProxiBlue Oct 16 '18 at 22:53
I agree with the comments, and you can read that in the last few sentences in my original answer that I understand the concern. In the code snippet in the question, writeProductForStores() seems to imply that we need all the product attributes, and there may be observers listening to the load events, so loading within a loop may not be a horrible option. But if optimization is the goal, I'll go one step further and use only the interfaces in the resource singleton, which has a more direct access to the DB and can be made to bypass all sorts of steps including events. – kiatng Oct 17 '18 at 03:12

How to prevent memory consumption during collection enumeration?

3 Answers3