Quick script to prewarm cache using sitemap.xml

Question

I was wondering if there is a small (nightly optimization) script out there that can

1. clean all cache, including fpc (https://github.com/GordonLesti/Lesti_Fpc) [check]
2. open the sitemap of corresponding store [check]
3. visit every link in there (to warm cache and FPC) [check, taken from W3 TC]

Pref. all from 1 script. And with the ability to run it *per domain, or *loop through all store views. I found this: Pre-Warming the Magento Enterprise Full Page Cache & other snippets.

ps. We know caching != performance and would really appreciate you help

Thanks, Sean

(added working code to my own question haha - took me 3 hours but here you go)

Answer 1

This seems to work for us

File: RefreshCache.php in root Magento folder, called from CRON

Could be made shell compliant

<?php

// 1 LOAD INITIALS
define('MAGENTO_ROOT', getcwd());
$mageFilename = MAGENTO_ROOT . '/app/Mage.php';
require_once $mageFilename;
//umask(0);

Mage::log('RefreshCache started @ ' . date('d-m-Y H:i:s'));

ini_set('display_errors', 1);

/* Store or website code */
$mageRunCode = isset($_SERVER['MAGE_RUN_CODE']) ? $_SERVER['MAGE_RUN_CODE'] : '';

/* Run store or run website */
$mageRunType = isset($_SERVER['MAGE_RUN_TYPE']) ? $_SERVER['MAGE_RUN_TYPE'] : 'store';

Mage::app($mageRunCode,$mageRunType);

// 2 CLEAN CACHE
Mage::app()->cleanCache();

try {
    $allTypes = Mage::app()->useCache();
    foreach($allTypes as $type => $blah) {
      Mage::app()->getCacheInstance()->cleanType($type);
    }
} catch (Exception $e) {
    echo $e->getMessage();
}

// 3 FIND SITEMAPS
// todo: per store selector yes/no, now it runs 1 domain and for all stores

$collection = Mage::getModel('sitemap/sitemap')->getCollection();

//echo $collection->getSelect();

foreach($collection as $sitemap) {
    $url = substr_replace(Mage::getBaseUrl() ,"",-1) . $sitemap->getData('sitemap_path') . $sitemap->getData('sitemap_filename');
    crawlUrl($url);
}

// 4 CRAWL SITEMAPS
// Take from http://www.pixelenvision.com/1572/php-cache-warmer-preloader-for-w3-total-cache/
// W3 TOTAL CACHE WARMER (PRELOADER) by Pixel Envision (E.Gonenc)
// Version 2.1 - 21 August 2011

function crawlUrl($sitemap_url) {

    //Configuration options
    $delay = 0.5;               // Delay in seconds between page checks, default is half a second
    $quiet = true;              // Do not output process log (true/false)
    $trailing_slash = false;    // Add trailing slash to URL's, that might fix cache creation problems (true/false)

    $sitemap = $sitemap_url;    //Path to sitemap file relative to the warm.php

    //Do not change anything below this line unless you know what you are doing
    ignore_user_abort(TRUE);
    set_time_limit(600);

    $xml = simplexml_load_file($sitemap);
    foreach ($xml->url as $url_list) { 

        $url = $url_list->loc;

        if($trailing_slash==true) {$url = rtrim($url,"/")."/";}

        $ch = curl_init();
        curl_setopt ($ch, CURLOPT_URL, $url);
        curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 15);
        curl_setopt ($ch, CURLOPT_HEADER, true);
        curl_setopt ($ch, CURLOPT_NOBODY, true);
        $ret = curl_exec ($ch);
        curl_close ($ch);

        usleep($delay*1000000);

        if($quiet!=true) {echo "Warmed: ".$url."\n";}

    }
    unset($xml);

if($quiet!=true) {echo "Crawled: " . $sitemap . "\n";}

}

Mage::log('RefreshCache finished @ ' . date('d-m-Y H:i:s'));

exit;

Answer 2

Here's a tool that does it without needing scripts: https://pagespeedplus.com/blog/cache-warmer. It's a bit easier to use.