I use Varnish to speed up Magento but my catalog is 450K+ SKU's so it takes time to Apache to serve the page to the browser. Of course after the page has been cached by Varnish, delivery time is millisecond stuff so, to avoid customers to wait too long for a NON-cached page to be delivered, I was wondering if there is a way to make Varnish cache all the categories pages which are the heavier to generate on the server.
thanks
The best option is on cold re/start is a cache warmer (aka crawler) to simply traverse the site and get the cache hot.
I've used something like this simple script in the past for such (originally from OnePica):
<?php
set_time_limit(0);
ini_set('max_execution_time', 0);
error_reporting(E_ALL);
ini_set('display_errors', 1);
$_CONFIG['domain'] = ''; // ie: www.youdomain.com
$_CONFIG['host'] = ''; // database server
$_CONFIG['username'] = ''; // database user
$_CONFIG['password'] = ''; // database password
$_CONFIG['database'] = ''; // database Magento uses
$_CONFIG['concurrency'] = 8; // default is 8
/*** YOU DO NOT NEED TO EDIT BELOW THIS LINE ***/
if(isset($_GET['start'])) {
echo "Initial memory usage: ".memory_get_usage()."\n";
$crawler = new crawler();
$crawler->connect($_CONFIG['host'], $_CONFIG['username'], $_CONFIG['password'], $_CONFIG['database']);
$crawler->start($_CONFIG['concurrency'], 'http://' . $_CONFIG['domain'] . '/');
} else {
echo 'For best results, you should run System > Cache Management > Catalog Rewrites Refresh first<br />';
echo '<a href="?start=1">Click here</a> to start crawling';
}
class Crawler {
private $urls = array();
private $numOfUrls = 0;
private $numOfChunks = 0;
private $totalTime = 0;
public function __construct() {
set_time_limit(0);
ob_start();
$this->flushOutput();
}
public function connect($host, $user, $pass, $database) {
mysql_connect($host, $user, $pass) or exit('Could not connect to the database');
mysql_select_db($database) or exit('Could not connect to the database');
}
public function start($concurrency, $baseUrl) {
//category urls
$result = mysql_query("SELECT request_path AS url FROM core_url_rewrite WHERE category_id IS NOT NULL AND product_id IS NULL GROUP BY category_id ORDER BY request_path");
while($row = mysql_fetch_assoc($result)) $this->urls[] = $row['url'];
//product urls
$result = mysql_query("SELECT request_path AS url FROM core_url_rewrite WHERE product_id IS NOT NULL GROUP BY product_id ORDER BY category_id, request_path");
while($row = mysql_fetch_assoc($result)) $this->urls[] = $row['url'];
$chunks = array_chunk($this->urls, $concurrency, true);
$this->numOfUrls = count($this->urls);
$this->numOfChunks = count($chunks);
$this->totalTime = 0;
echo '<pre>';
foreach($chunks as $chunk) {
$this->getChunk($chunk, $baseUrl);
}
echo "\n--------------------------------------------------\n";
echo 'TOTAL URLS: ' . $this->numOfUrls . "\n";
echo 'TOTAL SETS: ' . $this->numOfChunks . "\n";
echo 'TOTAL TIME: ' . round($this->totalTime, 3) . " seconds\n";
echo 'TIME / SET: ' . round($this->totalTime/$this->numOfChunks, 3) . " seconds\n";
echo 'TIME / URL: ' . round($this->totalTime/$this->numOfUrls, 3) . " seconds\n";
echo '</pre>';
$this->flushOutput();
}
private function getChunk($chunk, $baseUrl) {
static $chunkCount = 0;
$mh = curl_multi_init();
$time = microtime(true);
$chunkCount++;
echo "\n";
foreach($chunk as $x=>$URL) {
$url = $baseUrl . $URL;
echo $url . "\n";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_ENCODING, 'gzip,deflate');
curl_multi_add_handle($mh, $ch);
curl_close($ch);
unset($ch);
}
$running = 0;
do { curl_multi_exec($mh, $running); } while ($running > 0);
curl_multi_close($mh);
unset($mh);
$time = microtime(true) - $time;
$this->totalTime += $time;
echo 'Memory used:' . memory_get_usage() . " | ";
echo 'SET ' . $chunkCount . ' OF ' . $this->numOfChunks . ' FINISHED: ' . round($time, 3) . ' seconds';
$this->flushOutput();
}
private function flushOutput() {
echo "\n<script type='text/javascript'> setTimeout('document.body.scrollTop = document.body.scrollHeight;', 100); </script>";
ob_end_flush();
ob_flush();
flush();
ob_start();
}
}
NOTE: Enterprise has this feature built-in:
System->Configuration->Advanced "System"->Full Page Cache Auto Generation
You can build a cache warmup script just using wget, sed and grep. I use this script build up my cache:
wget -O – subdomain.test.de/site.html | grep -o '<a href="http[^"]*"' | sed 's/<a href="//;s/"$//' | grep "subdomain.test.de" | while read line; do
wget -P /tmp/warmup/output --delete-after $line
done
It is gets all the links in subdomain.test.de/site.html and follows them. I use it for Magento Enterprise FPC and for Varnish-based sites. wget -l should do the same. For me it did not work for FPC on the same server. A detailed description (unfortunately in German) is available here: http://www.code4business.de/magento-full-page-cache-warmup/.
If you have a page listing all categories or your categories have a specific structure you can limit the cached pages to the categories. To do so you have to adjust the part
sed 's/<a href="//;s/"$//' | grep "subdomain.test.de"
I also tried the Magento Enterprise built-in Cache Warmup. However, it did not seem to be as configurable as I needed it. I think that it is only useful for a very limited number of products.
