8

Ever since we applied the SUPEE-6788 patch on a client's site, about once a day the site has gone down & the only thing that seems to bring it back is to clear the cache. We've looked at the logs, & a bunch of them seem to include "Front controller reached 100 router match iterations". This problem wasn't occurring before the patch was applied. Anyone have any idea what could be causing this? Some people say it could be a cache bug in magento issue, but I can't tell. Any input would be helpful!

Some additional notes:

  • There hasn't been any heavy loads on the server right when it goes down, so that's not a factor.
  • Yes all previous patches were applied successfully.
  • We are using memcache to store the cache.
Daryl Gochnauer
  • 386
  • 4
  • 12
  • Not sure if this is related but this module is specific for performance with the new blocks and variables added to SUPEE-6788 https://github.com/EcomDev/SUPEE6788-PerformanceFix – David Manners Nov 13 '15 at 17:12
  • As another data point, we have a site which has also had this issue bring down the site twice so far with the 100 router match iterations error. It did not start until SUPEE-6788. After the first time I applied the AmpersandHQ patch (SUPEE-4755) but the issue still occurred a few days later, so that patch did not fix the issue for us. We are running Magento 1.7.0.2 with the Redis cache. – Nick Nov 21 '15 at 02:45

6 Answers6

3

Myself and another dev have been experiencing a similar issue, however we appear to have solved it by applying the patch present in this GitHub: https://github.com/AmpersandHQ/magento-ce-ee-config-corruption-bug

The cause appears to be some sort of race condition where the cache is cleared by one process while being re-instantiated by another, I've been able to reproduce it by following the steps also listed on that GitHub.

I've opened a support ticket with Magento for this issue and have my suspicions about what has started to cause it since the patch, but am waiting to hear back.

You can read more about it on the following question: Problems with Unstyled Page, Bad Paths, loss of layout configuration after application of SUPEE-6788 Patch.

Community
  • 1
Persata
  • 151
  • 4
  • Has this fix been tested on 1.8.1.0 w/ SUPEE-6788 installed? – Daryl Gochnauer Nov 13 '15 at 18:57
  • @dgwexdev13 I've not tested it on 1.8, but I developed the patch on 1.9 and 1.13 simultaneously. I dont think the module in question (Mage_Core_Model_Config) has changed in a LONG while so the patch should pretty much apply to all versions. I've seen this patch running happily with 1,12, 1.13, 1.14 systems with SUPEE 6788 installed. – Luke Rodgers Nov 13 '15 at 22:16
  • Ps - Please do update here if/when you have heard back from Magento. Thanks – Daryl Gochnauer Nov 21 '15 at 16:52
  • I'm afraid applying SUPEE-4755 along with SUPEE-6788 is not doing much to stop the "100 iterations reached" errors. I've applied both to a number of unrelated websites and I keep seeing occasional crashes on all of them. Has anyone had more luck? – Jan Tomka Dec 17 '15 at 23:58
1

We have the same issue with 3 sites version 1.8.1. It started to appear after SUPEE 6788. The fix from above didn't solve the problem. Actually, it seems there is some change. Before the fix the sites were crashing twice a day, now the last crash was after 2 days. But may be it is related to the load. The 3 sites are small and not very loaded. This problem doesn't appear with a big site which is version 1.6.2 and SUPEE 6788 applied. All sites are on the same server - the 3 with version 1.8.1 and the big one with version 1.6.2

Dimitar Dimitrov
  • 11
  • 1
  • This doesn't provide an answer, more suitable for a comment. You should ask some good questions and provide some good answers on the site. When you earn enough reputation, you'll be able to post comments as well. – Prateek Nov 19 '15 at 12:21
  • 1
    yes, I understand, but when I try to comment I have a message "You must have 50 reputation to comment". And I think it is important information that this happens to other sites as well. And seems to be version specific. – Dimitar Dimitrov Nov 19 '15 at 12:35
  • @DimitarDimitrov basically the same thing - we had a busy day on Tuesday and the site went down about once per hour. By moving the config cache off of Redis and just using file base caching (I'm still using Redis for FPC), I was able to stabilize the store. – Phil Birnie Nov 20 '15 at 14:40
  • After the big store with version 1.6.2. crashed many times with different error: "Illegal scheme supplied, only alphanumeric characters are permitted" we were forced to revert the patch. 24 hours since then no crashes and all our stores are stable. I really don't like the idea to work without the patch, we are trying to find the reason with one test installation, but the problem is that it doesn't crash. Probably real actions are needed to crash it and I don't know which exactly. We tried to provoke the crash with ab but it seems not to be load related. – Dimitar Dimitrov Nov 20 '15 at 15:16
  • I forgot to mention we are using simple file based caching. The server is with php 5.4 and opcache, but disabling any php caching doesn't help – Dimitar Dimitrov Nov 20 '15 at 15:26
1

We had the site cache switched from memcache to Redis & then added a cronjob to dump the cache every 12 hours. It went from crashing once a day to going about 4-5 days before it went down again. We then tweaked the cronjob to dump every 6 hours & it hasn't gone down since (it's been about 3-4 days since). Neither us or the hosting company can track down the actual issue, but this seems to be a working fix for us. Hope that helps someone.

Daryl Gochnauer
  • 386
  • 4
  • 12
  • I recommend you input the logging form here: https://github.com/AmpersandHQ/magento-ce-ee-config-corruption-bug#debugging-the-issue

    That way you'll see what piece of code is continually triggering configuration cache corruption saves.

     – Luke Rodgers Nov 20 '15 at 16:00
1

I added the AmpersandHQ debug code this morning and just now had the "Front controller reached 100 router match iterations" exception happen about 75 times in a 2 minute period. But this time, presumably because of the debug code not saving the corrupt cache entry, the site is still up without everyone getting exceptions (I did not flush the cache).

I have not dug into this yet to investigate but corrupt-cache.log contains:

2015-11-22T03:42:42+00:00 DEBUG (7):
#0 app/code/core/Mage/Core/Model/App.php(1147): Mage_Core_Model_Cache->save('<admin><design>...', 'config_global_s...', Array, NULL)
#1 app/code/core/Mage/Core/Model/Config.php(552): Mage_Core_Model_App->saveCache('<admin><design>...', 'config_global_s...', Array, NULL)
#2 app/code/core/Mage/Core/Model/Config.php(474): Mage_Core_Model_Config->_saveCache('<admin><design>...', 'config_global_s...', Array, NULL)
#3 app/code/core/Mage/Core/Model/App.php(421): Mage_Core_Model_Config->saveCache()
#4 app/code/core/Mage/Core/Model/App.php(343): Mage_Core_Model_App->_initModules()
#5 app/Mage.php(683): Mage_Core_Model_App->run(Array)
#6 index.php(87): Mage::run('', 'store')
#7 {main}

This is on Magento 1.7.0.2 with Redis cache and AmpersandHQ's SUPEE-4755 patch already applied.


Dec 2, 2015 update: Here is another error with the full stack trace:

2015-12-02T20:02:27+00:00 DEBUG (7):
#0 app/code/local/Mage/Core/Model/App.php(1156): save('<admin><design><package><name>default</name></package><theme><default>find</default></theme></design></admin>', 'config_global_stores_admin', Array, NULL)
#1 app/code/local/Mage/Core/Model/Config.php(552): saveCache('<admin><design><package><name>default</name></package><theme><default>find</default></theme></design></admin>', 'config_global_stores_admin', Array, NULL)
#2 app/code/local/Mage/Core/Model/Config.php(474): _saveCache('<admin><design><package><name>default</name></package><theme><default>find</default></theme></design></admin>', 'config_global_stores_admin', Array, NULL)
#3 app/code/local/Mage/Core/Model/App.php(430): saveCache()
#4 app/code/local/Mage/Core/Model/App.php(343): _initModules()
#5 app/Mage.php(683): run(Array)
#6 index.php(87): run('', 'store')
Nick
  • 121
  • 7
  • Fantastic mate. Thanks for posting yoru stack trace. Could you please read this gist? https://gist.github.com/convenient/2a30572d6d4bcae9796c

    I've an idea for narrowing it down to whether it is a useCache = true object cache error or something else entirely.

     – Luke Rodgers Nov 22 '15 at 16:30
  • OK I patched those additional files. Thank YOU for the work on the patches. Last night after I posted, the error happened two more times in a 30 minute period. But then after that it hasn't happened in 15 hours. So I'm not really sure how to predict when it might happen again. – Nick Nov 22 '15 at 20:47
  • Okay cool, please keep me up to date. Thanks – Luke Rodgers Nov 22 '15 at 20:48
  • I got 2 more 100 router match errors after applying the additional patch you gave me in the gist. So that didn't fix the problem. After the first one, I modified the debug code slightly to give the entire stack trace instead of the truncated PHP one. I didn't have room in my comment here so I modified my original post above to include the new trace. – Nick Dec 02 '15 at 20:45
  • So it is erroring in the "find" module's theme. Our site doesn't use the find module, and it looks like that company is now defunct anyway (but the module used to be shipped with Magento by default), so I disabled the module going forward. Not sure if that will fix the problem though or if it will just show up again listing a different theme. – Nick Dec 02 '15 at 20:52
  • Had it happen 2 more times. It is strange how when it happens it seems to happen several times within a couple hours, and then it goes days without happening. Now that I disabled the "find" module it is producing the same stack trace but this time shows the "default" theme, so the "find" module wasn't the problem. – Nick Dec 02 '15 at 22:59
  • Hey Nick. I'll have a look at your debug data next week. I'm on holiday from all things Magento this week, sorry it's still bugging you and I hope you can get it sorted. – Luke Rodgers Dec 02 '15 at 23:02
  • No problem, at least with the patch when it goes down it is just temporary instead of corrupting the cache and staying down. Have a great holiday. – Nick Dec 03 '15 at 02:54
  • I'm in the exact same boat. Nothing has really seemed to solve the issue for me. Our site typically stays up for about a week, but my stack trace is identical to yours - very frustrating. I'm going to try to upgrade to EE 1.14.2 to see if that somehow addresses the issue - it's a stab in the dark. I'm also going to try to modify @Luke Rodgers' patch so that it outputs the malformed XML to a log file so I can take a look at it. – Phil Birnie Dec 11 '15 at 02:26
  • @PhilBirnie Good luck outputting the malformed xml :) I've been down that route before and had never been able to make head nor tale of it haha. One of my 1.13.0.1 instances has started producing the same stack traces, it's on my todo list to review it further. – Luke Rodgers Dec 11 '15 at 10:45
  • @LukeRodgers did you find anything interesting that you could share ? I'm having this same issue that started out of nowhere. Store was running fine for almost 2 years, now it is crashing every 5 mins. Got it stable after disabling the config cache from admin, but that has slowed down the site significantly. – Prateek Jan 18 '17 at 18:33
1

We have been experiencing the same issue for weeks now with various Magento CE websites. However, none of the suggestions posted here have helped. After several frustrating debug sessions over several weeks we have finally managed to pin this down.

In summary we found the problem to be due to a combination of the SUPEE-6788 patch, Magento < 1.9.2.0 and PHP >= 5.5.22, with potential attackers or even security scanners able to take the sites down on demand. We have posted full details, including a fix, on our blog. I truly hope this helps any other poor souls suffering with the same issue.

mustdobetter
  • 318
  • 1
  • 7
0

We're experiencing this issue and our websites since we release SUPEE6788 and it seems that fraudulous calls to xmlrpc webservices could be responsible for cache corruption.

We're blocking webservice calls from our front servers since we don't use them + applying SUPEE 4755, I'll keep you posted.

SMASHED
  • 121
  • 2
  • This patch did modify xml validation to use libxml_disable_entity_loader which is not thread safe. In some instance this can cause Magento to redirect to the installation page, however i believe it is also possible that before errors like this, that it would miss the loadDB step of the config generation, saving corrupted data into the cache. See http://magento.stackexchange.com/questions/30071/warning-simplexml-load-file-i-o-magento-warning-failed-to-load-external-en – Luke Rodgers Jan 22 '16 at 17:17