Caching in Craft 3 causes out of memory when including grouped for loop

Question

I have a problem with using {% cache %}in Craft 3, although the code runs fine without caching, and it works in Craft 2.

I have narrowed the problem down to the following code:

{% cache %}
  {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
  {% for date, shows in upcomingShows|group("showTime|date_modify('-3 hours')|date('Y-m-d')") %}
    {% for show in shows %}
      {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
    {% endfor %}
  {% endfor %}
{% endcache %}

It gives me the following fatal error:

Allowed memory size of 268435456 bytes exhausted (tried to allocate 71002178 bytes)

If I skip the Twig |groupfilter, the cache works again:

{% cache %}
  {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
  {% for shows in upcomingShows %}
    {% for show in shows %}
      {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
    {% endfor %}
  {% endfor %}
{% endcache %}

(I first thought the problem was the relatedTo() function, but it turns out it's the |group is what triggers it, without it it works also with the relatedTo().)

Is this a bug in the cache, or is there a way around it?

UPDATE: After some more testing, it seems like it works either with the |group filter or the relatedTo() function, but not both at the same time.

Answer 1

After looking through this, it would probably be better off as a Github issue than a Stack Exchange post, but answering here for posterity.

In short, this comes down to a change we had to make in Craft 3 to fix some other caching issues where we started to serialize element queries when saving them, so that when retrieving them later to determine if a cache should be busted, we would unserialize and execute the saved element query.

It turns out that serialization requires significantly more memory than not serializing.

In the case of your database, with your custom fields and plugins on both shows and movies sections, it takes about 1.1MB of memory to save each element query inside of your {% cache %} tags.

Given your current template code, with the nested for loops and the number of elements you're accessing in the cache tag, you're looking at about 7,000 element queries that Craft has to track, or 7.7GB of memory - hence the error you're hitting.

It's on our list to completely revamp the template caching engine to help mitigate scenarios like this (as well as other template caching issues people have come across)

In the meantime, you've got a few options.

1. Revamp your template logic so you're not executing O(n2) database queries. That could involve pure Twig/templates or a custom plugin where you can grab exactly the data your template needs from PHP/Querybuilder.

2. Set the cacheElementQueries config setting to false. Craft will no long attempt to record/bust template caches, so you can come up with your own logic for busting them (cron job, plugin, etc.).

3. Use a plugin like Cache Flag (or a similar one). It also doesn't attempt to save element queries, but provides a way to bust caches based on keywords.