Eager-Loading Elements

When you are working with a list of elements and need to access their nested or related elements, eager-loading can help optimize your element queries.

Let’s look at a template that loops through a list of news posts (entries), and displays an image (asset) attached to each one.

{% set posts = craft.entries()
  .section('news')
  .all() %}

{% for post in posts %}
  <article>
    {% set image = post.featureImage.one() %}

    {% if image %}
      {{ image.getImg() }}
    {% endif %}

    {# ... #}
  </article>
{% endfor %}

In addition to the main query for the entry elements, an additional query is executed for each article to find the attached asset. The number of queries will be N (the number of entries) + 1 (the initial entries query). If there are 50 entries, this innocent-looking template will cost the page 51 queries!

If we were to then display the name of the user who uploaded each image (i.e. image.uploader.fullName) or a list of categories for each entry (i.e. post.topics.all()), we would introduce a 2N+1 query, whereby each entry in the initial query would trigger two additional queries, for a minimum of 101. Supposing each query takes a couple of milliseconds, this can quickly add up to 250–500ms just waiting for data.

Eager-loading allows Craft to proactively load related or nested elements in bulk, while keeping track of where they all belong.

Not every relationship needs to be eager-loaded! We have a list of recommendations for how to identify opportunities for optimization.

# Identifying Candidates

Auditing your templates for performance problems solvable with eager-loading can be tricky, but you may be able to narrow down what you’re looking for with these common bottlenecks:

  • Nested {% for %} loops;
  • Accessing or outputting data from nested blocks in a Matrix field;
  • Getting information about an entry’s author, within a loop;
  • Using multiple asset transforms;
  • Using relational fields within a loop;

The exact patterns here may differ template-to-template, and don’t always involve an explicit call to a query execution method (like .one() in the example above)! Implicit queries may be masked by filters like |first or |slice, or when iterating over a relational field’s value in a for loop.

Not all of these situations will require (or benefit from) eager-loading—the goal is only to consider which of your projects’ features may be candidates.

# Lazy Eager-Loading New!

Element queries in Craft 5 have a new .eagerly() query param that simplifies eager-loading of relational fields. Most projects can replace custom eager-loading maps (via the .with() method) with a call to .eagerly() on down-stream element queries.



 
 
 
 






















{% set posts = craft.entries()
  .section('news')
  .with([
    ['featureImage', { withTransforms: ['large'] }],
    ['topics'],
  ])
  .all() %}

{% for post in posts %}
  {% set image = post.featureImage|first %}

  <article>
    <h2>{{ post.title }}</h2>

    {% if image %}
      {{ image.getImg('large') }}
    {% endif %}

    {{ post.summary|md }}

    <ul>
      {% for topic in post.topics %}
        <li>{{ topic.getLink() }}</li>
      {% endfor %}
    </ul>
  </article>
{% endfor %}

In this example, our primary query (for entries in the News section) no longer needs to be concerned about what nested or related queries might be used—instead, each query that stems from an entry returned by the primary query should use .eagerly() to let Craft know that there is an opportunity to fetch a set of elements as though they had been eager-loaded, up-front. The main difference between this strategy and .with() is that—as the name “lazy” suggests—nested and related elements aren’t loaded until Craft actually evaluates logic that needs them; templates (like element partials) can request additional relational data without impacting the upstream query, while avoiding optimistic-but-unnecessary up-front queries.

After calling .eagerly(), lazily eager-loaded relations are used just like normal relational field values. This means your templates can continue chaining .one(), .all(), or applying filters like |first to get results.

Native attributes currently do not support lazy eager-loading due to the way many of their values are accessed, but they can be eager loaded using .with().

# The .with() Query Param

The purpose of the .with() param is to tell Craft which related or nested elements you are apt to need, in advance. With that information, Craft is able to fetch them all up front, in as few queries as possible.

Here’s how to apply the with param to our example:

{% set entries = craft.entries()
  .section('news')
  .with(['assetsField'])
  .all() %}

{% for entry in entries %}
  {# Get the “first” (or only) eager-loaded asset: #}
  {% set image = entry.assetsField|first %}

  {# Make sure we actually have an asset element: #}
  {% if image %}
    <img src="{{ image.url }}" alt="{{ image.title }}">
  {% endif %}
{% endfor %}

This template code will only cost three queries (one to fetch the entries, one to determine which assets should be eager-loaded, and one to fetch the assets), and the number of queries will not grow proportional to the number of elements being displayed. The entries are then are automatically populated with their respective related asset(s).

# Accessing Eager-Loaded Elements

Eager-loaded elements are returned as a collections—or more specifically, an craft\elements\ElementCollection (opens new window). This means that (in most cases) eager-loaded and non-eager-loaded elements can be treated in the same way.

There are some differences in the methods exposed by element queries and

Accessing eager-loaded elements works a little differently than regular elements, under certain circumstances. Take a look at how we assigned the image variable in our examples, before and after applying the with param:

{# Before: #}
{% set image = entry.assetsField.one() %}

{# After: #}
{% set image = entry.assetsField|first %}

When the assets aren’t eager-loaded, entry.assetsField gives you a preconfigured asset query to return the related assets.

However, when the assets are eager-loaded, entry.assetsField gets overwritten with an array of the eager-loaded assets. So one(), all(), and other element query methods are not available. Instead you must stick to the standard array syntaxes. In our example, we’re grabbing the first asset with entry.assetsField[0], and we’re using Twig’s null-coalescing operator (??) to define a default value (null) in case there is no related asset.

# Eager-Loading Multiple Sets of Elements

If you have multiple sets of elements you wish to eager-load via the primary query, add those field handles to your .with() parameter:




 
 












{% set posts = craft.entries
  .section('news')
  .with([
    'topics',
    'featureImage',
  ])
  .all() %}

{% for post in posts %}
  <article>
    {# Use the eager-loaded elements normally: #}
    {% for topic in post.topics %}
      {{ topic.getLink() }}
    {% endfor %}
  </article>
{% endfor %}

Lazy eager-loading works the same for any number of eager-loaded fields. Each query that stems from an element in the primary query should use .eagerly() to signal that it will be used similarly in later iterations of a loop.

# Eager-Loading Nested Sets of Elements

It’s also possible to optimize loading of elements nested two or more levels deep, using a special syntax. Suppose the topics in our news feed from previous examples each had a thumbnail or icon:




 












 
 














{% set posts = craft.entries()
  .section('news')
  .with([
    'topics.thumbnail',
  ])
  .all() %}

{# Main loop: #}
{% for post in posts %}
  <article>
    <h2>{{ post.title }}</h2>

    {# ... #}

    {# Nested topics loop: #}
    <ul>
      {% for topic in post.topics %}
        {% set icon = topic.thumbnail|first %}

        <li>
          <a href="{{ topic.url }}">
            {% if icon %}
              {{ icon.getImg() }}
            {% endif %}
            <span>{{ topic.title }}</span>
          </a>
        </li>
      {% endfor %}
    </ul>
  </article>
{% endfor %}

A dot (.) in the eager-loading map indicates that another set of elements on the eager-loaded results also needs to be loaded. Craft expands this notation into a multi-stage eager-loading “plan” and fetches each layer of elements in a batch. Each segment of an eager-loading path will be a field handle.

The same optimization is possible with lazy eager-loading—the .with() param can be omitted from the main query, and .eagerly() can be added to all the nested queries:










 
 






{% set posts = craft.entries()
  .section('news')
  .all() %}

{% for post in posts %}
  <article>
    {# ... #}

    <ul>
      {% for topic in post.topics.eagerly().all() %}
        {% set icon = topic.thumbnail.eagerly().one() %}

        {# ... #}
      {% endfor %}
    </ul>
  </article>

Elements that are the target of multiple relations (as you might expect with a set of categories or tags) are only loaded once per batch. For example, if we were displaying a list of 100 news articles that were each assigned one of 10 categories, eager-loading the categories would never fetch more than 10 elements—Craft assigns the returned categories to all the articles they were related to.

# Defining Custom Parameters on Eager-Loaded Elements

You can define custom parameters that will get applied as elements are being eager-loaded, by replacing its handle with an array that has two values: the handle, and a hash that defines the parameters that should be applied.

{% set entries = craft.entries()
  .section('news')
  .with([
    ['assetsField', { kind: 'image' }],
  ])
  .all() %}

When eager-loading nested sets of elements, you can apply parameters at any level of the eager-loading path.

{% set entries = craft.entries()
  .section('news')
  .with([
    ['featureImage', { dateUploaded: ">= #{now|date_modify('-1 month')}" }],
    ['authors', { group: 'staff' }],
    ['topic.thumbnail', { kind: 'image' }],
  ])
  .all() %}

The accepted parameters correspond to the kind of element that each relationship uses—this example uses the asset query’s dateUploaded and kind params, and the user query’s group param.

You can also provide nested with directives via these hashes. Here’s an example where we’re loading each topic’s thumbnail and sectionEditor, as well as all their descendants (presuming that our topic category group is a multi-level structure):

{% set entries = craft.entries()
  .section('news')
  .with([
    ['topic', {
      with: [
        ['thumbnail', { kind: image }],
        ['descendants'],
        ['sectionEditor'],
      ]
    }]
  ])
  .all() %}

The equivalent “flat” eager-loading map would look like this—note the repetition:

[
  ['topic.thumbnail', { kind: 'image' }],
  ['topic.descendants'],
  ['topic.sectionEditor'],
]

Ultimately, Craft normalizes both maps to something more like the first example, then loads each “layer” in sequence—the topics have to be loaded before anything attached to them!

# Eager-Loading Image Transforms

Another N+1 problem occurs when looping through a set of assets, and applying image transforms to each of them. For each transform, Craft needs to execute a query to see if the transform already exists.

This problem can be solved with the withTransforms asset criteria parameter:

{% set assets = entry.assetsField
  .withTransforms([
    'heroImage',
    { width: 100, height: 100 }
  ])
  .all() %}

Note that each transform definition you want to eager-load can either be a string (the handle of a transform defined in Settings → Assets → Image Transforms) or a hash that defines the transform properties.

Using the withTransforms param has no effect on how you’d access image transforms further down in the template.

Image transform indexes can be eager-loaded on assets that are also eager-loaded:

{% set entries = craft.entries()
  .with([
    ['assetsField', {
      withTransforms: ['heroImage']
    }]
  ])
  .all() %}

# Eligible Fields and Attributes

Relational fields are the most common candidate for eager-loading. Fields are always eager-loaded using their handle. In this example, manufacturer is an entries field attached to each “car” entry:

{% set carsWithManufacturers = craft.entries()
  .section('cars')
  .with([
    ['manufacturer'],
  ])
  .all() %}

Element types support a variety of unique eager-loadable attributes, as well:

All Element Types
  • Immediate children: children
  • All descendants: descendants
  • Immediate parent: parent
  • All ancestors: ancestors
  • Same element in other sites: localized
  • Drafts: drafts
  • Draft creators: draftCreator
  • Revisions: revisions
  • Revision creators: revisionCreator
Assets
  • Uploader: uploader
Entries
  • First selected author: author
  • All authors: authors
Users
  • Addresses: addresses
  • Profile photo: photo

Fields and attributes that don’t use element queries (or don’t ultimately resolve to one or more elements) are generally not eager-loadable—and trying to eager-load them may result in an error.

Some elements may also expose query methods that optimize the loading of associated non-element records (like asset transforms), but their interfaces are not part of the core element-only eager-loading system.

# Reverse Relationships

While related elements can be queried in either direction using sourceElement and targetElement options, it is only possible to declaratively eager-load elements that are the target of a relationship. If you need to query for other elements that point to an element in the main query, the queries must be written separately:

{# We can’t load these poets *and* their poems in one query: #}
{% set poets = craft.users()
  .group('poets')
  .all() %}

{# This second query takes the list of poets and loads all their poems: #}
{% set poems = craft.entries()
  .section('poems')
  .authorId(poets|column('id'))
  .all() %}

{# Then, we can group those poems by their author’s ID: #}
{% set poemsByAuthorId = poems|group('authorId') %}

{% for poet in poets %}
  <article>
    <h2>Poems by: {{ poet.fullName }}</h2>
    <ul>
      {% for poem in poemsByAuthorId[poet.id] ?? [] %}
        <li>{{ poem.title }}</li>
      {% endfor %}
    </ul>
  </article>
{% endfor %}

Of course, if each poet user element had a custom field in which administrators selected “Featured Poems,” that would be eager-loadable:

{% set poets = craft.users()
  .group('poets')
  .with([
    ['featuredPoems'],
  ])
  .all() %}