Add new comment

Drupal 8 Performance: Moving the service container cache away from the database

Difficulty: 
Come Get Some

Drupal relies on pluggable cache backends to store cache data such as Memcache, Wincache, Database, etc. The default storage backend is the Database, but Drupal being a very cache intensive application (even more in Drupal 8) you want to get better performance by using faster backends that will yield lower latency and scale better.

Moving caching away from the database is done by replacing the caching services by ones that do not rely on the database. You define the services in your services.yml file and the binaries routing in settings.php:

Services.yml

[...]

services:
  cache.backend.wincache:
    class: Drupal\wincache\WincacheBackendFactory 
    arguments: ['@app.root', '@site.path', '@cache_tags.invalidator.checksum']
  cache_tags.invalidator.checksum:
    class: Drupal\wincache\WincacheTagChecksum
    tags:
      - { name: cache_tags_invalidator}

Settings.php

$settings['cache']['bins']['bootstrap'] = 'cache.backend.wincache';
$settings['cache']['bins']['render'] = 'cache.backend.wincache';
$settings['cache']['bins']['discovery'] = 'cache.backend.wincache';
$settings['cache']['bins']['data'] = 'cache.backend.wincache';
$settings['cache']['bins']['config'] = 'cache.backend.wincache';
$settings['cache']['bins']['container'] = 'cache.backend.wincache';
$settings['cache']['bins']['toolbar'] = 'cache.backend.wincache';
$settings['cache']['bins']['menu'] = 'cache.backend.wincache';
$settings['cache']['bins']['entity'] = 'cache.backend.wincache';
$settings['cache']['bins']['default'] = 'cache.backend.wincache';

With the above code we are moving a big bunch of caching backends to in-memory. One thing I noticed is that even after routing "everything" to non database storage the service container was still being loaded from the database.

This happens because a streamlined service container (the bootstrap service container) is used before loading the service container itself, and this container has no knowledge about our services.

The bootstrap container is hardcoded into Drupal Core but can be overriden in settings.php:

if (class_exists(\Composer\Autoload\ClassLoader::class)) {
  $loader = new \Composer\Autoload\ClassLoader();
  $loader->addPsr4('Drupal\\wincachedrupal\\', 'modules/wincachedrupal/src');
  $loader->register();
  
  $settings['bootstrap_container_definition'] = [
    'parameters' => [],
    'services' => [
      'cache.container' => [
        'class' => 'Drupal\wincachedrupal\Cache\WincacheBackend',
        'arguments' => ['container', $settings['hash_salt'] ,'@cache_tags_provider.container'],
      ],
      'cache_tags_provider.container' => [
        'class' => 'Drupal\wincachedrupal\Cache\WincacheTagChecksum',
      ],
    ],
  ];
}

Notice that we had to manually register the namespace for the clases that are being used in the bootstrap container because Drupal's autoloading service is still not available in such an early bootstrap phase.

We could also modify the composer.json file so that Wincache's namespace was included in composer's autoload, but that just seems to fragile.

Another posiblity is to move the service container to Couchbase:

$settings['bootstrap_container_definition'] = [
    'parameters' => [],
    'services' => [
      'settings' => [
        'class' => 'Drupal\Core\Site\Settings',
        'factory' => 'Drupal\Core\Site\Settings::getInstance',
      ],
      'couchbase.manager' => [
        'class' => 'Drupal\couchbasedrupal\CouchbaseManager',
        'arguments' => ['@settings'],
      ],
      'cache.rawbackend.couchbase' => [
        'class' => 'Drupal\couchbasedrupal\Cache\CouchbaseRawBackendFactory',
        'arguments' => ['@couchbase.manager', $settings['hash_salt'],  $settings['hash_salt']],
      ],
      'cache.container' => [
        'class' => 'Drupal\supercache\Cache\CacheRawBackendInterface',
        'factory' => ['@cache.rawbackend.couchbase', 'get'],
        'arguments' => ['container'],
      ],
    ],
  ];

if (class_exists(\Composer\Autoload\ClassLoader::class)) {
  $loader = new \Composer\Autoload\ClassLoader();
  $loader->addPsr4('Drupal\\wincachedrupal\\', 'modules/wincachedrupal/src');
  $loader->addPsr4('Drupal\\couchbasedrupal\\', 'modules/couchbasedrupal/src');
  $loader->addPsr4('Drupal\\supercache\\', 'modules/supercache/src');
  $loader->register();
}

// Couchbase server.
$settings['couchbase_settings'] =
  array('servers' =>
    array('default' =>
      array(
        'uri' => 'couchbase://127.0.0.1'
         )
	)
  );

From questions in the comments

I did not measure the exact performance improvement of this change. But I did evaluate the overal performance improvent of moving everything - including the TagChecksum which we had to reimplement - off the database. The biggest performance gain came from moving TagChecksum storage off the database. The problem is that you can't store your tag checksum in a volatile storage, so this was quite an experimental performance test. Hiting the database as little as possible and using proximity in-memory storage is the key to get an application properly performing on cloud setups.

Seriusly, I don't think this can be considered a bug. Understand what you are deploying and how it works, and be happy that you have the flexibility to change and override things.