In a previous article on the Magento performance I have talked about the ways which let to achieve acceptable performance of oscommerce based on Magento. In this article I will talk about how we have achieved even more speed of the system making it much more horizontally scalable. But first a few words concerning the tools mentioned in the previous article. Unfortunately, upon a closer view it turned out that there are errors in the memcached-tags project, after a time the demon fell in with a «segmentation fault» error what is unacceptable of course.  Also we did not like the fact that there was no memcached increment and decrement commands support, we planned to use them, at least this probability was. Besides from the result of testing it has emerged that in the process of cache resetting on any tag all the cache is locked. It is not a problem if a small number of entries in the cache is marked by the tag but in the case of a large number of records invalidation delays measured in seconds, so performance is lost. After the memcached-tags source investigating  it was decided to take the latest stable version of memcached and to implement tags support from scratch. As a result, all the tasks have been successfully completed, the project was named  the  “extcached”. Here are some characteristics:

  • Full multithreading support;
  • Entries on the tag are immediately reseted and without cache blocking;
  • There are special options for management of additional memory for tags and structures storage which provides communication between tags and records;
  • A new command “add_tags ()” has been added, the command allows to add an array of tags to a record;
  • Increment and decrement commands support;
  • Caching servers cluster support;

 We also have improved php_memcache for new features support. The only thing we have decided is not to implement binary protocol support, having decided that such refinements are needed unless the Facebook, having maid a point that if it is necessary, this will not be a difficulty.

Now, with tools for data in memory caching, which would provide at the same time cache validity it was possible to enter upon Magento cache system implementation. We immediately rejected full-page caching, having decided that it is not optimally to make online-store on Magento  and curtail seriously its functionality both at once. It was decided to focus on the HTML blocks cache. By default, each block in Magento can be cached, if you implement getCacheKey (), getCacheTags () and getCacheLifetime () functions which must return the key, tags and cache lifetime. But what`s to be done if in a block another one supposed to be cached, well for example with another lifetime?. Besides we have found that to configure caching by editing the source code  is not correct. We also decided to drop distinction between an authorized user and guest (as is often recommended), having decided that authorized users may be at least not less than the guests in the online-store. Therefore, the module, which would be responsible for caching have several basic requirements: first, cache settings should be realized without editing the code, and secondly, it should be possible to cache blocks in the other blocks, third, caching should be done regardless of whether guest visitor or authorized user.

 As a result, we have developed a module Magento Extmage_Extcache , with all the foregoing  requirements have been met.

Let's start with the cache settings. All the settings are made in a special xml file and consist of two parts - description of the cached block through which determined whether the block is cached, as well as cache configuration  defines  and description of the cache configuration. By the means of the cache configuration description we define  cache key for the block, lifetime and tags set. One of the major difficulty in cache organizing is to choose key of the cache entry. This difficulty  we went around by introducing the “scope block” concept. Each configuration description has its own scope, namely: "All", "User Group", "User", "Session". For each scope the key of the cache is created in a special function that allows us to describe the creation logic of a key in one place.

 At the same time such concept as an “authorized user” and “guest” are erode as guest is a user with “0” ID and the  user group ID is “0” too. Each setting can be inherited from the parent and modify the last one. Thus, we can configure each block caching without source code editing, at the same time due to that some elements of the configuration are inherited from the other, we can change the cache logic of the multiple blocks by changing the configuration of the parent element. Each module can contain the xml-file with the configuration of the cache blocks of the module, and cache configurations can be inherited from the configurations described in other files.

 Now lets talk about block fo the cache inside another block. Why do we need it? Let`s suppose that we decided to cache such page element as a website header. It is the same for all users (the scope  is "All") except block which is responsible for the shopping cart information output which is individual  for each visitor (the scope is "User").

When the caching module works, the following happens: if the module determines that the parent block is cached it gives the description in a special format for output instead of child at that it saves html of the child block in extcached and in a local cache of the process. Further, html of the parent block is checked before output for the presence of a specific child blocks descriptions, present at child block html getting from the local cache and extcached attempts. If it is not found block generates html in the usual way. In the case of website header it turns out that the header generates only once for all visitors but blocks with description of the cart are generating individually for each visitor. Cart block reset happens when it changes due to the fact that the cache entry is marked by special tag, so generation of the new html of the cart block happens only when you change the content of the cart. Nesting of the blocks one inside the other is not restricted, all the page can be in the capacity of the root parent. After setting up of the blocks on the test system  we got the following results: page generation speed rate has doubled, the load on the database fell by 75% (on average). That's why I wrote earlier in this article about the horizontal scale. If it is necessary, it does not cause any difficulties to organize HTTP cluster of the servers. In the case of the database need to make a lot of effort to run the database in the cluster. So the effect of this module integration is significant. This is despite the fact that a similar scheme can be used  for models and collections cache. Customers that we asked said - "The pages open quickly,we do really like it."

This system has one more potential for growth. You can use SSI (Server Side Includes) instruction  as  a  child block description at the same time nginx must be the agent of the http-frontend: it can get data from memcached and from the extcahed too as well as processing SSI instructions. In this case, after filling the cache, Magento will generate only the blocks that are not in the cache and reset them for the tag if it is necessary, so as a result  we have  complete refill of «full page cache» without functionality losing. By the way, Magento already has the possibility of fractional initialization when only the modules you need for the inquiry processing for the exact block are initialized that will positively affect on the performance of this scheme. But, in our view, such solutions should be applied to complex, high load projects.