Note:

You are reading the Adobe Experience Manager 6.x version of performance tuning tips. This article is also available for version 5.x.

Response time is slow for editing and visitor requests

Response time is poor when authors edit content, or websites respond slowly to visitor requests.

Cause

The following factors influence performance problems in AEM:

  • Improper design
  • Application code
  • Bad disk I/O configuration
  • Network bandwidth and latency
  • AEM installed on some select windows 2008 and 2012 version where memory management is an issue
  • Modifying out-of-the-box configurations as described below can help improve performance in AEM.

Preventing performance issues

Here are some steps you can take to ensure that you find and fix performance issues before they have an impact on your users:

  1. Implement and execute load tests that simulate realistic scenarios in both author and publish instances. Researching and defining expected load is a crucial step in this process. This step helps you demonstrate whether the AEM application, architecture, and AEM installation will perform well once it is live in a production environment. Results of this exercise help determine whether there is a misconfiguration, application issue, sizing, hardware problem or other issue affecting the system performance.

    1. In addition to load testing, stress testing helps to define the maximum load the system can handle. This test can help you prepare for traffic spikes. More information on performance testing can be found here.
  2. If you are using Windows server, then review this article.

  3. If you are planning to load large amounts of Assets (images, videos, and so on) into AEM then review this article to avoid memory issues.

Note:

This article applies to AEM 6.0, for AEM/CQ 5.6.1 (or earlier version) performance tuning tips, see http://helpx.adobe.com/experience-manager/kb/performancetuningtips.html.

Enabling transient workflows

To optimize high ingestion loads, Adobe suggests switching the DAM Update and XMP Metadata Writeback workflow to a transient workflow. As the name implies, runtime data related to the intermediate worksteps in transient workflows are not persisted in the JCR when they run (the output renditions are persisted of course). It causes a 10% reduction in the workflow processing time and significantly reduces repository growth. No more CRUD workflows are required to purge. In addition, it reduces the number of TAR files to compact. If your business dictates persisting/archiving workflow runtime data for audit purposes, do not enable this feature.

Note:

If you intend to enable Transient Workflows, be aware of the following:

  • Transient workflows do not generate workflow events. As such, features and customers that depend on events should not enable Transient Workflows.
  • Transient workflows do not support workflow offloading, which depends on workflow events.
  • Transient workflows do not support legacy CQ workflow events. The reason is because the compatibility layer in CQEventDispatcher does not work.
  • Transient workflows do not support workflow email notification. The reason is because EmailNotificationService is based on CQEventDispatcher, which does not work with transient workflows.
  • Transient workflows do not support workflow statistics. The reason is because CQWorkflowStatisticsService is based on CQEventDispatcher, which does not work with transient workflows.

In cases where you cannot use Transient Workflow, you must run workflow purging regularly to delete archived DAM Update Asset workflows. Doing so helps maintain system performance. To configure workflow purging, add a new Adobe Granite Workflow Purge Configuration by way of the OSGi console. Configure and schedule it as part of your weekly maintenance window.

Note:

Switching the DAM Update workflow to a transient workflow disables the asset status updates in the asset browser.

  1. Navigate to /miscadmin in the AEM instance to be configured (for example, http://<Server>:<Port>/miscadmin).

  2. From the navigation tree, expand Tools > Workflow > Models > dam.

  3. Double-click DAM Update Asset.

  4. On the floating tools panel, switch to the Page tab, and then click Page Properties.

  5. Select the Transient Workflow check box, and click OK.

Tuning Sling Job Queues

Bulk upload of large assets is typically a very resource intensive process. By default, the number of concurrent threads per job queue is equal to the number of CPU cores. As such, this value setting may cause an overall performance impact and high Java heap consumption.

Adobe recommends that you do not exceed 50% of the CPU cores. To adjust this value, go to the following:

http://<host>:<port>/system/console/configMgr/org.apache.sling.event.jobs.QueueConfiguration

Set queue.maxparallel to a value that represents 50% of the CPU cores of the server that hosts your AEM instance. For example, for 8 CPU cores, set the value to 4.  

Tuning your Oak Repository

First make sure that you have the latest Oak version installed for your AEM 6 instance. Check the recommended hot fixes page mentioned above.

Oak Query Engine/Index optimizations

  1. Install recommended indexes (only for AEM 6.0)

    With the recent improvements in Oak and in QueryBuilder, it's necessary to define new Oak indexes to ensure that your system is at optimal performance. For more information about Oak indexing, see Apache Jackrabbit Oak-related page.

    Below are some sample packages that contain such new index definitions. There is also a consolidated AEM 6.0 index definition package (NPR-6340) available upon request to AEM Customer Care.

    * damLuceneProperty.zip 
    Recommended index definition for better DAM classic user interface search performances (updated on 13.03.2015)

    * productsIndex.zip 
    Recommended index definitions for better Content Finder products search

    For AEM 6.1, the 6.0 consolidated index definitions are available out of the box, except for rep:Token. As such, the following is required:

    • Add the value rep:Token to the declaringNodeTypes property of the node /oak:index/nodetype.
    • For the same node, set the value of the reindex property to true.
    • Click Save to reindex.
  2. Create custom oak indexes for all frequently used search queries.

    • For information on how to analyze slow queries see this article.
    • Create the custom indexes under the oak:index node for all search properties that you want to search with by following this article.
    • For each custom Lucene-based index, try to set includedPaths (String[]) setting to restrict the index to only apply to certain content paths. Then restrict applicable searches to those paths that are included by the index.
  3. JVM parameters

    Add these JVM parameters in the AEM start script to prevent expansive queries from overloading the systems.

    • -Doak.queryLimitInMemory=500000 (see also Oak documentation)
    • -Doak.queryLimitReads=100000 (see also Oak documentation)
    • -Dupdate.limit=250000 (only for DocumentNodeStore, eg. MongoMK, RDBMK)

    The following option might as well improve performance, but changes the meaning of the result size call. Specially, only query restrictions that are part of the used index are considered when calculating the size. Additionally, ACLs are not applied to the results, so nodes which are not visible to the current session will still be included in the count returned. As such,  the count returned can be higher than the actual number of results and the accurate count can only be determined by iterating through the results:

  4. Lucene index configuration

    Open /system/console/configMgr/org.apache.jackrabbit.oak.plugins.index.lucene.LuceneIndexProviderServiceand

    • enable CopyOnRead (enabled by default since AEM 6.2)
    • enable CopyOnWrite (enabled by default since AEM 6.2)
    • enable Prefetch Index Files (enabled by default since AEM 6.2)

    See http://jackrabbit.apache.org/oak/docs/query/lucene.html for more information about the available parameters

    Because some paths do not need to be indexed, you can do the following:

    In CRXDE Lite, go to /oak:index/lucene, set a multivalue string property (String[])named excludedPaths with these values /var, /etc/workflow/instances, /etc/replication.

  5. Data Store

    If you use AEM Assets or have an AEM application that uses binary files extensively, Adobe recommends that you use an external datastore. Using an external datastore helps ensure maximum performance.  See documentation for detailed instructions

    When using a FileDataStore, tune cacheSizeInMB to a percentage of your available heap. A conservative value is 2% of the max heap.  For example, for an 8 GB heap:

    maxCachedBinarySize=1048576
    cacheSizeInMB=164

    Note that maxCachedBinarySize is set to 1 MB (1048576). As such, it only caches files that are a maximum of 1 MB.  Tuning this setting to a smaller value may also make sense.

    When dealing with a large number of binaries, you want to maximize performance. Therefore, Adobe recommends that an external datastore is used instead of the default node stores. In addition, Adobe recommends you tune the following parameters:

    • maxCachedBinarySize=10485760
    • cacheSizeInMB=4096

Note:

The cacheSizeInMB setting can cause the Java process to run out of memory if it is set too high. For example, if you have the max heap size set to 8 GB (-Xmx8g) and you expect AEM and your application to utilize a combined heap of 4 GB, then it makes sense to set cacheSizeInMB to 82 instead of 164. In the range of 2-10% of the max heap is a safe configuration. However, Adobe recommends that you validate setting changes with load testing while also monitoring memory utilization. 

Mongo storage tuning

  1. MongoBlobStore cache size:  The blobstore is used to store and read large binary objects. Internally, the store with cache is implemented which splits the binaries in relatively small blocks (data or hash code or indirect hash), so that each block fits in memory.  In a default setup, the MongoBlobStore uses a fixed cache size of 16MB.  For deployments where more RAM is available and blob storage is frequently accessed (for example, when Lucene index is large), increase the cache size. This cache size only applies when you use MongoBlobStore (default), not when using an external blobstore.

    • You can configure the cache size (MB) by way of the blobCacheSize setting on DocumentNodeStoreService.
      For example:
        blobCacheSize=1024
  2. Document cache size: To optimize the performance of reading nodes from MongoDB you need to tune the caches sizes of DocumentNodeStore.  The default size of the cache is set to 256 MB, which is distributed among various caches used in DocumentNodeStore. See http://jackrabbit.apache.org/oak/docs/nodestore/documentmk.html#cache

    • You can configure the cache size (MB) by way of the cache setting on DocumentNodeStoreService. For example, cache=2048.
    • Set all of the following cache configurations in crx-quickstart/install/org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreService.configand then load test with various values to see what the optimal configuration is for your environment. Note that remaining cache percent is given to the document cache:
      • cache=2048
      • nodeCachePercentage=35
      • childrenCachePercentage=20
      • diffCachePercentage=30
      • docChildrenCachePercentage=10
    • With the above configuration, the percentages total 95%.  The remaining 5% of the cache is given to documentCache.
        documentCache = cache - nodeCache - childrenCache - diffCache - docChildrenCache
    • While distributing the cache percentages, ensure that cache left for documentCache is not very large. That is, keep it to ~500 MB max or less; a large documentCache can lead to increase in the time taken to perform cache invalidation.
  3. Cache settings in AEM 6.2 with Oak 1.4.x:

    • In AEM 6.2, changes were made to the way these cache settings work. In AEM 6.2 with Oak 1.4, there is a new cache: prevDocCache.  You can configure this cache using the setting prevDocCachePercentage. Default is 4.
    • The documentCache uses the remaining cache MB (cache setting minus the size of all other caches):
        documentCache = cache - nodeCache - childrenCache - diffCache - docChildrenCache - prevDocCache
  4. Compound index (AEM 6.0 with Oak 1.0.x and AEM 6.1 with Oak 1.2.x only)

    • Create a compound index in the background. Do this during off hours, if possible. It is best to stop AEM instances while this is running so that it can complete faster. Use the following command:
        nohup mongo localhost:27017/aem-author --eval "db.nodes.createIndex( {_modified:1, _id:1}, { background: true } )" &
    • Add this JVM parameter to AEM's start script:
        -Doak.mongo.disableIndexHint=true  
  5. Implement the MongoDB production checklist
    http://docs.mongodb.org/manual/administration/production-checklist/ - according to Mongo DB support, many of the items there have a large impact on performance. For any questions, contact MongoDB Support directly.

  6. Read performance: Add this query string parameter to your Mongo DB URL on each AEM node:  ?readPreference=secondaryPreferred
    This parameter tells the system to do reads from the secondary which gives some added read performance.

  7. Increase thread pool for oak-observation: open /system/console/configMgr/org.apache.sling.commons.threads.impl.DefaultThreadPool.factory
    Set the name to oak-observation and set the min and max pool size to 20.

  8. Increase observation queue length: Create a file named com.adobe.granite.repository.impl.SlingRepositoryManager.cfg containing parameter oak.observation.queue‐length=50000 . Place it under the /crx-­‐quickstart/install folder.

  9. Avoid long running queries: Set the system property in the JVM parameters :  -Doak.mongo.maxQueryTimeMS=60000 to avoid queries running longer than 1 minute.

Tar storage

Micro kernels do not call memory-mapped files directly. However, JDK internally uses memory-mapped files for efficient reading. On certain Windows 64-bit operating system could fail to clean up memory-mapped files and consume all the native OS memory. Make sure to install the performance-related patches/hotfix from microsoft (see KB 2731284) and Oracle.

  • An alternative option is to disable the memory map mode by adding tarmk.mode=32 in SegmentNodeStoreService.config until the operating system issue is resolved. The downside of disabling makes I/O intensive. Make sure to raise the I/O Page Lock Limit.

TarMK online compaction

On TarMK based AEM 6 instances, a good understanding and perfectly configured TarMK online compaction is essential to achieve a high-performing and healthy AEM environment.

Adobe recommends that you use offline compaction, by way of the oak-run tool as documented at the following: http://docs.adobe.com/docs/en/aem/6-0/deploy/upgrade/microkernels-in-aem-6-0.html#Maintaining the Repository

The online compaction--also known as RevisionGC--is disabled by default. See this page for more details about the online compaction.

What next?

For further help concerning your AEM performance issues:

  1. Join the discussion at the forums AEM 6 performance tuning tips and tricks  OR

  2. Get official support

    1. Collect the following data:
      1. Profiler output
      2. Thread Dumps
      3. All the log files
    2. Contact AEM Customer Care

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License  Twitter™ and Facebook posts are not covered under the terms of Creative Commons.

Legal Notices   |   Online Privacy Policy