AEM Version Purge Maintenance taxes system resources and runs past maintenance window

Version Purge is running and response times from AEM are slow. Thread dumps [6] show that version purge is running and blocking other threads.

Environment

AEM 6.0, 6.1 and 6.2

Cause

Version Purge maintenance task runs past its scheduled window of time if it takes too long.  The cause of this issue is likely due to bulk version creations in AEM that caused Version Purging to take too long.

Actions that cause versions to be created are (non-exhaustive list):

  1. Page creation, versioning, or activation
  2. Asset upload, modification, or activation
  3. Workflow model update and save

Solution

To fix the issue, follow the steps below:

1. Install latest Cumulative Fix Pack (AEM6.2 only)

If you are on AEM6.2 then install the AEM6.2 Cumulative Fix Pack 2 or later.  After installing it, when version purge maintenance runs it would put less load on the AEM server.

If you are unable to apply the Cumulative Fix Pack then Contact AEM Customer Care and request AEM6.2 hotfix 15186.

2. Modify the maintenance window

Move the version purge maintenance schedule earlier to avoid it from running during business hours.  See this documentation for how to modify maintenance schedules

3. Investigate what application code is causing the versions to be created and optimize it.

If versions are being created due to activations then modify the code so it suppresses version creation.
Use the ReplicationOptions [1] class, call ReplicationOptions.setSuppressVersions(true) [2]  and pass the object as a parameter when calling Replicator.replicate [3]. 

[1] https://docs.adobe.com/docs/en/aem/6-2/develop/ref/javadoc/com/day/cq/replication/ReplicationOptions.html
[2] https://docs.adobe.com/docs/en/aem/6-2/develop/ref/javadoc/com/day/cq/replication/ReplicationOptions.html#setSuppressVersions(boolean)
[3] https://docs.adobe.com/docs/en/aem/6-2/develop/ref/javadoc/com/day/cq/replication/Replicator.html

3. Disable Versioning on replication agents

a. Go to http://aem-host:port/etc/replication/agents.author.html
b. Open each replication agent configuration
c. Select the "Triggers" tab
d. Enable "No Versioning" (see [1] below)
e. Save

Or do this for all agents in one configurations:
a. Go to http://aem-host:port/system/console/configMgr/com.day.cq.wcm.core.impl.VersionManagerImpl
b. Disable "Create Version on Activation"
c. Save

4. Disable Versioning on Companion app and WebDav uploads

a. Go to http://aem-host:port/system/console/configMgr/com.adobe.cq.dam.webdav.impl.io.AssetIOHandler
b. Disable "create version" (see [2] below)
c. Save

5. Enable auto-purging on version creation

The Version Manager has a feature where it can purge old versions as you add new ones to pages and assets.  Follow these steps to enable this feature.
a. Go to http://aem-host:port/system/console/configMgr/com.day.cq.wcm.core.impl.VersionManagerImpl
b. Enable "Enable Purging"
​c. Save

6. Run Offline Tar Compaction

Make sure you are running offline tar compaction on a regular basis as a lack of compaction maintenance can affect system performance overall.

 

[4] Replication Agent configuration for disabling auto-versioning

[5] OSGi Configuration to disable version creation on WebDAV / ("Companion") Desktop App uploads

[6] Thread showing long running version purge:

pool-20-thread-17 - priority:5 - threadId:0x00007fbc5c00c800 - nativeId:0x341b - state:RUNNABLE
stackTrace:
java.lang.Thread.State: RUNNABLE
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:377)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:436)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.readEntry(TarWriter.java:186)
- locked <0x0000000679fe3910> (a org.apache.jackrabbit.oak.plugins.segment.file.TarWriter)
at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:847)
- locked <0x00000005c0c49560> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:182)
at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:108)
- locked <0x00000004dcafb448> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:142)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:534)
- locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:532)
- locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:532)
- locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:784)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1172)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1174)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1155)
at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1146)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1155)
at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1146)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:85)
at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:214)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:81)
at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.remove(MemoryNodeBuilder.java:355)
at org.apache.jackrabbit.oak.plugins.index.property.strategy.OrderedContentMirrorStoreStrategy.prune(OrderedContentMirrorStoreStrategy.java:242)
at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.remove(ContentMirrorStoreStrategy.java:118)
at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.update(ContentMirrorStoreStrategy.java:90)
at org.apache.jackrabbit.oak.plugins.index.property.PropertyIndexEditor.leave(PropertyIndexEditor.java:274)
at org.apache.jackrabbit.oak.plugins.index.property.OrderedPropertyIndexEditor.leave(OrderedPropertyIndexEditor.java:176)
at org.apache.jackrabbit.oak.spi.commit.CompositeEditor.leave(CompositeEditor.java:74)
at org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:63)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:176)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeDeleted(MapRecord.java:449)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:481)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:531)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583)
at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
at org.apache.jackrabbit.oak.spi.commit.EditorHook.processCommit(EditorHook.java:54)
at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:458)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:489)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:545)
at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:208)
at org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.merge(ProxyNodeStore.java:43)
at org.apache.jackrabbit.oak.core.MutableRoot.commit(MutableRoot.java:247)
at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:339)
at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:364)
at org.apache.jackrabbit.oak.jcr.version.ReadWriteVersionManager.removeVersion(ReadWriteVersionManager.java:239)
at org.apache.jackrabbit.oak.jcr.delegate.VersionManagerDelegate.removeVersion(VersionManagerDelegate.java:226)
at org.apache.jackrabbit.oak.jcr.delegate.VersionHistoryDelegate.removeVersion(VersionHistoryDelegate.java:209)
at org.apache.jackrabbit.oak.jcr.version.VersionHistoryImpl$11.performVoid(VersionHistoryImpl.java:240)
at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:266)
at org.apache.jackrabbit.oak.jcr.version.VersionHistoryImpl.removeVersion(VersionHistoryImpl.java:236)
at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:380)
at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:346)
at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:338)
at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:569)
at com.day.cq.wcm.core.impl.VersionManagerImpl.onEvent(VersionManagerImpl.java:329)
at org.apache.jackrabbit.commons.observation.ListenerTracker$1.onEvent(ListenerTracker.java:164)
at org.apache.jackrabbit.oak.jcr.observation.ChangeProcessor.contentChanged(ChangeProcessor.java:316)
at org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:131)
at org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:125)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000005c23fd400> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000005c24ad9d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
- <0x00000005c6039780> (a java.util.concurrent.ThreadPoolExecutor$Worker)

 Adobe

Get help faster and easier

New user?