Version Purge is running and response times from AEM are slow. Thread dumps [6] show that version purge is running and blocking other threads.
Environment
AEM 6.0, 6.1 and 6.2
Cause
Version Purge maintenance task runs past its scheduled window of time if it takes too long. The cause of this issue is likely due to bulk version creations in AEM that caused Version Purging to take too long.
Actions that cause versions to be created are (non-exhaustive list):
- Page creation, versioning, or activation
- Asset upload, modification, or activation
- Workflow model update and save
Solution
To fix the issue, follow the steps below:
1. Install latest Cumulative Fix Pack (AEM6.2 only)
If you are on AEM6.2 then install the AEM6.2 Cumulative Fix Pack 2 or later. After installing it, when version purge maintenance runs it would put less load on the AEM server.
If you are unable to apply the Cumulative Fix Pack then Contact AEM Customer Care and request AEM6.2 hotfix 15186.
2. Modify the maintenance window
Move the version purge maintenance schedule earlier to avoid it from running during business hours. See this documentation for how to modify maintenance schedules
3. Investigate what application code is causing the versions to be created and optimize it.
If versions are being created due to activations then modify the code so it suppresses version creation.
Use the ReplicationOptions [1] class, call ReplicationOptions.setSuppressVersions(true) [2] and pass the object as a parameter when calling Replicator.replicate [3].
[1] https://docs.adobe.com/docs/en/aem/6-2/develop/ref/javadoc/com/day/cq/replication/ReplicationOptions.html
[2] https://docs.adobe.com/docs/en/aem/6-2/develop/ref/javadoc/com/day/cq/replication/ReplicationOptions.html#setSuppressVersions(boolean)
[3] https://docs.adobe.com/docs/en/aem/6-2/develop/ref/javadoc/com/day/cq/replication/Replicator.html
3. Disable Versioning on replication agents
a. Go to http://aem-host:port/etc/replication/agents.author.html
b. Open each replication agent configuration
c. Select the "Triggers" tab
d. Enable "No Versioning" (see [1] below)
e. Save
Or do this for all agents in one configurations:
a. Go to http://aem-host:port/system/console/configMgr/com.day.cq.wcm.core.impl.VersionManagerImpl
b. Disable "Create Version on Activation"
c. Save
4. Disable Versioning on Companion app and WebDav uploads
a. Go to http://aem-host:port/system/console/configMgr/com.adobe.cq.dam.webdav.impl.io.AssetIOHandler
b. Disable "create version" (see [2] below)
c. Save
5. Enable auto-purging on version creation
The Version Manager has a feature where it can purge old versions as you add new ones to pages and assets. Follow these steps to enable this feature.
a. Go to http://aem-host:port/system/console/configMgr/com.day.cq.wcm.core.impl.VersionManagerImpl
b. Enable "Enable Purging"
c. Save
6. Run Offline Tar Compaction
Make sure you are running offline tar compaction on a regular basis as a lack of compaction maintenance can affect system performance overall.
[4] Replication Agent configuration for disabling auto-versioning
[5] OSGi Configuration to disable version creation on WebDAV / ("Companion") Desktop App uploads
?$png$&jpegSize=100&wid=574 "rtaImage (1)")
[6] Thread showing long running version purge:
pool-20-thread-17 - priority:5 - threadId:0x00007fbc5c00c800 - nativeId:0x341b - state:RUNNABLE stackTrace: java.lang.Thread.State: RUNNABLE at java.io.RandomAccessFile.readBytes(Native Method) at java.io.RandomAccessFile.read(RandomAccessFile.java:377) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:436) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416) at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.readEntry(TarWriter.java:186) - locked <0x0000000679fe3910> (a org.apache.jackrabbit.oak.plugins.segment.file.TarWriter) at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:847) - locked <0x00000005c0c49560> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore) at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:182) at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:108) - locked <0x00000004dcafb448> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId) at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82) at org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:142) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:534) - locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:532) - locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:532) - locked <0x00000005c0c57640> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:784) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1172) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1174) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1155) at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1146) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1155) at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399) at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1146) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:85) at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:214) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:81) at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.remove(MemoryNodeBuilder.java:355) at org.apache.jackrabbit.oak.plugins.index.property.strategy.OrderedContentMirrorStoreStrategy.prune(OrderedContentMirrorStoreStrategy.java:242) at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.remove(ContentMirrorStoreStrategy.java:118) at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.update(ContentMirrorStoreStrategy.java:90) at org.apache.jackrabbit.oak.plugins.index.property.PropertyIndexEditor.leave(PropertyIndexEditor.java:274) at org.apache.jackrabbit.oak.plugins.index.property.OrderedPropertyIndexEditor.leave(OrderedPropertyIndexEditor.java:176) at org.apache.jackrabbit.oak.spi.commit.CompositeEditor.leave(CompositeEditor.java:74) at org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:63) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:176) at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171) at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171) at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171) at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171) at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171) at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171) at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstBaseState(EmptyNodeState.java:143) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeDeleted(EditorDiff.java:171) at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeDeleted(MapRecord.java:449) at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:481) at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:436) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:531) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148) at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148) at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148) at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148) at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148) at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:418) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:583) at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52) at org.apache.jackrabbit.oak.spi.commit.EditorHook.processCommit(EditorHook.java:54) at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61) at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:458) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:489) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:545) at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:208) at org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.merge(ProxyNodeStore.java:43) at org.apache.jackrabbit.oak.core.MutableRoot.commit(MutableRoot.java:247) at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:339) at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:364) at org.apache.jackrabbit.oak.jcr.version.ReadWriteVersionManager.removeVersion(ReadWriteVersionManager.java:239) at org.apache.jackrabbit.oak.jcr.delegate.VersionManagerDelegate.removeVersion(VersionManagerDelegate.java:226) at org.apache.jackrabbit.oak.jcr.delegate.VersionHistoryDelegate.removeVersion(VersionHistoryDelegate.java:209) at org.apache.jackrabbit.oak.jcr.version.VersionHistoryImpl$11.performVoid(VersionHistoryImpl.java:240) at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:266) at org.apache.jackrabbit.oak.jcr.version.VersionHistoryImpl.removeVersion(VersionHistoryImpl.java:236) at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:380) at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:346) at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:338) at com.day.cq.wcm.core.impl.VersionManagerImpl.purgeVersions(VersionManagerImpl.java:569) at com.day.cq.wcm.core.impl.VersionManagerImpl.onEvent(VersionManagerImpl.java:329) at org.apache.jackrabbit.commons.observation.ListenerTracker$1.onEvent(ListenerTracker.java:164) at org.apache.jackrabbit.oak.jcr.observation.ChangeProcessor.contentChanged(ChangeProcessor.java:316) at org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:131) at org.apache.jackrabbit.oak.spi.commit.BackgroundObserver$1$1.call(BackgroundObserver.java:125) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x00000005c23fd400> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) - <0x00000005c24ad9d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) - <0x00000005c6039780> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Adobe MAX
The Creativity Conference
Oct 14–16 Miami Beach and online
Adobe MAX
The Creativity Conference
Oct 14–16 Miami Beach and online