Last updated on
16 May 2021
Running Offline Tar Compaction is slow on large Oak repositories and causes OutOfMemoryException. There is no way to track progress. Sometimes it is observed that offline compaction increases repository size after completion.
Environment
Adobe Experience Manager 6.x
Oak 1.2.x and 1.4.x
Resolution
Use the following command:
sudo -u crx nohup /usr/java/latest/bin/java -Dtar.memoryMapped=true -Doak.compaction.eagerFlush=true -server -Xmx30g -Dcompaction-progress-log=5000000 -Dlogback.configurationFile=logback-compaction.xml -Dcompress-interval=150000000 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/dumps -jar oak-run-1.2.16-R1750358.jar compact /path/to/segmentstore >> oak-tar-compact-a0.log 2>>oak-tar-error-a0.log
Note: There are a few parameters above for which paths need to be adjusted.
There are a few new parameters included in oak-run versions 1.2.17 for AEM6.1 and 1.4.4 for AEM6.2 or later:
- -Dlogback.configurationFile=logback-compaction.xml - This logs the progress of the compaction. You must add the logback-compaction.xml file at the same location as the oak-run jar.
logback-compaction.xml:
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
<appender name="STDERR" class="ch.qos.logback.core.FileAppender">
<file>/mnt/dumps/compaction.log</file>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.jackrabbit.oak.plugins.segment.Compactor" level="DEBUG"/>
<root level="warn">
<appender-ref ref="STDERR" />
</root>
</configuration>
- -Doak.compaction.eagerFlush=true - This is the new flag that enables the compactor to use persisted transactions. It prevents an Out of Memory exception.
Track Progress:
compaction.log:
18:18:34.531 [main] WARN o.a.j.o.p.segment.file.TarReader - Could not find a valid tar index in [/mnt/crx/author/crx-quickstart/repository/segmentstore/data01624a.tar], recovering... 18:18:36.569 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Eager flush enabled. 18:19:36.910 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 5000000 nodes, 1987504 properties, 57 binaries in 60328 ms. 18:22:06.905 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 10000000 nodes, 4771516 properties, 294 binaries in 149994 ms. 18:23:46.708 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 15000000 nodes, 6505235 properties, 323 binaries in 99803 ms. 18:36:38.014 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 20000000 nodes, 25559366 properties, 430789 binaries in 771305 ms. 18:40:46.736 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 25000000 nodes, 62174573 properties, 442755 binaries in 248722 ms. 18:46:51.053 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 30000000 nodes, 96729247 properties, 1040072 binaries in 364317 ms.
Adobe MAX
The Creativity Conference
Oct 14–16 Miami Beach and online
Adobe MAX
The Creativity Conference
Oct 14–16 Miami Beach and online