Oak Offline Compaction - Improve performance, avoid memory issues and track progress

Running Offline Tar Compaction is slow on large Oak repositories and causes OutOfMemoryException. There is no way to track progress. Sometimes it is observed that offline compaction increases repository size after completion.

Environment

Adobe Experience Manager 6.x

Oak 1.2.x and 1.4.x

Resolution

Use the following command:

sudo -u crx nohup /usr/java/latest/bin/java -Dtar.memoryMapped=true -Doak.compaction.eagerFlush=true -server -Xmx30g -Dcompaction-progress-log=5000000
-Dlogback.configurationFile=logback-compaction.xml -Dcompress-interval=150000000 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/dumps
-jar oak-run-1.2.16-R1750358.jar compact /path/to/segmentstore >> oak-tar-compact-a0.log 2>>oak-tar-error-a0.log

Note: There are a few parameters above for which paths need to be adjusted.

There are a few new parameters included in oak-run versions  1.2.17 for AEM6.1 and 1.4.4 for AEM6.2 or later:

  • -Dlogback.configurationFile=logback-compaction.xml - This logs the progress of the compaction. You must add the logback-compaction.xml file at the same location as the oak-run jar.

logback-compaction.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
 
  <appender name="STDERR" class="ch.qos.logback.core.FileAppender">
    <file>/mnt/dumps/compaction.log</file>
    <encoder>
      <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
    </encoder>
  </appender>
 
  <logger name="org.apache.jackrabbit.oak.plugins.segment.Compactor" level="DEBUG"/>
 
  <root level="warn">
    <appender-ref ref="STDERR" />
  </root>
</configuration>
  • -Doak.compaction.eagerFlush=true - This is the new flag that enables the compactor to use persisted transactions. It prevents an Out of Memory exception.

Track Progress:

compaction.log:

18:18:34.531 [main] WARN  o.a.j.o.p.segment.file.TarReader - Could not find a valid tar index in [/mnt/crx/author/crx-quickstart/repository/segmentstore/data01624a.tar], recovering...
18:18:36.569 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Eager flush enabled.
18:19:36.910 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 5000000 nodes, 1987504 properties, 57 binaries in 60328 ms.
18:22:06.905 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 10000000 nodes, 4771516 properties, 294 binaries in 149994 ms.
18:23:46.708 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 15000000 nodes, 6505235 properties, 323 binaries in 99803 ms.
18:36:38.014 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 20000000 nodes, 25559366 properties, 430789 binaries in 771305 ms.
18:40:46.736 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 25000000 nodes, 62174573 properties, 442755 binaries in 248722 ms.
18:46:51.053 [main] DEBUG o.a.j.oak.plugins.segment.Compactor - Compacted 30000000 nodes, 96729247 properties, 1040072 binaries in 364317 ms.

 Adobe

Saņemiet palīdzību ātrāk un vienkāršāk

Jauns lietotājs?

Adobe MAX 2024

Adobe MAX
Radošuma konference

14.–16. oktobris Maiami pludmalē un tiešsaistē

Adobe MAX

Radošuma konference

14.–16. oktobris Maiami pludmalē un tiešsaistē

Adobe MAX 2024

Adobe MAX
Radošuma konference

14.–16. oktobris Maiami pludmalē un tiešsaistē

Adobe MAX

Radošuma konference

14.–16. oktobris Maiami pludmalē un tiešsaistē