Observed SegmentNotFound Exceptions in the logs.


Observed SegmentNotFound Exceptions in the logs. For example:

*ERROR* [FelixStartLevel] org.apache.sling.event [org.apache.sling.event.impl.jobs.queues.QueueManager(1431)] The activate method has thrown an exception (org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment da5bcb95-d00a-4c04-a9d9-0f10f2b14e5e not found)

*ERROR* [pool-6-thread-3] org.apache.sling.commons.scheduler.impl.QuartzScheduler Exception during job execution of org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@1dc173f9 : Segment e669f30b-e886-4b7a-b161-56432601ec6b not found
org.apache.jackrabbit.oak.plugins.segment.SegmentNotFoundException: Segment e669f30b-e886-4b7a-b161-56432601ec6b not found


AEM 6.*


Due to some older issues in Oak or some inconsistencies in the repository, a segment can go missing and repository might be inconsistent.


Run consistency check on the repository and find the last good revision [healthy state] and revert back to it. Please do the following:

  1. Download a version of oak-run that matches your oak core version from https://mvnrepository.com/artifact/org.apache.jackrabbit/oak-run

  2. To revert a corrupt segment store to its latest good state change into CQ's working directory (the one containing the crx-quickstartfolder) and backup all files in ./crx-quickstart/repository/segmentstore/.

  3. Run the consistency check,
    java -Xmx6000m -jar oak-run-*.jar check --bin=-1 /path/to/crx-quickstart/repository/segmentstore

    This searches backwards through the revisions until it finds a consistent one:
    Look for message like below:
    [main] INFO o.a.j.o.p.s.f.t.ConsistencyChecker - Found latest good revision afdb922d-ba53-4a1b-aa1b-1cb044b535cf:234880

  4. Revert the repository to this revision by editing ./crx-quickstart/repository/segmentstore/journal.log and deleting all lines after the line containing the latest good revision.

  5. Remove all ./crx-quickstart/repository/segmentstore/*.bak files.

  6. Run checkpoint clean-up to remove orphaned checkpoints:

    java -Xmx6000m -jar oak-run-*.jar checkpoints /path/to/crx-quickstart/repository/segmentstore rm-unreferenced

  7. Finally compact the repository:

    java -Xmx6000m -jar oak-run-*.jar compact /path/to/crx-quickstart/repository/segmentstore/