AEM restart takes a long time due to sling jobs

Issue

AEM restart takes a long time. The thread dumps taken during the restart show the following threads delaying AEM startup:

"Apache Sling Job Topology Listener Thread" #452 daemon prio=5 os_prio=0 tid=0x00007fd7556ac000 nid=0x5477 pcpu%=25.7  memory%=60.0 -- running for 8401 seconds --   runnable [0x00007fd204297000]
   java.lang.Thread.State: RUNNABLE
 at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
 at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
 at sun.nio.ch.IOUtil.read(IOUtil.java:197)
 at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:740)
 at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:726)
 at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.readEntry(TarWriter.java:200)
 at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:1262)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.readSegment(SegmentTracker.java:242)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:149)
 - locked <0x0000000790d24550> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
 at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
 at org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:142)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:298)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:296)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:219)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:724)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:726)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$ChildNodeCollectorDiff.childNodeChanged(SegmentWriter.java:877)
 at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$ChildNodeCollectorDiff.diff(SegmentWriter.java:892)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:714)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$ChildNodeCollectorDiff.childNodeChanged(SegmentWriter.java:877)
 at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$ChildNodeCollectorDiff.diff(SegmentWriter.java:892)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:714)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:111)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:36)
 at org.apache.jackrabbit.oak.spi.commit.EditorHook.processCommit(EditorHook.java:57)
 at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
 at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:494)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:525)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:581)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:238)
 at org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.merge(ProxyNodeStore.java:43)
 at org.apache.jackrabbit.oak.core.MutableRoot.commit(MutableRoot.java:247)
 at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:347)
 at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.save(SessionDelegate.java:494)
 at org.apache.jackrabbit.oak.jcr.session.SessionImpl$8.performVoid(SessionImpl.java:419)
 at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:274)
 at org.apache.jackrabbit.oak.jcr.session.SessionImpl.save(SessionImpl.java:416)
 at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:497)
 at org.apache.sling.jcr.base.SessionProxyHandler$SessionProxyInvocationHandler.invoke(SessionProxyHandler.java:116)
 at com.sun.proxy.$Proxy7.save(Unknown Source)
 at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.commit(JcrResourceProvider.java:480)
 at org.apache.sling.resourceresolver.impl.providers.stateful.AuthenticatedResourceProvider.commit(AuthenticatedResourceProvider.java:215)
 at org.apache.sling.resourceresolver.impl.helper.ResourceResolverControl.commit(ResourceResolverControl.java:411)
 at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.commit(ResourceResolverImpl.java:1262)
 at org.apache.sling.event.impl.support.ResourceHelper.getOrCreateResourceInternal(ResourceHelper.java:410)
 at org.apache.sling.event.impl.support.ResourceHelper.getOrCreateResource(ResourceHelper.java:328)
 at org.apache.sling.event.impl.support.ResourceHelper.getOrCreateResource(ResourceHelper.java:271)
 at org.apache.sling.event.impl.jobs.tasks.CheckTopologyTask$2.handle(CheckTopologyTask.java:258)
 at org.apache.sling.event.impl.jobs.JobTopicTraverser.traverse(JobTopicTraverser.java:145)
 at org.apache.sling.event.impl.jobs.JobTopicTraverser.traverse(JobTopicTraverser.java:102)
 at org.apache.sling.event.impl.jobs.tasks.CheckTopologyTask.assignJobs(CheckTopologyTask.java:243)
 at org.apache.sling.event.impl.jobs.tasks.CheckTopologyTask.assignUnassignedJobs(CheckTopologyTask.java:207)
 at org.apache.sling.event.impl.jobs.tasks.CheckTopologyTask.fullRun(CheckTopologyTask.java:332)
 at org.apache.sling.event.impl.jobs.config.JobManagerConfiguration.startProcessing(JobManagerConfiguration.java:511)
 at org.apache.sling.event.impl.jobs.config.JobManagerConfiguration.handleTopologyEvent(JobManagerConfiguration.java:585)
 - locked <0x000000074b4236a8> (a java.util.ArrayList)
 at org.apache.sling.event.impl.jobs.config.TopologyHandler.run(TopologyHandler.java:98)
 at java.lang.Thread.run(Thread.java:745)


"FelixStartLevel" #16 daemon prio=5 os_prio=0 tid=0x00007fd7ad954800 nid=0x514d pcpu%=21.2  memory%=60.0 -- running for 8529 seconds --   waiting for monitor entry [0x00007fd78c125000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.sling.event.impl.jobs.config.JobManagerConfiguration.removeListener(JobManagerConfiguration.java:608)
    - waiting to lock <0x000000074b4236a8> (a java.util.ArrayList)
    at org.apache.sling.event.impl.jobs.scheduling.JobSchedulerImpl.deactivate(JobSchedulerImpl.java:114)
    at org.apache.sling.event.impl.jobs.JobManagerImpl.deactivate(JobManagerImpl.java:144)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222)
    at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37)
    at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615)
    at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499)
    at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:295)
    at org.apache.felix.scr.impl.manager.SingleComponentManager.disposeImplementationObject(SingleComponentManager.java:342)
    at org.apache.felix.scr.impl.manager.SingleComponentManager.deleteComponent(SingleComponentManager.java:157)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:783)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:757)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:1014)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:915)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136)
    at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175)
    at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:883)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:857)
    at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:140)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:925)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:774)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:757)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:1014)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:915)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136)
    at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175)
    at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryActivator.unregisterFactory(ResourceResolverFactoryActivator.java:646)
    at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryActivator.unregisterFactory(ResourceResolverFactoryActivator.java:637)
    at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryActivator.access$300(ResourceResolverFactoryActivator.java:83)
    at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryActivator$1.providerRemoved(ResourceResolverFactoryActivator.java:580)
    at org.apache.sling.resourceresolver.impl.providers.ResourceProviderTracker.unregister(ResourceProviderTracker.java:254)
    at org.apache.sling.resourceresolver.impl.providers.ResourceProviderTracker.access$100(ResourceProviderTracker.java:59)
    at org.apache.sling.resourceresolver.impl.providers.ResourceProviderTracker$1.removedService(ResourceProviderTracker.java:114)
    at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:967)
    at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:870)
    at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341)
    at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:909)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.sling.resourceresolver.impl.legacy.LegacyResourceProviderWhiteboard.unbindResourceProviderFactory(LegacyResourceProviderWhiteboard.java:165)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222)
    at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37)
    at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615)
    at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499)
    at org.apache.felix.scr.impl.helper.BindMethod.invoke(BindMethod.java:41)
    at org.apache.felix.scr.impl.manager.DependencyManager.invokeUnbindMethod(DependencyManager.java:1774)
    at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeUnbindMethod(SingleComponentManager.java:393)
    at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.removedService(DependencyManager.java:364)
    at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.removedService(DependencyManager.java:295)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136)
    at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175)
    at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:883)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:857)
    at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:140)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:925)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:774)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:757)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:1014)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:915)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136)
    at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175)
    at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:883)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:857)
    at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:140)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:925)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:774)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:757)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.dispose(AbstractComponentManager.java:550)
    at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.disposeComponents(ConfigurableComponentHolder.java:725)
    at org.apache.felix.scr.impl.BundleComponentActivator.dispose(BundleComponentActivator.java:530)
    at org.apache.felix.scr.impl.Activator.disposeComponents(Activator.java:414)
    at org.apache.felix.scr.impl.Activator.access$300(Activator.java:53)
    at org.apache.felix.scr.impl.Activator$ScrExtension.destroy(Activator.java:273)
    at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:290)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:312)
    at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
    at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:916)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:835)
    at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:517)
    at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4541)
    at org.apache.felix.framework.Felix.stopBundle(Felix.java:2600)
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1389)
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x000000074c4e7a10> (a java.util.concurrent.locks.ReentrantLock$FairSync)

Environment

AEM 6.2

Cause

There were a massive amount of sling jobs (replication, workflow, etc) pending in queue when AEM was shut down.  During restart of AEM, the jobs get reassigned to their queues. This operation can take a long time to complete if there are thousands of jobs in queue.

I. Clear out excess sling jobs

After AEM finally starts up, try clearing the replication queues and purging workflows.  Make sure that minimal users are actively using the system during this time as this operation would make the system slow.

Check the replication queues

  1. Go to http://aem-host:port/etc/replication/agent.author.html
  2. Browse to any agents with a large amount of pending items
  3. Clear the queue if it is not processing       

Check workflow instances:

  1. Go to http://aem-host:port/workflow UI
  2. Check the Instances tab to see if there are a lot of piled up workflows in running state
  3. Go to http://aem-host:port/system/console/jmx/com.adobe.granite.workflow%3Atype%3DMaintenance
  4. Use the purgeActive JMX method to clear out the active workflows (if it is safe to do so)

Purge job nodes:

If neither of the above options help with startup and shutdown times, then delete the Sling jobs manually:

  1. Go to http://aem-host:port/crx/explorer/index.jsp and log in as admin
  2. Open Content Explorer
  3. Browse to /var/eventing/jobs
  4. See which jobs are piled up under /var/eventing/jobs/assigned/{sling id}
  5. Right click a sub-folder there that you don't need to preserve and select Delete Recursively
  6. Enable Disable Preliminary Scan option
  7. Click Delete

WARNING: During the deletion AEM may be very slow, it is recommended that you take an outage window of a few hours to perform this deletion.

II. Install Latest AEM Cumulative Fix Pack

To lessen the impact of sling jobs loading on startup, there is a fix included in AEM6.2 CFP1 that makes these jobs load in a background thread.  See the 6.2 hotfix page to obtain this fix.

 Adobe

Saņemiet palīdzību ātrāk un vienkāršāk

Jauns lietotājs?

Adobe MAX 2024

Adobe MAX
Radošuma konference

14.–16. oktobris Maiami pludmalē un tiešsaistē

Adobe MAX

Radošuma konference

14.–16. oktobris Maiami pludmalē un tiešsaistē

Adobe MAX 2024

Adobe MAX
Radošuma konference

14.–16. oktobris Maiami pludmalē un tiešsaistē

Adobe MAX

Radošuma konference

14.–16. oktobris Maiami pludmalē un tiešsaistē