AEM restart takes a long time due to sling jobs

ค้นหา

อัปเดตครั้งล่าสุดเมื่อ 21 ธ.ค. 2021

Issue

AEM restart takes a long time. The thread dumps taken during the restart show the following threads delaying AEM startup:

"Apache Sling Job Topology Listener Thread" #452 daemon prio=5 os_prio=0 tid=0x00007fd7556ac000 nid=0x5477 pcpu%=25.7  memory%=60.0 -- running for 8401 seconds --   runnable [0x00007fd204297000]
   java.lang.Thread.State: RUNNABLE
 at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
 at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
 at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
 at sun.nio.ch.IOUtil.read(IOUtil.java:197)
 at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:740)
 at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:726)
 at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.readEntry(TarWriter.java:200)
 at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:1262)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.readSegment(SegmentTracker.java:242)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:149)
 - locked <0x0000000790d24550> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
 at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
 at org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:142)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:298)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:296)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:219)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:724)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:726)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$ChildNodeCollectorDiff.childNodeChanged(SegmentWriter.java:877)
 at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$ChildNodeCollectorDiff.diff(SegmentWriter.java:892)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:714)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$ChildNodeCollectorDiff.childNodeChanged(SegmentWriter.java:877)
 at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$ChildNodeCollectorDiff.diff(SegmentWriter.java:892)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:714)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:111)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:36)
 at org.apache.jackrabbit.oak.spi.commit.EditorHook.processCommit(EditorHook.java:57)
 at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
 at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:61)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:494)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:525)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:581)
 at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:238)
 at org.apache.jackrabbit.oak.spi.state.ProxyNodeStore.merge(ProxyNodeStore.java:43)
 at org.apache.jackrabbit.oak.core.MutableRoot.commit(MutableRoot.java:247)
 at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:347)
 at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.save(SessionDelegate.java:494)
 at org.apache.jackrabbit.oak.jcr.session.SessionImpl$8.performVoid(SessionImpl.java:419)
 at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:274)
 at org.apache.jackrabbit.oak.jcr.session.SessionImpl.save(SessionImpl.java:416)
 at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:497)
 at org.apache.sling.jcr.base.SessionProxyHandler$SessionProxyInvocationHandler.invoke(SessionProxyHandler.java:116)
 at com.sun.proxy.$Proxy7.save(Unknown Source)
 at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.commit(JcrResourceProvider.java:480)
 at org.apache.sling.resourceresolver.impl.providers.stateful.AuthenticatedResourceProvider.commit(AuthenticatedResourceProvider.java:215)
 at org.apache.sling.resourceresolver.impl.helper.ResourceResolverControl.commit(ResourceResolverControl.java:411)
 at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.commit(ResourceResolverImpl.java:1262)
 at org.apache.sling.event.impl.support.ResourceHelper.getOrCreateResourceInternal(ResourceHelper.java:410)
 at org.apache.sling.event.impl.support.ResourceHelper.getOrCreateResource(ResourceHelper.java:328)
 at org.apache.sling.event.impl.support.ResourceHelper.getOrCreateResource(ResourceHelper.java:271)
 at org.apache.sling.event.impl.jobs.tasks.CheckTopologyTask$2.handle(CheckTopologyTask.java:258)
 at org.apache.sling.event.impl.jobs.JobTopicTraverser.traverse(JobTopicTraverser.java:145)
 at org.apache.sling.event.impl.jobs.JobTopicTraverser.traverse(JobTopicTraverser.java:102)
 at org.apache.sling.event.impl.jobs.tasks.CheckTopologyTask.assignJobs(CheckTopologyTask.java:243)
 at org.apache.sling.event.impl.jobs.tasks.CheckTopologyTask.assignUnassignedJobs(CheckTopologyTask.java:207)
 at org.apache.sling.event.impl.jobs.tasks.CheckTopologyTask.fullRun(CheckTopologyTask.java:332)
 at org.apache.sling.event.impl.jobs.config.JobManagerConfiguration.startProcessing(JobManagerConfiguration.java:511)
 at org.apache.sling.event.impl.jobs.config.JobManagerConfiguration.handleTopologyEvent(JobManagerConfiguration.java:585)
 - locked <0x000000074b4236a8> (a java.util.ArrayList)
 at org.apache.sling.event.impl.jobs.config.TopologyHandler.run(TopologyHandler.java:98)
 at java.lang.Thread.run(Thread.java:745)


"FelixStartLevel" #16 daemon prio=5 os_prio=0 tid=0x00007fd7ad954800 nid=0x514d pcpu%=21.2  memory%=60.0 -- running for 8529 seconds --   waiting for monitor entry [0x00007fd78c125000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.sling.event.impl.jobs.config.JobManagerConfiguration.removeListener(JobManagerConfiguration.java:608)
    - waiting to lock <0x000000074b4236a8> (a java.util.ArrayList)
    at org.apache.sling.event.impl.jobs.scheduling.JobSchedulerImpl.deactivate(JobSchedulerImpl.java:114)
    at org.apache.sling.event.impl.jobs.JobManagerImpl.deactivate(JobManagerImpl.java:144)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222)
    at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37)
    at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615)
    at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499)
    at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:295)
    at org.apache.felix.scr.impl.manager.SingleComponentManager.disposeImplementationObject(SingleComponentManager.java:342)
    at org.apache.felix.scr.impl.manager.SingleComponentManager.deleteComponent(SingleComponentManager.java:157)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:783)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:757)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:1014)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:915)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136)
    at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175)
    at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:883)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:857)
    at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:140)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:925)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:774)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:757)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:1014)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:915)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136)
    at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175)
    at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryActivator.unregisterFactory(ResourceResolverFactoryActivator.java:646)
    at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryActivator.unregisterFactory(ResourceResolverFactoryActivator.java:637)
    at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryActivator.access$300(ResourceResolverFactoryActivator.java:83)
    at org.apache.sling.resourceresolver.impl.ResourceResolverFactoryActivator$1.providerRemoved(ResourceResolverFactoryActivator.java:580)
    at org.apache.sling.resourceresolver.impl.providers.ResourceProviderTracker.unregister(ResourceProviderTracker.java:254)
    at org.apache.sling.resourceresolver.impl.providers.ResourceProviderTracker.access$100(ResourceProviderTracker.java:59)
    at org.apache.sling.resourceresolver.impl.providers.ResourceProviderTracker$1.removedService(ResourceProviderTracker.java:114)
    at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:967)
    at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:870)
    at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341)
    at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:909)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.sling.resourceresolver.impl.legacy.LegacyResourceProviderWhiteboard.unbindResourceProviderFactory(LegacyResourceProviderWhiteboard.java:165)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222)
    at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37)
    at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615)
    at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499)
    at org.apache.felix.scr.impl.helper.BindMethod.invoke(BindMethod.java:41)
    at org.apache.felix.scr.impl.manager.DependencyManager.invokeUnbindMethod(DependencyManager.java:1774)
    at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeUnbindMethod(SingleComponentManager.java:393)
    at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.removedService(DependencyManager.java:364)
    at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.removedService(DependencyManager.java:295)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136)
    at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175)
    at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:883)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:857)
    at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:140)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:925)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:774)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:757)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:1014)
    at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.removedService(DependencyManager.java:915)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1241)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1136)
    at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.untrack(ServiceTracker.java:996)
    at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1175)
    at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120)
    at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:991)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:839)
    at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:546)
    at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4557)
    at org.apache.felix.framework.Felix.access$000(Felix.java:106)
    at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:420)
    at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:170)
    at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:144)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:883)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.unregister(AbstractComponentManager.java:857)
    at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:140)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.unregisterService(AbstractComponentManager.java:925)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:774)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:757)
    at org.apache.felix.scr.impl.manager.AbstractComponentManager.dispose(AbstractComponentManager.java:550)
    at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.disposeComponents(ConfigurableComponentHolder.java:725)
    at org.apache.felix.scr.impl.BundleComponentActivator.dispose(BundleComponentActivator.java:530)
    at org.apache.felix.scr.impl.Activator.disposeComponents(Activator.java:414)
    at org.apache.felix.scr.impl.Activator.access$300(Activator.java:53)
    at org.apache.felix.scr.impl.Activator$ScrExtension.destroy(Activator.java:273)
    at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:290)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:312)
    at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
    at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:916)
    at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:835)
    at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:517)
    at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4541)
    at org.apache.felix.framework.Felix.stopBundle(Felix.java:2600)
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1389)
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x000000074c4e7a10> (a java.util.concurrent.locks.ReentrantLock$FairSync)

Environment

AEM 6.2

Cause

There were a massive amount of sling jobs (replication, workflow, etc) pending in queue when AEM was shut down. During restart of AEM, the jobs get reassigned to their queues. This operation can take a long time to complete if there are thousands of jobs in queue.

I. Clear out excess sling jobs

After AEM finally starts up, try clearing the replication queues and purging workflows. Make sure that minimal users are actively using the system during this time as this operation would make the system slow.

Check the replication queues

Go to http://aem-host:port/etc/replication/agent.author.html
Browse to any agents with a large amount of pending items
Clear the queue if it is not processing

Check workflow instances:

Go to http://aem-host:port/workflow UI
Check the Instances tab to see if there are a lot of piled up workflows in running state
Go to http://aem-host:port/system/console/jmx/com.adobe.granite.workflow%3Atype%3DMaintenance
Use the purgeActive JMX method to clear out the active workflows (if it is safe to do so)

Purge job nodes:

If neither of the above options help with startup and shutdown times, then delete the Sling jobs manually:

Go to http://aem-host:port/crx/explorer/index.jsp and log in as admin
Open Content Explorer
Browse to /var/eventing/jobs
See which jobs are piled up under /var/eventing/jobs/assigned/{sling id}
Right click a sub-folder there that you don't need to preserve and select Delete Recursively
Enable Disable Preliminary Scan option
Click Delete

WARNING: During the deletion AEM may be very slow, it is recommended that you take an outage window of a few hours to perform this deletion.

II. Install Latest AEM Cumulative Fix Pack

To lessen the impact of sling jobs loading on startup, there is a fix included in AEM6.2 CFP1 that makes these jobs load in a background thread. See the 6.2 hotfix page to obtain this fix.