Last updated on
16 May 2021
Issue
After using the out-of-the-box online backup feature to back up and restore AEM on a system, which uses FileDataStore, it displays the following errors [1] and [2]. These errors indicate missing files in the restored datastore.
[1] 28.11.2016 14:49:09.521 *ERROR* [pool-6-thread-5] org.apache.sling.commons.scheduler.impl.QuartzScheduler No discovery available, therefore not executing job org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@62ac5133 with name Registered Service.342 and config SINGLE. 28.11.2016 14:49:14.521 *ERROR* [pool-6-thread-2] org.apache.sling.commons.scheduler.impl.QuartzScheduler No discovery available, therefore not executing job org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate@62ac5133 with name Registered Service.342 and config SINGLE. [2] 8.11.2016 14:48:54.257 *WARN* [FelixStartLevel] org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore Error occurred while loading bytes from steam while fetching for id 1b38777ffff709284d3feb12099f24983b6f3594#112232 java.util.concurrent.ExecutionException: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 1b38777ffff709284d3feb12099f24983b6f3594 does not exist at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:915) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:877) at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:222) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:280) at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47) at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:265) at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:83) at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.loadBlob(OakDirectory.java:270) at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.readBytes(OakDirectory.java:320) at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.readBytes(OakDirectory.java:423) at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:254) at org.apache.lucene.store.Directory.copy(Directory.java:186) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.copyFilesToLocal(IndexCopier.java:436) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.prefetchIndexFiles(IndexCopier.java:410) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.<init>(IndexCopier.java:322) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(IndexCopier.java:138) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:53) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:179) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:154) at org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:220) at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:957) at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:902) at org.apache.jackrabbit.oak.query.ast.SelectorImpl.prepare(SelectorImpl.java:288) at org.apache.jackrabbit.oak.query.QueryImpl.prepare(QueryImpl.java:584) at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:183) at org.apache.jackrabbit.oak.security.user.UserProvider.getAuthorizableByPrincipal(UserProvider.java:234) at org.apache.jackrabbit.oak.security.user.UserManagerImpl.getAuthorizable(UserManagerImpl.java:121) at org.apache.jackrabbit.oak.security.principal.PrincipalProviderImpl.getAuthorizable(PrincipalProviderImpl.java:140) at org.apache.jackrabbit.oak.security.principal.PrincipalProviderImpl.getPrincipal(PrincipalProviderImpl.java:69) at org.apache.jackrabbit.oak.spi.security.principal.CompositePrincipalProvider.getPrincipal(CompositePrincipalProvider.java:50) at org.apache.jackrabbit.oak.spi.security.principal.PrincipalManagerImpl.getPrincipal(PrincipalManagerImpl.java:47) at com.adobe.granite.repository.impl.SlingRepositoryManager.setupPermissions(SlingRepositoryManager.java:1005) at com.adobe.granite.repository.impl.SlingRepositoryManager.createRepository(SlingRepositoryManager.java:429) at com.adobe.granite.repository.impl.SlingRepositoryManager.acquireRepository(SlingRepositoryManager.java:299) at org.apache.sling.jcr.base.AbstractSlingRepositoryManager.start(AbstractSlingRepositoryManager.java:304) at com.adobe.granite.repository.impl.SlingRepositoryManager.activate(SlingRepositoryManager.java:276) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222) at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37) at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615) at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499) at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:295) at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:302) at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:113) at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:847) at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:814) at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:724) at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:950) at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:911) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167) at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120) at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987) at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4547) at org.apache.felix.framework.Felix.registerService(Felix.java:3521) at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) at org.apache.sling.commons.threads.impl.Activator.start(Activator.java:55) at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:697) at org.apache.felix.framework.Felix.activateBundle(Felix.java:2223) at org.apache.felix.framework.Felix.startBundle(Felix.java:2141) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1368) at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 1b38777ffff709284d3feb12099f24983b6f3594 does not exist at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:484) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.access$000(DataStoreBlobStore.java:69) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:284) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:280) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:911) ... 70 common frames omitted Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record 1b38777ffff709284d3feb12099f24983b6f3594 does not exist at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:47) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:493) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:478) ... 74 common frames omitted 28.11.2016 14:48:54.259 *ERROR* [FelixStartLevel] com.adobe.granite.repository.impl.SlingRepositoryManager start: Uncaught Throwable trying to access Repository, calling stopRepository() java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId [1b38777ffff709284d3feb12099f24983b6f3594#112232] at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49) at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:265) at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:83) at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.loadBlob(OakDirectory.java:270) at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexFile.readBytes(OakDirectory.java:320) at org.apache.jackrabbit.oak.plugins.index.lucene.OakDirectory$OakIndexInput.readBytes(OakDirectory.java:423) at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:254) at org.apache.lucene.store.Directory.copy(Directory.java:186) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.copyFilesToLocal(IndexCopier.java:436) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.prefetchIndexFiles(IndexCopier.java:410) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier$CopyOnReadDirectory.<init>(IndexCopier.java:322) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexCopier.wrapForRead(IndexCopier.java:138) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexNode.open(IndexNode.java:53) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.findIndexNode(IndexTracker.java:179) at org.apache.jackrabbit.oak.plugins.index.lucene.IndexTracker.acquireIndexNode(IndexTracker.java:154) at org.apache.jackrabbit.oak.plugins.index.lucene.LucenePropertyIndex.getPlans(LucenePropertyIndex.java:220) at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:957) at org.apache.jackrabbit.oak.query.QueryImpl.getBestSelectorExecutionPlan(QueryImpl.java:902) at org.apache.jackrabbit.oak.query.ast.SelectorImpl.prepare(SelectorImpl.java:288) at org.apache.jackrabbit.oak.query.QueryImpl.prepare(QueryImpl.java:584) at org.apache.jackrabbit.oak.query.QueryEngineImpl.executeQuery(QueryEngineImpl.java:183) at org.apache.jackrabbit.oak.security.user.UserProvider.getAuthorizableByPrincipal(UserProvider.java:234) at org.apache.jackrabbit.oak.security.user.UserManagerImpl.getAuthorizable(UserManagerImpl.java:121) at org.apache.jackrabbit.oak.security.principal.PrincipalProviderImpl.getAuthorizable(PrincipalProviderImpl.java:140) at org.apache.jackrabbit.oak.security.principal.PrincipalProviderImpl.getPrincipal(PrincipalProviderImpl.java:69) at org.apache.jackrabbit.oak.spi.security.principal.CompositePrincipalProvider.getPrincipal(CompositePrincipalProvider.java:50) at org.apache.jackrabbit.oak.spi.security.principal.PrincipalManagerImpl.getPrincipal(PrincipalManagerImpl.java:47) at com.adobe.granite.repository.impl.SlingRepositoryManager.setupPermissions(SlingRepositoryManager.java:1005) at com.adobe.granite.repository.impl.SlingRepositoryManager.createRepository(SlingRepositoryManager.java:429) at com.adobe.granite.repository.impl.SlingRepositoryManager.acquireRepository(SlingRepositoryManager.java:299) at org.apache.sling.jcr.base.AbstractSlingRepositoryManager.start(AbstractSlingRepositoryManager.java:304) at com.adobe.granite.repository.impl.SlingRepositoryManager.activate(SlingRepositoryManager.java:276) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222) at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37) at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615) at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499) at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:295) at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:302) at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:113) at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:847) at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:814) at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:724) at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:950) at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:911) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945) at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881) at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167) at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120) at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987) at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4547) at org.apache.felix.framework.Felix.registerService(Felix.java:3521) at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) at org.apache.sling.commons.threads.impl.Activator.start(Activator.java:55) at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:697) at org.apache.felix.framework.Felix.activateBundle(Felix.java:2223) at org.apache.felix.framework.Felix.startBundle(Felix.java:2141) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1368) at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 1b38777ffff709284d3feb12099f24983b6f3594 does not exist at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:484) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:299) at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47) ... 66 common frames omitted Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record 1b38777ffff709284d3feb12099f24983b6f3594 does not exist at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:47) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:493) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:478) ... 68 common frames omitted
Cause
Online backup currently does not back up the FileDataStore properly. It is a known product bug, fixed in the latest AEM 6.1 Oak cumulative fix pack 1.2.23. This bug will also be fixed in AEM6.3 and in the later AEM versions.
Currently, the bug is only fixed in a hotfix for AEM 6.1, not AEM 6.2.
Resolution
- Back up the datastore separate from the crx-quickstart folder:
- Stop AEM.
- To point to a location outside the AEM installation directory, modify the path configured in the FileDataStore configuration under crx-quickstart/install.
- Move your crx-quickstart/repository/repository/datastore to the new location you have configured.
- Start AEM.
- After running an online backup, use some other backup technology to back up the datastore folder. Incremental backup is acceptable as files are only added to the datastore. Remove the files only after running the Datastore Garbage Collection.
- Or, if you are using AEM6.1, apply Oak 1.2.23 CFP. Continue to use the online backup feature.