To restore the missing blobs, follow the steps to recover missing datastore files.
Last updated on
May 18, 2021
This article discusses how to restore missing S3 datastore blobs.
Issue
An error similar to the following is observed in the AEM logs:
27.03.2017 13:43:39.698 *WARN* [qtp120214359-3936] org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore Error occurred while loading bytes from steam while fetching for id 857ee6b8664cea7e092775a2ec22dd6138342800#23435 java.util.concurrent.ExecutionException: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800 at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1010) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:971) at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:282) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:298) at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47) at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:265) at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:83) at org.apache.jackrabbit.oak.plugins.value.ValueImpl.getStream(ValueImpl.java:297) at org.apache.jackrabbit.vault.fs.PropertyValueArtifact$PVAInputStream.assertOpen(PropertyValueArtifact.java:300) at org.apache.jackrabbit.vault.fs.PropertyValueArtifact$PVAInputStream.read(PropertyValueArtifact.java:312) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) at org.apache.jackrabbit.vault.fs.io.JarExporter.writeFile(JarExporter.java:128) at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:216) at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214) at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214) at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214) at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214) at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214) at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:184) at org.apache.jackrabbit.vault.packaging.impl.PackageManagerImpl.assemble(PackageManagerImpl.java:131) at org.apache.jackrabbit.vault.packaging.impl.PackageManagerImpl.assemble(PackageManagerImpl.java:85) at org.apache.jackrabbit.vault.packaging.impl.JcrPackageManagerImpl.assemble(JcrPackageManagerImpl.java:471) at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.snapshot(JcrPackageImpl.java:549) at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.extract(JcrPackageImpl.java:420) at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.extract(JcrPackageImpl.java:483) at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.install(JcrPackageImpl.java:392) at com.day.jcr.vault.packaging.impl.JrVltJcrPackageAdapter.install(JrVltJcrPackageAdapter.java:107) at com.day.crx.packaging.impl.J2EEPackageManager.consoleInstall(J2EEPackageManager.java:336) at com.day.crx.packaging.impl.J2EEPackageManager.doPost(J2EEPackageManager.java:175) at com.day.crx.packaging.impl.PackageManagerServlet.doPost(PackageManagerServlet.java:152) at javax.servlet.http.HttpServlet.service(HttpServlet.java:644) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:85) at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:79) at com.adobe.granite.license.impl.LicenseCheckFilter.doFilter(LicenseCheckFilter.java:308) at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135) at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74) at org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:89) at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135) at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74) at org.apache.sling.security.impl.ReferrerFilter.doFilter(ReferrerFilter.java:290) at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135) at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74) at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:129) at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135) at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74) at org.apache.sling.featureflags.impl.FeatureManager.doFilter(FeatureManager.java:116) at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135) at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74) at org.apache.sling.engine.impl.log.RequestLoggerFilter.doFilter(RequestLoggerFilter.java:75) at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135) at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74) at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:124) at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:61) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800 at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:534) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:302) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:298) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1006) ... 73 common frames omitted Caused by: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800 at org.apache.jackrabbit.oak.blob.cloud.aws.s3.S3Backend.getLength(S3Backend.java:485) at org.apache.jackrabbit.core.data.CachingDataStore.getLength(CachingDataStore.java:669) at org.apache.jackrabbit.core.data.CachingDataStore.getRecord(CachingDataStore.java:467) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:543) at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:528) ... 76 common frames omitted Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 84B4B960EBF4F10B) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3826) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1015) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:991) at org.apache.jackrabbit.oak.blob.cloud.aws.s3.S3Backend.getLength(S3Backend.java:478) ... 80 common frames omitted
Environment
AEM 6.2 / Oak 1.4.0-1.4.9
Cause
A known bug OAK-5005 is responsible for this issue.
Resolution
-
-
Upload the same missing blobs to S3 using the "aws s3 cp" command line tool.
-
Install the latest AEM 6.2 Oak hot fix.