This article discusses how to restore missing S3 datastore blobs.

Issue

An error similar to the following is observed in the AEM logs:

27.03.2017 13:43:39.698 *WARN* [qtp120214359-3936] org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore Error occurred while loading bytes from steam while fetching for id 857ee6b8664cea7e092775a2ec22dd6138342800#23435
java.util.concurrent.ExecutionException: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800
at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1010)
at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:971)
at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:282)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:298)
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47)
at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:265)
at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:83)
at org.apache.jackrabbit.oak.plugins.value.ValueImpl.getStream(ValueImpl.java:297)
at org.apache.jackrabbit.vault.fs.PropertyValueArtifact$PVAInputStream.assertOpen(PropertyValueArtifact.java:300)
at org.apache.jackrabbit.vault.fs.PropertyValueArtifact$PVAInputStream.read(PropertyValueArtifact.java:312)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at org.apache.jackrabbit.vault.fs.io.JarExporter.writeFile(JarExporter.java:128)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:216)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:184)
at org.apache.jackrabbit.vault.packaging.impl.PackageManagerImpl.assemble(PackageManagerImpl.java:131)
at org.apache.jackrabbit.vault.packaging.impl.PackageManagerImpl.assemble(PackageManagerImpl.java:85)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageManagerImpl.assemble(JcrPackageManagerImpl.java:471)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.snapshot(JcrPackageImpl.java:549)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.extract(JcrPackageImpl.java:420)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.extract(JcrPackageImpl.java:483)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.install(JcrPackageImpl.java:392)
at com.day.jcr.vault.packaging.impl.JrVltJcrPackageAdapter.install(JrVltJcrPackageAdapter.java:107)
at com.day.crx.packaging.impl.J2EEPackageManager.consoleInstall(J2EEPackageManager.java:336)
at com.day.crx.packaging.impl.J2EEPackageManager.doPost(J2EEPackageManager.java:175)
at com.day.crx.packaging.impl.PackageManagerServlet.doPost(PackageManagerServlet.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:644)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:85)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:79)
at com.adobe.granite.license.impl.LicenseCheckFilter.doFilter(LicenseCheckFilter.java:308)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:89)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.sling.security.impl.ReferrerFilter.doFilter(ReferrerFilter.java:290)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:129)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.sling.featureflags.impl.FeatureManager.doFilter(FeatureManager.java:116)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.sling.engine.impl.log.RequestLoggerFilter.doFilter(RequestLoggerFilter.java:75)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:124)
at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:61)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:534)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:302)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:298)
at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1006)
... 73 common frames omitted
Caused by: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800
at org.apache.jackrabbit.oak.blob.cloud.aws.s3.S3Backend.getLength(S3Backend.java:485)
at org.apache.jackrabbit.core.data.CachingDataStore.getLength(CachingDataStore.java:669)
at org.apache.jackrabbit.core.data.CachingDataStore.getRecord(CachingDataStore.java:467)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:543)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:528)
... 76 common frames omitted
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 84B4B960EBF4F10B)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3826)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1015)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:991)
at org.apache.jackrabbit.oak.blob.cloud.aws.s3.S3Backend.getLength(S3Backend.java:478)
... 80 common frames omitted

Environment

AEM 6.2 / Oak 1.4.0-1.4.9

Cause

A known bug OAK-5005 is responsible for this issue.

Resolution

  1. To restore the missing blobs, follow the steps to recover missing datastore files.

  2. Upload the same missing blobs to S3 using the "aws s3 cp" command line tool.

  3. Install the latest AEM 6.2 Oak hot fix.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License  Twitter™ and Facebook posts are not covered under the terms of Creative Commons.

Legal Notices   |   Online Privacy Policy