Missing S3 datastore blobs | AEM 6.2 Assets

This article discusses how to restore missing S3 datastore blobs.

Issue

An error similar to the following is observed in the AEM logs:

27.03.2017 13:43:39.698 *WARN* [qtp120214359-3936] org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore Error occurred while loading bytes from steam while fetching for id 857ee6b8664cea7e092775a2ec22dd6138342800#23435
java.util.concurrent.ExecutionException: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800
at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1010)
at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:971)
at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:282)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:298)
at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47)
at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:265)
at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.getNewStream(SegmentBlob.java:83)
at org.apache.jackrabbit.oak.plugins.value.ValueImpl.getStream(ValueImpl.java:297)
at org.apache.jackrabbit.vault.fs.PropertyValueArtifact$PVAInputStream.assertOpen(PropertyValueArtifact.java:300)
at org.apache.jackrabbit.vault.fs.PropertyValueArtifact$PVAInputStream.read(PropertyValueArtifact.java:312)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at org.apache.jackrabbit.vault.fs.io.JarExporter.writeFile(JarExporter.java:128)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:216)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:214)
at org.apache.jackrabbit.vault.fs.io.AbstractExporter.export(AbstractExporter.java:184)
at org.apache.jackrabbit.vault.packaging.impl.PackageManagerImpl.assemble(PackageManagerImpl.java:131)
at org.apache.jackrabbit.vault.packaging.impl.PackageManagerImpl.assemble(PackageManagerImpl.java:85)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageManagerImpl.assemble(JcrPackageManagerImpl.java:471)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.snapshot(JcrPackageImpl.java:549)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.extract(JcrPackageImpl.java:420)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.extract(JcrPackageImpl.java:483)
at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.install(JcrPackageImpl.java:392)
at com.day.jcr.vault.packaging.impl.JrVltJcrPackageAdapter.install(JrVltJcrPackageAdapter.java:107)
at com.day.crx.packaging.impl.J2EEPackageManager.consoleInstall(J2EEPackageManager.java:336)
at com.day.crx.packaging.impl.J2EEPackageManager.doPost(J2EEPackageManager.java:175)
at com.day.crx.packaging.impl.PackageManagerServlet.doPost(PackageManagerServlet.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:644)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:85)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:79)
at com.adobe.granite.license.impl.LicenseCheckFilter.doFilter(LicenseCheckFilter.java:308)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:89)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.sling.security.impl.ReferrerFilter.doFilter(ReferrerFilter.java:290)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:129)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.sling.featureflags.impl.FeatureManager.doFilter(FeatureManager.java:116)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.sling.engine.impl.log.RequestLoggerFilter.doFilter(RequestLoggerFilter.java:75)
at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)
at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)
at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:124)
at org.apache.felix.http.base.internal.DispatcherServlet.service(DispatcherServlet.java:61)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:534)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:302)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore$2.call(DataStoreBlobStore.java:298)
at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1006)
... 73 common frames omitted
Caused by: org.apache.jackrabbit.core.data.DataStoreException: Could not length of dataIdentifier 857ee6b8664cea7e092775a2ec22dd6138342800
at org.apache.jackrabbit.oak.blob.cloud.aws.s3.S3Backend.getLength(S3Backend.java:485)
at org.apache.jackrabbit.core.data.CachingDataStore.getLength(CachingDataStore.java:669)
at org.apache.jackrabbit.core.data.CachingDataStore.getRecord(CachingDataStore.java:467)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:543)
at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:528)
... 76 common frames omitted
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: 84B4B960EBF4F10B)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3826)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1015)
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:991)
at org.apache.jackrabbit.oak.blob.cloud.aws.s3.S3Backend.getLength(S3Backend.java:478)
... 80 common frames omitted

Environment

AEM 6.2 / Oak 1.4.0-1.4.9

Cause

A known bug OAK-5005 is responsible for this issue.

Resolution

  1. To restore the missing blobs, follow the steps to recover missing datastore files.

  2. Upload the same missing blobs to S3 using the "aws s3 cp" command line tool.

  3. Install the latest AEM 6.2 Oak hot fix.

 Adobe

Get help faster and easier

New user?

Adobe MAX 2024

Adobe MAX
The Creativity Conference

Oct 14–16 Miami Beach and online

Adobe MAX

The Creativity Conference

Oct 14–16 Miami Beach and online

Adobe MAX 2024

Adobe MAX
The Creativity Conference

Oct 14–16 Miami Beach and online

Adobe MAX

The Creativity Conference

Oct 14–16 Miami Beach and online