Workaround for SIGSEGV occurs in com.adobe.cq.social.datastore.as.impl.UGCCDataServiceImpl

Sometimes there are JVM crashes with a SEGV, all occurring within JIT compiled code in com.adobe.cq.social.datastore.as.impl.UGCCDataServiceImpl. It is possible that there is a Java bug here, but one that seems to affect a very localized set of methods.

There is a work-around by specifying JIT parameters for this class that appears to alleviate the problem. This task is to capture and document the workaround and the circumstances where it is applicable.

Who needs this workaround?

• You are using Social Communities Extension with a configured Social Cloud Storage provider.
• You are running AEM on 64-bit Linux using the Oracle 64-bit JVM version 1.7.
• You are experiencing JVM crashes with Adobe Experience Manager.
• The JVM crashes are reported in com.adobe.cq.social.datastore.as.impl.UGCCDataServiceImpl
  JVM crashes are accompanied by a file hs_err_pid<<PROCESS-ID>>.log appearing in the crx-quickstart directory of CQ. The basic information about the cause of the JVM crash will appear in this file. Look for something like this in the case of crashes this workaround addresses:

The critical information is the last line in this example that shows the problematic frame involves com.adobe.cq.social.datastore.as.impl.UGCCDataServiceImpl, a component of the cloud service connector.

What is the workaround?

The workaround uses two approaches: use a JVM version less prone to experience the problem, and use a JVM option to disable native compilation of the specific problem code.
At time of publishing this note, the latest Java 1.7 version available from Oracle is 1.7_65. The recommendation of this workaround is to move to 1.7_65 from any earlier version.
Using 1.7_65, the workaround is further to disable just in time (JIT) compilation of a specific problem class and method via the Java -XX:CompileCommand=exclude option. Specifically, -XX:CompileCommand=exclude,com.adobe.cq.social.datastore.as.impl.UGCCDataServiceImpl::readDocument"

How do I implement this workaround in my environment?

This implementation will require verifying and possibly updating the version of Java you have installed and updating the startup script you use to start AEM. To upgrade Java, please see the Oracle documentation regarding choosing, downloading, installing and configuring jdk1.7.0_65 64-bit for Linux x86-64.

Generally it is recommended that AEM is started via the start scripts located in crx-quickstart/bin/start. It is common, however, for customized startup scripts to be employed. In the case where crx-quickstart/bin/start is being used, the following option must be added:

CQ_JVM_OPTS="${CQ_JVM_OPTS} -XX:CompileCommand=exclude,com.adobe.cq.social.datastore.as.impl.UGCCDataServiceImpl::readDocument"

NOTE: need to look at the default bin/start and see where this would belong, exactly.