Article summary

Summary

Discusses how to integrate AEM and Solr server. A special thank you to Kalyan Venkat for contributing content used in this HELPX AEM community article.

Also a special thank you to community member Ranta Kumar Kotla for testing this Community Article to ensure it works.

Digital Marketing Solution(s) Adobe Experience Manager (Adobe CQ)
Audience
Developer (intermediate)
Required Skills
JCR nodes, JavaScript, HTML
Version Adobe Experience Manager 6.2
Video https://youtu.be/WeSFxsAjwzA

Note:

Download and unzip this file as described in this article.

Download

Note:

Do not install this ZIP file using Package Manager. Instead, unzip it and build it via Maven as instructed in this article.

Introduction

You can integrate SOLR with Adobe Experience Manager to improve searching. Solr is the popular, blazing-fast, open source enterprise search platform built on Apache Lucene. Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated fail-over and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world's largest internet sites. For more information, see http://lucene.apache.org/solr/.

The following illustration shows the SOLR client.

SOLRClient
The SOLR web client

This AEM Community article walks you though how to integrate AEM and SOLR so that an AEM component can use SOLR to perform searches, as shown in the following illustration. 

AEMSOLR
An AEM component uses SOLR to perform searches

The following list compares developing an external search platform using Solr to Oak indexing. The following are benefits:

• Full control over the Solr document model
• Control over boosting specific fields in Solr document
• Real time indexing is within your control
• Comes handy when multiple heterogeneous systems are contributing for indexing

The following is a con:
• Implementation effort is required

Install the AEM Package

Download the ZIP file named aem-solr-article.zip and unzip it. Open the command prompt and change the working directory to the aem-solr-article folder. Next run the following Maven command:

mvn -PautoInstallPackage install

This installs the package to your AEM instance. Notice that you have the following nodes under /apps.  

crxdelite
SOLR files under /apps

Note:

It is recommended that you use Maven 3.0.3 or greater. For more information about setting up Maven and the Home variable, see: Maven in 5 Minutes.

The package also contains an OSGi bundle that uses classes located in the org.apache.solr.client.solrj.impl.HttpSolrClient package. This OSGi bundle performs tasks related to indexing values with the Solr server (this is performed later in this article). For information about these API, see Class HttpSolrClient.

Note:

The package that you just built and deployed to AEM contains an OSGi bundle that contains Solr APIs located in the package org.apache.solr.client.solrj.impl.HttpSolrClient. This article discusses how to build the OSGi bundle in case you want to modify the logic. If you do not want to modify the bundle, then you do no have to build it. Skip the section titled Building the Solr OSGi bundle.

Building the Solr OSGi bundle

This section discusses how to build the OSGI Solr bunlde. You only need to build it if you want to modify the bundle. The package installed at the start of this article does deploy this bundle. This section also lists the Java code in case you want to understand it.

Note:

Make sure that you setup your Maven to use the Adobe Repository. For information, see "Setup Maven in your development environment" in Creating an AEM HTML Template Language component that uses the WCMUsePojo class.

Convert the aem-solr-article project to an Eclipse project

Convert the Java project to an Eclipse project by opening the command prompt and changing the working directory to the folder where you extracted the aem-solr-article file. Next, enter the following command.

mvn eclipse:eclipse

After you run this command, you can import the project into Eclipse as discussed in the next section.

Add Java files to the Maven project using Eclipse

To make it easier to work with the Maven generated project, import it into the Eclipse development environment, as shown in the following illustration.

solrProject
Eclipse Import Project Dialog

The following Java files are located in this project:

  • SolrSearchService (a Java interface)
  • SolrServerConfiguration (a Java interface)
  • SolrUtils (add to a package named com.adobe.aem.core.util)
  • DeleteIndexesFromSolr (add to a package named com.adobe.aem.core.servlets)
  • IndexContentToSolr (add to a package named com.adobe.aem.core.servlets)
  • SolrSearchServiceImpl (add to a package named  com.adobe.aem.core.impl)
  • SolrServerConfigurationImpl (add to a package named  com.adobe.aem.core.impl)

SolrSearchService interface

The SolrSearchService interface describes the operations exposed by this service. The following Java code represents this interface. 

package com.adobe.aem.core;

import java.io.IOException;

import javax.jcr.RepositoryException;

import org.apache.sling.api.resource.Resource;
import org.apache.sling.commons.json.JSONArray;
import org.apache.sling.commons.json.JSONException;
import org.apache.sling.commons.json.JSONObject;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpSolrClient;

import com.day.cq.search.result.SearchResult;

public interface SolrSearchService {

	JSONArray crawlContent(String resourcePath, String resourceType);

	JSONArray createPageMetadataArray(SearchResult results)
			throws RepositoryException;

	JSONObject createPageMetadataObject(Resource pageContent);

	boolean indexPageToSolr(JSONObject indexPageData, HttpSolrClient server)
			throws JSONException, SolrServerException, IOException;

	boolean indexPagesToSolr(JSONArray indexPageData, HttpSolrClient server)
			throws JSONException, SolrServerException, IOException;

}

SolrServerConfiguration interface

The SolorServerConfiguration interface descibes operations exposed by this configuration service. The following Java code represents this interface. 

 

package com.adobe.aem.core;

public interface SolrServerConfiguration {

	public String getSolrProtocol();

	public String getSolrServerName();

	public String getSolrServerPort();

	public String getSolrCoreName();
	
	public String getContentPagePath();
	
}

SolrUtils class

The SolrUtils class contains helper methods. For example, the getPageTags method returns tags located in the current page. The following code represents this Java class that is located in the com.adobe.aem.core.utils package.  

package com.adobe.aem.core.utils;

import java.text.SimpleDateFormat;
import java.util.Calendar;

import org.apache.commons.lang3.StringUtils;
import org.apache.sling.api.resource.Resource;

import com.day.cq.tagging.Tag;
import com.day.cq.wcm.api.Page;

public final class SolrUtils {
	// private constructor to avoid unnecessary instantiation of the class
	private SolrUtils() {
	}


	/**
	 * This method is used to extract the tags from the content page
	 * 
	 * @param pageContent
	 * @return Array of tags which are attached to the page. Empty array if no
	 *         tags are attached
	 */
	public static String[] getPageTags(Resource pageContent) {
		Page page = pageContent.getParent().adaptTo(Page.class);
		Tag tags[] = page.getTags();
		String tagsArray[] = new String[tags.length];
		for (int i = 0; i < tags.length; i++) {
			Tag tag = tags[i];
			tagsArray[i] = tag.getTitle();
		}
		return tagsArray;
	}


	/**
	 * This method converts jcr formatted date to Solr specification format
	 * 
	 * @param Takes input as Calendar
	 * @return Solr formatted date of type string
	 */
	public static String solrDate(Calendar cal) {
		SimpleDateFormat dateFormat = new SimpleDateFormat(
				"YYYY-MM-DD'T'hh:mm:ss");
		return dateFormat.format(cal.getTime()) + "Z";
	}

	/**
	 * This method returs "" if string is null.
	 * 
	 * @param Takes
	 *            input as string
	 * @return String value. if string value is "null" then ""
	 */
	public static String checkNull(String property) {
		if (StringUtils.isEmpty(property)) {
			return "";
		}
		return property;

	}

}

IndexContentToSolr Servlet

The following Java code represents the IndexContentToSolr sling servlet. 

package com.adobe.aem.core.servlets;

import java.io.IOException;

import javax.servlet.Servlet;
import javax.servlet.ServletException;

import org.apache.felix.scr.annotations.Component;
import org.apache.felix.scr.annotations.Properties;
import org.apache.felix.scr.annotations.Property;
import org.apache.felix.scr.annotations.Reference;
import org.apache.felix.scr.annotations.Service;
import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.servlets.SlingAllMethodsServlet;
import org.apache.sling.commons.json.JSONArray;
import org.apache.sling.commons.json.JSONException;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.adobe.aem.core.SolrSearchService;
import com.adobe.aem.core.SolrServerConfiguration;

/**
 * 
 * This servlet acts as a bulk update to index content pages and assets to the
 * configured Solr server
 *
 */
@Component(immediate = true, metatype = true)
@Service(Servlet.class)
@Properties({
		@Property(name = "sling.servlet.methods", value = "GET"),
		@Property(name = "sling.servlet.paths", value = "/bin/solr/push/pages") })
public class IndexContentToSolr extends SlingAllMethodsServlet {

	private static final long serialVersionUID = 1L;
	private static final Logger LOG = LoggerFactory
			.getLogger(IndexContentToSolr.class);

	@Reference
	SolrServerConfiguration solrConfigurationService;

	@Reference
	SolrSearchService solrSearchService;

	@Override
	protected void doGet(SlingHttpServletRequest request,
			SlingHttpServletResponse response) throws ServletException,
			IOException {
		doPost(request, response);

	}

	@Override
	protected void doPost(SlingHttpServletRequest request,
			SlingHttpServletResponse response) throws ServletException,
			IOException {
		response.setContentType("text/html");
		String indexType = request.getParameter("indexType");
		final String protocol = solrConfigurationService.getSolrProtocol();
		final String serverName = solrConfigurationService.getSolrServerName();
		final String serverPort = solrConfigurationService.getSolrServerPort();
		final String coreName = solrConfigurationService.getSolrCoreName();
		final String pagesResourcePath = solrConfigurationService
				.getContentPagePath();
		String URL = protocol + "://" + serverName + ":" + serverPort
				+ "/solr/" + coreName;
		HttpSolrClient server = new HttpSolrClient(URL);
		if (indexType.equalsIgnoreCase("indexpages")) {
			try {
				JSONArray indexPageData = solrSearchService.crawlContent(
						pagesResourcePath, "cq:PageContent");
				boolean resultindexingPages = solrSearchService
						.indexPagesToSolr(indexPageData, server);
				if (resultindexingPages == true) {
					response.getWriter()
							.write("<h3>Successfully indexed content pages to Solr server </h3>");
				} else {
					response.getWriter().write("<h3>Something went wrong</h3>");
				}
			} catch (Exception e) {
				LOG.error("Exception due to", e);
				response.getWriter()
						.write("<h3>Something went wrong. Please make sure Solr server is configured properly in Felix</h3>");
			}

		} else {
			response.getWriter().write("<h3>Something went wrong</h3>");
		}

	}

}

DeleteIndexesFromSolr

The following Java code represents the DeleteIndexesFromSolr sling servlet. 

package com.adobe.aem.core.servlets;

import java.io.IOException;

import javax.servlet.ServletException;

import org.apache.felix.scr.annotations.Reference;
import org.apache.felix.scr.annotations.sling.SlingServlet;
import org.apache.sling.api.SlingHttpServletRequest;
import org.apache.sling.api.SlingHttpServletResponse;
import org.apache.sling.api.servlets.SlingAllMethodsServlet;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.adobe.aem.core.SolrServerConfiguration;


/**
 * 
 * This servlet deletes all the indexes from the configured Solr server
 *
 */
@SlingServlet(paths = "/bin/solr/delete/all/indexes", methods="POST")
public class DeleteIndexesFromSolr extends SlingAllMethodsServlet {
	private static final long serialVersionUID = 1L;
	private static final Logger LOG = LoggerFactory
			.getLogger(DeleteIndexesFromSolr.class);
	@Reference
	SolrServerConfiguration solrConfigurationService;
	
	@Override
    protected void doPost(final SlingHttpServletRequest reqest,
            final SlingHttpServletResponse response) throws ServletException, IOException {
		response.setContentType("text/html");
		final String protocol = solrConfigurationService.getSolrProtocol();
		final String serverName = solrConfigurationService.getSolrServerName();
		final String serverPort = solrConfigurationService.getSolrServerPort();
		final String coreName = solrConfigurationService.getSolrCoreName();
		String URL = protocol + "://" + serverName + ":" + serverPort
				+ "/solr/" + coreName;
		HttpSolrClient server = new HttpSolrClient(URL);
		try {
			server.deleteByQuery("*:*");
			server.commit();
			server.close();
			response.getWriter().write("<h3>Deleted all the indexes from solr server </h3>");
		} catch (SolrServerException e) {
			LOG.error("Exception due to", e);
		}
		
        
    }
}

Next, modify the POM file located at C:\AdobeCQ\aem-solr-article\core. The following code represents this POM file. 

Note:

The POM files contains the dependencies you need to build the OSGi bundle. 

Build the OSGi bundle using Maven

To build the OSGi bundle by using Maven, perform these steps:

  1. Open the command prompt and go to aem-solr-article.
  2. Run the following maven command: mvn clean install.
  3. The OSGi component can be found in the following folder: aem-solr-article\core\target. The file name of the OSGi component is solr.core-1.0-SNAPSHOT.jar.
  4. Install the OSGi using the Felix console.

 

Setup the Solr Server

Download and install the Solr server (solr-6.2.0.zip ) from the following URL:

http://archive.apache.org/dist/lucene/solr/

Ensure that you select version 6.2.0, as shown in the following illustration.

solrserverfiles
Solr server files

Note:

To install Solr server, you need to have Java 1.8 installed.  

Unzip the file so that you are able to view the following file structure.

zip
The contents of the Solr zip file

You can start the Solr server by opening the command prompt and changing the working directory to the bin folder located in the directory in which you extracted the ZIP file. Run the following command:

solr start -e cloud -noprompt

Once the server successfully starts, you will see a message in the command prompt similiar to the following illustration.

SOLRServer
Solr Server started

To verify that Solr is running, go to the following URL:

http://localhost:8983/solr/#/

You should see the Solr web client.

SOLRClient
Solr web client

Create a new collection

The next task is to create a new collection by using the web client. The collection that you create is the single search index that is used in this article. For more information, see Apache Solr Reference Guide.

To create a collection,

1. From the web client, select  Collection in the right hand column (shown above).

2 . Click Add Collection.

3. Name the new collection: collection.

 

Configure AEM to use Solr Server

Configure AEM to use Solr server. Go to the following URL:

http://localhost:4502/system/console/configMgr

Search for AEM Solr Search - Solr Configuration Service and enter the following values:

  • Protocol - http
  • Solr Server name - localhost
  • Solr Server Port: 8983
  • Solr Core Name - collection (references the collection you created)
  • Content page path - /content/geometrixx

The following illustration shows these values.

AEMCon2
Solr configuration values

Next, index values with Solr by going to the following AEM page.

http://localhost:4502/etc/solr/solrindexer.html

This is the admin console that lets you create a SOLR document from the content pages. You can also delete the index documents from this page. Select Index Pages from the Index Options drop-down control and click Index to Solr as shown in the following illusration.

solrsuccess
Indexing with Solr

This page is invoking the IndexContentToSolr sling servlet (you can see the source code of this servlet in the OSGi section of this article). If the index operation is successful, a message confirming the operation was successful appears, as shown in the previous illustration.  

You can validate if the SOLR docs are created by going to the following URL:

http://localhost:8983/solr/#/collection/query

Select collection from the drop-down control and select Query. Then click the Execute button. If successful, you will see the result set.

Solor_ResultSet
Solr result set

View Solr results in an AEM component

In CRXDE lite, open the following HTML file:

/apps/solr/components/content/solrsearch/solrsearch.html

and add the following script at the top of the page:

<script src="https://code.jquery.com/jquery-3.1.0.js" integrity="sha256-slogkvB1K3VOkzAI8QITxV3VzpOnkeNVsKvtkYLMjfk=" crossorigin="anonymous"></script>  

To access the component that display Solr values, enter the following URL: 

http://localhost:4502/content/solr/en.html

Specify the term content in search bar and click search. You will see the following values returned by Solr.

 

SOLR_WORKED
Solr result set in an AEM component

See also

Congratulations, you have just created an AEM sample application that integrates with Solr server. Please refer to the AEM community page for other articles that discuss how to build AEM services/applications.

This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License  Twitter™ and Facebook posts are not covered under the terms of Creative Commons.

Legal Notices   |   Online Privacy Policy