If you have made extensive use of permissions / ACLs in CQ or CRX it can affect the CRX search performance because when search results are being retrieved they must be filtered against the permissions in the CRX repository.
In cases where this is an issue, if you take thread dumps you will see that there is a lot of lock contention related to stack traces containing the CacheEntryCollector class.
To see if you have a enough permissions (rep:ACE nodes) set in your repository to greatly affect performance, do the following:
If that results in more than 5000 nodes then see below for details on how to improve system performance.
1. First of all, research whether or not you can reduce your total number of ACLs. Here are a few tips that may help:
Note: If you don't know how to view ACLs in CRX then see this article. You can also search for ACLs using the search mentioned above //element(*,rep:ACL) order by @jcr:score.
2. Once you reduce the total number of ACEs, if you still have a performance problem then see if you can implement a caching layer in your application code for commonly retrieved search results.
3. Finally to increase the ACL cache size.