alfresco5.2 and solr4 performance Issue

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Member II

alfresco5.2 and solr4 performance Issue

I am getting the following issue in alfresco and tomcat logs while indexing. The memory is shooting upto 23 GB, though there is very less nodes and transactions on the server.

I guess it is not releasing the memory. Upon restart to it shoots up to the same figure which prevents us to log into share application or even open the login page.

2017-12-06 00:00:09,752 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-433] Exception from executeScript: 11050380 Wrapped Exception (with status template): Java heap space
org.springframework.extensions.webscripts.WebScriptException: 11050380 Wrapped Exception (with status template): Java heap space
at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:1138)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
at org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:512)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:464)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:587)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:656)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:355)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:308)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:399)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:210)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.module.aosmodule.service.ContextRootFilter.doFilter(ContextRootFilter.java:93)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.web.scripts.servlet.X509ServletFilterBase.doFilter(X509ServletFilterBase.java:144)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at org.json.JSONObject.toString(JSONObject.java:1377)
at org.json.JSONObject.toString(JSONObject.java:1340)
at org.alfresco.repo.web.scripts.solr.NodesMetaDataGet$FreemarkerNodeMetaData.<init>(NodesMetaDataGet.java:297)
at org.alfresco.repo.web.scripts.solr.NodesMetaDataGet$1.handleNodeMetaData(NodesMetaDataGet.java:203)
at org.alfresco.repo.solr.SOLRTrackingComponentImpl$NodeMetaDataQueryRowHandler.processResult(SOLRTrackingComponentImpl.java:1356)
at org.alfresco.repo.solr.SOLRTrackingComponentImpl.getNodesMetadata(SOLRTrackingComponentImpl.java:1151)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.alfresco.repo.management.subsystems.SubsystemProxyFactory$1.invoke(SubsystemProxyFactory.java:79)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy105.getNodesMetadata(Unknown Source)
at org.alfresco.repo.web.scripts.solr.NodesMetaDataGet.executeImpl(NodesMetaDataGet.java:191)
at org.springframework.extensions.webscripts.DeclarativeWebScript.executeImpl(DeclarativeWebScript.java:235)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:64)
at org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:512)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:464)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:587)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:656)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:355)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:308)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:399)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:210)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

Do anyone have any idea?

7 Replies
Highlighted
Master

Re: alfresco5.2 and solr4 performance Issue

So what are the numbers really that we are talking about? How many nodes, transactions and ACLs do exist? "very less" is not an information that is specific enough....

Do you have any customisations / 3rd party addons installed in the system that may interfere in some way or even be prone to a memory leak?

Highlighted
Member II

Re: alfresco5.2 and solr4 performance Issue

Nodes = 15000
Transactions = 50000

ACLs = 50000

No customization or add ons. 

Highlighted
Senior Member

Re: alfresco5.2 and solr4 performance Issue

Hi

There is not really much here to go on. I would suspect an in memory transformation from a giant pdf...

Andy 

Highlighted
Member II

Re: alfresco5.2 and solr4 performance Issue

So is there any size limit in uploading documents in alfresco? 

how should I identify these large documents?

Highlighted
Master

Re: alfresco5.2 and solr4 performance Issue

Best bet would be to create a heap dump and analyse that for the biggest consumers / dominator classes etc. to find the actual culprit.

One of my customers once had a memory issue dealing with inbound SMTP when one of the emails included a 600 MiB video in Base64 encoding, which was loaded in-memory. Another had an issue with a PDF catalog file containing ~500 other PDF files. Again, you should see this best in a heap dump where you could also investigate the context and identify the node being processed at the time.

Highlighted
Member II

Re: alfresco5.2 and solr4 performance Issue

Heap analysis dump also does not hint out at anything major.

Highlighted
Master

Re: alfresco5.2 and solr4 performance Issue

You have 33 instances of ArrayList with 3 GiB heap usage, and 3.5 GiB of String values in arrays (each apparently over 300 MiB). That certainly looks like a major issue. You need to check the path to GC roots to determine which components contain those data structures and for what they are used.