alfresco5.2 and solr4 performance Issue

[deepak9batra]

I am getting the following issue in alfresco and tomcat logs while indexing. The memory is shooting upto 23 GB, though there is very less nodes and transactions on the server.

I guess it is not releasing the memory. Upon restart to it shoots up to the same figure which prevents us to log into share application or even open the login page.

2017-12-06 00:00:09,752 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] [http-bio-8443-exec-433] Exception from executeScript: 11050380 Wrapped Exception (with status template): Java heap space
org.springframework.extensions.webscripts.WebScriptException: 11050380 Wrapped Exception (with status template): Java heap space
at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:1138)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
at org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:512)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:464)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:587)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:656)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:355)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:308)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:399)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:210)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.module.aosmodule.service.ContextRootFilter.doFilter(ContextRootFilter.java:93)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.web.scripts.servlet.X509ServletFilterBase.doFilter(X509ServletFilterBase.java:144)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at org.json.JSONObject.toString(JSONObject.java:1377)
at org.json.JSONObject.toString(JSONObject.java:1340)
at org.alfresco.repo.web.scripts.solr.NodesMetaDataGet$FreemarkerNodeMetaData.<init>(NodesMetaDataGet.java:297)
at org.alfresco.repo.web.scripts.solr.NodesMetaDataGet$1.handleNodeMetaData(NodesMetaDataGet.java:203)
at org.alfresco.repo.solr.SOLRTrackingComponentImpl$NodeMetaDataQueryRowHandler.processResult(SOLRTrackingComponentImpl.java:1356)
at org.alfresco.repo.solr.SOLRTrackingComponentImpl.getNodesMetadata(SOLRTrackingComponentImpl.java:1151)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.alfresco.repo.management.subsystems.SubsystemProxyFactory$1.invoke(SubsystemProxyFactory.java:79)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy105.getNodesMetadata(Unknown Source)
at org.alfresco.repo.web.scripts.solr.NodesMetaDataGet.executeImpl(NodesMetaDataGet.java:191)
at org.springframework.extensions.webscripts.DeclarativeWebScript.executeImpl(DeclarativeWebScript.java:235)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:64)
at org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:512)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:464)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:587)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:656)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScriptInternal(RepositoryContainer.java:355)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:308)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:399)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:210)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

Do anyone have any idea?

[afaust]

So what are the numbers really that we are talking about? How many nodes, transactions and ACLs do exist? "very less" is not an information that is specific enough....

Do you have any customisations / 3rd party addons installed in the system that may interfere in some way or even be prone to a memory leak?

[deepak9batra]

Nodes = 15000
Transactions = 50000

ACLs = 50000

No customization or add ons. 

[andy1]

Hi

There is not really much here to go on. I would suspect an in memory transformation from a giant pdf...

Andy 

[deepak9batra]

So is there any size limit in uploading documents in alfresco? 

how should I identify these large documents?

[afaust]

Best bet would be to create a heap dump and analyse that for the biggest consumers / dominator classes etc. to find the actual culprit.

One of my customers once had a memory issue dealing with inbound SMTP when one of the emails included a 600 MiB video in Base64 encoding, which was loaded in-memory. Another had an issue with a PDF catalog file containing ~500 other PDF files. Again, you should see this best in a heap dump where you could also investigate the context and identify the node being processed at the time.

[deepak9batra]

Heap analysis dump also does not hint out at anything major.

[afaust]

You have 33 instances of ArrayList with 3 GiB heap usage, and 3.5 GiB of String values in arrays (each apparently over 300 MiB). That certainly looks like a major issue. You need to check the path to GC roots to determine which components contain those data structures and for what they are used.