Alfresco is frequently getting hung due to Java-level deadlock

[fouada]

Hi All,

I am using Alfresco 4.1 and it is getting hung frequently .It is production environment and have two legs

and load balancer is configured using round robbin mechanism.

I have analysed logs and found deadlock errors.

Please guide me how to investigate for the cause these errors.

Below are the errors getting in catalina.out

"SolrTrackerScheduler_QuartzSchedulerThread" daemon prio=10 tid=0x00002b76187c4000 nid=0x3dd8 in Object.wait() [0x00002b76152a6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:296)
- locked <0x0000000740b08480> (a java.lang.Object)

"SolrTrackerScheduler_Worker-3" daemon prio=10 tid=0x00002b761858b800 nid=0x3dd7 runnable [0x00002b76151a5000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:830)
- locked <0x00000007f3966978> (a java.lang.Object)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:787)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
- locked <0x00000007f3967258> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x00000007f3967270> (a java.io.BufferedInputStream)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:130)
at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:106)
at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:356)
at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1035)
at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1851)
at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1137)
at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:483)
at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

"SolrTrackerScheduler_Worker-2" daemon prio=10 tid=0x00002b76185b5000 nid=0x3dd6 in Object.wait() [0x00002b76150a4000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
- locked <0x0000000740afd8e0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)

"SolrTrackerScheduler_Worker-1" daemon prio=10 tid=0x00002b76187f3800 nid=0x3dd5 runnable [0x00002b7614fa2000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:830)
- locked <0x00000007f280a328> (a java.lang.Object)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:787)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
- locked <0x00000007f280b210> (a com.sun.net.ssl.internal.ssl.AppInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x00000007f280f750> (a java.io.BufferedInputStream)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:130)
at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:106)
at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:356)
at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1035)
at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1851)
at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1137)
at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:483)
at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

"RMI TCP Accept-0" daemon prio=10 tid=0x00000000544f9000 nid=0x3d6a runnable [0x00002b7614488000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <0x0000000740b10c10> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x000000005435f000 nid=0x3d69 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x000000005435c800 nid=0x3d68 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x0000000054357000 nid=0x3d67 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0000000054355000 nid=0x3d66 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0000000054333800 nid=0x3d65 in Object.wait() [0x00002b76106f1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x0000000740aeec98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x0000000054331800 nid=0x3d64 in Object.wait() [0x00002b76105f0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x0000000740508c08> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00000000542ca800 nid=0x3d60 runnable [0x00002b760bc7c000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <0x0000000750dd3dc0> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:431)
at org.apache.catalina.startup.Catalina.await(Catalina.java:676)
at org.apache.catalina.startup.Catalina.start(Catalina.java:628)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

"VM Thread" prio=10 tid=0x000000005432a800 nid=0x3d63 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000542dd800 nid=0x3d61 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000542df800 nid=0x3d62 runnable

"VM Periodic Task Thread" prio=10 tid=0x00000000543ed000 nid=0x3d6b waiting on condition

JNI global references: 1641

Found one Java-level deadlock:
=============================
"http-8080-200":
waiting for ownable synchronizer 0x0000000747965898, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "http-8443-14"
"http-8443-14":
waiting for ownable synchronizer 0x0000000747350d08, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "http-8080-12"
"http-8080-12":
waiting for ownable synchronizer 0x0000000747705928, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "http-8080-7"
"http-8080-7":
waiting for ownable synchronizer 0x0000000747350d08, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "http-8080-12"

Java stack information for the threads listed above:
===================================================
"http-8080-200":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000747965898> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at org.alfresco.repo.web.scripts.RepositoryContainer.getRegistry(RepositoryContainer.java:541)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:164)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:877)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:594)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1675)
at java.lang.Thread.run(Thread.java:662)
"http-8443-14":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000747350d08> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getDictionaryRegistry(DictionaryDAOImpl.java:1040)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getDictionaryRegistry(DictionaryDAOImpl.java:1022)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getUriToModels(DictionaryDAOImpl.java:1206)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getModelsForUri(DictionaryDAOImpl.java:462)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getProperty(DictionaryDAOImpl.java:737)
at org.alfresco.repo.dictionary.DictionaryComponent.getProperty(DictionaryComponent.java:357)
at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.getValue(NodeRefPropertyMethodInterceptor.java:289)
at org.alfresco.repo.node.NodeRefPropertyMethodInterceptor.invoke(NodeRefPropertyMethodInterceptor.java:199)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy14.getProperty(Unknown Source)
at sun.reflect.GeneratedMethodAccessor315.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:196)
at $Proxy14.getProperty(Unknown Source)
at org.alfresco.service.cmr.repository.Path.toDisplayPath(Path.java:209)
at org.alfresco.repo.web.scripts.RepoStore.getPath(RepoStore.java:297)
at org.alfresco.repo.web.scripts.RepoStore.getBaseDir(RepoStore.java:262)
at org.alfresco.repo.web.scripts.RepoStore.access$000(RepoStore.java:73)
at org.alfresco.repo.web.scripts.RepoStore$3$1.execute(RepoStore.java:434)
at org.alfresco.repo.web.scripts.RepoStore$3$1.execute(RepoStore.java:431)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
at org.alfresco.repo.web.scripts.RepoStore$3.doWork(RepoStore.java:430)
at org.alfresco.repo.web.scripts.RepoStore$3.doWork(RepoStore.java:427)
at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:529)
at org.alfresco.repo.web.scripts.RepoStore.getDocumentPaths(RepoStore.java:426)
at org.springframework.extensions.webscripts.DeclarativeRegistry.initWebScripts(DeclarativeRegistry.java:223)
at org.springframework.extensions.webscripts.DeclarativeRegistry.reset(DeclarativeRegistry.java:178)
at org.alfresco.repo.web.scripts.RepositoryContainer.getRegistry(RepositoryContainer.java:565)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:164)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
"http-8080-12":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000747705928> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at org.alfresco.cmis.dictionary.CMISAbstractDictionaryService.init(CMISAbstractDictionaryService.java:409)
at org.alfresco.cmis.dictionary.CMISAbstractDictionaryService.afterDictionaryInit(CMISAbstractDictionaryService.java:512)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.initDictionaryRegistry(DictionaryDAOImpl.java:274)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.access$000(DictionaryDAOImpl.java:58)
at org.alfresco.repo.dictionary.DictionaryDAOImpl$1.doWork(DictionaryDAOImpl.java:214)
at org.alfresco.repo.dictionary.DictionaryDAOImpl$1.doWork(DictionaryDAOImpl.java:209)
at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:529)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.initDictionary(DictionaryDAOImpl.java:208)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getDictionaryRegistry(DictionaryDAOImpl.java:1080)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.init(DictionaryDAOImpl.java:148)
at org.alfresco.repo.dictionary.NamespaceDAOImpl.getNamespaceRegistry(NamespaceDAOImpl.java:397)
at org.alfresco.repo.dictionary.NamespaceDAOImpl.getPrefixesCtx(NamespaceDAOImpl.java:560)
at org.alfresco.repo.dictionary.NamespaceDAOImpl.getPrefixesCtx(NamespaceDAOImpl.java:547)
at org.alfresco.repo.dictionary.NamespaceDAOImpl.getNamespaceURI(NamespaceDAOImpl.java:297)
at org.alfresco.repo.dictionary.DictionaryNamespaceComponent.getNamespaceURI(DictionaryNamespaceComponent.java:73)
at org.alfresco.service.namespace.QName.createQName(QName.java:96)
at org.alfresco.service.namespace.QName.createQName(QName.java:121)
at org.alfresco.repo.descriptor.RepositoryDescriptorDAOImpl$RepositoryDescriptor.getDescriptor(RepositoryDescriptorDAOImpl.java:522)
at org.alfresco.repo.descriptor.RepositoryDescriptorDAOImpl$RepositoryDescriptor.getName(RepositoryDescriptorDAOImpl.java:474)
at org.alfresco.repo.web.scripts.RepositoryServerModel.getName(RepositoryServerModel.java:67)
at sun.reflect.GeneratedMethodAccessor2171.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at freemarker.ext.beans.BeansWrapper.invokeMethod(BeansWrapper.java:870)
at freemarker.ext.beans.BeanModel.invokeThroughDescriptor(BeanModel.java:277)
at freemarker.ext.beans.BeanModel.get(BeanModel.java:184)
at freemarker.core.Dot._getAsTemplateModel(Dot.java:76)
at freemarker.core.Expression.getAsTemplateModel(Expression.java:89)
at freemarker.core.Expression.getStringValue(Expression.java:93)
at freemarker.core.StringBuiltins$StringBuiltIn._getAsTemplateModel(StringBuiltins.java:71)
at freemarker.core.Expression.getAsTemplateModel(Expression.java:89)
at freemarker.core.Expression.getStringValue(Expression.java:93)
at freemarker.core.DollarVariable.accept(DollarVariable.java:76)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.Environment.process(Environment.java:199)
at org.alfresco.repo.template.FreeMarkerProcessor.process(FreeMarkerProcessor.java:202)
at org.springframework.extensions.webscripts.AbstractWebScript.renderTemplate(AbstractWebScript.java:879)
at org.springframework.extensions.webscripts.DeclarativeWebScript.renderFormatTemplate(DeclarativeWebScript.java:267)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:147)
at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:417)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:479)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:517)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:333)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:377)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:877)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:594)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1675)
at java.lang.Thread.run(Thread.java:662)
"http-8080-7":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000747350d08> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:941)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getDictionaryRegistry(DictionaryDAOImpl.java:1040)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getDictionaryRegistry(DictionaryDAOImpl.java:1022)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getCompiledModels(DictionaryDAOImpl.java:1182)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getCompiledModels(DictionaryDAOImpl.java:869)
at org.alfresco.repo.dictionary.DictionaryDAOImpl.getModels(DictionaryDAOImpl.java:798)
at org.alfresco.repo.dictionary.DictionaryComponent.getAllModels(DictionaryComponent.java:67)
at org.alfresco.repo.dictionary.DictionaryComponent.getAllTypes(DictionaryComponent.java:111)
at org.alfresco.cmis.dictionary.CMISStrictDictionaryService.createDefinitions(CMISStrictDictionaryService.java:48)
at org.alfresco.cmis.dictionary.CMISAbstractDictionaryService.init(CMISAbstractDictionaryService.java:418)
at org.alfresco.cmis.dictionary.CMISAbstractDictionaryService.getRegistry(CMISAbstractDictionaryService.java:233)
at org.alfresco.cmis.dictionary.CMISAbstractDictionaryService.findProperty(CMISAbstractDictionaryService.java:361)
at org.alfresco.cmis.mapping.CMISServicesImpl.getProperty(CMISServicesImpl.java:901)
at org.alfresco.repo.cmis.rest.CMISPropertyValueMethod.exec(CMISPropertyValueMethod.java:118)
at freemarker.core.MethodCall._getAsTemplateModel(MethodCall.java:93)
at freemarker.core.Expression.getAsTemplateModel(Expression.java:89)
at freemarker.core.Assignment.accept(Assignment.java:90)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.Macro$Context.runMacro(Macro.java:172)
at freemarker.core.Environment.visit(Environment.java:614)
at freemarker.core.UnifiedCall.accept(UnifiedCall.java:106)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.MixedContent.accept(MixedContent.java:92)
at freemarker.core.Environment.visit(Environment.java:221)
at freemarker.core.Environment.process(Environment.java:199)
at org.alfresco.repo.template.FreeMarkerProcessor.process(FreeMarkerProcessor.java:202)
at org.springframework.extensions.webscripts.AbstractWebScript.renderTemplate(AbstractWebScript.java:879)
at org.springframework.extensions.webscripts.DeclarativeWebScript.renderFormatTemplate(DeclarativeWebScript.java:267)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:147)
at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:417)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:479)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:517)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:333)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:377)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:877)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:594)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1675)
at java.lang.Thread.run(Thread.java:662)

Found 1 deadlock.

Heap
PSYoungGen total 846784K, used 465884K [0x00000007c0000000, 0x00000007fb170000, 0x0000000800000000)
eden space 810432K, 53% used [0x00000007c0000000,0x00000007da37b8d0,0x00000007f1770000)
from space 36352K, 99% used [0x00000007f1770000,0x00000007f3aeb890,0x00000007f3af0000)
to space 78656K, 0% used [0x00000007f64a0000,0x00000007f64a0000,0x00000007fb170000)
PSOldGen total 2097152K, used 2054806K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
object space 2097152K, 97% used [0x0000000740000000,0x00000007bd6a58b0,0x00000007c0000000)
PSPermGen total 240320K, used 240197K [0x0000000720000000, 0x000000072eab0000, 0x0000000740000000)
object space 240320K, 99% used [0x0000000720000000,0x000000072ea914a8,0x000000072eab0000)

[andy1]

Hi

This looks like an issue from the dim and distant past. What is the exact version you are using?
I suspect you can get this fixed by moving to a supported version or a more recent supported/community version.

Andy

[fouada]

Hi Andy,

Thanks for your suggestion. I am using alfresco-enterprise-4.1.4. 

So it is a bug of Alfresco. How can I confirm it is failing due to Alfresco and not by our custom code.

Thanks,

Fouad

[fouada]

HI All,

Same error came again. Is there any way other than migrating to new version.

Thanks,

Fouad

[andy1]

Hi

It looks like [MNT-11098] 'Failed to get lock java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock' error oc...  or similar
The fix is to get the fix....

Are you adding models?

Andy

[fouada]

Hi Andy,

Yes, We have added Custom content model.

Fouad