Search stops due to solr4 not working properly

hardik_thakkar · ‎9 Oct 2020

Hi Team,

I have restarted server and solr4 stops working:
Below are the error logs from solr.log file:

2020-10-09 02:18:32,330 ERROR [org.alfresco.solr.tracker.AbstractTracker] Tracking failed
org.alfresco.solr.TrackerStateException: 09090000 The trackers work was rolled back by another tracker error
	at org.alfresco.solr.SolrInformationServer.canUpdate(SolrInformationServer.java:3238)
	at org.alfresco.solr.SolrInformationServer.commit(SolrInformationServer.java:773)
	at org.alfresco.solr.tracker.ContentTracker.doTrack(ContentTracker.java:111)
	at org.alfresco.solr.tracker.AbstractTracker.track(AbstractTracker.java:190)
	at org.alfresco.solr.tracker.TrackerJob.execute(TrackerJob.java:54)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
2020-10-09 03:34:45,363 ERROR [org.alfresco.solr.tracker.AbstractTracker] Tracking failed
java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:210)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)

Below are the logs from alfresco.log file and based on the logs I can see solr backup job runs everyday at 2 AM and that is failing, I have verified that solr backup was done 3 days back.

2020-10-09 02:00:41,110 ERROR [org.quartz.core.JobRunShell] [SearchScheduler_Worker-2] Job DEFAULT.search.alfrescoCoreBackupJobDetail threw an unhandled Exception: 
org.alfresco.error.AlfrescoRuntimeException: 09091799 Backup for core alfresco failed .... 
	at org.alfresco.repo.search.impl.solr.SolrBackupClient.executeImpl(SolrBackupClient.java:203)
	at org.alfresco.repo.search.impl.solr.SolrBackupClient.execute(SolrBackupClient.java:140)
	at org.alfresco.repo.search.impl.solr.SolrBackupJob.execute(SolrBackupJob.java:65)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.apache.solr.client.solrj.SolrServerException: java.net.SocketTimeoutException: Read timed out
	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:483)
	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
	at org.alfresco.repo.solr.SOLRAdminClient.query(SOLRAdminClient.java:203)
	at org.alfresco.repo.search.impl.solr.SolrBackupClient.executeImpl(SolrBackupClient.java:192)
	... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
	at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)

For the socket time out exception, I have added below configuration in custom-slingshot-application-context.xml (/tomcat/shared/classes/alfresco/web-extension)

<bean id="connector.remoteclient" parent="connector.remoteclient.abstract" 
  class="org.alfresco.web.scripts.SlingshotRemoteClient" scope="prototype" > 
<!-- the http.connection.timeout value in milliseconds to apply to HTTP connections  --> 
    <property name="connectTimeout"><value>420000</value></property>
<!-- the http.socket.timeout value in milliseconds to apply to HTTP connections    -->
	<property name="readTimeout"><value>420000</value></property> 
</bean>

In the alfresco.log, I can see read timed out error frequently.

What is the maximum timeout values we can set?
I have Postgres DB on other node, does that creates any issue?

I have set solr.suggester.enabled=false in solrcore.properties file before restart, does that creates any issue?

Need help/guidance to troubleshoot this issue.

Thanks,

Hardik

hardik_thakkar · ‎12 Oct 2020

Does anyone has faced this type of issue?

vincent-kali · ‎13 Oct 2020

Did you try to:

- stop alf

- remove existing model data from alf_data/solr4/model folder

- restart

(model files will be reloaded by solr model tracker)

hardik_thakkar · ‎15 Oct 2020

Hi @vincent-kali

I have tried the steps suggested by you, after restart models are reloaded in alf_data/solr4/model folder.

After restart solr works properly for sometime and then it stops working.
I have checked alfresco.log and solr.log file, it continuesly printing connection timeout error.