AsyncExecutor multiple instances causing deadlock

[francop]

Hi, We are currently experiencing out of memory(OOM) situation in our application. Thread dump shows blocked threads on activiti's AsyncJobExecutor with 2 instances of the same being created concurrently to acquire the jobs, blocking on each other causing deadlock situation. This results in spinning numerous timer threads in timed waiting state and eventually crashing the app with out of memory conditions. 5.20.0 is the activiti version we use.  Can you please shed some light on why 2 instances of async threads are created and provide a solution on how to fix this problem?.

Attaching screenshot showing 2 threads created by AquireTimerJobsDueRunnable  and AquireAsyncJobsDueRunnable and are blocked on each other while acquiring timer jobs from DB using UCP pooling.

[gdharley]

Hello Franco,

An interesting problem and certainly not one I have ever come across, which suggests it to be an environmental issue.
The fact that Activiti creates two threads that both monitor the Job Queue (ACT_RU_JOBS) is completely expected and in fact a performance enhancement made around the 5.17 release. Prior to this, a single Job Executor thread was responsible for both async and timer jobs.

The change introduced a second thread, now we have a thread managing async jobs and a second managing timer jobs. Both share the ACT_RU_JOBS table where lock details, retry counts etc are stored.

Likely this has something to do with your Oracle connection pool setup, transaction isolation level or something like that but without a full repro environment I can't really give you much more than that.

Greg

‌

[francop]

Thanks for the response Greg. I will dig into at our end to see if we can find something.