NullPointerExceptions for parallel & asynchronous continuation

cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
sdeutschmann
Member II

NullPointerExceptions for parallel & asynchronous continuation

Hello everybody!

We have problems under heavy load (8 requests per second) with our setup. Currently we don't know, what is the cause of the problem, but I will describe our observations. We are using Activiti 5.21.


1. NullPointerExceptions in logfile
2017-05-08T10:40:34.255 ERROR [pool-4-thread-9] Job 46568299 failed
java.lang.NullPointerException: null
at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:96)
at org.activiti.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:35)
at org.activiti.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:85)
at org.activiti.engine.impl.persistence.entity.MessageEntity.execute(MessageEntity.java:34)
at org.activiti.engine.impl.cmd.ExecuteAsyncJobCmd.execute(ExecuteAsyncJobCmd.java:52)
at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
at org.activiti.engine.impl.asyncexecutor.ExecuteAsyncRunnable.executeJob(ExecuteAsyncRunnable.java:95)
at org.activiti.engine.impl.asyncexecutor.ExecuteAsyncRunnable.run(ExecuteAsyncRunnable.java:50)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2. Jobs in act_ru_job table with less than 0 retries_ (down to -8)
3. These jobs contain the message "JobEntity [id=4431434] was updated by another transaction concurrently"

Exception Stacktrance found in DB:

org.activiti.engine.ActivitiOptimisticLockingException: JobEntity [id=4431434] was updated by another transaction concurrently
at org.activiti.engine.impl.db.DbSqlSession$CheckedDeleteOperation.execute(DbSqlSession.java:298)
at org.activiti.engine.impl.db.DbSqlSession.flushRegularDeletes(DbSqlSession.java:929)
at org.activiti.engine.impl.db.DbSqlSession.flushDeletes(DbSqlSession.java:895)
at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:620)
at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:212)
at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:138)
at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:66)
at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
at org.activiti.engine.impl.asyncexecutor.ExecuteAsyncRunnable.executeJob(ExecuteAsyncRunnable.java:95)
at org.activiti.engine.impl.asyncexecutor.ExecuteAsyncRunnable.run(ExecuteAsyncRunnable.java:50)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

We have one Activiti engine running, with a configured AsyncJobExecutor. The process itself looks as following:
3 parallel subprocesses, each of them contains a JavaServiceTask and UserTask. The ServiceTasks are all configured as async = true.
Before and after these parallel processes we have parallel gateways to fork/join. The first parallel gateway is also set to async = true.
All elements are also exclusive.

I have the feeling, that our configuration is not well set, because 99,9 % of the failing jobs are on the first parallel gateway. But currently I simply don't understand why this happens. I read the section about AsyncJobExecutor as well as the one about Exclusive Jobs. With the knowledge I gained there, I would expect, that such concurrency cannot happen.

Any hint is appreciated.

Best wishes,
Steffi

4 Replies
gdharley
Intermediate

Re: NullPointerExceptions for parallel & asynchronous continuation

You will need to use the exclusive flag on any service tasks that flow into the join.

Likely this is the cause of your problem.

Greg

sdeutschmann
Member II

Re: NullPointerExceptions for parallel & asynchronous continuation

Hi Greg,

I already have the exclusive flag set on all my service tasks. That's why I don't understand why that happens. I would expect, that all tasks marked as exclusive are executed by one thread, so that I don't get problems with concurrency.

Regards, Steffi

jdp
Active Member

Re: NullPointerExceptions for parallel & asynchronous continuation

Hi,

I am facing same type of problem, is this resolved ? if yes, can you please share fix for the same.

Thanks

sdeutschmann
Member II

Re: NullPointerExceptions for parallel & asynchronous continuation

Hi Paresh,

in our case it seemed to be indeed wrong configuration. Accidentally we set lower values than the default ones for these variables (we forgot one 0):

  • activiti.asyncExecutor.timer.lockTime.millis
  • activiti.asyncExecutor.asyncJob.lockTime.millis

When we were setting them to the default value of 300.000, everything runs smooth.

One other problem we had at the beginning was caused by our database configuration: It was not vacuumed regularly.

Maybe it helps :-)