How to avoid book multiple tickets for async task

cancel
Showing results for 
Search instead for 
Did you mean: 
dean
Member II

How to avoid book multiple tickets for async task

I read the docs https://www.activiti.org/userguide/#_what_are_exclusive_jobs 

and saw the comment 

If a job has non-transactional side effects, those will not be rolled back by the failing transaction. For instance, if the "book concert tickets" service does not share the same transaction as Activiti, we might book multiple tickets if we retry the job.

Say we have some aysnc task that invoke webservice e.g. book ticket from webservice, in this kind of task if ActivitiOptimisticLockingException occurred, we may got 3 tickets. But we do want only 1 call. how to avoid retry for these scenario?

Plus, we still need retry for non-webserivce task, so how do we deal with it ?

Thx.

Actually, we already got lots of ActivitiOptimisticLockingException and Mysql Deadlock now.

Async Service Task executed multi-times in cluster  

3 Replies
warper
Established Member II

Re: How to avoid book multiple tickets for async task

Hi Dean!

There is workaround for simple scenarios - set exclusive flag for async step. Every critical section should be with exclusive flag, this way asyncExecutor locks process before execution writing timestamp of seizure. After each "critical" step use another async step, so process will be presisted right after successful booking of ticket. 

For long calls you may need to increase timerLockTimeInMillis, asyncJobLockTimeInMillis timers in asyncExecutor settings. Default timers can be not enough for some long thinking services.

Exclusive flag does not reduce overall performance of high load system, since overall resources are utilized. It does increase time for execution for particular processes, but while one process is doing its job in one thread, another thread can be used by another process.

There is another possibility - cache calls to outer services, stall/reject calls with identical parameters if there is one in progress, save results in separate DB on successful call, read result for second identical call instead of proceeding to outer service. You need to make proxies for outer services, make separate DB, design requests so that they allow you to identify if the request is repeated attempt or not.

The main drawback here is excessive successful request if later your process fails.

The proposed way to do processes is to make every service you use transactional and properly add rollback steps in process. So every time your second booking process execution fails, it calls corresponding booking web service to cancel excessive reservation. It's good idea to use this approach with exclusive flags anwyay. 

Retry for non-webservice tasks can happen naturally if you do DB manipulations in the same transaction as for activiti process, for example. 

P.S. MySql deadlock is not supposed to happen here, though.

dean
Member II

Re: How to avoid book multiple tickets for async task

Thank your for the reply, I have add redis counter (outer service) to avoid the duplicated ticket booking.

Regarding the deadlock, we got lots of and paste blew, it's really appreciate if you can give us some clue.

$ cat ~/Downloads/instanc1.log | grep 'Deadlock' -c
17945

-----------------------------

org.apache.ibatis.exceptions.PersistenceException:
### Error updating database. Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
### The error may involve org.activiti.engine.impl.persistence.entity.JobEntity.updateMessage-Inline
### The error occurred while setting parameters
### SQL: update ACT_RU_JOB SET REV_ = ?, LOCK_EXP_TIME_ = ?, LOCK_OWNER_ = ?, RETRIES_ = ?, EXCEPTION_STACK_ID_ = ?, EXCEPTION_MSG_ = ?, DUEDATE_ = ? where ID_= ? and REV_ = ?
--
### The error may involve org.activiti.engine.impl.persistence.entity.JobEntity.updateMessage-Inline
### The error occurred while setting parameters
### SQL: update ACT_RU_JOB SET REV_ = ?, LOCK_EXP_TIME_ = ?, LOCK_OWNER_ = ?, RETRIES_ = ?, EXCEPTION_STACK_ID_ = ?, EXCEPTION_MSG_ = ?, DUEDATE_ = ? where ID_= ? and REV_ = ?
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:172)
at org.activiti.engine.impl.db.DbSqlSession.flushUpdates(DbSqlSession.java:875)
at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:616)
at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:212)
at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:138)
at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:66)
at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
at org.activiti.engine.impl.asyncexecutor.AcquireAsyncJobsDueRunnable.run(AcquireAsyncJobsDueRunnable.java:52)
at java.lang.Thread.run(Thread.java:745)
--
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:172)
at org.activiti.engine.impl.db.DbSqlSession.flushUpdates(DbSqlSession.java:875)
at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:616)
at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:212)
at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:138)
at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:66)
at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
at org.activiti.engine.impl.asyncexecutor.AcquireAsyncJobsDueRunnable.run(AcquireAsyncJobsDueRunnable.java:52)
at java.lang.Thread.run(Thread.java:745)

### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:172)
at org.activiti.engine.impl.db.DbSqlSession.flushUpdates(DbSqlSession.java:875)
at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:616)
at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:212)
at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:138)
at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:66)
at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
at org.activiti.engine.impl.asyncexecutor.AcquireAsyncJobsDueRunnable.run(AcquireAsyncJobsDueRunnable.java:52)
at java.lang.Thread.run(Thread.java:745)
--
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:172)
at org.activiti.engine.impl.db.DbSqlSession.flushUpdates(DbSqlSession.java:875)
at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:616)
at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(Comman^CckException: Deadlock found when trying to get lock; try restarting transaction

warper
Established Member II

Re: How to avoid book multiple tickets for async task

I've seen something similar on WildFly 10 + Oracle, when messed with jta and ccm settings of server JDBC connection pool. Can't say I understand what I was doing there Smiley Sad Anyway, somehow it did work after few tries of available settings.

There was another case of deadlocks, JobExecutor and AsyncExecutor were active on the same database. 

Can't say anything special about your case, unfortunately - I have never run activiti on mySql.