Error in Manage > Jobs - "illegal access: this web application instance has stopped already. could not load {org.codehuse.goovy.runtime.callsite.SaticMetaMethodSite}."

[msaballa]

Hi guys,

In the activiti explorer under Manage > Jobs, I have a lot of failed Message job that says "illegal access: this web application instance has stopped already. could not load {org.codehuse.goovy.runtime.callsite.SaticMetaMethodSite}."

I'm not sure what this means really but I do remember that we initiated one process 3 times.

Do you guys know what this means? Is it because the initial process (probably the first one) was abandoned and these jobs are related to that?

Thanks in advance!

Regards,

Mark

[gdharley]

Not really enough to go on here.
Don't s'pose you have a full stack?

Thx,

Greg

‌

[msaballa]

Hi Greg,

Sorry about that. Here is the full stack.

01-18 10:21:41,204 [pool-4-thread-13] ERROR org.activiti.engine.impl.asyncexecutor.ExecuteAsyncRunnable  - Job 102188339 failed
java.lang.NoClassDefFoundError: org/codehaus/groovy/runtime/callsite/StaticMetaMethodSite
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
 at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.define(ClassLoaderForClassArtifacts.java:41)
 at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(ClassLoaderForClassArtifacts.java:74)
 at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(ClassLoaderForClassArtifacts.java:72)
 at java.security.AccessController.doPrivileged(Native Method)
 at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.defineClassAndGetConstructor(ClassLoaderForClassArtifacts.java:72)
 at org.codehaus.groovy.runtime.callsite.CallSiteGenerator.compileStaticMethod(CallSiteGenerator.java:243)
 at org.codehaus.groovy.reflection.CachedMethod.createStaticMetaMethodSite(CachedMethod.java:285)
 at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.createStaticMetaMethodSite(StaticMetaMethodSite.java:111)
 at groovy.lang.MetaClassImpl.createStaticSite(MetaClassImpl.java:3382)
 at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallStaticSite(CallSiteArray.java:74)
 at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:159)
 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:122)
 at Script37.run(Script37.groovy:5)
 at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:344)
 at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:145)
 at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
 at org.activiti.engine.impl.scripting.ScriptingEngines.evaluate(ScriptingEngines.java:87)
 at org.activiti.engine.impl.scripting.ScriptingEngines.evaluate(ScriptingEngines.java:73)
 at org.activiti.engine.impl.bpmn.behavior.ScriptTaskActivityBehavior.execute(ScriptTaskActivityBehavior.java:78)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationActivityExecute.execute(AtomicOperationActivityExecute.java:60)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerStart.java:52)
 at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:49)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionCreateScope.execute(AtomicOperationTransitionCreateScope.java:49)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerTake.execute(AtomicOperationTransitionNotifyListenerTake.java:80)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionDestroyScope.execute(AtomicOperationTransitionDestroyScope.java:116)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerEnd.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerEnd.java:35)
 at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:49)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.take(ExecutionEntity.java:453)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.take(ExecutionEntity.java:431)
 at org.activiti.engine.impl.bpmn.behavior.BpmnActivityBehavior.performOutgoingBehavior(BpmnActivityBehavior.java:140)
 at org.activiti.engine.impl.bpmn.behavior.BpmnActivityBehavior.performDefaultOutgoingBehavior(BpmnActivityBehavior.java:66)
 at org.activiti.engine.impl.bpmn.behavior.FlowNodeActivityBehavior.leave(FlowNodeActivityBehavior.java:44)
 at org.activiti.engine.impl.bpmn.behavior.FlowNodeActivityBehavior.execute(FlowNodeActivityBehavior.java:36)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationActivityExecute.execute(AtomicOperationActivityExecute.java:60)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationActivityStart.eventNotificationsCompleted(AtomicOperationActivityStart.java:26)
 at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.executeActivity(ExecutionEntity.java:458)
 at org.activiti.engine.impl.bpmn.behavior.SubProcessActivityBehavior.execute(SubProcessActivityBehavior.java:55)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationActivityExecute.execute(AtomicOperationActivityExecute.java:60)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerStart.java:52)
 at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:49)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:636)
 at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:629)
 at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionCreateScope.execute(AtomicOperationTransitionCreateScope.java:49)
 at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
 at org.activiti.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:35)
 at org.activiti.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:85)
 at org.activiti.engine.impl.persistence.entity.MessageEntity.execute(MessageEntity.java:34)
 at org.activiti.engine.impl.cmd.ExecuteAsyncJobCmd.execute(ExecuteAsyncJobCmd.java:52)
 at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
 at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
 at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)
 at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
 at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
 at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
 at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
 at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
 at org.activiti.engine.impl.asyncexecutor.ExecuteAsyncRunnable.executeJob(ExecuteAsyncRunnable.java:96)
 at org.activiti.engine.impl.asyncexecutor.ExecuteAsyncRunnable.run(ExecuteAsyncRunnable.java:50)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1318)
 at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1181)
 at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1142)
 at org.codehaus.groovy.runtime.callsite.CallSiteClassLoader.loadClass(CallSiteClassLoader.java:45)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.loadClass(ClassLoaderForClassArtifacts.java:57)
 ... 106 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1328)
 at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1316)
 ... 111 more

Although, it looked like this problem started with a connection issue to a 3rd party application. The activiti process connects to the 3rd party application via web service call. The web service call to the 3rd party application was not responding due to disk space. When the disk space issue was resolved, the activiti process didn't seem to re-establish the connection back to the web service and the jobs were not processed automatically. I'm not sure what happened there but I do notice that clicking the "Execute" button manually does the trick. However, there were a lot of jobs in queue.

I had to undeploy my BAR file from activity (because I want the processes to be removed) and restart the tomcat (this is where activiti sits currently) and redeploy the BAR. Afterwards, everything was working fine.

[gdharley · Accepted solution]

It looks as though a service task (likely a script task) was trying to persist state during shutdown at some time.

It will try three times then give up.

Because the scripting engine could be any JSR-223 compatible module, reflection is used to retrieve the specific engine for the language selected. Likely the classnotfound exception comes from the static classes being unloaded before the job engine shuts down.

I think you can just delete these jobs as the process instance state should have been reset.

Hope this helps,

Greg

‌

[msaballa]

Thanks Greg. It does seem that way. The jobs were deleted automatically when I undeployed the BAR file so that's alright now.

I appreciate your help though!