ACS 7.4 Community keystore issue

[atTheBeach]

Yet another keystore question. These are everywhere, and Abhinav has gone through this in detail on his blog, but something seems to be missing.

I've got mTLS working, but the issue is that the Alfresco log is complaining about its own encryption keystore:

org.alfresco.error.AlfrescoRuntimeException: 08280000 Keystores are invalid
...
Caused by: org.alfresco.encryption.InvalidKeystoreException: The key with alias metadata has been changed, re-instate the previous keystore

This is the community edition, so there's no encryption, and I'm using the default keystore from alfresco-content-services-community-distribution-7.4.0.zip. This is two files:

-rw-r--r-- 1 foo foo 645 May  3 20:59 keystore
-rw-r--r-- 1 foo foo 359 May  3 20:59 keystore-passwords.properties

where 'keystore' is a 'Java JCE KeyStore'. These two files are copied to the server inside directory /opt/tomcat/updated/shared/classes/alfresco/extension/keystore/metadata-keystore/, and:

# shared/classes/alfresco-global.properties
dir.keystore=/opt/tomcat/updated/shared/classes/alfresco/extension/keystore

# bin/catalina.sh
export JAVA_TOOL_OPTIONS="\                                                                                                                                                                             
 -Dencryption.keystore.type=JCEKS \                                                                                                                                                                     
 -Dencryption.cipherAlgorithm=DESede/CBC/PKCS5Padding \                                                                                                                                                 
 -Dencryption.keyAlgorithm=DESede \                                                                                                                                                                     
 -Dencryption.keystore.location=/opt/tomcat/updated/shared/classes/alfresco/extension/keystore/metadata-keystore/keystore \                                                                             
 -Dmetadata-keystore.password=mp6yc0UD9e \                                                                                                                                                              
 -Dmetadata-keystore.aliases=metadata \                                                                                                                                                                 
 -Dmetadata-keystore.metadata.password=oKIWzVdEdA \                                                                                                                                                     
 -Dmetadata-keystore.metadata.algorithm=DESede"

In other words, JAVA_TOOL_OPTIONS points to the actual JCE keystore, and dir.keystore points to two directories above this keystore. This doesn't work, giving the error above. There are lots of variations on the net of exactly what JAVA_TOOL_OPTIONS and dir.keystore should point to (ie. the JCE KeyStore, or the metadata directory,  or the keystore directory above the metadata directory), and I think I've tried pretty much all of these. I've also tried locating the keystore directory in alf_data, but I think this is now obsolete.

I'm pretty sure that I have had this working recently, with a very minor change somewhere, but I don't know what. I can't find any way to convince ACS that I'm just using the default keystore and haven't changed anything. Any ideas?

[atTheBeach · Accepted solution]

Fixed. The procedure above works, but you have to be careful not to start ACS too early during the install, because it may then automatically create its own keystore, which is inconsistent with your setup. You can recover from any of these keystore issues by flushing the database, which was the missing link.