First of all your mail server seems to be using a corporate certificate issued by a Certificate Authority which is unknown to the JVM. It could be a certificate issued by a private PKI, a public cert from a somewhat "exotic" CA, or a self-signed certificate. In any cases, you would need to make sure the Alfresco JVM knows about the self-signed certificate or the signing CA by importing the certificate (either the self-signed one or the public certificate from the signing CA) inside the java keystore set as the truststore.
You'll find plenty of tutorial on internet on how to use keytool for that.
Second, I see you're using an SMTPS server on port 465. Usually mail servers talking on port 465 are using SSL wrapping more that TLS. So you should not need to `mail.smtps.starttls.enable`... However this depends really ooon how your own server is configured and doesn't seem related with your exception.
To be honest, I don't understand with certificate things you said. LOL. But as I know, my mail server is not using any certificate, just a free mail server (postfix server). Do you have other suggestions?
For the second point,I'll delete it. I just copy it from the internet, I don't know it's not necessary.
Check the content of the /tmp/mail.mycompany.com.crt, it should contain a PEM encoded certificate you can later import in the JVM using keytool (this command will prompt you for a password, you can find it in alf_data/keystore/ssl-truststore-passwords.properties):
The commands above are making a lot of assumptions and may not work as is on your setup.
If you're not confortable with what's above, the simple way is to go without encryption, but of course it means your server has to allow it and you are fine with your emails going clear text over the wire... including username and password for SMTP authentication (if any)!
In this case you need to find out the right TCP port for you mail server unencrypted communicatios (usually port 25) and switch protocol to "smtp" instead of "smtps"
Note that you anyway need to understand how your SMTP server is configured.