You can use a self-signed certificate for your Alfresco inbound email server capabilties IF you know for sure that all email servers that will route emails to Alfresco accept a self-signed certificate. E.g. one of the ways I have used the inbound email server in the past is by having a Microsoft Exchange mail server - which dealt with all the emails for the customer domain - forward emails for specific addresses to Alfresco. The Exchange server was under the control of the customer and configured to accept self-signed certificates from the Alfresco server, and all the publich email servers would only talk with the Exchange mail server, which used a proper, publicly signed certificate.
I am jus saying that using a self-signed certificate for Alfresco in a situation where any mail server can send emails to it might not work when those mail servers don't accept self-signed certificates. E.g. my company email server has been set up in a way to not accept self-signed certificates at all (when using TLS).
I googled a little bit and this is how Gmail works:
"You have most likely enabled inbound TLS within the MailEnable SMTP properties window under the "General" tab which is failing. Try and disable the TLS option for SMTP and then try to send from Gmail again to your server and Gmail will fall back to use non TLS. TLS requires that you set a valid SSL certificate within MailEnable do be used for the TLS connection."
You can disable TLS for inbound SMTP by setting email.server.enableTLS=false
But honestly: In 2017 no one should sacrifice security for the reason of being too lazy to generate a properly signed certificate, especially when those are available for free from Let's Encrypt. I use a certificate from Let's Encrypt for my mail server and it is extremely easy to create / renew the certification.