Alfresco will use the local host name / address to announce itself to the mail server via HELO. In a Docker environment, this will be some arbitrary, Docker-internal address. If your email server is configured to validate the sender host identity, it may fail and thus reject the sender. Alfresco does not provide a default configuration property to specify an alternative name (e.g. the Docker host's name) instead. But it is technically possible to configure a proper host name - it requires that the javaMailProperties (https://github.com/Alfresco/alfresco-repository/blob/master/src/main/resources/alfresco/subsystems/e...) be extended with a "mail.smtp.localhost" entry, which specifies the effective name to use for announcing Alfresco via HELO.
Alternatively, one might relax the HELO validation on the mail server, or use a relay / proxy on the Docker host instead of connecting directly.
It might be worth creating a JIRA for Alfresco to add such a configuration option by default. Apparently no one else so far encountered this kind of issue, which is actually quite natural for this new default Docker-deployment method.