Just as a starting point, take a gander at this thread - a previous user was attempting to do the same thing you have been, but after removing the configuration from engine.properties and only using the activiti-custom-context.xml they were able to get everything working.
Shout out to gurolakman _, for actually including his configuration files in the original thread also. Should be pretty simple to get it going
I tried to set the email properties in "activiti-custom-context.xml" but still its not working.
I was able to solve the issue with activiti-rest by referring to the thread Email configuration - Activiti REST . As per this thread email related logic is not present in class ActivitiEngineConfiguration.java. After adding the code i was able to run email functionality for activiti-rest.
So can you please confirm whether this is right approach? Or in order to get email functionality working no source code changes is required?