Alfresco Photo Analyzer Module

[david_anton]

By popular request, I'm going to explain a bit the project that we developed for the last Global Hack-a-thon. The purpose of this project was simply to develop something different using Alfresco as content management and integrate it with external processes or tools.

This Alfresco module analyze people photos and extract information about the gender, age, face expressions, emotions and others. So you can make a bulk photo import to Alfresco, and then make searches(for example, to know how many 30 age people photos are in the repository).

The list of possible information to extract from a photo is:

• Number of persons
• Ages
• Gender  
• Hair      
• Facial Hair   
• Emotions        
• Accessories   

The result you will see is a box with the properties extracted from the metadata photo. 

And a friendly search interface.

For more  details about the user experience, you can check the Youtube Demo.

For this module we integrated Microsoft Azure with Alfresco 5.2 community version.

To configure the Azure API connection, change the following properties in the alfresco-global.properties file.

// Depending on the geographical zone that you are   photo_analyzer.azure.api.url=https://westeurope.api.cognitive.microsoft.com/face/v1.0/detect
// List of attributes that we want to analyze in the response   photo_analyzer.azure.api.attribute_list=age,gender,emotion,hair,facialHair,accessories,glasses
// Subscription key trial(*) https://azure.microsoft.com/es-es/try/cognitive-services/ 
photo_analyzer.azure.api.subscription_key=<subsription-key>

(*) Limitations of Trial subscription key: 30.000 transactions, 20 per minute.

And about technical things, to extract all the information from the photos, we created a behavior that was executed when a new photo is uploaded/created in the repository. The execution of this behaviour was based on the integration with Microsoft Cognitive Services, so we needed to make a request and collected a response from the external service.

The following method is for construct the request to Azure:

public HttpPost prepareRequest(File file) throws URISyntaxException {
    URIBuilder builder = new URIBuilder(Constants.getPropertyValue(Constants.AZURE_FACE_API_URL));
    builder.setParameter("returnFaceId", "true");
    builder.setParameter("returnFaceRectangle", "true");
    builder.setParameter("returnFaceAttributes", Constants.getPropertyValue(Constants.FACE_ATTRIBUTES_LIST_TO_EXTRACT));

    URI uri = builder.build();
    HttpPost request = new HttpPost(uri);
    request.setHeader("Content-Type", Constants.APPLICATION_OCTET_STREAM);
    request.setHeader("Ocp-Apim-Subscription-Key", Constants.getPropertyValue(Constants.API_SUBSCRIPTION_KEY));

    FileEntity reqEntity = new FileEntity(file, ContentType.APPLICATION_OCTET_STREAM);
    request.setEntity(reqEntity);
    return request;
}‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Then we only need to analyze the JSON response and match with each metadata field of our Alfresco model.

[
    {
        "faceId": "c5c24a82-6845-4031-9d5d-978df9175426",
        "faceRectangle": {
            "width": 78,
            "height": 78,
            "left": 394,
            "top": 54
        },
        "faceLandmarks": {
            "pupilLeft": {
                "x": 412.7,
                "y": 78.4 
            },
            "pupilRight": {...},
            "noseTip": {...},
            "mouthLeft": {...},
            ...
            "underLipBottom": {...}
        },
        "faceAttributes": {
            "age": 71.0,
            "gender": "male",
            "smile": 0.88,
            "facialHair": {
                "moustache": 0.8,
                "beard": 0.1,
                "sideburns": 0.02
            },
            "glasses": "sunglasses",
            "headPose": {
                "roll": 2.1,
                "yaw": 3,
                "pitch": 0
            },
            "emotion":{
                "anger": 0.575,
                "contempt": 0,
                "disgust": 0.006,
                "fear": 0.008,
                "happiness": 0.394,
                "neutral": 0.013,
                "sadness": 0,
                "surprise": 0.004
            },
            "hair": {
                "bald": 0.0,
                "invisible": false,
                "hairColor": [
                    {"color": "brown", "confidence": 1.0},
                    {"color": "blond", "confidence": 0.88},
                    {"color": "black", "confidence": 0.48},
                    {"color": "other", "confidence": 0.11},
                    {"color": "gray", "confidence": 0.07},
                    {"color": "red", "confidence": 0.03}
                ]
            },
            "makeup": {
                "eyeMakeup": true,
                "lipMakeup": false
            },
            "occlusion": {
                "foreheadOccluded": false,
                "eyeOccluded": false,
                "mouthOccluded": false
            },
            "accessories": [
                {"type": "headWear", "confidence": 0.99},
                {"type": "glasses", "confidence": 1.0},
            ],
            "blur": {
                "blurLevel":"Medium",
                "value":0.51
            },
            "exposure": {
                "exposureLevel":"GoodExposure",
                "value":0.55
            },
            "noise": {
                "noiseLevel":"Low",
                "value":0.12
            }
        }
    }
]‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

In this blog post you saw how Alfresco Photo Analyzer Module works, integrating Alfresco Content Services and the Microsoft Cognitive Services. I attach you a list of useful links to find more material about this topic:

Face API: Reconocimiento facial | Microsoft Azure 

Emotion API: detector de emociones | Microsoft Azure 

Documentación de Microsoft Azure | Microsoft Docs 

Azure Code Samples | Microsoft Azure 

And some links about the project:

GitHub: https://github.com/davidantonlou/alfrescoPhotoAnalyzer

Youtube Demo: http://youtu.be/TZdIdTk6D3U?t=20m32s

Other related projects using Google API:

GitHub - keensoft/alfresco-google-vision: Google Vision API integration in Alfresco 

GitHub - deas/contentreich-ecm-google-ms: Clojure Microservice for Google