AWS DeepLens TechConnect IoT Rule

AWS DeepLens: Creating an IoT Rule (Part 2 of 3)

This post is the second in a series on getting started with the AWS DeepLens. In Part 1, we introduced a program that could detect faces and crop them by extending the boilerplate Greengrass Lambda and pre-built model provided by AWS. This focussed on the local capabilities of the device, but the DeepLens device is much more than that. At its core, DeepLens is a fully fledged IoT device, which is just one part of the 3 Pillars of IoT: devices, cloud and intelligence.

All code and templates mentioned can be found here. This can be deployed using AWS SAM, which helps reduces the the complexity for creating event-based AWS Lambda functions.

Sending faces to IoT Message Broker

AWS DeepLens Device Page

AWS DeepLens Device Console Page

When registering a DeepLens device, AWS creates all things associated with the IoT cloud pillar . If you have a look for yourself in the IoT Core AWS console page, you will see existing IoT groups, devices, certificates, etc.. This all simplifies the process of interacting with the middle-man, the MQTT topic that is displayed on the main DeepLens device console page. The DeepLens (and others if given authorization) has the right to publish messages to the IoT topic within certain limits.

Previously, the AWS Lambda function responsible for detecting faces only showed them on the output streams and was only publishing to the MQTT topic the threshold of detected faces. We can modify this by including cropped face images as part of the packets that are sent the topic.

The Greengrass function below extends the original version by publishing a message for each detected face. Encoded cropped face images are set in the “image_string” key of the object. IoT messages have a size limit of 128 KB, but the images will be well within the limit and encoded in Base64.


# File "src/greengrassHelloWorld.py" in code repository
from threading import Thread, Event
import os
import json
import numpy as np
import awscam
import cv2
import greengrasssdk

class LocalDisplay(Thread):
    def __init__(self, resolution):
    ...
    def run(self):
    ...
    def set_frame_data(self, frame):
    ....

    def set_frame_data_padded(self, frame):
        """
        Set the stream frame and return the rendered cropped face
        """
        ....
        return outputImage

def greengrass_infinite_infer_run():
    ...
    # Create a local display instance that will dump the image bytes
    # to a FIFO file that the image can be rendered locally.
    local_display = LocalDisplay('480p')
    local_display.start()
    # The sample projects come with optimized artifacts,
    # hence only the artifact path is required.
    model_path = '/opt/awscam/artifacts/mxnet_deploy_ssd_FP16_FUSED.xml'
    ...
    while True:
        # Get a frame from the video stream
        ret, frame = awscam.getLastFrame()
        # Resize frame to the same size as the training set.
        frame_resize = cv2.resize(frame, (input_height, input_width))
        ...
        model = awscam.Model(model_path, {'GPU': 1})
        # Process the frame
        ...
        # Set the next frame in the local display stream.
        local_display.set_frame_data(frame)

        # Get the detected faces and probabilities
        for obj in parsed_inference_results[model_type]:
            if obj['prob'] > detection_threshold:
                # Add bounding boxes to full resolution frame
                xmin = int(xscale * obj['xmin']) \
                       + int((obj['xmin'] - input_width / 2) + input_width / 2)
                ymin = int(yscale * obj['ymin'])
                xmax = int(xscale * obj['xmax']) \
                       + int((obj['xmax'] - input_width / 2) + input_width / 2)
                ymax = int(yscale * obj['ymax'])

                # Add face detection to iot topic payload
                cloud_output[output_map[obj['label']]] = obj['prob']

                # Zoom in on Face
                crop_img = frame[ymin - 45:ymax + 45, xmin - 30:xmax + 30]
                output_image = local_display.set_frame_data_padded(crop_img)

                # Encode cropped face image and add to IoT message
                frame_string_raw = cv2.imencode('.jpg', output_image)[1]
                frame_string = base64.b64encode(frame_string_raw)
                cloud_output['image_string'] = frame_string

                # Send results to the cloud
                client.publish(topic=iot_topic, payload=json.dumps(cloud_output))
        ...

greengrass_infinite_infer_run()

Save faces to S3 with an IoT Rule

The third IoT pillar intelligence interacts with the cloud pillar, which uses insights to perform actions on other AWS and/or external services. Our goal is to have all detected faces saved to an S3 bucket in the original JPEG format before we encoded it to Base64. To achieve this, we need to create an IoT rule that will launch an action to do so.

IoT Rules listen for incoming MQTT messages of a topic and when a certain condition is met, it will launch an action. The messages from the queue are analysed and transformed using a provided SQL statement. We want to act on all messages, passing on data captured by the DeepLens device and also inject the “unix_time” property. The IoT Rule Engine will allow us to construct statements that do just that, calling the timestamp function within a SQL statement to add it to the result, as seen in the statement below.


# MQTT message
{
    "image_string": "/9j/4AAQ...",
    "face": 0.94287109375
}

# SQL Statement 
SELECT *, timestamp() as unix_time FROM '$aws/things/deeplens_topic_name/infer'

# IoT Rule Action event
{
    "image_string": "/9j/4AAQ...",
    "unix_time": 1540710101060,
    "face": 0.94287109375
}

The action is an AWS Lambda function (seen below) that is given an S3 Bucket name and an event. At a minimum, the event must contain properties: “image_string” representing the encoded image and “unix_time” which used for the name of the file. The last property is not something that is provided when the IoT message is published to the MQTT topic but instead is added by the IoT rule that calls the action.


# File "src/process_queue.py" in code repository
import os
import boto3
import json
import base64

def handler(event, context):
    """
    Decode a Base64 encoded JPEG image and save to an S3 Bucket with an IoT Rule
    """
    # Convert image back to binary
    jpg_original = base64.b64decode(event['image_string'])

    # Save image to S3 with the timestamp as the name
    s3_client = boto3.client('s3')
    s3_client.put_object(
        Body=jpg_original,
        Bucket=os.environ["DETECTED_FACES_BUCKET"],
        Key='{}.jpg'.format(event['unix_time']),
    )

Deploying an IoT Rule with AWS SAM

AWS SAM makes it incredibile easy to deploy an IoT Rule as it is a supported event type for Serverless function resources, a high-level wrapper for AWS Lambda. By providing only the DeepLens topic name as a parameter for the template below, an fully event-driven and least privalege AWS architecture is deployed.


# File "template.yaml" in code repository
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'

Parameters:
  DeepLensTopic:
    Type: String
    Description: Topic path for DeepLens device "$aws/things/deeplens_..."

Resources:
  ProcessDeepLensQueue:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python2.7
      Timeout: 30
      MemorySize: 256
      Handler: process_queue.handler
      CodeUri: ./src
      Environment:
        Variables:
          DETECTED_FACES_BUCKET: !Ref DetectedFaces

      Policies:
        - S3CrudPolicy:
            BucketName: !Ref DetectedFaces

      Events:
        DeepLensRule:
          Type: IoTRule
          Properties:
            Sql: !Sub "SELECT *, timestamp() as unix_time FROM '${DeepLensTopic}'"

  DetectedFaces:
    Type: AWS::S3::Bucket
AWS DeepLens TechConnect

AWS DeepLens: Getting Hands-on (Part 1 of 3)

TechConnect recently acquired two AWS DeepLens to play around with. Announced at Re:Invent 2017, the AWS DeepLens is a small Intel Atom powered Deep Learning focused device with an embedded High-Definition video camera. The DeepLens runs AWS Greengrass, allowing quick compute for local events without having to send a large amount of data for processing on the cloud. This can substantially help businesses reduce costs, sensitive information transfer, and latency response for local events.

Zero to Hero (pick a sample project)

AWS DeepLens Face Detection Project

Creating a face detection AWS DeepLens project

What I think makes the DeepLens special, is how easy it is to get started using computer vision models to process visual surroundings on the device itself. TechConnect has strong capabilities in Machine Learning, but I myself haven’t had much of a chance to play around with Deep Learning frameworks like MxNet or Tensorflow. Thankfully, AWS provides a collection of pre-trained models and projects to help anyone get started. But if you are quite savvy already in those frameworks, you can train and use your own models too.

Face Detection Lambda Function

AWS DeepLens Face Detection Lambda Function

AWS DeepLens Face Detection Lambda Function


An AWS DeepLens project consists of a trained model and an AWS Lambda function (written in Python) at its core. These are deployed to the device to run via AWS Greengrass, where the AWS Lambda function continually processes each frame of coming in from the video feed using the awscam module.

The function can access the model that is downloaded as an accessible artifact on the device at the path. This location and others (example: “/tmp”) have permissions granted to the function from the AWS Greengrass group which is associated with the DeepLens project. I chose the face detection sample project, which processes faces in a video frame captured from the DeepLens camera and draws a rectangle around them.


from threading import Thread, Event
import os
import json
import numpy as np
import awscam
import cv2
import greengrasssdk

class LocalDisplay(Thread):
    def __init__(self, resolution):
    ...
    def run(self):
    ...
    def set_frame_data(self, frame):
    ....

def greengrass_infinite_infer_run():
    ...
    # Create a local display instance that will dump the image bytes
    # to a FIFO file that the image can be rendered locally.
    local_display = LocalDisplay('480p')
    local_display.start()
    # The sample projects come with optimized artifacts,
    # hence only the artifact path is required.
    model_path = '/opt/awscam/artifacts/mxnet_deploy_ssd_FP16_FUSED.xml'
    ...
    while True:
        # Get a frame from the video stream
        ret, frame = awscam.getLastFrame()
        # Resize frame to the same size as the training set.
        frame_resize = cv2.resize(frame, (input_height, input_width))
        ...
        model = awscam.Model(model_path, {'GPU': 1})
        # Process the frame
        ...
        # Set the next frame in the local display stream.
        local_display.set_frame_data(frame)
        ...

greengrass_infinite_infer_run()

Extending the original functionality: AWS DeepLens Zoom Enhance!

AWS DeepLens Face Detection Enhance

AWS DeepLens Face Detection Enhance

I decided to have a bit of fun and extend the original application functionality by cropping and enhancing a detected face. The DeepLens project video output set to 480p definition, but the camera frames from the device are much higher than this! So reusing the code from the original sample that drew a rectangle around each detected face, I was able to capture a face and display that on the big screen. The only difficult thing was centring the captured face and adding padding, bringing back bad memories of how hard centring an image in CSS used to be!


from threading import Thread, Event
import os
import json
import numpy as np
import awscam
import cv2
import greengrasssdk

class LocalDisplay(Thread):
    def __init__(self, resolution):
    ...
    def run(self):
    ...
    def set_frame_data(self, frame):
        # Get image dimensions
        image_height, image_width, image_channels = frame.shape

        # only shrink if image is bigger than required
        if self.resolution[0] < image_height or self.resolution[1] < image_width:
            # get scaling factor
            scaling_factor = self.resolution[0] / float(image_height)
            if self.resolution[1] / float(image_width) < scaling_factor:
                scaling_factor = self.resolution[1] / float(image_width)

            # resize image
            frame = cv2.resize(frame, None, fx=scaling_factor, fy=scaling_factor, interpolation=cv2.INTER_AREA)

        # Get image dimensions and padding after scaling
        image_height, image_width, image_channels = frame.shape

        x_padding = self.resolution[0] - image_width
        y_padding = self.resolution[1] - image_height

        if x_padding <= 0:
            x_padding_left, x_padding_right = 0, 0
        else:
            x_padding_left = int(np.floor(x_padding / 2))
            x_padding_right = int(np.ceil(x_padding / 2))

        if y_padding  detection_threshold:
            # Add bounding boxes to full resolution frame
            xmin = int(xscale * obj['xmin']) \
                   + int((obj['xmin'] - input_width / 2) + input_width / 2)
            ymin = int(yscale * obj['ymin'])
             max = int(xscale * obj['xmax']) \
                   + int((obj['xmax'] - input_width / 2) + input_width / 2)
            ymax = int(yscale * obj['ymax'])

            # Show Enhanced Face
            crop_img = frame[ymin - 45:ymax + 45, xmin - 30:xmax + 30]
            local_display.set_frame_data(crop_img)
            time.sleep(5)
        ...

def greengrass_infinite_infer_run():
    ...
    while True:
        # Get a frame from the video stream
        ret, frame = awscam.getLastFrame()
        # Resize frame to the same size as the training set.
        frame_resize = cv2.resize(frame, (input_height, input_width))
        ...
        model = awscam.Model(model_path, {'GPU': 1})
        # Process the frame
        ...
        # Set the non-cropped frame in the local display stream.
        local_display.set_frame_data(frame)
        
        # Get the detected faces and probabilities
        for obj in parsed_inference_results[model_type]:
           if obj['prob'] > detection_threshold:
               # Add bounding boxes to full resolution frame
               xmin = int(xscale * obj['xmin']) \
                      + int((obj['xmin'] - input_width / 2) + input_width / 2)
               ymin = int(yscale * obj['ymin'])
               xmax = int(xscale * obj['xmax']) \
                      + int((obj['xmax'] - input_width / 2) + input_width / 2)
               ymax = int(yscale * obj['ymax'])

               # Add face detection to iot topic payload
               cloud_output[output_map[obj['label']]] = obj['prob']

               # Zoom in on Face
               crop_img = frame[ymin - 45:ymax + 45, xmin - 30:xmax + 30]
               local_display.set_frame(crop_img)

        # Send results to the cloud
        client.publish(topic=iot_topic, payload=json.dumps(cloud_output))

greengrass_infinite_infer_run()
Intensive Care Unit - Data Collection

Precision Medicine Data Platform

Recently TechConnect and IntelliHQ attended the eHealth Expo 2018. IntelliHQ are specialists in Machine Learning in the health space, and are the innovators behind the development of a cloud-based precision medicine data platform. TechConnect are IntelliHQ’s cloud technology partners, and our strong relationship with Amazon Web Services and the AWS life sciences team has enabled us to deliver the first steps towards building out the precision medicine data platform.

This video certainly sums up the goals of IntelliHQ and how TechConnect are partnering to deliver solutions in life sciences on the Amazon Web Services cloud platform.

Achieving this level of integration with the General Electric Carescape High Speed Data Interface is a first in Australia and potentially a first outside of America. TechConnect have designed a lightweight service to connect to the GE Carescape and push the high fidelity data to Amazon Kinesis Firehose and then to persisted cost effective storage on Amazon S3.

With the raw data stored on Amazon S3, data lake principles can be applied to enrich and process data for research and ultimately help save more lives in a proactive way. The diagram below shows a high level architecture that supports the data collection and machine learning capability inside the precision medicine data platform.

 

GE Carescape HSDI to Cloud Connector

This software, named Panacea, will be made available as an open source project.

Be sure to explore the following two sources of further information:

Check out Dr Brent Richards’ presentation at the recent eHealth Expo 2018 as well as a selection of other speakers located here.

AIkademi seeks to develop the capabilities of individuals, organisations and communities to embrace the opportunities emerging from machine learning.