Using Computer Vision to know if I have a package at home.

Many of the shipping companies, FedEx, UPS, USPS and the like offer notification if a package has been delivered.  Some of these are timely, some are not.

Wouldn’t it be great if, using the standard home security cameras and the power of Machine Learning YOU could detect a package on your front door step, instead of waiting for others?

Well, today is your day!  Microsoft has released “Custom Computer Vision” https://www.customvision.ai/ and with just a little work, we can make that happen.

I have a less than $200 home security surveillance system, similar to: https://www.amazon.com/Floureon-1000TVL-Waterproof-Camera-Security/dp/B01M03THKX.  Most of these systems allow you to stream video from them, using the correct URL.  For my system, I can use the URL:

rtsp://192.168.1.1:8854/live0.264?user=myuser&passwd=mypassword

Change the IP, user id and password to match your settings.

Putting these in VLC, we can remotely see what my front door camera sees:

VLC

So using the RTSP protocol, we can stream data, but Computer Vision needs images, not a streaming protocol.

Using the ffmpeg http://www.ffmpeg.org/ application, we can grab a picture and download it as a JPEG using the following command:

ffmpeg.exe -i "rtsp://192.168.1.1:8854/live0.264?user=myuser&passwd=mypassword" -f image2 -frames 1 myPicture.JPEG

This is important, because to ‘train’ our Computer Vision, we need LOTS of pictures.  So our first step is to collect LOTS of pictures.

Gathering Images of “No Packages”:

Using the following DOS script (yep, old school) I collected over 10,000 pictures throughout the day, so I can sample day and night images to set a baseline of what my front porch looks like without any packages.  The script below samples 100 pictures every 10 minutes.  It ran for 24 hours.

@echo off
SET counter=0
:start
 SET /A counter+=1
 ffmpeg -i "rtsp://192.168.1.1:8854/live0.264?user=myuser&passwd=mypassword" -f image2 -frames 100 sampling\picture%counter%--%%d.jpg
 timeout /t 600
goto start

:endloop

Gathering Images of “Packages”:

So to train the Computer Vision model on what packages looks like, my daughter and I spent several quality minutes taking pictures of packages on my front porch.  I simply put a package in the camera view area, and ran the ffmpeg command to collect the image.

Lots of images like:

Picture2

Creating a Custom Computer Vision Model:

Next I logged into: https://www.customvision.ai/ and created a project.

I added the thousands of images and tagged them “No Packages”.  Then I added my 101 images and tagged them “Package”, as shown below:

model

Training and Testing the Model:

So now we have images, we simply train the model and test it.  Now I can take a sample picture and ask the model if it sees a package: proof

As shown above, there is a 93.1% probability there is a package on my front door step.  I have highlighted in red where the actual package is, so we feel good.

Sampling my front door step when there is not a package, the model says there is a 96.5% probability there is no package, as shown below.

proof2

While Machine Learning is not an absolute science all the time, over time the model will be better.  In the above comparisons, simply take the highest probability.  🙂

Putting it all in action:

Lastly, lets make it real!  Imagine the following logic:

  • Ever 15 minutes, take a picture of the front door step
  • send the picture to the Computer Vision web service
  • If the probability is that there is a package, send a text message / mobile notification / email / (Instagram???) with the picture of the front porch.
  • (optional) if we do have a package add the image to the known packages to better train the model
  • (optional) if we have a known no package, take random samples of images to better train the model
  • (optional) export the finished model from Custom Computer Vision and add the exported model to a local ‘gateway’ / ‘application’ for edge processing

Sample Code:

The sample bash code below grabs a picture, calls the Computer Vision Model and sends a notification if this is the first time a package is detected:

#!/bin/bash

# used to not double notify a person
Notify=0

while [ 1 -eq 1 ]
do
  rm ./image.jpg
  ffmpeg -i 'rtsp://192.168.1.1:8854/live0.264?user=myuser&passwd=' -f image2 -frames 1 ./image.jpg
  JSONOUTPUT=$(curl -H "Prediction-Key: 956mykey699" -H "Content-type: application/octet-stream" -X POST --data-binary @image.jpg 'https://southcentralus.api.cognitive.microsoft.com/customvision/v1.1/Prediction/1d865b07-6836-4642-8fbd-ad493b4c3064/image?iterationId=fab00f85-3d20-43fe-9b93-cae1107282d0')

 echo "$JSONOUTPUT"

 PackageStatus=$(echo ${JSONOUTPUT} | jq '.Predictions[0].Tag')
 echo $PackageStatus

 if [ '$PackageStatus' == '"Packages"' ]; then
    echo Package
    echo "$(date)" >> packages.log
    if [ $Notify -eq 0 ]; then
     echo "Notify somebody"
     Notify=1
    fi
 else
    echo "$(date)" >> nopackages.log
    echo NoPackage
    Notify=0
 fi

 sleep 15m

done

And here we see the script working:

inaction

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s