This project lets you build an image recognition device that can see and identify objects, powered by TensorFlow’s machine learning models. All you need is a Raspberry Pi Zero W, a Raspberry Pi Camera 2, and a blank SD card.
Assembling the kit should take about an hour.
Open the box and verify you have all of the necessary components in your kit. You’ll also need a couple of tools for assembly.
The Raspberry Pi Zero W does not typically ship with headers installed. You will need to supply and solder your own 2x20 pin header or purchase a unit with them pre-soldered.
This guide shows you how to assemble the AIY Projects Vision Kit.
The kit contains a cardboard form, the VisionBonnet accessory board for image recognition, and some other connecting components. You’ll need a Raspberry Pi Zero W board, the Raspberry Pi Camera 2, and a blank SD card with at least 4 GB of space to build the kit.
By the end of this guide your Vision Kit will be assembled, configured, and ready to run!
The Vision Kit requires a special version of the Raspbian operating system that includes some extra AIY software. So before you begin, you need to download the Vision Kit system image and flash it to your MicroSD card.
.img.xz file from the releases page on GitHub..img.xz file onto your MicroSD card. (Etcher is free and works on Windows, Mac OS, and Linux.)Flashing the system image onto the card can take a several minutes. So while that’s going, start assembling the kit. Once the kit is assembled, you'll put the card into it.
If you have any issues while building the kit, contact us at support-aiyprojects@google.com.
Take your Raspberry Pi Zero W board and insert the two plastic standoffs into the two yellow holes opposite the 40-pin box header.
WARNING: First make sure your Raspberry Pi is disconnected from any power source and other components. Failure to do so may result in electric shock, serious injury, death, fire or damage to your board or other components and equipment.
Find the VisionBonnect cable connector next to the pin header. Pull up the black release lever.
Find the short MIPI flex cable and orient it with the serial number facing towards you.
On the right side, gently slide the MIPI flex cable into the VisionBonnet board until the cable hits the back of the connector.
Then flip the black lever into the parallel state to close it and secure the cable.
IMPORTANT: Ignore the white sticker that says “PI” or “VisionBonnet” on the MIPI flex cable. These may be incorrectly applied.
Now find your Raspberry Pi board and pull back the black release lever.
Slide the other side of the MIPI flex cable into the Raspberry Pi. The Raspberry Pi header should be facing toward you.
Be gentle and don’t push too hard; otherwise, you may damage the connectors.
WARNING: Failure to securely seat the ribbon in the connector may cause electric shock, short, or start a fire, and lead to serious injury, death, or damage to property.
Secure the release lever once it’s in place.
Carefully and gently bend the MIPI cable so that the board pin headers are facing each other (see the next step).
Connect the two boards via the 40-pin header. Gently push the MIPI cable so that it folds into the space between the two boards.
Press down to snap the standoffs opposite the header into place.
WARNING: Push gently: forcing the standoffs into place may cause them to break.
Find the button harness and plug it into the button connector on the VisionBonnet board.
Set your connected boards aside for now.
WARNING: Failure to securely seat the connector may cause electric shock, short, or start a fire, and lead to serious injury, death, or damage to property.
Well done! Set aside your hardware for now.
Find your Pi camera 2 board and the Pi0 camera flat flex cable.
Open the cable release lever and remove the white cable that may be attached to the board. Connect the gold Pi0 cable to the Pi camera 2 board. Close the release lever once the cable is secure.
WARNING: Failure to securely seat the connector may cause electric shock, short, or start a fire, and lead to serious injury, death, or damage to property.
Find the smaller cardboard piece. This piece holds the hardware components and fits inside the larger cardboard box.
In the middle of the cardboard piece is an upside-down U-shaped cutout. Push it out, then fold the tab upward (see the image).
Place the Pi camera 2 board onto the frame. The camera’s aperture will fit into a rectangular slot in the middle of the cardboard.
Remove the protective film from the camera lens.
Turn the cardboard over.
Fold the window-shaped cardboard tab over camera board.
Fold the tabs on the left and the right of the board toward you. There will be two small cardboard notches on each side that will secure each tab.
Turn the cardboard frame over
Find your Piezo buzzer and insert the wire into the cardboard hole below the camera aperture.
Remove the adhesive cover from the Piezo buzzer.
Stick the adhesive side of the Buzzer onto the Upside-down U-shaped tab you indented earlier.
Take the other end of your Pi0 camera flat flex cable and plug it into the VisionBonnet board connector. The Pi Zero board should be facing away from you.
WARNING: Failure to securely seat the connector may cause electric shock, short, or start a fire, and lead to serious injury, death, or damage to property.
Place the two boards onto the bottom tab of the cardboard frame. Fold the rest of the cardboard frame upward.
The cardboard also has two feet under the board. Fold those downward.
Gently fold the flex cable toward the board (otherwise it won’t fit into the cardboard box later).
Loop the flex camera cable through the cardboard indent.
Find the other cardboard piece and fold it into a box. Keep the bottom and top open.
Take your internal cardboard frame and slide it into the bottom of the cardboard box (the bottom side has the smaller hole).
Here’s another view.
Make sure the buzzer wire and button harness didn’t get stuck.
Also check that the board connectors are aligned with the box cutouts.
WARNING: Forcing connectors into misaligned ports may result in loose or detached connectors. Loose wires in the box can cause electric shock, shorts, or start a fire, which can lead to serious injury, death or damage to property.
Take the tripod mounting nut and place it between the flaps on the bottom of the cardboard box.
Fold the large flap over to close the bottom of the box.
Find your privacy LED and LED bezel.
Push the LED bezel into the hole above the camera aperture.
From the inside of the box, take the privacy LED and insert it into the LED bezel. It should snap into place.
Make sure the LED is peeking out on the other side.
Find your arcade button and unscrew the plastic nut. Insert the arcade button into the hole on the top flap of the cardboard box. Secure it with the plastic nut on the other side.
Using too much force may result in bending or ripping the box.
Check the arcade button’s board for the words Bonnet, LED, and Piezo.
Find your Piezo buzzer cable and plug it into the slot marked Piezo.
Same for the LED.
And plug the button harness into the slot marked Bonnet.
WARNING: Failure to securely seat the connector may cause electric shock, short, or start a fire, and lead to serious injury, death, or damage to property.
Carefully close the top of the box.
Find the camera lens mount washer and remove the adhesive strip from the back.
Attach the camera lens mount washer to the cardboard.
Attach the lens assembly to the lens washer. Look through the lens to ensure lens and camera are concentrically aligned.
WARNING: The lens contains magnets and should be kept out of reach of children and may interfere with medical implants and pacemakers.
Your Vision Kit is fully assembled. A couple more steps and you’re ready to turn it on.
Now that you’ve assembled your kit, plug in everything you need. Your Vision Kit can be used with or without a monitor, keyboard, and mouse. To begin, plug in both MicroUSB ports - one for power and one for data.
Insert your SD card with the Vision Kit SD image.
Plug in your micro-USB to USB cable. Connect the USB end to another computer.
Plug in the USB power supply, if you have one.
Note: Your Vision Kit will run with power from just the data cable connection, but we recommend using an external power source via the PiZero’s power MicroUSB port for reliability.
Your Vision Kit can be run without a monitor, keyboard, or mouse--the Joy Detector demo app will start automatically when the device boots up. But if you want to use the GUI and do some more tinkering, plugging in peripherals are required.
Point the Vision Kit toward someone’s face (or your own) to try out the Joy Detector demo. When the camera detects a face, the button illuminates.
The Joy Detector uses machine learningMachine learning is the science of making predictions based on patterns and relationships that've been automatically discovered in data. to detect if a person is smiling or frowning, and how much they are doing so. A smile turns the button to yellow, and a frown turns it blue.
If expressions are really big, a sound plays. If the camera sees more than one face, it evaluates each person’s face and sums the joy score of each face.
Not working?
Try holding the camera at least an arms length away from the face you’re pointing it at. Sometimes the camera has trouble if the subject is too close. And be sure the subject is well lit from the front. Keep this in mind for all the demos that you try.
At any point while Joy Detector is running, you can snap a photo by pressing the button.
If you take a photo while the camera detects a face (the button is illuminated), it saves a second version of the photo that's annotated with the joy score.
These photos are saved on the SD card in the ~/Pictures/ directory. You'll learn how to access these photos after you connect to your kit in the next step.
AIY Vision Kit is designed to work with a 5V 2A DC power supply with micro USB connector. We recommend the Raspberry Pi Power Supply. A power supply with less than a 2A rating will cause to the system to behave improperly.
AIY Vision Kit is designed to work with Raspberry Pi Zero W (version 1.1) and Raspberry Pi Zero (version 1.3). Please note the AIY Vision Kit requires the Raspberry Pi board to have a camera connector.
To find out which version of Raspberry Pi Zero you have, log into your system and type the following command and then refer to the ID in the following table:
cat /proc/cpuinfo | grep Revision
| ID | Revision |
|---|---|
| 900092 | Pi Zero without camera connector, no wireless (not compatible) |
| 900093 | Pi Zero with camera connector, no wireless |
| 9000c1 | Pi Zero W with camera connector + Wi-Fi/Bluetooth |
AIY Vision Kit is designed to work with Raspberry Pi Camera Module Version 2. Please note that Camera Module Version 1 is not supported and will not work. The Raspberry Pi Camera Module Version 2 NoIR will technically work (it uses the same sensor), however our ISP is not optimized for this unit.
Be sure to check both flex cable connections. The device will not boot if the cables are connected incorrectly.
IMPORTANT: Ignore the white sticker that says “PI” or “VisionBonnet” on the MIPI flex cable. These may be incorrectly applied.
Find the silkscreen serial number on the MIPI flex cable and orient it so the numbers are facing you. With this orientation, the right side of the flex cable connects to the VisionBonnet and the left side to the Raspberry Pi board.
You can run this command to check that the Vision Bonnet is connected and has booted successfully:
dmesg | grep googlevisionbonnet
If it has, you should see output similar to this:
[ 18.545995] googlevisionbonnet spi0.0: Initializing [ 18.712917] googlevisionbonnet spi0.0: Resetting myriad on probe [ 18.712940] googlevisionbonnet spi0.0: Resetting myriad [ 22.843448] googlevisionbonnet spi0.0: Writing myriad firmware [ 31.573648] googlevisionbonnet spi0.0: Myriad booting [ 31.818673] googlevisionbonnet spi0.0: Myriad ready
To try out other demos, you’ll connect to your Vision Kit so that you can give it commands. There are two options for connecting, explained in Meet your kit.
Follow the instructions below for one connection option, either with the AIY Projects app or with a monitor, mouse, and keyboard.
Go to the Google Play Store and download the AIY Projects app.
This app will allow you to connect your Vision Kit to a Wi-Fi network, and display an IP address which you’ll use to communicate with your Vision Kit wirelessly via a separate computer and SSHSSH stands for “secure shell.” It’s a way to securely connect from one computer to another..
Psst: This app only works on Android smartphones. If you don’t have one, please use the alternate connection method (which uses a monitor, keyboard, and mouse).
Open the app and follow the onscreen instructions to pair with your Vision Kit.
Take note of the IP addressThe Internet Protocol Address is a four-digit number that identifies a device on a network. Every device on your network (your computer, phone, your Vision Kit) will have a unique IP Address. Using this address, one device can talk to another. — you’ll need it later. The app will also remember and display it on the home screen.
Not working? Make sure your Vision Kit is connected to a power supply.
If you run into errors, quit the app and try again.
If the device won’t pair, make sure the green LED on the Vision Bonnet is flashing. If it’s not flashing, it may have timed out. Press and hold the Vision Bonnet button for 5 seconds, and try again. If that doesn’t work, try restarting your phone.
A monitor is not required to run these demos, but if you have one available, it can be useful so that you can see what your Vision Kit is seeing.
If you have a monitor and mini-HDMI cable (or HDMI cable + mini-HDMI adapter) available, follow these instructions:
When your kit is booted, you’ll see a desktop with the AIY logo on the background.
A pop-up will tell you that the password for the Raspberry Pi user is set to the default. This is important if you plan to use this kit in other projects or expose it to the internet, but for now, you can leave it (we’ll explain more later).
If you don’t have a monitor, or when your kit is powered back on, go to the next step.
Make sure your computer is on the same Wi-Fi network as your Vision Kit. This will allow you to connect to your kit through SSH.
We’re going to connect your computer to the Raspberry Pi using SSH in a terminalA terminal is a text window where you can issue commands to your Raspberry Pi. SSH allows you to do so from a separate computer..
If you’re familiar with using a terminal, start an SSH session with pi@192.168.0.0 (but using the Raspberry Pi's real IP address from above), then skip to step 10.
If you're not familiar with a terminal, download and install the Chrome browser and Secure Shell Extension, and proceed to the next step.
Once the extension is installed open it.
If you’re using Chrome on a Windows, Mac, or Linux computer, you should see the Secure Shell Extension icon 
in the toolbar (look to the right of the address bar). Click that icon and then select Connection Dialog in the menu that appears.
If you’re using Chrome on a Chromebook, go to the app menu and type "secure shell app extension".
In the top field, type pi@192.168.0.0, but replacing those numbers with the real IP address of your Raspberry Pi. After typing this in, click on the port field. The [ENTER] Connect button should light up.
Click [ENTER] Connect to continue.
Can’t connect? If you can’t connect, check to make sure the IP address you wrote down earlier is correct and that your Raspberry Pi is connected to the same Wi-Fi access point your computer is.
Note If you rewrite or replace your SD card, you will need to remove and add the Secure Shell Extension from Chrome. You can do this by right clicking on the icon in your toolbar and selecting "Remove", then re-add it by following the instructions above.
Click Allow.
This gives permission to the SSH extension to access remote computers like your Raspberry Pi.
You will only need to do this once when you add the extension into Chrome.
At the prompt, type yes and press enter to confirm that the displayed host keyThe SSH extension is designed to be secure, and because of this goal, it needs to identify that the computer you're trying to connect to is actually the computer you expect. To make this job easier, the computers generate a long number and present it to the extension for verification each time. The extension saves this key somewhere safe so that it can verify that the computer you're speaking to is actually the right one. matches what is stored on your Raspberry Pi. You will only have to do this the first time you connect to your kit.
Enter the Raspberry Pi’s password at the prompt. The default, case-sensitive password is raspberry
When you type, you won’t see the characters.
Note Your IP address might be different than the one shown in the example.
It’s okay if you see the warning line. It’s letting you know that the host key has been saved, and the extension will do the hard work of comparing what it just stored with what the Raspberry Pi provides automatically next time.
Having trouble? If it’s typed wrong, you’ll see “Permission denied, please try again” or “connection closed.” You’ll need to re-start your connection by pressing the R key.
If the password was entered correctly, you’ll see a message about SSH being enabled by default and the pi@raspberrypi:~ $ shellA shell is a program that runs on a computer that waits for instructions from you, and helps you to make your computer work for you. promptIt’s a response from the shell that indicates that it is ready to receive commands, and tells you what your current working directory is (the tilde, ~, in this case). It ends in a $ where you type your command. will be green.
You will also see a warning that the password for the Raspberry Pi user is set to the default. This is important if you plan to use this kit in other projects, or expose it to the internet, but for now it’s okay to proceed.
Congrats! You’re now connected to your Vision Kit. Skip to the Try More Demos section to explore more Vision Kit demos.
Do I need to change my password? You'll want to change the pi user's password if you plan on using this kit in a project that is exposed to the open internet. It's not safe to expose it with a password everybody knows. If you plan on doing this, you'll want to use the passwd program. This is an advanced step, so for the purposes of this guide, we will assume you haven't changed the password for the pi user.
Note If you do change the password, make sure you keep your password written down somewhere safe in case you forget. It's not easy to recover it if you change it.
How did connecting to your Vision Kit go?
Use connection option if you don’t have access to an Android smartphone and second computer, or if you prefer to connect directly to your Raspberry Pi.
You’ll need a set of peripherals to interact with your Raspberry Pi, including a monitor, keyboard and mouse. Check here for suggestions.
Before plugging in your peripherals, unplug your kit from power.
Plug your monitor into the HDMI port and your keyboard and mouse into the Data port on your Vision Kit using one of the adapters described in Meet your kit.
Plug your monitor into power if it’s not already.
Plug your Raspberry Pi back into power via the Power port. To confirm that it’s connected to power, look into the hole in the cardboard labeled SD Card. You’ll see a green LED flashing on the Raspberry Pi board.
You’ll also see the Raspberry Pi logo in the top left corner of the monitor.
Wait for your device to boot, which will take about two minutes. It’s okay if your screen goes black while it’s booting. Be patient! You’ll know when it’s booted when you hear it beep.
You’ll see a desktop with the AIY on the background. A pop-up will tell you the password for the Raspberry Pi user is set to the default. This is important if you plan to use this kit in other projects or expose it to the internet, but for now, it's safe to click OK.
Do I need to change my password? You'll want to change the pi user's password if you plan on using this kit in a project that is exposed to the open internet. It's not safe to expose it with a password everybody knows. If you plan on doing this, you'll want to use the passwd program. This is an advanced step, so for the purposes of this guide, we will assume you haven't changed the password for the pi user.
Note If you do change the password, make sure you keep your password written down somewhere safe in case you forget; it's not easy to recover if you change it.
Open the terminalA terminal is a text window where you can issue commands to your Raspberry Pi. by clicking the black rectangular icon on the taskbar at the top of the screen.
Now you’ll be able to issue commands to your Raspberry Pi.
You should now see the promptIt’s a response from the shell that indicates that it is ready to receive commands, and tells you what your current working directory is (the tilde, ~, in this case). It ends in a $ where you type your command. pi@raspberrypi: ~ $.
Congrats! You’re ready to start issuing commands to your Raspberry Pi.
What if my prompt looks different? If you clicked on the Start dev terminal icon, you’ll see the prompt “pi@raspberrypi: ~/AIY-projects-python $” instead. This is because the Start dev terminal shortcut is setup to open a terminal and then set your working directory to “~/AIY-projects-python”. That’s fine for the purpose of these instructions. We’ll show you in a few steps how you can use the cd command to change your working directory.
How did connecting to your Vision Kit go?
Now that you’ve connected to your Vision Kit, you can try out other demos to experiment with machine learning.
If you connected a monitor to your Vision Kit, you can now look at the photos you captured. (Unfortunately, you cannot view the photos if your Vision Kit isn't directly plugged into a monitor.)
To view the photos captured with the Joy Detector, you need to first navigate your terminal to the ~/Pictures directoryYou might have heard the terms "folder" or "directory" before. They are synonyms for the same thing: a data structure that contains a listing of filenames and the location of their contents on disk. Think of them like a table of contents: each time you run the ls command, you're "list"-ing the contents of one of these directories. using cdcd stands for “change directory.” Think of it as clicking through file folders. You should see the path in the command line in blue. Capitalization matters: it’s cd, not CD. If you ever get lost or curious, typing pwd and then pressing enter will display your current path.. So type the following into the terminal and press enter:
cd ~/Pictures
Now type ls"ls" is shorthand for "LiSt" and prints out all of the files in the current working directory. It's a great way to look around and see what changed on disk. and press enter to see what’s in the directory. (Hint: that’s an “l” as in lemon, not a #1.) You should see a list of filenames ending with .jpeg. So let's look at one of these.
Type the following command in your terminal and press enter, replacing <filename> with the filename you want to open (such as 2018-05-03_19.52.00.jpeg):
DISPLAY=:0 gpicview <filename>
This photo opens in a new window on the monitor that's plugged into the Vision Kit.
Tip: If you connected to your Vision Kit with monitor, mouse, and keyboard, you can enter the command without DISPLAY=:0.
To close the photo window from your terminal, press Ctrl-CCtrl-C interrupts a running process and returns control back to the terminal prompt..
What is gpicviewgpicview is an application that you can use to display an image. You need to type “DISPLAY=:0” when connecting to your Pi via SSH to tell gpicview how to display an image on the screen.?
Learn more about working in the terminal: Check out some guides from our friends at the Raspberry Pi Foundation: Conquer the Command Line and Linux Commands.
Not seeing anything on your monitor? If your monitor looks like it’s asleep, try typing Ctrl-C to interrupt your previous command and return to the prompt. Then type “DISPLAY=:0 xset s activate” and press enter. Then try to view the image again by typing the command above.
The Joy Detector runs by default, so you need to stop it before you can run another demo. To do this, type the following command and press enter:
sudo systemctl stop joy_detection_demo
After the demo stops, you are brought back to the command prompt. If you instead see an error, check the command for typos and try again.
Always stop any demos that are running before trying a new demo. If you don’t, you will run into errors.
However, the next time you reboot your kit, the Joy Detector demo will start running again. So if you want to disable it completely so that it does not start by default, type the following command into your prompt and press enter:
sudo systemctl disable joy_detection_demo
You can re-enable it later with:
sudo systemctl enable joy_detection_demo
For more information about these commands, see run your app at bootup.
To try out several other Vision Kit demos, move into the directory where they’re located. Type the following command into your prompt and press enter:
cd ~/AIY-projects-python/src/examples/vision
Your prompt should now say pi@raspberrypi:~/AIY-projects-python/src/examples/vision $.
Trying to Copy + Paste? Copying and pasting in a terminal is a little different than other applications you may be used to.
If you are using the Secure Shell Extension, to copy some text, highlight what you want by clicking and dragging with the left mouse button, and as soon as you let go of it, it'll copy it. To paste, click the right mouse button. On a touchpad this can be a little tricky, so try tapping or pressing in the lower right of the touchpad, or tapping with two fingers.
To copy text using the terminal on your Raspberry Pi: select the text, right-click, and select 'copy' from the menu. Left click where you want to paste the text, then right click and select 'paste' from the pop up menu.
Now that you’ve changed directories, type ls and press enter to see what’s inside your current directory.
Here you’ll see a list of files that end in .py. These are the example demos written in PythonPython is a programming language that we use for the majority of our demos and scripts. These files end in “.py”. It's a simple language and is very easy to learn. You can find out more about Python at https://www.python.org/ that you can run.
The image classification camera demo uses an object detection model to identify objects in view of the Vision Kit.
To start it, type the following command and press enter:
./image_classification_camera.py
It might take a moment to fire up.
If it's working, a camera window pops up on your monitor (if one is attached) and the output from the model starts printing to your terminal. If you are brought back to the prompt after seeing error text, check the Using the Vision Kit section of the help page for troubleshooting tips.
Help! The camera is blocking my terminal window. If you are connected directly to your Raspberry Pi via mouse, monitor, and keyboard, the camera window might block your terminal. That’s okay - your terminal is still there in the background. Press Ctrl-C after pointing your camera at a few objects to stop the demo and close the camera window. Then you can scroll up in your terminal window to see what the camera identified. If you want to see the terminal and camera preview at the same time, you can connect your Raspberry Pi to Wi-Fi and then connect to it from another computer via SSH. For information about that setup, see the login setup for the Voice Kit.
Point your Vision Kit at a few objects, such as some office supplies or fruit. Check your terminal screen to see what the modelA model is like a program for a neural network. It is a mathematical representation of all the different things the neural network can identify. But unlike a program, a model can't be written, it has to be trained from hundreds or thousands of example images. When you show your Vision Kit a new image, the neural network uses the model to figure out if the new image is like any image in the training data, and if so, which one. is guessing. The number next to each guess is its confidence scoreThe confidence score indicates how certain the model is that the object the camera is seeing is the object it identified. The closer the number is to 1, the more confident it is..
You might be surprised at the kinds of objects the model is good at guessing. What is it bad at? Try different angles of the same object and see how the confidence score changes.
The image classification camera demo will run indefinitely until you interrupt it.
When you’re done playing with the image classifier demo, press Ctrl-C to end it. This will bring you back to the prompt.
Don’t worry, you can always start the demo again.
This demo enables your Vision Kit to identify faces. It prints out how many faces it sees in the terminal, and if you have a monitor attached, it draws a box around each face it identifies.
To start the face detection demo, type the following command and press enter:
./face_detection_camera.py
If it's working, you will see a camera window pop up on your monitor (if one is attached) and the output from the model will start printing to your terminal. If you are brought back to the prompt after seeing error text, check out the Using the Vision Kit section of the help page for troubleshooting tips.
Help! The camera is blocking my terminal window. If you are connected directly to your Raspberry Pi via mouse, monitor, and keyboard, the camera window might block your terminal. That’s okay - your terminal is still there in the background. Press Ctrl-C after pointing your camera at a few objects to stop the demo and close the camera window. Then you can scroll up in your terminal window to see what the camera identified. If you want to see the terminal and camera preview at the same time, you can connect your Raspberry Pi to Wi-Fi and then connect to it from another computer via SSH. For information about that setup, see the login setup for the Voice Kit
Point the camera toward some faces and watch the demo output. Iteration tells you the number of times the model has run. num_faces is the model’s best guess at how many faces are in view of the camera.
Try moving the camera quickly, or farther away. Does it have a harder time guessing the number of faces?
When you’re done experimenting with the face detection demo, press Ctrl-C to end it. This will bring you back to the prompt.
With this demo, your Vision Kit automatically takes a photo when it detects a face. To start it, type the following command and press enter:
./face_camera_trigger.py
If you have a monitor attached, you’ll see a blinking cursor and a camera window pops up. It will remain in this state until the camera sees a face and captures a photo.
Point the camera at yourself or a friend. Try making a bunch of faces and experiment with what the machine considers to be a face.
When it sees a face, it will take a photo and create an image called faces.jpg in your current directory, and then close the camera window and bring you back to the prompt.
Seeing an error? Check out the Using the Vision Kit section of the help page for troubleshooting tips.
To verify that a photo was created, type ls at the prompt and press enter.
You should see a file called faces.jpg listed in your current directory.
To open the photo, see the instructions for how to View an image on your Pi.
Hint:
Each time the face_camera_trigger demo captures a photo, it overwrites faces.jpg. If you want to rename the last photo you took so that you don’t overwrite it, type the following command and press enter:
mv faces.jpg newname.jpg
The following demos show you how to use existing image files as input (instead of using the live camera feed). So you need to first capture a photo with the camera (or save a file into the same directory).
To capture a new photo named image.jpg, type the following command and press enter:
raspistill -w 1640 -h 922 -o image.jpg
The camera will wait 5 seconds, and then take a photo.
What should I name my file?
image.jpg is the name of the file we are telling the command to write to in the screenshot to the left. You can name your file anything you want, as long as you use only letters, numbers, dashes, and underscores. You should end your filename with .jpg because this command is saving the image in the JPEG format.
What does this command mean?
raspistill is a command that lets you capture photos using your Raspberry Pi camera module. The -w flag and -h flags specify the width and height for the image. The -o flag specifies the filename. For more information, see the raspistill documentation.
To verify that a photo was created, type ls at the prompt and press enter.
You should see the filename you used in the step above.
To open the photo, see the instructions for how to View an image on your Pi.
Tip: Press the up and down arrow keys at the prompt to scroll through a history of commands you've run. To rerun a command, it's easier to press the arrows until the one you want is shown. You can edit the command if needed, then press enter.
Now let’s use a photo you captured above with the face detection model. If you skipped that step, go back and take a photo or make sure you have a photo with a face on your SD card.
To run the demo, type the following command in your terminal and press enter:
./face_detection.py --input image.jpg
If you named your image file something different, replace image.jpg with the name of the file you want to use.
Seeing an error? Check out the Using the Vision Kit section of the help page for troubleshooting tips.
When it’s done, you should get something like this:
Face #0: face_score=0.989258, joy_score=0.969556, bbox=(632.0, -15.0, 782.0, 782.0)
face_score is how certain the model is that it’s found a face, and joy_score is how happy it appears the person is (both scores are out of 1). bbox tells you where the face is located in the image.
Nothing happened: If you’re brought back to the prompt and don’t see any output in white, the model didn’t detect a face in the photo. Try taking a new photo and then running the command again. Be sure your subject is well lit from the front and there are no bright lights directly behind them.
The object detection demo takes an image and checks whether it’s a cat, dog, or person.
First, you need an image ready: take a photo with the camera or save a photo on the SD card.
Then type the following command and press enter, replacing image.jpg with the file you want to use:
./object_detection.py --input image.jpg
Seeing an error? Check out the Using the Vision Kit section of the help page for troubleshooting tips.
When it’s done, you should get something like this:
Object #0: kind=PERSON(1), score=0.959231, bbox=(359, 108, 896, 808)
kind is the type of object detected and score is how confident the model is about the
result it gave. bbox is where that object is located in the image.
Nothing happened: If you’re brought back to the prompt and don’t see any output in white, the model didn’t detect a face, cat, or dog in the photo. Try taking a new photo and then running the command again.
The dish classifier model can identify food from an image.
First, you need an image ready: take a photo with the camera or save a photo on the SD card.
Then type the following command and press enter, replacing image.jpg with the file you want to use:
./dish_classification.py --input image.jpg
Seeing an error? Check out the Using the Vision Kit section of the help page for troubleshooting tips.
When it’s done processing (it may take a minute), you’ll get a list of results, along with the type of food identified and a probability score indicating how confident the model is of its answer (out of 1).
Nothing happened: If you’re brought back to the prompt and don’t see any output in white, the model didn’t detect anything, womp. Try again with a different photo.
This is the same image classifier from above but now running against a captured image.
First, you need an image ready: take a photo with the camera or save a photo on the SD card.
Then type the following command and press enter, replacing image.jpg with the file you want to use:
./image_classification.py --input image.jpg
Seeing an error? Check out the Using the Vision Kit section of the help page for troubleshooting tips.
Like the camera image classifier, you will get a list of results, which includes the kind of object and the model’s level of confidence.
Nothing happened: If you’re brought back to the prompt and don’t see any output in white, the model didn’t detect anything, womp. Try taking a new photo and then running the command again.
When you’re done with your Vision Kit for the day, it’s important to shut it down properly before unplugging it, to make sure you don’t corrupt the SD card.
If you've connected your kit to a monitor, mouse, and keyboard, you can shut it down by opening the applications menu (the Raspberry Pi icon in the top-left corner of the desktop) and then clicking Shutdown.
Otherwise, if you're connected to the kit with an SSH terminal, type the following command and press enter:
sudo poweroff
After a few moments, the green LED on the Raspberry Pi will turned off (look through the hole labeled SD Card), indicating that the kit is powered off.
You can then safely unplug the power supply from your kit.
To reconnect your kit, plug your kit back into the power supply and wait for it to boot up (about 2 minutes).
If you’re using a monitor, mouse, and keyboard, make sure they’re connected before you plug in your kit. Once the kit is booted, open up a terminal and you’re good to go.
If you’re using SSH, wait until the green LED stops flickering before connecting via SSH. Once your kit is booted, reconnect via the Secure Shell Extension (review the steps to connect to your kit). Note: You might have to re-pair your kit via the app.
Congrats! You’ve setup your very own intelligent camera.
Now that you’ve got a taste for the Vision Kit can do, you can start hacking the kit to build your own intelligent vision projects.
In the following Maker's Guide, you'll find documentation about the Python APIs and hardware features available in the Vision Kit. It also describes how you can train your own TensorFlow model to perform new machine vision tasks.
Share your creations with the maker community at #aiyprojects
To support various features in the Vision Kit, we've built a Python library that handles a lot of programming dirty work for you. It makes it easy to perform an inference with a vision model and draw a box around detected objects, and to use kit peripherals such as the button, LEDs, and extra GPIO pins.
These APIs are built into a Python package named aiy, which is pre-installed in the kit's system image. Just be sure that you've installed the latest system image.
To learn more about these APIs, refer to the API reference. In particular, the following APIs will be of interest for use with your Vision Kit:
aiy.toneplayer:
A simple melodic music player for the piezo buzzer.aiy.trackplayer:
A tracker-based music player for the piezo buzzer.aiy.vision.annotator:
An annotation library that draws overlays on the Raspberry Pi’s camera preview.aiy.vision.inference:
An inference engine that communicates with the Vision Bonnet from the Raspberry
Pi side.aiy.vision.models:
A collection of modules that perform ML inferences with specific types of image
classification and object detection models.aiy.board:
APIs to use the button that’s attached to the Vision Bonnet’s button connector.aiy.leds:
APIs to control certain LEDs, such as the LEDs in the button and the privacy
LED.aiy.pins:
Pin definitions for the bonnet's extra GPIO pins, for use with gpiozero.You might find it easier learn the aiy Python API if you start with an existing demo and modify it to do what you want.
You've seen some of these demos above, so they're already installed on your kit at ~/AIY-projects-python/src/examples/. You can also browse the examples on GitHub, where you'll find the source code for all the examples and more.
For instance, to learn more about the aiy.vision.inference and face_detection API, try running the face_detection.py example:
cd ~/AIY-projects-python/src/examples/vision ./face_detection.py --input image.jpg --output result.jpg
For each face detected in image.jpg, the demo prints information such as the face score and joy score. It also creates an image to the output location, which is a copy of the image that includes a box around each face.
To see how it works, open this file on your Raspberry Pi or see the source code here. Then start tweaking the code.
If you're more interested in programming hardware such as buttons and servos, see the section below about the GPIO expansion pins, which includes some other example code.
To further customize your project, you can train a TensorFlow model to recognizes new types of objects, and use our Vision Bonnet compiler to convert the model into binary file that's compatible with the Vision Bonnet.
Give it a try right now by following our tutorial to retrain a classification model.
If you want to build your own TensorFlow model, beware that due to limited hardware resources on Vision Bonnet, there are constraints on what type of models can run on device. We have tested and verified that the following model structures are supported on the Vision Bonnet.
| Model type | Supported Configuration |
|---|---|
| MobileNetV1 | input size: 160x160, depth multiplier = 0.5 input size: 192x192, depth multiplier = 1.0 |
| MobileNetV1 + SSD | input size: 256x256, depth multiplier = 0.125 |
| SqueezeNet | input size: 160x160, depth multiplier = 0.75 |
For an example of how to retrain and compile a TensorFlow model for the Vision Bonnet, follow this Colab tutorial to retrain a classification model for the Vision Kit.
The tutorial uses Google Colab to run all the code in the cloud, so you don't need to worry about installing and running TensorFlow on your computer.
At the end of the tutorial, you'll have a new TensorFlow model that's trained to recognize five types of flowers and compiled for the Vision Bonnet, which you can download and run on the Vision Kit (as explained in the tutorial).
You can also modify the code directly in the browser (or download the code) to adjust the training parameters and provide your own training data. For example, you can replace the flowers training data with something else, like photos of different animals to train a pet detector.
Beware that although this script retrains an existing classification model, it still requires a large amount of training data to produce accurate results (usually hundreds of photos for each class). You can often find good, freely-available datasets online, such as from the Open Images Dataset.
Download the Vision Bonnet model compiler here.
To unzip the file, run tar -zxvf bonnet_model_compiler_yyyy_mm_dd.tgz. This should give you bonnet_model_compiler.par (you might need to chmod u+x bonnet_model_compiler.par to run it).
You can also download the TensorFlow models shipped with Vision Kit (except FaceDetection) in frozen graph format.
Here's the basic command to compile a model:
./bonnet_model_compiler.par \
--frozen_graph_path=<frozen_graph_path> \
--output_graph_path=<output_graph_path> \
--input_tensor_name=<input_tensor_name> \
--output_tensor_names=<output_tensor_names> \
--input_tensor_size=<input_tensor_size>
Take mobilenet_v1_160res_0.5_imagenet.pb as an example. Put mobilenet_v1_160res_0.5_imagenet.pb in the same folder as bonnet_model_compiler.par and run:
./bonnet_model_compiler.par \
--frozen_graph_path=./mobilenet_v1_160res_0.5_imagenet.pb \
--output_graph_path=./mobilenet_v1_160res_0.5_imagenet.binaryproto \
--input_tensor_name="input" \
--output_tensor_names="MobilenetV1/Predictions/Softmax" \
--input_tensor_size=160
input_tensor_name is the input node’s name of the inference part of TensorFlow graph. Similarly, output_tensor_names are the output nodes’ names of the inference part of TensorFlow graph. README.md of the downloaded file contains this information.
Due to the Vision Bonnet model constraints, it's best to make sure your model can run on Vision Bonnet before you spend a lot of time training the model. You can do this as follows:
Use the checkpoint generated at training step 0 and export as a frozen graph; or export a dummy model with random weights after defining your model in TensorFlow.
Use our compiler to convert the frozen graph into binary format, and copy it onto the Vision Kit.
~/AIY-projects-python/src/examples/vision/any_model_camera.py \ --model_path <path_to_model> \ --input_height <h> \ --input_width <w>
Model takes square RGB image and input image size must be a multiple of 8.
Note: Vision Bonnet handles down-scaling, therefore, when doing inference, you can upload image that is larger than model's input image size. And inference image's size does not need to be a multiple of 8.
Model's first operator must be tf.nn.conv2d.
Model should be trained in NHWC order.
Model's structure should be acyclic.
When running inference, batch size is always 1.
The following subset of TensorFlow operators can be processed by the model compiler and run on device. There are additional constraints on the inputs and parameters of some of these ops, imposed by the need for these ops to run efficiently on the Vision Bonnet processor.
| TF operators | Supported on device configuration |
|---|---|
| tf.nn.conv2d | Input tensor depth must be divisible by 8 unless it is the first operator of the model. filter: [k, k, in_channels, out_channels], k = 1, 2, 3, 4, 5; strides: [1, s, s, 1], s = 1, 2; padding: VALID or SAME; data_format: NHWC; |
| tf.nn.depthwise_conv2d | filter: [k, k, in_channels, channel_multiplier], k = 3, 5, channel_multiplier = 1; strides: [1, s, s, 1], s = 1, 2; padding: VALID or SAME; data_format: NHWC; |
| tf.nn.max_pool | Input tensor depth must be divisible by 8. ksize: [1, k, k, 1], k = 2, 3, 4, 5, 6, 7; strides: [1, s, s, 1], s <= k; padding: VALID or SAME; data_format: NHWC |
| tf.nn.avg_pool | ksize: [1, k, k, 1], k = 2, 3, 4, 5, 6, 7; strides: [1, s, s, 1], s <= k; padding: VALID or SAME; data_format: NHWC |
| tf.matmul | Suppose a is MxK matrix, b is KxN matrix, K must be a multiple of 8. a: rank-1 or rank-2 tensor; b: rank-1 or rank-2 tensor; transpose_a: False; transpose_b: False; adjoint_a: False; adjoint_b: False; a_is_sparse : False; b_is_sparse: False; |
| tf.concat | axis: 1, or 2, or 3 |
| tf.add | Yes |
| tf.multiply | Yes |
| tf.nn.softmax | dim: -1. |
| tf.sigmoid | x: tensor's shape must be [1, 1, 1, k]. |
| tf.nn.l2_normalize | Input tensor depth must by a multiple of 8. dim: -1. |
| tf.nn.relu | Yes |
| tf.nn.relu6 | Yes |
| tf.tanh | Yes |
| tf.reshape | First dimension tensor can not be reshaped. That is shape[0] = tensor.shape[0]. |
The pretrained MobileNet based model listed here is based on 300x300 input and depth multiplier of 1.0, which is too big to run on Vision Kit. You can train a smaller model with supported configuration (MobileNet + SSD, input 256x256, depthwise multiplier 0.125), this requires changing the input size and depth multiplier. Unfortunately, if you are following their retraining tutorial, you cannot retrain (fine tune) a depth multiplier 1.0 model to use a different depth multiplier. At this point, you have to train from scratch.
Known supported architecture, MobileNet + SSD
Verified configuration:
| Input height x width | Depthwise multiplier |
|---|---|
| 256 x 256 | 0.125 |
Use embedded version of training configuration embedded_ssd_mobilenet_v1_coco.config
Let’s take training PASCAL VOC dataset locally as an example.
fine_tune_checkpointtrained_checkpoint_prefix, it is usually model.ckpt-${CHECKPOINT_NUMBER}. bonnet_model_compiler to compile the model./bonnet_model_compiler.par \ --frozen_graph_path=frozen_inference_graph.pb \ --output_graph_path=customized_detector.binaryproto \ --input_tensor_name="Preprocessor/sub" \ --output_tensor_names="concat,concat_1" \ --input_tensor_size=256 \ --debug
~/AIY-projects-python/src/examples/vision/any_model_camera.py \ --model_path <path_to_model> \ --input_height <h> \ --input_width <w>
If you plan to take your project beyond the cardboard box, you might be wondering which GPIO pins are available for your other hardware. So figure 1 shows exactly which pins from the Raspberry Pi are used by the Vision Bonnet.
Figure 1. GPIO pins used by the Vision Bonnet (highlighted pins are used)
The Vision Bonnet also includes a dedicated microcontroller (MCU) that enables the following additional features:
The extra GPIO pins are provided on the top of the Vision Bonnet (see figure 2). You can control the GPIOs and LEDs with the gpiozero library, using pin names PIN_A, PIN_B, PIN_C, PIN_D, LED_1, and LED_2.
Figure 2. GPIO expansion pins on the Vision Bonnet
The gpiozero-compatible pin definitions are provided by the aiy.pins package. You can use these definitions to construct standard gpiozero devices like LEDs, Servos, and Buttons.
If you want to dig deeper into these pins, checkout the MCU (SAM D09) docs—the bonnet GPIO pin names correspond to the MCU pins as follows:
Also see how to read the analog voltages.
WARNING: Before you connect any wires to the Vision Bonnet, be sure your Raspberry Pi is disconnected from any power source. Failure to do so could result in electric shock, serious injury, death, fire or damage to your board or connected components and equipment.
Note: The following example code might not be installed on your SD card right out of the box. Be sure that you are running the latest system image.
Although the LEDs on the bonnet are easy to use, you probably want your light to appear somewhere else. So connect an LED to PIN_A and GND as shown in figure 3. (Be sure the long/bent leg of the LED is connected to PIN_A; the resistor can be any size over 50 ohms.)
Then run the led_chaser.py example code:
cd ~/AIY-projects-python/src/examples/gpiozero ./led_chaser.py
It takes several seconds for the script to begin. Once it does, your light will blink on and off. To stop, press Control+C.
If the light does not blink, continue to wait another 15 seconds. If it still does not blink, look for any errors in the terminal window. Then press Control+C to stop the script, power off the kit, and double check all wiring. Then try again.
Figure 3. An LED connected to the Vision Bonnet
The led_chaser.py script is designed to light up 4 LEDs in sequence, as shown here:
from time import sleep from gpiozero import LED from aiy.pins import (PIN_A, PIN_B, PIN_C, PIN_D) leds = (LED(PIN_A), LED(PIN_B), LED(PIN_C), LED(PIN_D)) while True: for led in leds: led.on() sleep(0.5) led.off()
Of course, the code works fine with just one LED connected. But once you have the one LED working, try connecting LEDs to PIN_B, PIN_C, and PIN_D in the same way, and run the code again.
Because the GPIO pins on the Vision Bonnet are controlled by an on-board MCU, they perform pulse-width modulation (PWM) more precisely than the Raspberry Pi. So these pins are great for controlling servos.
To try it out, connect a servo to the GND, PIN_B, and 5V pins as shown in figure 4, and then run the servo_example.py script:
cd ~/AIY-projects-python/src/examples/gpiozero ./servo_example.py
It takes several seconds for the script to begin. Once it does, your servo should rotate back and forth between the minimum, maximum, and neutral position. But each servo can be a little different, so you might need to tune the parameters of the code to achieve a perfect alignment with your servo's full range of motion.
If the servo does not respond, continue to wait another 15 seconds. If it still does nothing, look for any errors in the terminal window. Then press Control+C to stop the script, power off the kit, and double check all wiring. Then try again.
Figure 4. A servo connected to the Vision Bonnet
The servo_example.py script uses the gpiozero Servo object to control the servo. The important parts of the script look like this:
from gpiozero import Servo from aiy.pins import PIN_B # Create a servo with the custom values to give the full dynamic range. tuned_servo = Servo(PIN_B, min_pulse_width=.0005, max_pulse_width=.0019) # Move the Servos back and forth until the user terminates the example. while True: tuned_servo.max() sleep(1) tuned_servo.mid() sleep(1) tuned_servo.min() sleep(1)
To adjust the rotation range of your servo, open the Python script and adjust the parameters of the Servo() constructor. Also see the Servo API documentation.
For more examples using the GPIO pins, see the AIY GitHub examples.
All of these example files are already available on your Vision Kit in the direcory ~/AIY-projects-python/src/examples/. Just be sure you have the latest system image on your SD card.
If you want to modify the button interface (such as to change the actual button), be sure to follow the wiring pinout as shown in figure 5.
Figure 5. Pinout for the bonnet button connector
Note: The push button built onto the board functions exactly the same as the button connected to the button connector. They both activate GPIO23 on the Raspberry Pi.
By default, your Vision Kit runs the Joy Detector demo when it boots up. This is enabled using a systemd service, which is defined with a .service configuration file at ~/AIY-projects-python/src/examples/vision/joy/joy_detection_demo.service, and it looks like this:
[Unit]
Description=AIY Joy Detection Demo
Requires=dev-vision_spicomm.device
After=dev-vision_spicomm.device
Wants=aiy-board-info.service
After=aiy-board-info.service
[Service]
Type=simple
Restart=no
User=pi
Environment=AIY_BOARD_NAME=AIY-Board
EnvironmentFile=-/run/aiy-board-info
ExecStart=/usr/bin/python3 /home/pi/AIY-projects-python/src/examples/vision/joy/joy_detection_demo.py --enable_streaming --mdns_name "${AIY_BOARD_NAME}" --blink_on_error
[Install]
WantedBy=multi-user.target
The .service file accepts a long list of configuration options, but this example provides everything you need for most programs you want to run at bootup.
To create a service like this to start your own app at bootup, just copy this configuration to a new file such as my_program.service (the name must end with .service). Then change ExecStart so it points to your program's Python file (and passes it any necessary parameters), and change Description to describe your program.
Then you need to put this file into the /lib/systemd/system/ directory. But instead of moving this file there, you can keep it with your program files and create a symbolic link (a "symlink") in /lib/systemd/system/ that points to the file. For example, let's say your config file is at ~/Programs/my_program.service. Then you can create your symlink as follows:
# Create the symlink sudo ln -s ~/Programs/my_program.service /lib/systemd/system # Reload the service files so the system knows about this new one sudo systemctl daemon-reload
Now tell the system to run this service on bootup:
sudo systemctl enable my_program.service
All set! You can try rebooting now to see it work.
Or manually run it with this command:
sudo service my_program start
If you want to stop the service from running on bootup, disable it with this command:
sudo systemctl disable my_program.service
And to manually stop it once it's running, use this command:
sudo service my_program stop
You can check the status of your service with this command:
sudo service my_program status
If you'd like to better understand the service configuration file, see the .service config manual.
If you need to see more logs to help with debugging (or you're simply curious to see more output), you can view system logs and program-specific logs using the journalctl tool.
By default, this prints a lot of system information that won't be useful to you, so it's best if you launch your program as a service and then tell journalctl to print only the logs from that service.
For example, if you start the Joy Detector demo as a service (or it's already running, as usual), you can begin printing all log output for that service with this command:
sudo journalctl -u joy_detection_demo -f
The -f option continuously prints new log entries as they occur. To stop printing the log, press Control+C.
The Face Detector model locates and identifies faces from an image. It also provides a “joy score” for each face.
The Dog / Cat / Human Detector can identify whether there’s a dog, cat, or person in an image and draw a box around the identified objects. It’s based on the MobileNet model architecture.
The Dish Classifier model is designed to identify food in an image. It’s based on the MobileNet model architecture and trained to recognize over 2,000 types of food.
The Image Classifier demo is designed to identify 1,000 different types of objects. This demo can use either the SqueezeNet model or Google's MobileNet model architecture.
Nature explorer has 3 machine learning models based on MobileNet, trained on photos contributed by the iNaturalist community. These models are built to recognize 4,080 different species (~960 birds, ~1020 insects, ~2100 plants).
To get the latest bug fixes and features, update the system image for your kit as follows:
.img.xz file from our releases page on GitHub..img.xz file you
downloaded above.When flashing is done, put the microSD card back in your kit and you're good to go!
If you're having trouble assembling your kit or running the demos, try the following resources:
You did it! Whether this was your first hackable project or you’re a seasoned maker, we hope this project has sparked new ideas for you. Keep tinkering, there’s more to come.