Control Camera position with smartphone gyroscope

Beginning

In the previous tutorial (here) I showed how to display the image from the Raspberry Pi camera in the browser window. In this tutorial I will show you how to control the position of the camera. We will control it in two ways

By moving the mouse / finger on the video component (window)
Tilting the phone and using the built-in gyroscope

All this on the website opened in the browser (I used Chrome for both the desktop and smartphone, because it has the best support for the webRTC protocol).

Don’t forget to checkout the video tutorial :

How it’s going to work

Just as the camera image is sent via the webRTC protocol, we will use the same channel to send instructions to control the position of the camera. Messages from the browser will go to the Python script on Raspberry Pi, and this script will drive the servo driver accordingly. It sounds simple and so it will be – sending messages is completely on the RemoteMe side, in the Python script you only need to implement a function that is called after changing the value of the variable – more about variables here

What is needed:

Camera preview
- Raspberry PI Zero W
- Camera Module with Ribbon cable
Control servo
- 16-bit PWM Adafruit Channel controller I2C
- Two 9g servos
- Servo Pan-tilt Mechanism (Pan-tilt with servos included)
Power – it does not give specific applications – because they depend on the motors and on what kind of batteries we have at our disposal. I used LiPo 4 * 1.2V batteries with a step-down converter

Connecting servo mechanisms

To control the servo mechanisms, we will use a 12-channel LED driver 16-bit PWM I2C compatible with Adafruit. It communicates with Rpi via the I2C interface. Adafruit provides a python library that works out-of-the-box, If in the second step RemoteMe installation on RPi you have chosen the [Yes] option, you have this library already installed;).

Connections:

SDA, SCL, and ground pins connect to RPI as shown:

Rpi pins:

RPI pins image here

Mounting the webcam tilt handle – it should not be difficult, the only thing you need to pay attention to is the assembly of the holder, so that it looks straight ahead and what is most important – the servo mechanisms were in a central position.

In the pictures and the movie you can see the green PCB to which the RPI and the servo driver are mounted, the files you can download from here it was designed in such a way that it can be easily performed using the thermal transfer method. In the repository there are also gerber files for easy ordering, eg from allpcb.com where I ordered my PCB.

The plate will be more useful when we add engine control to our car in the next tutorial.

How its going to work:

For camera control we will use a variable of the type SMALL_INTEGER_2, which sends two numerical values, one of them will control the axis x with the second axis y.
The variable will be changed by phone or computer where the image will be displayed
- By moving the mouse / finger around the area where the video is displayed
- Using a gyroscope in the smartphone where the rotation from the starting position will control the camera axes

Let’s start:

We will start by creating the variables – tab variables -> add the appeared window we complete:

We create by clicking Submit more about variables here

Then, connect our Raspberry Pi to RemoteMe.org, how to connect here. Next, on the bar of the connected raspberry, click the burger menu and “Add External script with wizard”

In the first step, select the variable cameraPos

Thanks to this, in the generated code we will have functions ready to be implemented to interact with our variable.

In the second step, we name our device “python” and any unused Id. In the last step, click “Create device”. More about python devices here.

In the open window, we will add a code to support our servo mechanisms:

import logging
import socket

import struct
import sys
import os

os.chdir(sys.argv[1])
sys.path.append('../base')

import remoteme

import Adafruit_PCA9685 # Added
import time # Added
import RPi.GPIO as GPIO # Added

pwm = None; # Added

def setCameraPos(i1, i2):
    remoteMe.getVariables().setSmallInteger2("cameraPos", i1, i2)



def onCameraPosChange(i1, i2):
    global pwm # Added
    logger.info("on camera change {} , {}".format(i1, i2)) # Added
    pwm.set_pwm(1, 0, i1) # Added
    pwm.set_pwm(0, 0, i2) # Added
    pass # Added

def setupPWM(): # Added
    global pwm # Added
    pwm = Adafruit_PCA9685.PCA9685() # Added
    pwm.set_pwm_freq(80) # Added
    
try:

    logging.basicConfig(level=logging.DEBUG,
    format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
    datefmt='%d.%m %H:%M',
    filename="logs.log")

    logger = logging.getLogger('application')
    setupPWM() # Added
    remoteMe = remoteme.RemoteMe()

    remoteMe.startRemoteMe(sys.argv)
    remoteMe.getVariables().observeSmallInteger2("cameraPos" ,onCameraPosChange);

    remoteMe.wait()

finally:
    pass

import logging

import socket

import struct

import sys

import os

os.chdir(sys.argv[1])

sys.path.append('../base')

import remoteme

import Adafruit_PCA9685 # Added

import time # Added

import RPi.GPIO as GPIO # Added

pwm = None; # Added

def setCameraPos(i1, i2):

remoteMe.getVariables().setSmallInteger2("cameraPos", i1, i2)

def onCameraPosChange(i1, i2):

global pwm # Added

logger.info("on camera change {} , {}".format(i1, i2)) # Added

pwm.set_pwm(1, 0, i1) # Added

pwm.set_pwm(0, 0, i2) # Added

pass # Added

def setupPWM(): # Added

global pwm # Added

pwm = Adafruit_PCA9685.PCA9685() # Added

pwm.set_pwm_freq(80) # Added

try:

logging.basicConfig(level=logging.DEBUG,

format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',

datefmt='%d.%m %H:%M',

filename="logs.log")

logger = logging.getLogger('application')

setupPWM() # Added

remoteMe = remoteme.RemoteMe()

remoteMe.startRemoteMe(sys.argv)

remoteMe.getVariables().observeSmallInteger2("cameraPos" ,onCameraPosChange);

remoteMe.wait()

finally:

pass

The added lines are marked with the #Added comment

Code overview

import Adafruit_PCA9685 # Added
import time # Added
import RPi.GPIO as GPIO # Added

pwm = None; # Added

import Adafruit_PCA9685 # Added

import time # Added

import RPi.GPIO as GPIO # Added

pwm = None; # Added

Adding libraries and global variable declaration pwm to control the PWM controller (Adafruit_PCA9685)

def onCameraPosChange(i1, i2):
    global pwm 
    logger.info("on camera change {} , {}".format(i1, i2)) 
    pwm.set_pwm(1, 0, i1) 
    pwm.set_pwm(0, 0, i2)
    pass

def onCameraPosChange(i1, i2):

global pwm

logger.info("on camera change {} , {}".format(i1, i2))

pwm.set_pwm(1, 0, i1)

pwm.set_pwm(0, 0, i2)

pass

This function will be called when you want to change the position of the camera with a smart phone or computer. In the parameters, we get the values of the x and y axes, and set the servo mechanisms connected to the PWM controller at position 0 and 1. We will already receive variables, with appropriate values, for example, the value (534.234) will mean the central position of the camera – therefore we simply enter the received numbers into the server driver without formatting them.

def setupPWM(): 
    global pwm 
    pwm = Adafruit_PCA9685.PCA9685()
    pwm.set_pwm_freq(80)

def setupPWM():

global pwm

pwm = Adafruit_PCA9685.PCA9685()

pwm.set_pwm_freq(80)

Function for PWM controller library initialization. The value of 80 was chosen by trial and error to my servo mechanisms. This function is called in the main program block.

After saving the code, the program on Raspberry Pi will be reset to load the new script.

Web Page

Now we will add a web page with two sliders to find the central position of the camera and maximum swing. In the ideal world, the numbers would be, for example, 0.100 for both values, i.e. sending a variable of value (50.50), we would set our camera in the central position, and the value (100,100) would direct it up and right. Unfortunately, these numbers will certainly be different and we have to find them. For this purpose, we will create a website with two sliders that will send variables and we will know what are the variables.

To add a website – choose “New Device” -> “Webpage” and complete it as follows:

After adding, expand the beam, click index.html and “edit with wizard”. Then “Insert component” and complete:

More about components here

The most important information

we will control two sliders of the CameraPos variable
the minimum and maximum number that we will send is 0 – 1000 for both axes

Click the insert, close the wizard window, and open the page in the new tab by clicking index.html and selecting Open in new tab.

We will get our website. The first slider is responsible for the X-axis of the camera (horizontal movements – ie looking left right) and the second one for the Y axis (ie looking up, down). When the sliders control differently, we replace the servo plugs with mechanisms in places. When the servos do not respond, check to see if they have been plugged into the right place.

Moving the sliders set the camera in a central position for me it is x = 564 and y = 474.

Then we tilt the camera up to the minimum values for me then x = 298. y = 223.

Then, calculate the maximum deflection for the camera, the calculation is to make the central position in the middle of the interval. So simple math

X_max = X_center-X_min + X_center = 564 – 298 + 564 = 830
Y_max = Y_center-Y_min + Y_center = 474 – 223+ 474 = 725

That is, X_min = 298, X_center = 564, X_max = 830. has an axis looks like this:

The point is that the distances between the values were the same and additionally the X_center value indicated the central position of the camera in the horizontal plane.

Y-axe : Y_min=223, Y_center=474, Y_max=725

On the sliders, check if the maximum values set the camera in the right position without any problem and the servos do not block – if they block, increase X_min and repeat the calculation (or calculate X_min based on X_max).

We are going back to the index.html page and delete the sliders removing the line from the page’s sources:

<variable  component="slider"  type="SMALL_INTEGER_2"  name="cameraPos" label="camera" min="0" max="1000" valuebox="true" ></variable>

1	<variable component="slider" type="SMALL_INTEGER_2" name="cameraPos" label="camera" min="0" max="1000" valuebox="true" ></variable>

And we re-open the Components Wizard and insert the components

camera:

Status – in the upper right corner it will display the connection status:

Finger / mouse control by moving on the image from the camera:

Please note that I have supplemented the Xmin values with the calculated values.

and Gyroscope – so that we can control the camera position by tilt the smartphone :

I also put the calculated values here. More about components and configurations here

It’s also worth changing the size of the video in index.html sources instead of width and height, enter style = "width: calc (100vmin); height: calc (75vmin)" – this will fill the camera image when the phone is in a vertical position

We open our website in Smartphone – the easiest way is by clicking on index.html -> Get anonymous Link -> click on the QR code icon and scan it with a smartphone.

We test by sliding the finger on the screen, when the camera instead of pointing up its moving down in the component <variable component = "cameraMouseTrack" index.html change invertX = "false" into invertX = "true"

If the same applies to the Y axis, we change the value of invertY in the same component

We turn on the gyroscope by clicking the Gyroscope button – it remembers the position of the smartphone and changes the position of the camera – when titl the phone , when moving is opposite then should be we changed similar like in previous step invertX invertY but now in component <variable component = "gyroscope". If we want the gyroscope to react more gently, we increase the values of the xRange and yRange properties

That’s all for today in the video at the beginning of the course are all steps.

Greetings,

Maciej

10 February 2019 Uncategorized Maciej