Abstract- In this paper we present the integration of a mobile robotic system with a novel color based object tracking mobile robot control algorithm based on spatial coordinates under non-ideal environments. For object tracking we use a visual Wireless sensor system used in the field of robotics for identification and tracking of the object. The program is designed to capture a color based object through a Wireless Surveillance Camera interfaced with PCI Based Image Grabber TV Tuner card integrated with LABVIEW and MATLAB software for tracking the object based on the Spatial Coordinates with RGB Color. The complete algorithm is integrated in LABVIEW Mathscript the resultant data are send serial COM port for controlling the Robot action based on the Threshold values. The hardware design including the microcontroller description and the wireless data link is discussed.
Keywords-Spatial Coordinates, Wireless Vision,Object tracking,Microcontroller,Wireless RF Link,Mobile Robot
Get your grade
or your money back
using our Essay Writing Service!
Robotic systems are becoming ever more complex everyday and the need for visual feedback based systems is increasing by the hour for the development of efficient self-adaptive mobile robotic systems. In the system we present here, we amalgamate an established color based tracking algorithm and a robotic system. The tracking and is done on a desktop computer and the appropriate signals are sent to a micro-controller through a wireless interface.
The system attempts to keep the object being tracked in the center of its field of view. Therefore the robot motion is discrete in nature rather than continuous. Incremental signals are sent to the micro-controller, i.e. if the object has moved farther away from the center of field of view the magnitude of the required motion will be greater and vice versa. Electronic systems onboard the robot has two primary tasks, to communicate with the PC and to control the motion of the robot. A wireless serial link is established between the computer and the micro-controller using wireless RF modules. Motion control is achieved by interfacing two geared DC motors to a standard H-Bridge motor driver circuit. The microcontroller interprets the signals received from the computer and accordingly controls the motion of the robot by sending signals to the motor driver circuitry.
The mechanical structure is designed keeping in perspective the weight and size requirements, and the required degree of mobility. Figure 1 illustrates the various logical and physical connections between the systems. The paper does not aim designing of mobile robot this system describes about the integration of simple mobile robot interface with LABVIEW Mathscript for tracking the Object based on Spatial Coordinates. In this paper we describe about image capturing and processing techniques, followed by an introduction to actual robotic application to trace the red ball using the serial COM port of the PC through wireless interface.
Section II describes about the robotic vision and control. Section III describes about the RGB and Binary Images for detection of object based on RGB Values. Section IV describes the details of object detection, positioning and RGB Threshold values settings. Section V describes the LABVIEW Mathscript interfacing details with system model implementation details and its results are discussed in the following in section VI.
Robotic vision and control
The whole system of making a robot to follow a red ball can be divided into four blocks: image acquisition, processing of image, decision-making and motion control. Image acquisition can be achieved by using a PC-based webcam or a digital video camera. This device will capture the image and send it to the camera processor for further processing in the computer. Its main function is to convert the light energy received into electrical signals. Image processing involves conversion of RGB color images into grayscale images, setting of threshold levels, saturation of the features into binary images and setting of cut-off values to remove noise in the binary image. Decision-making is done through the software program and motion control through either software or constant monitoring by the operator from the keyboard. Before going to the software program in detail, let us see the coordinate systems to understand the image acquisition process.
Block Diagram of Robotic Vision and Control System
Always on Time
Marked to Standard
Generally, the most convenient method for expressing locations in an image is to use pixel coordinates. Here, the image is treated as a grid of discrete elements, ordered from top to bottom and left to right. That is, the first component 'r' (the row) increases downward, while the second component 'c' (the column) increases to the right (see Fig. 2). For example, the data for the pixel in the third row, second column is stored in the matrix element (3, 2). When the syntax for a function uses 'r' and 'c,' it refers to the pixel coordinate system. MATLAB matrix subscripting is used to access the values of individual pixels.
Consider a pixel as a square patch. From this perspective, a location such as (3.3, 2.2) is a spatial coordinate. Here, locations in an image on a plane are described in terms of 'x' and 'y.' When the syntax uses 'x' and 'y,' it refers to the spatial coordinate system. Fig. 3 shows the coordinate convention of the spatial coordinate system. Notice that 'y' increases downward. An image may be defined as a two dimensional f(x, y) function, where 'x' and 'y' are spatial coordinates, and the amplitude of 'f' at any pair of coordinates (x, y) is called the intensity or gray level of the image at the point. The finite discrete values of the coordinates (x, y) and amplitude of 'f' are the digital images. An intensity image is a data matrix whose values have been scaled to represent intensities. When the elements of an intensity image are of class 'uint8' or class 'uint16,' they have integer values in the range [0, 255] and [0, 65535], respectively. The class of the elements used in this program is 'uint8.'
Pixel and Spatial Coordinate images
BINARY AND RGB IMAGES
Binary images (B&W).
A binary image is a logical array of 0's and 1's. Numeric-array images consisting of 0's and 1's are converted into binary, with '1' indicating white color (maximum intensity) and '0' indicating black color (minimum intensity):
>>imBw= [1 0 1; 0 0 1; 1 1 0]
A red, green and blue (RGB) image is an MxNx3 array of color pixels, where each color pixel is a triplet corresponding to the red, green and blue components of an RGB image at a specific spatial location. An RGB image may be viewed as a stack of three gray-scale images that, when fed into the RGB color monitors, produce a color image. Eight bits are used to represent the pixel values of each component of the image. Thus an RGB image corresponds to 24 bits.
object detection and positioning
Capture a frame and store it in a variable, say, 'rgb_image'
Extract the red, green and blue components of the images and store them in variables fR, fG and fB:
fR= rgb_image ( : , : , 1);%extracts the red component.
fG= rgb_image (:,: , 2);%extracts the green component.
fB= rgb_image (:, : , 3);%extracts the blue component.
Here, fR, fG and fB are image matrices.
Next, find the red object in the image. (R_THRESHOLD=)140, (G_THRESHOLD=)105 and (B_ THRESHOLD=)100 are specific numbers called 'threshold.' The technique for finding these numbers is described later on. The following statement creates a B&W image array 'I':
I= ((fRâ‰¥140) & (fGâ‰¤105) & (fBâ‰¤100));
That is, the result of logically 'ANDed' image matrices fR, fG and fB is stored in 'I.' If the following three conditions are satisfied, the pixel value of the image is set to '1':
FRâ‰¥140 if the value of the red component of the pixel is greater than 140.
fGâ‰¤105 if the value of the green component of the pixel is less than 105
fBâ‰¤100 if the value of the blue component of the pixel is less than 100
After making the B&W image, we would find that apart from the region of the red ball there are also some unwanted white regions in the image. These unwanted white regions are called 'noise.' Before you plot the centre of the image, filter the noisy parts of the image as follows:
This Essay is
a Student's Work
This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.Examples of our work
Se = strel ('disk', 20); % creates a flat, disk-shaped structuring element with radius 20
B= imopen (I, se);%morphological opening
Final= imclose (B, se);%morphological closing
Morphological opening removes those regions of an object which cannot contain the structuring element; smoothes object contours, breaks thin connections and removes thin protrusions. Morphological closing also tends to smooth the contours of objects besides joining narrow breaks and filling long, thin gulfs and holes smaller than the structuring element:
By obtaining the desired part, find the centre of the ball. The following statement computes all the connected components in a binary image:
[L, n]= bwlabel(Final),
Here 'n' is the total number of connected components and 'L' is the label matrix. (Each connected component is given a unique number.) The following statement:
[r, c]= find (L= = K) % K= 1, 2 â€¦n returns the row and column indices for all pixels belonging to the Kth object:
rbar= mean (n); cbar= mean(c);
Variables 'rbar' and 'cbar' are the coordinates of the centre of mass. As you have already filtered the image, the final image contains only one white region. But in case there is a computational fault due to excessive noise, you might have two connected components. So form a loop from '1' to 'n' using 'for' statement, thus calculating the centre of mass for all objects. If there are no components in a frame, the control doesn't enter the loop and 'rbar' and 'cbar' remain initialised to zero. For checking the output on the computer, use the following instructions:
plot (cbar, rbar, 'marker', ' * ', 'MarkerEdgeColor', B );
These statements pop-up a window with a 'blue' mark is plotted on the detected centre of mass of the red ball. They have been commented out in the main program for testing.
Position of the object
The position of the red ball is plotted as described below: Divide the frame captured by the camera (refer Fig. 4) into five parts by means of points 'x1' and 'x2' on X-axis and points 'y1' and 'y2' on Y-axis.
Calculate x1, x2, y1 and y2 by the following method:
x1= x/2-numx; x2=x/2+numx
'numx' and 'numy' are arbitrary numbers which you have to find out. These depend on the size of your ball. We have taken numx=120 and numy=30 in our program. Calculate the coordinates of the centre of the frame, which is nothing but (x/2, y/2). 'x' is the maximum dimension of X-axis (in the program it is 640) and 'y' is the maximum dimension of Y-axis (in the program it is 480).
Various conditions for detecting the position of the ball are:
If the ball is in region 5, it is at the centre of the frame. The robot should stop moving.
If the ball is in region 3, it is at the left of the frame. The robot should move left.
If the ball is in region 4, it is at the right of the frame. The robot should move right.
If the ball is in region 1, it is at the upper part of the frame. The robot should move forward.
If the ball is in region 2, it is at the lower part of the frame. The robot should move backward.
Define the above conditions in the code for decision making as given below:
(i) cbarâ‰¥x1 (output either '0' or '1')
(ii) cbarâ‰¤x2 (output either '0' or '1')
(iii) rbarâ‰¤y2 (output either '0' or '1')
(iv) rbarâ‰¥y1 (output either '0' or '1')
The above four conditions are considered to be four bits, which can be easily convert the information into a number for easy computation. This can be done by the following code:
e=((cbar>= x1)*2*2*2 % bit number 3
+(cbar<= x2)*2*2 % bit number 2
+(rbar>= y1)*2 % bit number 1
+(rbar<= y2)) % bit number 0
Thereafter, we can generate a simple switch-case code for outputting the appropriate data through serial port, to control the external device such as a robot. The decision table derived from the above conditions is shown in Table I. 'X' denotes a "don't care" situation. Some of the cases are imaginary in the practical world. The robot will stop in such situations. For calculating the dimensions of the frame, use the sizeof( ) function. This function returns the size of the frame.
Decision making rule based on threshold
cbar â‰¥ x1
cbar â‰¤ x2
rbar â‰¥ y1
rbar â‰¤ y2
Setting the RGB threshold values
The red ball is of particular interest. The steps for setting the RGB threshold values of the red ball follow:
1. Take at least ten snaps of the red ball at various angles
2. Read each image
3. Display the image by using 'imview' function
4. Note down the pixel values of the red ball by placing the mouse cursor at various portions of the red ball. The threshold for red component (R_THRESHOLD) should be the least value of the red component found in the red ball. The threshold for green component (G_THRESHOLD) should be the maximum value of the green component found in the red ball. The threshold for blue component (B_THRESHOLD) should be the maximum value of the blue component found in the red ball whose screenshots shown in Figure 7, 8, 9, 10 and Figure.11 consecutively. The least value of the red component is '144.' So R_THRESHOLD can be taken as '144.' The maximum value of the green component is '115.' So G_THRESHOLD can be taken as '115.' The maximum value of the blue component is '100.' So B_THRESHOLD can be taken as '100.' Syntax of the functions specified can be had from the MATLAB Help. If the image file is not in the current directory, specify the whole path of the file.
This paper focuses on the implementation of a wireless Mobile Robot control with Object tracking based on Spatial Coordinates and to process the images using MATLAB. To transfer the data to the mobile robot serial port COM is used for transferring the data byte by byte so that Wireless control of the robot is done from LABVIEW with MATHSCRIPT function with system overview is shown in the below Figure 3. Wireless control eliminates the constraint in distance between the color object and robot for better tracking from remote location. Wireless technique uses Radio Frequency waves for transmitting the control signals. The transmitter module uses the ON-OFF Key Modulation scheme for its transmission at the frequency of 433.92 MHZ.
LABVIEW with MATHSCRIPT
Wireless TV Receiver Module
Wireless Surveillance CAMERA
PCI based TV TUNER card is used for grabbing the images from the remote location for tracking the object based on the color with its spatial coordinated, monitoring the activities from remote place through LABVIEW Mathscript which is running the algorithm mentioned above. Since the wireless camera is placed on Mobile robot, it's necessary to use PCI or USB based TV tuner card for acquiring the remote images from camera. The PCI card is being programmed by means of MATLAB TOOL, in which DLL file act as a bridge between the MATLAB and PCI card for grabbing the images from camera. These images are used for future processing.
The serial binary data is received by the max 232 IC from the computer. The data is transferred to the microcontroller. Depending Upon the data received, the controller transmits 4-bit data to the encoder as shown in the below Figure 4. The encoder converts the received parallel data into serial data and transmits it to TWS 434. The transmitter then transmits the data at 433.9 MHz. The receiver RWS 434 receives the transmitted data serially and it is given to the decoder IC. The decoder converts the serial data into parallel data and is transmitted to the microcontroller. Depending upon the received data the controller drives the motor to perform the function. Below Figure.5 shows the RF Receiver interfacing with Microcontroller for mobile robot control. Transmitting though walls of buildings will reduce the range; how much will depend on the number of walls, and what they are made of.
Wireless RF-433Mhz Interfacing circuit with PC
Wireless RF-433Mhz Interfacing circuit with Robot
Motion control of the robot
The computer processes the image of the red ball and sends out five different data through its serial COM port. The data depends on the location of the red ball, upper, lower, left, right and centre of the frame, as captured by the camera. By mounting Wireless surveillance camera in front of the robot, so that it acts as a visual sensor. To control the Wireless Mobile robot serial data are transferred from PC by running the MATHLAB code on the PC integrated with LABVIEW for serial data transfer.
Here the robot will be controlled by sending the following codes to the serial port:
1. Key 'F' for forward movement
2. Key 'L' for left movement
3. Key 'R' for right movement
4. Key 'B' for backward movement
5. Key 'S' for Stop robot movement
Any other code will make the robot stop. The code for implementing motion control of the robot is given at the end of this article.
Image Acquisition using MATLAB
The Image Acquisition Toolbox 1.0 is a new product from The MathWorks that enables you to connect to an image acquisition device from within a MATLAB session. Based on MATLAB object technology, the Image Acquisition Toolbox provides functions for creating objects that represent the connection to the device.
The wireless camera acquires the video of the surrounding and it transmits through a wireless module which is received through a TV receiver module. It is fed to the computer through TV tuner PCI card and the video can be viewed in the computer. For acquiring the real time images from remote location we have invoked the vcap2.dll with MATLAB.
Serial Communication with LABVIEW
Some of the VISA functions on this palette for GPIB communication. The VISA Write and VISA Read functions work with any type of instrument communication and are the same whether you are doing GPIB or serial communication. However, because serial communication requires you to configure extra parameters, you must start the serial port communication with the VISA Configure Serial Port VI. The VISA Configure Serial Port VI initializes the port identified by VISA resource name to the specified settings. The timeout sets the timeout value for the serial communication. Baud rate, data bits, parity, and flow control specify those specific serial port parameters.
Figure 5 shows how to send the identification query command *IDN? to the instrument connected to the COM2 serial port. The VISA Configure Serial Port VI opens communication with COM2 and sets it to 9,600 baud, eight data bits, odd parity, one stop bit, and XON/XOFF software handshaking. Then the VISA Write function sends the command. The VISA Read function reads back up to 200 bytes into the read buffer, and the Simple Error Handler VI checks the error condition
Serial Communication under LABVIEW Environment
For controlling the wireless mobile robot based on the spatial co-ordinates we need to consider the following RGB_THRESHOLD image values of the real time image acquired from the Wireless Surveillance Camera via PCI based Image Grabber i.e., FRONT TECH TV Tuner Card. The following values help the system to take the exact decision rules based on these values
(G_THRESHOLD=) 105 and
(B_ THRESHOLD=) 100
These values help us to obtain the exact object detection for the frame 320x240 image size. Once the object is detected, we need to find the position of the object located in the frame size mentioned above. In order to obtain the exact position of the object we need to divide the frame into 4 equally spaced quadrants. Based on the object detected on the quadrants the robot control is done through wireless RF Modules. Various conditions for detecting the position of the ball based on the quadrants are already discussed in the section IV.
The decision making is done by implementing simple switch case for robot movement based on different position of the object located on the frame and these coding are executed in MATLAB. In order to interface with Wireless Surveillance camera we have used the VCAP2.DLL files for interfacing with the MATLAB since this VCAP2.DLL is capable of accessing up to nearly 6 Image grabbing card simultaneously.
The complete MATLAB code is executed under LABVIEW environment by incorporating the MATLAB code along with the LABVIEW. Since LABVIEW has a toolbox known as LABVIEW Mathscript which is used for interfacing the MATLAB and LABVIEW for better efficient application building. LABVIEW is used for serial communication to transfer the data from PC to the Mobile robot for controlling the action based on the decision making Rule as described in the section IV under Table I. The output results are shown in below Figures 7, 8,9,10 and 11 consecutively which clearly shows the robot movements based on the spatial coordinates by executing the MATLAB code under LABVIEW Mathscript.
Robot Backward Movement
Robot Forward Movement
Robot Left Movement
Robot Right Movement
Robot Null action
LABVIEW Program for Object tracking under LABVIEWMathscript
The equivalent robot control data are sent from the LABVIEW environment through serial communication COM port for controlling the robot actions. The robot control command is already discussed in the above Section V for better understanding.
Vision based object tracking has been challenging research area for many of the researchers for so many decades. In this paper our ultimate goal is to interface low cost PCI based image grabber card for visual robot control from remote location through Wireless means.One of the major advantages of implementing such system is that the low cost Machine vision applications such as Image processing based Inspection system would be build with such PCI based Image grabber cards. Any low cost image grabber cards can be used for interfacing with LABVIEW rather than purchasing the customized Frame Grabber Cards with expenditures. Even more applications such as Biometric, Human Tracking and GIAT analysis would also be performed with such low cost cards.
The current test and its results motivate towards doing further research on this project. The roadmap for further work is outlined below. Object detection would be carried out by background subtraction using appropriate thresholds which could be used for implementing Autonomous Vision based Turret triggering for military applications. Use of two cameras to employ stereovision to assist in 3D reconstruction of the object which would able to track the object on both the front and rear vision for object tracking A study of use of multiple robots in a network to detect and track down multiple, big objects co-operatively. Study on the effect of frame rate of the camera and motion parameters on the performance of the system can be provoked.