Robot Programming Using Augmented Reality Computer Science Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

General Problem: How to program efficiently industrial robot? Robots are increasingly present in our world; they help us to accomplish tasks that were once complicated and long for a man. In the industrial world, they are now becoming unavoidable. But the engineers and programmers are always trying to push the boundaries of these steel monsters. Thus, a natural question arises, is it possible to program industrial robots more effectively?

The programming of robots is not an easy task. This requires a precise methodology in a specific context. It is a complex task requiring specific language in almost every robot. Thus, the rapid development of robots forced engineers to ask whether there are methods of programming to make these machines more efficient, more accurate, and safer [1]. In this paper, different existing methods will be explained, and then further study will be done on the robot programming using augmented reality. The process itself is explained as well as the main problems encountered by this method.

Existing methods:

When we want to programming a robot, we have the choice between 3 basic programming techniques:

In these methods, we can differentiate 2 types of programming: on-line and off-line methods.

The Off-line method consists of writing the program in a remote computer, to simulate it in a virtual environment and to check the behavior of the robot in the virtual reality. When the performance is satisfying, this one is downloaded or implemented in the robot and the robot can start the production of a new product.

This method has many advantages: it allows the operator to program a robot from a computer separate of the production area; we can program it when the robot is still in production and we can use some programming tools. Some tools are developed to optimize the process, to assist in the cell design or to facilitate the programming.

The drawbacks of this method are that we need to have a virtual representation of the robot and of the environment to realize a correct simulation. We need to have a personal qualified to program a robot, like a robot programmer or an application engineer, because it is complex to do that.

So this method is costly because we need to create the virtual reality and to have a robot programmer. This method permits to have a higher utilization of the robot because the downtime is reduced. [2]

The first basic and on-line method is called «lead through» programming.

As we can see in the name of this method, the trainer manipulates the robot in the desired sequence of event. During this phase the robot is in «teach mode» and the operator uses a remote teach-pendant to move the robot. In the production phase, the robot plays back the task which is saved in its memory.

This method is easy and convenient. The drawback is when the trainer has some hesitations or makes some inaccuracies it cannot be edited out easily without reprogramming the whole task.

The last basic and on-line method is called «walk through» programming

This method is also known as «direct teaching» or «manual lead through». The main idea is the same that the lead through programming, we want to move the robot to do a task and the robot should repeat this movement. In this case, we do not use the teach-pendant to move the robot but we move the robot by hand. In other word, we move manually the end-effector and the robot saves the position of all joints.

This method can be used when each work piece has a unique dimension with some repeated patterns because this method is easy and fast. So we can use this method for spray painting and welding [3]. Robots which use this method can be used by unskilled or untrained workers because the operator can operate it out of his own common sense.

The drawback of on-line programming methods is that it is risky either for the human operator or the robot.

Virtual Reality

There is no uniform and single definition, but many different can be found. Most of them consider giving the sight of the user the impression of a real environment the most important one. This requires tracking of the users movements (especially head) in order to display the environment. Also the ability to change the virtual environment is important. Stimulating other senses, most often hearing, but also tactile or smell can be part of it. It can be used to try to emulate the real world or to build up a Fantasy world.

Augmented Reality


Augmented Reality (AR) stands for the combination of real world and virtual reality elements, so as to enable a user to take effect on this hybrid world.

Azuma [6] defines AR by the following three characteristics:

-Combines real and virtual

-Interactive in real time

-Registered in 3-D


In order to see the AR the user needs to have a special display. The most common variant are head mounted displays (HMD). They look like glasses but have displays instead of lenses.

Both cameras are used and their images are displayed as well, or see through devices to show the real part of the environment. For the virtual part overlapping and view angle are calculated with the data acquired by position sensors attached to the user, a view is computed and then rendered and the data is sent to the display.

To stimulate the hearing standard stereo surround systems can be used.

For the tactile/force feedback to be possible (and also for easy steering with natural movements), the user has to wear specific clothes, like for example gloves to enable his fingers to move objects and get feedback from servos mounted on there.


Not the whole environment has to be included in the computer simulation, and the user can even help to include reality. It can be used directly in existing environments to check in real time if the task is solved right, and for example no unwanted collisions occur.

It is much more intuitive than lead-through or VR, because one can see the robot in its real environment and check right there for example if no collisions occurs.

VR and AR don't need an actual robot, therefore a lot of money can be saved by using these methods, since normal robots are expensive, need a lot of space, and downtimes also lead to much lower production and thus reduced revenue. Also since virtual robots can't do any damage to the environment, there are no safety issues, and the flexibility is much higher, the robot can just be displayed in another place.

Class I Task Definition

In that part, we will define the class I task which simply consist of moving from a point to another without hearting obstacles. This is typically a task where the robot is at a certain position and wants to take a piece in another position. Then it moves this piece from the second position to a third one. The entire movement is a combination of two class I tasks.

Class I Task Methodology

First, the user has to know how the robot reacts and moves by moving it with the help of a probe.

Then, the user generates a collision free volume (CFV) by the use of a sphere attached to the probe. This volume will be the one inside which the end-effector of the robot will be allowed to move. To generate this CFV, the user moves the sphere in the space, where he/she wants the CFV. The generated CFV is showed to the user to help him/her in this task.

After that, the virtual robot can be moved inside the CFV to check if there is no collision between the robot and the environment with the generated CFV. If there is, the user has to restart the CFV generation, if not he/she confirms the generated volume.

Once the volume generated, the user gives the information about which point should be the start point and which one should be the end of the movement, always by the use of the probe.

The last step is done by the computer itself and consists of finding the more efficient path that corresponds to the path with the smallest cost. This research of optimal path is done by a beam search algorithm which uses a heuristic to find this path as fast as possible.

Class I Task Results

A class I task can be planned within 2-3 minutes with the AR method. This method is very simple and intuitive to use, making it possible to plan robot movements in a very short time and without a lot of advanced knowledge.

Class II Task Definition

A class II task is defined as a task that requires the end-effector to follow a specific path defined by the user, at a specific orientation with respect to the path. Some examples of class II tasks include arc wielding and laser cutting.

Class II Task Methodology

The user first has to use the end-effector probe to perform a few demonstrations of the desired path for the robot to follow. As this is done, data points of the positions of the end-effector probe are generated and are visible to the user. This data is then processed using a Bayesian neural network to produce a curve from the data points.

Alternatively, instead of using the end-effector probe to demonstrate the entire path, sequential paths with straight lines and sharp edges can be more easily drawn by specifying the start and end points, and connecting them using linear interpolation.

Next, the user should then familiarize with moving the virtual robot around in the environment. He will then "paint" the Collision Free Volume (CFV) around the curve using the probe. This is used to ensure that the there is no collision between the robot's end-effector and any obstacles along the curve.

The user then uses the AR interface to specify the orientation of the end-effector with respect to the curve.

Finally, a simulation is done to calculate the end-effector transformations with respect to the robot base for all points along the curve. Using inverse kinematics, the angular profile for the robots joints can be obtained.

If the end-effector is not within the CFV at all times, collision is possible. The orientation of the end-effector has to be adjusted and the simulation is repeated until a successful path is found.

Class II Task Results

On average, complex curves can be programmed within 6-8 minutes from data collection to simulation, using three good demonstrations.

Problem with AR:

Today, the augmented reality systems use modern technologies for tracking more accurately. These technologies are optical sensors, digital cameras, accelerometers, GPS, gyroscopes, solid state compass, RFID and wireless sensors [4]. The problem that ensues is that each of these technologies has a different precision and as a result a different cost.

Mainly problems related to augmented reality are due to path planning. This means specifically looking for a continuous collision-free volume (CFV), which defines a subset of the entire free space. Thus, path finding algorithm are perform in this CFV in order to find a way in which the robot to not touch any other element. In addition, finding a good CFV avoid the fact to model the complete environment. The robot does only need to know that there is nothing to touch inside the bounded volume of the CFV. Indeed, it is useless to try to model the entire world. Therefore, finding a good CFV is very important. Regardless this observations, "The quality of the CFV can be evaluated based on its compactness and regularity in shape." [5]

[1], [10/04/20]


[3] A walk-through programmed robot for welding in shipyards, Marcelo H. Ang Jr, Wei Lin and Ser-Yong Lim, Industrial Robot: An industrial Journal Volume 26, Number 5, 1999, pp.377-388

[4], [10/04/20]

[5] Robot programming using augmented reality: An interactive method for planning collision-free paths, J.W.S. Chong, S.K. Ong, A.Y.C. Nee, K. Youcef-Youmi; Robotics and Computer-Integrated Manufacturing 25 (2009) 689- 701

[6] A survey of Augmented Reality; Ronald T. Azuma; Presence: Teleoperators and Virtual Environments 6, 4 (August 1997), 355-385


What are the advantages of AR compared to older methods (VR, walk-through, lead-through)?

Difference online-offline programming?

Difference AR - VR?

Explain CFV

Methods currently used for programming?

Problems of AR?

Why does AR seem promising to be used in medical applications?

Domains to use AR?

How to use AR to implement a pick and place task for a roboto?

A: method in paper

Why robot path planning algorithms instead of human demonstration?

A: human: not smooth; maybe not ideal path, restricted by human body.