How AVs Work
Autonomous vehicles (AVs) have a computer that can control the speed and direction of the vehicle, either to assist a human driver or to perform all driving tasks independently.
These vehicles are equipped with sensors that are used to take in different kinds of information about the surrounding world. This data goes to the computer, which then has to detect all surrounding objects, make a motion plan, and adjust the vehicle’s speed and direction accordingly.
Below, we describe some of the details of these sensors and the operations of the AV computer.
Jump to…
Sensors
The three most common types of sensors in AVs today are cameras, lidar, and radar. They all have their advantages and disadvantages, and a mix of sensors is used to maximize the safety of the autonomous vehicle.
Cameras
Cameras are used to rapidly take images of the world. The computer uses these images to read road signs and traffic lights and to identify vehicles, pedestrians, and objects.
Pros:
- Can see high detail and color
- Sensors are cheaper than other types of sensors
- Can work in light inclement weather (rain, snow, fog, etc.) as long as road markings are clearly visible.
Cons:
- Data processing is expensive, slow, and challenging
- Can be blinded by sudden bright light or glare
- Does not work in the dark
- Does not work in heavy inclement weather (rain, snow, fog, etc.)
Lidar
Lidar uses a laser pulse to measure the distance to an object. The laser can be rotated to rapidly create a full 360-degree map of all the objects around the vehicle.
Pros:
- Reliably detects objects around vehicle without any image processing
- Data processing is very cheap and quick
- Can be used in total darkness
- Can work in light inclement weather (rain, snow, fog, etc.)
Cons:
- Cannot read road signs or see color
- Sensors are expensive (though the costs have decreased significantly in recent years)
- Can only detect objects at shorter distances (200 meters or less)
- Does not work in heavy inclement weather (rain, snow, fog, etc.)
Radar
Radar uses a pulse of radio waves to detect the speed and distance of objects.Radar can be used for many purposes, but is most useful for detecting moving objects in the path of the vehicle (e.g., another vehicle or pedestrian).
Pros:
- Reliably detects moving objects in the path of the vehicle without any image processing
- Works well for long-distance object detection (less well at very short distances)
- Data processing is very cheap and quick
- Can be used in total darkness
- Sensors are moderately cheap
- Can work in light and heavy inclement weather (rain, snow, fog, etc.).
Cons:
- Cannot read road signs or see color
- Struggles to distinguish stationary objects (e.g., stopped vehicles, road signs and clutter, etc.) from the background
- Narrow field of view. Cannot easily make a 3D object map
See a comparison of how lidar and cameras see the world.
Computer Processing
Object Detection & Classification
As the computer drives a vehicle, the raw sensor data needs to be transformed into a consistent understanding of the world around it. This involves separating foreground objects from the background, and classifying the objects the computer detects in the scene. For example, the vehicle might see three pedestrians across the street standing still, one pedestrian on the side of the road walking at a slow speed, and an oncoming truck traveling in the adjacent lane at 25 mph. All of these objects must be correctly detected and classified for the vehicle to operate safely.
A comprehensive mix of sensors gives the computer the best ability to classify objects. Things can go wrong when the computer only has camera data to work with. For example, in the image below, a computer tried to convert the camera data into a collection of objects. It mistakenly saw a large person standing on the road and did not correctly identify the large truck completely blocking the vehicle’s path. With lidar and radar data also available, the computer would be able to “see” the size and shape of the truck and would be much more likely to identify the whole truck as a truck. Cases like this are a major reason why autonomous vehicles should be guided by a diverse mix of sensors.
Image credit: Simon Stent, Toyota Research Institute, 2016
Computer Training: AI Learning and Mapping
AV computers are trained to drive by studying large data sets of previously recorded human driving. Human-driven test vehicles collect data by driving around to build up a repository of what different vehicles look like, what different pedestrians look like, etc. This data is labeled by a human (“vehicle,” “pedestrian,” “cyclist,” “stop sign,” etc. are common labels) and then fed into AI algorithms that learn to recognize these objects when they appear on the road. As more data is used for training and the algorithm improves, the AV learns to recognize all the different shapes, sizes, and colors of different vehicles, pedestrians, and other objects.
AVs are also guided by maps of the environments that they drive in. These maps include landmarks used to determine the vehicle’s position when the sensors see them again, as well as how the lanes in the road are connected. This helps the AV know where it’s safe to perform a lane change, which traffic lights control which lanes, and how to get from one street to another.
Most AV companies extensively map the areas they will operate in so the vehicle recognizes where it can and cannot drive. For instance, Chevy Supercruise can only be activated on highways that have been pre-mapped, and Waymo taxis only operate on city streets that have been pre-mapped. Other systems, such as Tesla Autopilot and Full Self-Driving, can be operated in areas that have not been pre-mapped, though this can be problematic if the vehicle makes poor decisions based on incorrectly interpreted sensor data.
Stay Updated
Sign up for our newsletter to get updates delivered straight to your inbox.