The Self Driving Car Story

In October 2024, the first robotaxis without a safety driver began commercial operation in several US cities. The technology, more than fifteen years in development, combines cameras, lidar, radar, GPS, and high-definition maps, all processed by a deep neural network running on a powerful on-board computer. The story of how self-driving cars went from a 2004 DARPA Grand Challenge curiosity to a commercial reality is, in retrospect, a story of an AI winter followed by a deep-learning revolution, of billion-dollar bets, and of a regulatory and ethical landscape that is still being worked out.

The DARPA Grand Challenge and the birth of the field

The history of self-driving cars is usually traced to the DARPA Grand Challenge, a 142-mile off-road race in the Mojave Desert, organised by the US Defense Advanced Research Projects Agency in March 2004. The prize was $1 million, and none of the 15 entrants finished the course. The best team, from the Carnegie Mellon Red Team, completed just 7.3 miles before getting stuck on a switchback. The challenge was repeated in 2005, and this time 5 of the 23 finalists finished, with the Stanford Racing Team (led by Sebastian Thrun, using a Volkswagen Touareg named “Stanley”) winning the $2 million prize. The 2007 DARPA Urban Challenge added a more realistic urban environment, and the team from Carnegie Mellon (with Chris Urmson, who later became CTO of Waymo) won.[1, 2, 9]

The impact of the Grand Challenge was enormous. The 2005 and 2007 challenges demonstrated that self-driving was technically feasible, and they catalysed the formation of a new industry. Sebastian Thrun, the leader of the Stanford team, joined Google in 2009 to lead the Google self-driving car project (later renamed Waymo). Chris Urmson, who led the CMU team, became the CTO of Waymo and later the CEO of Aurora. The talent that the challenges attracted — engineers from robotics, computer vision, machine learning, and control theory — seeded the new industry. The leading self-driving companies, in 2024, are Waymo (owned by Alphabet), Cruise (owned by GM, though scaled back after a 2023 incident), Aurora, Motional, Zoox (owned by Amazon), Tesla (with its “Full Self-Driving” software, though the name is controversial), and a wave of Chinese companies, including Baidu’s Apollo, Pony.ai, WeRide, and AutoX.[1, 2, 9]

The transition from the DARPA challenges to commercial deployment took about 15 years. The challenges demonstrated feasibility, but the engineering effort to make the technology safe enough for public roads was, in retrospect, much larger than the participants had anticipated. The problem is the “long tail” of edge cases: the system must handle not just the normal cases (driving down a wide, well-marked road in good weather) but the rare cases (a child running into the road, a fallen tree, a construction zone with temporary signs, a policeman directing traffic, an unmarked lane closure, a heavy snowstorm, an emergency vehicle approaching from behind). The “long tail” is, in practice, the dominant engineering problem, and the dominant cost of the self-driving programme.[1, 2, 9]

The sensor stack: cameras, lidar, radar

A self-driving car must perceive its environment in real time, in all weather and lighting conditions, with centimetre-level accuracy. No single sensor can do this, so every commercial system uses a combination of cameras, lidar, radar, and (for localisation) GPS + IMU. The trade-off between these modalities is the central engineering decision in self-driving.[2, 3, 10]

Cameras are the most familiar sensor. A modern self-driving car has 8–12 cameras, mounted around the perimeter, providing a 360-degree field of view. The cameras are high resolution (8 megapixels or more), high frame rate (30–60 fps), and high dynamic range (necessary to handle the bright sun and the dark shadows in the same frame). The cameras are essential for reading signs and traffic lights, for recognising traffic light states, and for detecting and classifying objects (the colour and the texture of an object are important cues for identifying it as a pedestrian, a cyclist, or a car). The weakness of cameras is the dependence on light: in low light, in fog, in heavy rain, and in direct sun glare, camera performance degrades significantly.[2, 3, 10]

Lidar (Light Detection and Ranging) is a laser rangefinder. A lidar unit emits a pulse of laser light (typically at 905 nm or 1550 nm, in the infrared) and measures the time it takes to return, producing a 3D point cloud of the environment with centimetre accuracy. A modern automotive lidar (e.g. the Velodyne Puck, the Hesai Pandar, the Luminar Iris) has a range of 200–300 metres, a field of view of 120° × 25° (or 360° for a rotating unit), and a point rate of several million points per second. The advantage of lidar is that it is independent of light (it works in complete darkness) and that it provides accurate 3D geometry (the exact position of every object in the environment). The disadvantage is the cost (a high-performance lidar unit costs $1,000–$10,000 in 2024) and the performance in heavy rain and snow (the water droplets in the air scatter the laser light and reduce the effective range). The cost of lidar has been falling rapidly, and the dominant robotaxi companies, in 2024, use a roof-mounted lidar as the primary sensor.[2, 3, 10]

Radar (Radio Detection and Ranging) is the oldest of the three modalities. A radar unit emits a radio wave (typically at 76–77 GHz, in the millimetre-wave band) and measures the time it takes to return and the Doppler shift of the frequency. The result is a list of objects, with their range, their radial velocity, and (with multiple antennas) their angular position. Radar has the advantage of being robust to weather (it works in fog, heavy rain, and snow) and of providing velocity information directly (the Doppler shift is a direct measurement of the radial velocity). The disadvantage is the resolution: a typical automotive radar has a range resolution of about 1 metre and an angular resolution of a few degrees, much worse than lidar or cameras. The resolution is improving with imaging radar (which uses a large antenna array and signal processing to produce a 2D image), but the technology is still emerging.[2, 3, 10]

The perception stack: deep neural networks for everything

The perception stack — the part of the system that turns raw sensor data into a labelled, tracked set of objects — is dominated by deep neural networks. The dominant approach, in 2024, is an end-to-end neural network that takes the raw sensor data (cameras, lidar, radar) as input and produces a scene representation (a list of objects, with their class, position, velocity, and trajectory) as output. The network is trained on a large corpus of annotated driving data, with the annotations provided by a combination of human labelers and (more recently) by self-supervision (the network learns from the differences between the predicted and the actual outcomes of its decisions).[3, 4, 11]

The most important sub-task is object detection: given a camera image or a lidar point cloud, identify the objects in the scene and draw a bounding box around each one. The dominant architecture, in 2024, is a transformer-based network similar to those used in large language models, but applied to 2D images or 3D point clouds. The network is trained on millions of annotated images, with the annotations provided by human labelers. The performance, measured by mean average precision (mAP) on a standard benchmark, has improved from about 30 percent in 2010 (with hand-engineered features) to about 90 percent in 2024 (with deep learning). The remaining 10 percent is, in practice, the hard part: the rare, ambiguous, or unusual cases that the network has not seen enough of in training.[3, 4, 11]

The second most important sub-task is multi-object tracking: given a sequence of detections, associate them across frames to produce a coherent trajectory for each object. The dominant approach is a transformer that takes the detections from the current frame and the trajectories from the previous frame as input, and produces a set of updated trajectories. The transformer learns to handle occlusions (an object that disappears behind another object for a few frames), entry and exit (an object that enters or leaves the field of view), and identity switches (two objects that come close together and then separate). The performance, measured by the multi-object tracking accuracy (MOTA), is now above 80 percent on standard benchmarks, and is improving rapidly with the use of larger models and more data.[3, 4, 11]

HD maps: the silent partner

Every commercial self-driving system, in 2024, relies on a high-definition (HD) map. The HD map is a 3D representation of the road environment, accurate to a few centimetres, that includes the position of every lane line, every curb, every sign, every traffic light, and every relevant landmark. The HD map is built by a fleet of mapping vehicles, equipped with the same sensor stack as the self-driving cars, that drive every road in the operating area many times. The resulting map is then used by the self-driving system for two purposes: localisation (the car compares its sensor data to the map and computes its position with centimetre accuracy) and prior (the car knows that a stop sign is supposed to be at a particular location, so the perception system can use the map to disambiguate). The HD map is, in this sense, a kind of cheat sheet for the perception system, and it dramatically simplifies the perception problem.[4, 5, 12]

The cost of building and maintaining the HD map is significant. A single mapping vehicle costs $500,000 or more, and the process of mapping a single city takes months. The map must be kept up to date: a road closure, a new sign, a new building can all change the environment, and the map must be updated within hours or days. The dominant map providers, in 2024, are Here Technologies (owned by a consortium of German automakers), TomTom (the Dutch navigation company), Mapbox (a US-based mapping platform), and the in-house mapping teams of the robotaxi companies (Waymo, Cruise, Baidu). The trend, in 2024, is toward mapless systems that can drive in unmapped areas, but the performance is still significantly worse than the map-based systems, and the map is likely to remain an important part of the system for the foreseeable future.[4, 5, 12]

The HD map also raises interesting privacy questions. The mapping vehicle, equipped with cameras and lidar, captures detailed images of every house, every garden, every parked car in the operating area. The images are, in many jurisdictions, considered personal data, and the mapping companies must comply with data protection regulations (GDPR in Europe, CCPA in California). The data is typically anonymised and aggregated, but the risk of re-identification is real, and the regulatory framework is still being developed. The HD map, in this sense, is not just a technical asset but a legal and social one, and the way it is managed will shape the public’s acceptance of self-driving technology.[4, 5, 12]

Deployment: the state of the art in 2024

The first commercial robotaxi service without a safety driver was launched by Waymo in Phoenix, Arizona, in October 2020. The service, called Waymo One, was initially available to a small group of pre-approved users, and was gradually expanded. By 2024, Waymo is operating more than 700 robotaxis in Phoenix, San Francisco, and Los Angeles, providing more than 50,000 paid rides per week, and has accumulated more than 30 million miles of fully driverless operation. The service is widely used by the public, and the average rating is high. The expansion, however, is also limited: the robotaxis operate only in well-mapped urban areas (about 600 km² in total), only in good weather, and only during certain hours. The technology is, in 2024, somewhere between a demo and a mass-market product.[5, 6, 13]

The other commercial robotaxi service, in 2024, is operated by Baidu’s Apollo in several Chinese cities, including Beijing, Shanghai, and Shenzhen. Baidu has accumulated more than 100 million km of test driving, and the service is now available in more than 10 cities. The Chinese deployment is notable for the speed of the regulatory approval: the Chinese government has been actively supportive of the self-driving industry, and the regulatory framework is more permissive than in the US. Other Chinese companies, including Pony.ai, WeRide, and AutoX, are also operating robotaxi services in China.[5, 6, 13]

The dominant consumer-facing system, in 2024, is Tesla’s Full Self-Driving (FSD) software. FSD is a Level 2 system (the human must remain attentive and ready to take over), but it is sold under the “Full Self-Driving” name and is widely used. The technical approach is different from the robotaxi companies: Tesla relies on cameras alone (no lidar or radar, since 2021), and uses a neural network trained on data from the Tesla fleet (more than 5 million cars, more than 100 billion miles of driving data). The approach is controversial: the robotaxi companies argue that lidar is essential for safety, while Tesla argues that cameras are sufficient and that the data advantage is more important. The debate is unresolved, and the performance of the two approaches is, in 2024, comparable on highways but divergent in complex urban environments.[5, 6, 13]

The economics, the law, and the ethics

The economic case for self-driving is, in principle, strong. A robotaxi costs about $0.30–$0.50 per mile to operate (no driver, no salary, no tips), compared to about $1.50–$2.50 per mile for a human-driven taxi. The cost of the sensor stack, however, is still significant: a roof-mounted lidar costs $5,000–$10,000, the cameras and radars add another $5,000–$10,000, and the on-board computer adds another $10,000–$20,000. The total hardware cost, in 2024, is about $50,000–$100,000 per car, much more than the cost of a human-driven taxi. The hardware cost is falling, and the operating cost is increasing (the remote operators, the software updates, the insurance), but the economic case is still marginal at current costs. The path to a sustainable business is, in part, a path to lower hardware costs.[6, 7, 14]

The legal question is who is liable when a self-driving car causes an accident. The current framework, in most jurisdictions, is that the manufacturer of the car (or the operator of the service) is liable, but the details are still being worked out. In the US, the National Highway Traffic Safety Administration (NHTSA) has issued a series of guidelines and regulations, and several states (California, Arizona, Texas) have their own frameworks. In Europe, the UNECE has issued regulations for Level 3 systems (where the car can drive itself under certain conditions), and the EU is working on a comprehensive framework. The legal framework, in 2024, is still a work in progress, and the liability question is, in practice, the most important legal question facing the self-driving industry.[6, 7, 14]

The ethical question is the most difficult. When an accident is unavoidable, how should the car be programmed to act? The classic trolley problem — should the car swerve to avoid a pedestrian, even if it means hitting a cyclist, or should it continue straight and hit the pedestrian? — is a useful thought experiment, but the real ethical questions are more subtle. The car must make thousands of small ethical decisions every day, in the form of small adjustments to the trajectory: how close to pass a cyclist, how aggressively to merge, how to handle a jaywalker. The programming of these decisions, in the current systems, is largely hand-engineered, but the trend is toward learned behaviour. The question of who is responsible for the ethical decisions of a self-driving car — the manufacturer, the software provider, the regulator, the public — is unresolved, and will be one of the defining ethical questions of the next decade.[6, 7, 15]

Comparison: leading self-driving companies (2024)

Company	Headquarters	Sensor stack	Status (2024)	Operating area	Reported autonomous miles
Waymo	Mountain View, USA	Lidar + cameras + radar	Commercial robotaxi (no driver)	Phoenix, SF, LA (~600 km²)	~30 million
Cruise	San Francisco, USA	Lidar + cameras + radar	Scaled back after 2023 incident	Limited	~5 million (paused)
Aurora	Pittsburgh, USA	Lidar + cameras + radar	Trucking pilot	Texas (trucking)	~1 million (test)
Zoox	Seattle, USA	Lidar + cameras + radar	Test, no commercial service	Las Vegas, SF	<1 million
Baidu Apollo	Beijing, China	Lidar + cameras + radar	Commercial robotaxi (some driverless)	10+ Chinese cities	~100 million km (test)
Tesla FSD	Austin, USA	Cameras only	Level 2 consumer (driver required)	Global (with caveats)	~100 billion (fleet data)

Key numbers

• ~700: number of Waymo robotaxis in operation (2024).
• ~50,000: paid Waymo rides per week (2024).
• ~30 million: cumulative Waymo driverless miles (2024).
• ~$0.30–$0.50: estimated cost per mile for a robotaxi (2024).
• ~$50,000–$100,000: cost of a self-driving sensor and compute stack (2024).
• ~5 million: number of Tesla cars sending FSD training data (2024).
• 2004: year of the first DARPA Grand Challenge.
• 2020: year of the first commercial robotaxi service (Waymo, Phoenix).

References

1. Wikipedia, “DARPA Grand Challenge.” (Standard reference on the 2004, 2005, and 2007 DARPA Grand Challenges, which seeded the self-driving industry.) https://en.wikipedia.org/wiki/DARPA_Grand_Challenge
2. Wikipedia, “Self-driving car.” (Standard reference on self-driving cars, with a section on the history and the technology stack.) https://en.wikipedia.org/wiki/Self-driving_car
3. Wikipedia, “Lidar.” (Standard reference on lidar, the primary sensor for most self-driving systems.) https://en.wikipedia.org/wiki/Lidar
4. Wikipedia, “Waymo.” (Standard reference on Waymo, the leading robotaxi company.) https://en.wikipedia.org/wiki/Waymo
5. Wikipedia, “Cruise (company).” (Standard reference on Cruise, the GM-owned robotaxi company.) https://en.wikipedia.org/wiki/Cruise_(company)
6. Wikipedia, “Tesla Autopilot.” (Standard reference on Tesla Autopilot and FSD, the most widely deployed Level 2 system.) https://en.wikipedia.org/wiki/Tesla_Autopilot
7. Wikipedia, “Trolley problem.” (Standard reference on the trolley problem, the classic ethical thought experiment applied to self-driving cars.) https://en.wikipedia.org/wiki/Trolley_problem
8. Wikipedia, “Level 3 autonomous driving.” (Standard reference on the SAE levels of driving automation, with the regulatory framework.) https://en.wikipedia.org/wiki/Level_3_autonomous_driving
9. Wikipedia, “Sebastian Thrun.” (Standard reference on Sebastian Thrun, leader of the Stanford DARPA team and co-founder of Google X.) https://en.wikipedia.org/wiki/Sebastian_Thrun
10. Wikipedia, “Automotive radar.” (Standard reference on automotive radar, the third primary sensor for self-driving.) https://en.wikipedia.org/wiki/Automotive_radar
11. Wikipedia, “Multi-object tracking.” (Standard reference on multi-object tracking, a core perception task in self-driving.) https://en.wikipedia.org/wiki/Multi-object_tracking
12. Wikipedia, “Here Technologies.” (Standard reference on Here Technologies, one of the leading HD map providers.) https://en.wikipedia.org/wiki/Here_Technologies
13. Wikipedia, “Baidu.” (Standard reference on Baidu, the leading Chinese self-driving platform with Apollo.) https://en.wikipedia.org/wiki/Baidu