On a Wednesday morning last October, three friends exited their red Tesla Model S P85D after parking it in a garage on the east side of Manhattan and stretched their legs. Carl Reese, Deena Mastracci and Alex Roy had just finished one heck of a road trip, traveling from Redondo Beach, Calif., to New York City in 57 hours and 40 minutes, record time for an electric vehicle.

But the most impressive aspect of their trip was that the car had driven itself almost the entire 2,994 miles.

All three passengers had been on coast-to-coast “Cannonball Run” drives before, competing to complete the trip in the shortest time, but this voyage was different. The team used Tesla’s new “Autopilot” feature about 96 percent of the time and had averaged 51.8 miles per hour, including time spent at charging stations. That meant that for most of the trip, the car was ripping across the nation at 90 mph.

Tesla introduced its autopilot feature last October through an over-the-air software update that makes late-model Model S sedans and all Model X SUVs semi-autonomous, meaning that they can accelerate, brake and steer themselves without a driver’s input. They are not fully autonomous, however, meaning that the car still may signal to the driver to take the wheel in certain situations.

A 2015 report from Business Insider Intelligence estimated that by the end of 2020, there will be almost 10 million semi- or fully autonomous cars on the world’s roadways, with fully autonomous cars likely to be available for consumer purchase as early as 2019. And if they prove safe and popular, self-driving cars could eventually become the norm.

Tesla’s autopilot system, which has been installed in tens of thousands of vehicles, uses radar, cameras, GPS, and ultrasonic sensors to keep the car within its lane, maintain a safe following distance, and change lanes at the driver’s command. Rival car makers such as Audi, Volvo, Mercedes, BMW and Nissan are developing similar systems in a kind of automotive arms race.

The appeal of self-driving cars is obvious. Drivers could rest, work, read the paper or use electronics while on the road, making the morning commute more enjoyable. And the car’s high-tech sensory equipment could lead to fewer crashes by eliminating the potential for human error.

The car itself would never fail to pay attention, fall asleep at the wheel, spill its coffee or become distracted. Plus, fully autonomous cars could essentially end drunk driving, which represents about one-third of the approximately 33,000 annual U.S. deaths from car crashes. And they could allow the elderly, blind people and those with developmental disabilities to become increasingly engaged in their communities and lead fuller lives.

But for now, self-driving cars are creating some white-knuckle moments. During the cross-country Tesla trip, there were several moments when one of the three passengers had to take the wheel to avoid a crash. In most of the cases, it was because the car was traveling quickly on a curving road and was simply following the lane lines, failing to adequately adjust its speed and direction.

“If I hadn’t had my hands there, ready to take over, the car would have gone off the road and killed us,” Roy said in an interview with Wired.com.

Self-driving cars’ lack of a proven track record has left many consumers with mixed feelings. According to a survey conducted last year by the World Economic Forum in conjunction with the Boston Consulting Group, 58 percent of global respondents said they would be either likely or very likely to try an autonomous car, including 52 percent of U.S. respondents, though only 35 percent would let their children ride alone in one.

One of the leaders in the field is Google, whose autonomous cars have traveled more than 1.5 million miles, with ongoing pilot programs on the public streets of Mountain View, Calif.; Austin, Texas; Kirkland, Wash.; and Phoenix. The testing fleet includes modified Lexus SUVs and new prototype vehicles that Google has designed from the ground up to be fully self-driving, though they currently always have a safety driver aboard.

From September 2014 to November 2015, Google’s fleet drove 423,000 miles, and its safety drivers had to assume control of the cars only 341 times, the company announced in January. Of those 341 instances, 272 stemmed from equipment or system failures, the kind of hardware flaws that can be eliminated as the company continues to improve its product.

The other 69 instances stemmed from safety issues – times when the driver wasn’t confident that the car would make the correct decision and therefore had to take the wheel. In such instances, data from the car is later analyzed to determine what decision the car would have made. Google found that 13 of those 69 incidents would have resulted in a crash.

Those 13 potential crashes came at a rate of one every 74,000 miles, whereas human drivers record one accident per 238,000 miles, according to statistics from the Virginia Tech Transportation Institute. In other words, Google’s cars are getting good but aren’t quite good enough yet. So far, Google’s smart cars have been involved in 17 crashes, none of which were deemed to be the fault of the self-driving cars.

With so many companies looking to create self-driving cars, the intense competition is likely to spur rapid improvements in the technology.

In 2015, Uber announced a partnership with Carnegie Mellon University to develop self-driving cars. The company, which has quickly become a chief rival to traditional taxicab companies, hired 40 robotics researchers from Carnegie Mellon to work at its Advanced Technologies Center in Pittsburgh.

Even the terrorist group ISIS is looking to develop self-driving cars that could drive into a crowded area before detonating a bomb, according to Jamie Shea, NATO’s deputy assistant secretary general for emerging security threats. Speaking to reporters in London in April, Shea said ISIS is exploring the technology at its bomb-making factory in Raqqa, Syria.

“It is not just Google that is producing the autonomous car,” Shea said. “ISIS is also trying to do the same.”

Autonomous cars are creating a dilemma for regulatory agencies, which must come up with laws to govern their use. Since the driver must take control of the car in hazardous situations, the driver still must be held accountable for distracted driving. But requiring the driver to be attentive for every moment of the trip essentially cancels out the benefit of having a self-driving car.

In Canada, Minister of Transport Marc Garneau solicited feedback from various stakeholders upon taking office last November. The Canadian press obtained the notes he was given under the country’s Access to Information Act and found that they included media reports of Tesla drivers engaging in activities such as reading the newspaper or brushing their teeth while using the autopilot feature.

Barrie Kirk, the co-founder and executive director of the Canadian Automated Vehicles Centre of Excellence consulting firm, even brought up a concern that politicians might be too embarrassed to address publicly: sex behind the wheel.

“I am predicting that once computers are doing the driving, there will be a lot more sex in cars,” Kirk said in the notes. “That’s one of several things people will do which will inhibit their ability to respond quickly when the computer says to the human, ‘Take over.’ “

In the U.S., auto experts gathered at a forum in April at Stanford University organized by the National Highway Traffic Safety Administration, which is soliciting comments to help it establish guidelines for companies developing autonomous cars. At the forum, Chris Urmson, the chief architect of Google’s 7-year-old autonomous car program, said he hoped federal agencies would set the rules governing self-driving cars because state agencies might lack the technical know-how to regulate the industry.

Also in April, Google, Ford, Volvo, Uber and Lyft teamed up to form the Self-Driving Coalition for Safer Streets, whose stated mission is to “work with lawmakers, regulators, and the public to realize the safety and societal benefits of self-driving vehicles.” The group will push for federal uniform rules governing self-driving vehicles.

Under current U.S. federal regulations, fully autonomous cars that lack human controls such as steering wheels, gas and brake pedals and a licensed driver are illegal.

One country that is just about ready to put autonomous cars to the test is Sweden, where next year Volvo plans to place 100 self-driving cars in the hands of customers on selected roads in the company’s hometown of Gothenburg. The public pilot program is a collaboration between legislators, transport authorities, city officials and the carmaker.

“We are entering uncharted territory in the field of autonomous driving,” Dr. Peter Mertens, Volvo’s senior vice president of research and development, said in a statement. “Taking the exciting step to a public pilot, with the ambition to enable ordinary people to sit behind the wheel in normal traffic on public roads, has never been done before.”