The Future Of The In-Vehicle Experience

It’s 2030 and you’re taking a road trip.

A quick fingerprint scan unlocks your car. Once you’re settled in the driver’s seat, you tell your in-vehicle voice assistant to pull up directions to your destination.

As you travel, turn-by-turn navigation instructions are projected onto an AR display over the windshield, and sensors track your eye and head movements to monitor for fatigue. Meanwhile, your passengers use in-vehicle screens and VR headsets to shop and play video games.

But this is just the beginning.

Advancements in AI, voice, and mixed reality are bringing this futuristic vision of the in-vehicle experience closer to reality. Automakers and retailers are looking to capitalize on new tech solutions to enhance the in-vehicle experience for both drivers and passengers.

In-vehicle tech is also helping mobility service providers such as Uber and Lyft gain an edge in a highly competitive market. Shared vehicles — and eventually autonomous vehicles — could evolve into hubs for immersive entertainment and new channels for retail as the passenger experience becomes a priority.

In this report, we break down the latest emerging in-vehicle technologies, how they work, and how they could change how we interact with cars.

Table of contents

• The future of the in-vehicle experience
• Biometric-based authentication
• The in-car voice assistant
• AR-enabled heads-up displays
• Vision-based driver monitoring
• Shoppable content and entertainment
• AI-optimized parking

The future of the in-vehicle experience

Automakers prioritizing the in-car experience are looking to optimize across all facets of the car journey, from vehicle entry and ignition to parking, while ensuring passenger safety and vehicle security.

While many disruptive forces are expected to ultimately render the driver obsolete, most of the solutions discussed in this report can be adapted for a shared, autonomous future. For example, the sensors and computer vision software used for driver monitoring could eventually be used to monitor passengers in a shared robotaxi.

Below, we outline emerging focus areas for automakers as they prioritize safety, comfort, and convenience.

Please click to enlarge.   

Biometric-based authentication

Vehicle security is a serious priority for automakers, and though car theft is trending downward in countries including the US, studies have shown that car thieves are devising new methods for stealing.

This trend is especially relevant for cars with keyless ignition, which now make up more than half of the 17M cars sold annually in the US. Keyless cars are notably easier to break into, as hackers can replicate the signal that the key fob gives off when close enough to the vehicle.

As a result, automakers are looking for more secure technology for vehicle entry and ignition.

Biometric-based authentication — which verifies an individual’s identity using biological markers like a fingerprint, face, or pupil — offers a powerful alternative to the key fob. Biometric markers are unique to each individual and can be difficult to replicate when implemented properly.

While still nascent, biometric technology has gained traction over the past few years, as it holds promise across a number of industries, including payments, consumer electronics, and surveillance.

As a result, a number of OEMs (original equipment manufacturers) are experimenting with biometric-based authentication.

Major auto players test biometrics for vehicle entry and ignition

Several notable players in the automotive space are deploying biometric technology to assess its viability for vehicle entry and ignition.

Hyundai has already integrated fingerprint scanning into two of its models in China. The authentication system lets drivers enter the vehicle and turn on the ignition without a key. Unlike prior fingerprint-based systems, Hyundai’s system does not require a key fob, smartphone, or wireless key card.

Source: Hyundai

As computer vision technology matures and sees adoption across multiple industries, automakers and other players in the auto space are exploring facial recognition for biometric authentication.

Porsche, for example, is collaborating with Intel-backed edge computing startup FogHorn to build an automatic car unlocking system that uses an infrared camera. The system relies on multifactor authentication, meaning that users have to confirm their identity a second time using their phone or another device.

Apple is also apparently working on integrating its Face ID technology to secure vehicles, according to a patent published by the company.

Exploring alternative biometric indicators

One potential roadblock for biometric-based authentication systems is their vulnerability to hacking. These systems rely on databases of images that can easily be manipulated, and some biometric markers, such as fingerprints, can be forged and faked, carrying a high risk if they’re used to unlock and start a vehicle.

Fingerprints have also been difficult to use for unlocking vehicles given their sensitivity to conditions such as rain, snow, heat, and cold temperatures.

Companies like Hyundai are developing fingerprint unlocking technology that uses capacitance recognition, which detects fingerprints using electrical current rather than the typical means of using light to trace over the skin. This means it relies on the fingerprint’s physical shape instead of a visual impression, making it harder to forge.

Another approach is to use other biometric indicators that are harder to replicate. For example, electronics firm Gentex has developed an iris-based authentication system, which the company claims is the most secure form of biometric authentication.

Gentex’s iris-based driver identification technology (Source: CNET)

An even more nascent approach comes from Aerendir, which is developing a system that measures micro-vibrational patterns in a user’s muscles to detect a unique signature that originates in the brain. Aerendir says this approach is more secure than traditional biometric-based authentication methods.

Biometrics in a shared, autonomous world

With the rise of shared mobility services, identity verification is becoming an increasingly important issue, following incidents of driver misconduct and assault cases at major ride-hailing companies such as Uber, Lyft, and Didi Chuxing.

Biometric-based identification could help verify the identities of both drivers and passengers.

In China, computer vision startup Reconova is already seeing demand  for its facial recognition technology from ride-hailing firms, and it expects this trend to continue.

T3 Chuxing, a China-based ride-hailing firm formed by 3 major Chinese automakers (FAW Group, Dongfeng Motor, and Changan Automobile), is positioning itself as a safer alternative to Didi Chuxing. Didi holds roughly 90% of China’s ride-hailing market, but has come under scrutiny following 2 cases of passenger murder.

Facial recognition is central to T3’s safety technology. According to the firm’s CEO, connecting drivers with passengers via smartphones is harder to supervise. Instead of relying on a user’s smartphone for identification purposes, the company’s solution uses cameras and computer vision software installed in the vehicle.

BEYOND SECURITY: BIOMETRICS FOR WELLNESS TRACKING & personalization

Biometrics could also have applications in the automotive space beyond security.

Startups have been focused on reading an individual’s electrocardiogram (ECG/EKG) to provide real-time information on driver and passenger health, helping to identify potential medical issues before they occur.

UK-based B-Secur, for example, has developed a biometric-based authentication technology called HeartKey that identifies users by analyzing their heartbeat.

B-Secur’s HeartKey measuring an individual’s vital signs (Source: B-Secur)

The company’s sensor is integrated into the steering wheel, where it can track changes in heart rate and rhythm, stress levels, fatigue, respiration, and atrial fibrillation. These metrics can provide drivers and passengers with real-time insights into their health, and can also work with driver monitoring systems to detect and intervene when drivers are fatigued or fall ill.

Automakers such as Daimler, Ford, and Honda are also developing in-vehicle heart monitoring technology, thought it is still too nascent to incorporate into production vehicles.

In addition to health monitoring, a number of automakers, including Byton, are leveraging biometrics to enhance the appeal of personal vehicle ownership by personalizing the in-car experience with biometrics. For example, a vehicle could detect which of its owners is entering the vehicle based on his or her biometric indicator, then adjust comfort settings based on the driver’s personal preferences.

The in-vehicle voice assistant

As the car increasingly resembles a connected device, automakers and big tech firms are integrating voice technology into the vehicle.

Voice assistants allow drivers to access important information about navigation, detours, and potential pit stops without having to check their smartphones.

But voice technology goes beyond just accessing hands-free navigation: more advanced in-vehicle digital assistants can let drivers play music, search, and send text messages.

Source: Vox

Voice assistants also allow passengers to access information and other forms of media and entertainment in the vehicle with a simple voice command.

Big Tech looks to capitalize on automotive voice market

A survey published by JD Power in April found that the majority of car owners (76%) would prefer to have the same brand of in-home voice assistant in their next vehicle, to maintain consistency from home to car.

This suggests that tech firms like Google, Apple, and Amazon — all of which are integrating their voice assistants into cars today — could have a notable advantage over automakers and startups attempting to build new in-vehicle assistants.

These companies are already racing to integrate their digital systems across the consumer’s suite of connected devices to make their platforms stickier.

While Amazon’s Alexa currently dominates the smart speaker market, the company faces more competition from Google and Apple when it comes to the car. This is largely because both Google and Apple already have a presence in a number of vehicles with their operating systems, Android and Carplay, respectively.

That said, Amazon is working with major automakers such as BMW, Toyota, and Ford to integrate Alexa into their smart infotainment systems. The company also recently rolled out the Echo Auto, an aftermarket device that brings Alexa into the vehicle.

Tech firms in China are also looking to bring voice tech into the car. Alibaba announced in June that its voice assistant for the connected car, the Tmall Genie Auto, will be featured in Volkswagen’s Audi, Renault, and Honda vehicles.

The move suggests that Alibaba is looking at new markets for its voice assistant as smartphone shipments fall in China and as its smart speaker faces strong competition from Baidu, which has developed an automotive version of its conversational AI platform. Baidu’s platform DuerOS is already featured in vehicles made by Ford, Hyundai, and Kia, among others.

“You want to keep your hands on the wheel and eyes on the road, so using your voice makes more sense.” — Ned Curic, vice president of Amazon’s Alexa Auto division

Automakers and startups still building out their own tech

While Big Tech has a number of competitive advantages, including consistency and scale, some automakers are also developing their own voice assistants — and they have a better understanding of how the vehicle works. This is especially important for voice assistants used for driver assistance.

BMW showcased its digital personal assistant for cars in September 2018, just weeks after announcing that it would be integrating Amazon’s Alexa into its vehicles. Notably, the company stressed that the two voice technologies would complement each other rather than compete, as BMW’s assistant is more focused on the car. It can make suggestions, such as turning on headlights as it gets dark, and provide information about a car’s features, which grows more challenging as cars become increasingly software-oriented. Amazon’s Alexa, on the other hand, focuses more on infotainment and the passenger experience.

Automakers building out their own voice assistants are turning to natural language processing startups. Daimler, Hyundai, and Volkswagen have invested in SoundHound, Saltlux, and Mobvoi, respectively, to develop in-vehicle voice assistants. 

Given concerns around privacy in the smart speaker space, a number of startups are also working on in-vehicle voice assistants designed to ensure users’ data privacy by running commands locally rather than on the cloud. Companies working on this technology include Mycroft AI and Snips.

The future of automotive voice tech

Despite the majority of car drivers wanting a voice assistant in their vehicle, voice assistants remain one of the biggest pain points for drivers right now, as the AI used for them is not mature enough to handle a majority of commands.

As tech firms, automakers, and startups look to enhance in-vehicle voice assistants’ command recognition capabilities, a number of startups are looking ahead to conversational interfaces or chatbots that can serve more functional purposes, such as locking doors or opening the trunk.

Some companies are also looking to develop other car-specific capabilities, such as detecting an oil change and booking an appointment while in the vehicle, or serving as a spoken owner’s manual to answer car owner questions.

AR-enabled heads-up displays

With advancements in vehicle connectivity, OEMs are revolutionizing the cockpit to enhance driver safety and keep passengers entertained.

One facet of the cockpit of the future is the heads-up display (HUD), a transparent display that presents data on the windshield so that drivers can keep their eyes on the road. These displays reflect relevant information such as speed limits, time, and temperature — all within the driver’s line of sight.

While a number of luxury automakers currently offer optional heads-up displays that reflect information like speed and turn-by-turn directions onto the windshield, all of these displays are two-dimensional, meaning they can say very little about anything related to depth — such as when exactly to turn, or any impending detours.

In response, automakers and startups are looking to enhance current heads-up display technology so that they can provide more valuable information to drivers. HUDs can also create a medium for immersive passenger entertainment, as holographic displays can project enhanced visuals for TV and movies.

More advanced AR-enabled displays could integrate information about the roadway and present driver alerts in addition to navigation instructions, both of which would enhance driver safety in a number of challenging situations.

HUD startup WayRay, backed by Porsche and Hyundai, is pioneering the technology with its consumer product, Navion. Navion offers a built-in HD camera that maps a driver’s environment, projecting turn-by-turn directions onto the dashboard and also providing hazard warnings. The application allows for hands-free interaction.

WayRay’s holographic navigation systems (Source: TechCrunch)

Automakers and auto suppliers are also developing their own technology to enhance the functionality of the dashboard.

Jaguar Land Rover has launched a research effort around developing an immersive heads-up display. In collaboration with the Center for Advanced Photonics and Electronics at the University of Cambridge, the company is developing technology the projects a close reflection of the road to allow drivers to react more naturally to hazards and prompts.

Roadblocks still persist for AR in the vehicle

AR technology still remains nascent, and in automotive applications with incredibly high safety standards, it will likely take a while for the tech to reach commercial scale.

Further, HUDs will likely be substantially expensive. Companies such as Waymo see an opportunity to subsidize these displays by using them for advertising purposes, such as virtual billboards that can point to certain stops along a route, or advertisements that play during games or content streaming.

Vision-based driver monitoring

As the timeline for fully autonomous driving remains uncertain, automakers are looking for other ways to minimize the 95% of traffic accidents caused by human error.

One technology that stands to curb these accidents is driver monitoring, which tracks driver behavior, identifies if a driver is distracted or fatigued, and intervenes before an accident occurs.

Driver monitoring systems rely on in-car sensors to capture what’s going on in the vehicle’s interior. Applying computer vision and facial recognition to this sensor data enables these systems to track and process a driver’s facial expressions, eye movements, and head position, all of which can point to cases of driver distraction or fatigue.

Once a system detects a potential issue, it can send the driver an alert or, in more advanced vehicles, automatically pull over and bring the car to a stop.

DEMAND FOR DRIVER MONITORING IS ON THE RISE

Earnings call chatter around driver monitoring technology has picked up in recent quarters, reflecting a growing emphasis on driver safety technology as timelines for fully autonomous vehicles continue to be pushed back.

The companies that have been most vocal about driver monitoring technology include Tier 1 auto suppliers such as Visteon and Autoliv, which develop driver monitoring systems for auto OEMs. Semiconductor companies that develop the sensors that go into these systems, such as Ambarella and Melexis, are also noting strong demand for driver monitoring technology.

Ambarella CEO Fermi Wang spoke to this on the company’s Q1’19 earnings call:

“But even before that computer vision wave for ADAS, there’s another wave, which is DMS, Driver Monitor System. We continue to see huge momentum behind this market.”

The source of this demand is the increasing number of automakers looking to embed driver monitoring technology into their vehicles.

One of the first automakers to outline driver monitoring plans was Volvo, which in March announced that it would use in-vehicle cameras to monitor driver behavior and intervene if a driver is distracted or intoxicated.

EV manufacturer Rivian has also reportedly been working on its own system to monitor driver awareness levels in Level 3 autonomous vehicles. CB Insights’ patent search shows at least one patent filed by Rivian for “occupant awareness monitoring for autonomous vehicles.”

 

Other automakers are focused on this technology and will likely turn to computer vision startups to help build out the facial recognition capabilities required for these driver monitoring systems.

Computer vision startups are tapping into the driver monitoring opportunity

AI startups are also developing their own vision-based driver monitoring technology.

For example, eyeSight Technologies is developing in-vehicle dashcams that analyze a driver’s eyes, head position, and upper body movements. In the case of driver distraction or fatigue, the system will either send the driver an alert or work with the vehicle’s driver assistance system to automatically pull the vehicle over.

Emotion recognition startup Affectiva has already partnered with Aptiv, a Tier 1 automotive supplier that has also invested in the startup, to develop a driver monitoring system that can read nonverbal cues such as facial expression, body language, and tone of voice based on inputs from in-vehicle cameras. This information can then trigger alerts or automated intervention.

Source: Affectiva

Applications exist beyond driver safety

Even if the future is driverless, in-vehicle monitoring technology could still monitor passengers in shared robotaxis to better understand and improve the user experience.

One auto supplier focused on both drivers and on passengers is ZF, which has developed a cabin monitoring system that detects and classifies vehicle occupants to assess their size, location, and position. Knowing these details can better inform the car’s safety system to determine the best way to deploy an airbag based on a person’s size, or detect whether a child has been left behind in a vehicle.

Shoppable content and entertainment

In addition to focusing on drivers, OEMs are also looking at new ways to enhance the passenger experience, which is becoming more important as mobility service providers prioritize their customers and as automated driving creates more idle time for drivers in personal vehicles.

On the entertainment front, automakers and tech giants are capitalizing on advancements in connectivity technology that enable higher quality gaming and content streaming. These services could create new monetization opportunities for automakers, retailers, and mobility service providers alike, as they could all take a cut of any revenue generated.

In-vehicle retail is also gaining traction as improvements in mobile payment technology create opportunities for in-car purchases. As with gaming and streaming, in-vehicle retail creates new revenue channels for ride-hailing drivers and retailers, while enhancing the passenger experience.

Catering to the passenger with gaming and content streaming

To keep passengers entertained in a driverless future, auto OEMs and tech firms — such as Audi and Intel — are developing immersive in-vehicle experiences.

Audi made headlines at CES in January with Holoride, a new technology that integrates VR games and movies with real-time vehicle data to create a “motion-synchronized” VR experience for passengers.

Demos of Holoride’s in-vehicle VR experience (Souce: TechCrunch)

The company says that the technology also seems to reduce motion sickness, an especially desirable quality given that many people experience sickness when reading or watching videos in the car.

On the content streaming front, startups such as Portl Media and Vugo are developing in-vehicle video screens that generate ad revenue from which ride-sharing companies such as Uber and Lyft could take a cut.

Tesla is also talking about integrating content subscriptions into its next fleet of vehicles. The company announced in July that it will offer Netflix and YouTube streaming in some of its vehicles.

Source: Tesla

Some startups are also focusing on driver entertainment in an effort to keep drivers attentive and focused on the road. Amazon-backed Drivetime, which raised $11M in Series A funding in September, is developing interactive voice-based games for drivers. Other notable investors in Drivetime include Founders Fund and Index Ventures.

The car as an alternative channel for retail

With the growing adoption of ride-sharing services and the potential rise of shared robotaxis, the concept of the car as an alternative convenience channel is gaining traction.

Ride-sharing as a point-of-sale is slated to grow as mobility service providers increasingly look for ways to increase customer loyalty and enhance the in-vehicle experience.

As mobility service providers such as Uber and Lyft face intense pricing competition in the ride-hailing space, in-vehicle retail offerings could offer a new lever to sustain customer loyalty as they look to enhance stickiness on their platforms.

Furthermore, any revenues earned from in-car streaming or purchases can benefit drivers as well, as drivers can earn commission on any goods they sell during the ride.

In-vehicle commerce startup Cargo has developed transparent boxes filled with items such as electronics and snacks that ride-hailing passengers can purchase on their trip via the Cargo app. The company partnered with Southeast Asian ride-hailing firm Grab as well as with Uber in mid-2018 to deploy its boxes and help drivers earn more money per ride.

Source: Cargo

More recently, Uber and Cargo launched a shopping app for passengers in vehicles with Cargo consoles that features items curated by Uber, such as Glossier cosmetics, Nintendo Switches, and Apple hardware. The app will also provide in-car content streaming, showing movies from Universal Studio.

AI-optimized parking

Parking is one of the most notable pain points for car owners and fleet managers.

Studies have shown that the average US driver spends 17 hours every year looking for parking, and the total cost of wasted time, fuel, and emissions can add up to $345 a year for the average driver. Especially in dense urban areas, looking for a parking spot can sometimes take just as long as the commute itself.

These metrics are slated to worsen as cities grow more congested, with population growth and on-demand services such as ride-hailing, micromobility, and food delivery driving increased demand for curb space.

In response, a growing number of startups are developing tech-based solutions to make the parking process easier and more efficient.

The rise of connected parking technology

Startups are deploying a host of new technologies, such as IoT, computer vision, and machine learning, to better regulate demand for parking, optimize supply, and improve retrieval time.

Munich-based Cleverciti Systems focuses on reducing traffic in cities by deploying sensors that can monitor all parking spaces in a city and connect to screens that show drivers where to park. The company works not only with cities, but also in shopping malls, airports, and stadiums.

SpotHero, which raised $50M in Series D funding in August, is developing both hardware and software solutions to help users find, book, and access parking. The company has created a network of over 6,500 garages in over 300 cities to provide more visibility into parking inventory and better match supply with demand.

In June, the company announced a partnership with navigation app Waze to help users find the fastest route to their pre-booked spot.

Looking ahead: robotics in parking

Startups are also leveraging robotics to facilitate the parking process. For example, French startup Stanley Robotics is developing a self-driving parking robot for Lyon-Saint Exupéry airport.

Source: New Atlas

The company’s robots can physically pick up a vehicle by sliding a platform underneath it, lift it up, and carry it to a parking spot, saving drivers and passengers time and minimizing fuel costs and emissions.

Parking in an autonomous, shared future

With the rise of autonomous and shared vehicles, the future of parking is largely uncertain, and much of the parking infrastructure in place today will have to change to serve new ways of commuting and traveling. For example, shared parking venues could eventually fully replace city street parking. A number of cities are already utilizing street parking spots for more efficient uses, such as bike and scooter lanes or lanes for on-demand delivery vehicles.

Given the uncertainty around the future of mobility, parking startups will have to be ready to adapt their technology for new applications and a shifting customer base in urban areas. For example, some startups are already outfitting garages with sensors so that self-driving car operators can book and pay for parking spaces in advance.

That said, even with the rise of autonomous vehicles and mobility-as-a-service, cars will still need to be parked, and a number of startups are developing technology to help optimize this process. This includes ensuring access to the right number of spots, as well as managing relationships between garage owners, fleet companies, and cities by enabling the exchange of parking data.

Conclusion

Major disruptive forces, such as connectivity, autonomy, and shared mobility, are requiring automakers to prioritize the driver and passenger experience.

To adapt, automakers will have to continue digitizing the mobility ecosystem so that the car can be seamlessly integrated into the consumer’s entire suite of connected devices.

Ultimately, given that the future of the automotive space remains largely in flux, startups looking to disrupt the in-vehicle experience will need to master a difficult balancing act: remaining flexible and adaptable, while also complying with the stringent safety regulations characteristic of the automotive industry. 

If you aren’t already a client, sign up for a free trial to learn more about our platform.