The self-driving car industry is at a crossroads, facing challenges from safety concerns, regulatory issues, and public skepticism. In the midst of this, Ghost Autonomy, a startup focusing on autonomous driving software, believes it has found a breakthrough to address safety concerns and win over critics.

Ghost Autonomy recently announced plans to explore the use of multimodal large language models (LLMs) in self-driving technology, partnering with OpenAI through the OpenAI Startup Fund. This collaboration provides Ghost Autonomy with early access to OpenAI systems, Azure resources from Microsoft, and a $5 million investment.

The concept involves applying LLMs, which can understand both text and images, to enhance scene interpretation for autonomous vehicles. By using these models, Ghost aims to improve the decision-making process for self-driving cars based on complex scenes captured by car-mounted cameras.

John Hayes, Ghost Autonomy’s co-founder and CEO, emphasized the potential of LLMs in providing nuanced reasoning for scenes where current models fall short. He envisions LLM-based analysis becoming more crucial as these models evolve.

However, some experts express skepticism about Ghost Autonomy’s approach. Os Keyes, a Ph.D. candidate at the University of Washington, sees the use of LLMs as a marketing buzzword and questions their appropriateness for self-driving applications. Keyes argues that LLMs were not designed or trained for such purposes, potentially making them less efficient for addressing challenges in vehicular autonomy.

Mike Cook, a senior lecturer at King’s College London, echoes Keyes’ concerns, emphasizing that LLMs are not a panacea in computer science. He points out the challenges of validating the safety of LLMs, even for ordinary tasks, and questions the wisdom of applying this technology to autonomous driving.

Despite skepticism, Ghost Autonomy remains confident in its approach. Hayes argues that LLMs could enable autonomous driving systems to reason about scenes holistically, utilizing broad-based world knowledge to navigate complex and unusual situations. Ghost is actively testing multimodal model-driven decision-making with its development fleet and collaborating with automakers to integrate these models into its autonomy stack.

While the skepticism remains, Ghost Autonomy believes that a combination of rigorous testing, data-driven improvements, and collaboration with automakers will pave the way for the successful integration of LLMs into self-driving technology.

By Impact Lab