Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis Clemson University

Format:

Book

Conference/Event

Author/Creator:

Fernández, David, author.

MohajerAnsari, Pedram, author.

Salarpour, Amir, author.

Pese, Mert D., author.

Conference Name:

WCX SAE World Congress Experience (2026-04-14 : Detroit, Michigan, United States)

Language:

English

Subjects (All):

Automated driving systems.

Autonomous vehicles.

Visibility.

Local Subjects:

Automated driving systems.

Autonomous vehicles.

Visibility.

Physical Description:

1 online resource

Place of Publication:

Warrendale, PA SAE International 2026

Summary:

Vision-language models (VLMs) are increasingly used in autonomous driving because they combine visual perception with language-based reasoning, supporting more interpretable decision-making, yet their robustness to physical adversarial attacks, especially whether such attacks transfer across different VLM architectures, is not well understood and poses a practical risk when attackers do not know which model a vehicle uses. We address this gap with a systematic cross-architecture study of adversarial transferability in VLM-based driving, evaluating three representative architectures (Dolphins, OmniDrive, and LeapVAD) using physically realizable patches placed on roadside infrastructure in both crosswalk and highway scenarios. Our transfer-matrix evaluation shows high cross-architecture effectiveness, with transfer rates of 7391% (mean TR = 0.815 for crosswalk and 0.833 for highway) and sustained frame-level manipulation over 64.779.4% of the critical decision window even when patches are not optimized for the target model. We further find asymmetric architecture-level risk, with Dolphins most vulnerable to incoming transfer attacks (VS = 0.82) and LeapVAD producing the most transferable patches (TO = 0.882), while models sharing CLIP-based vision encoders exhibit stronger bidirectional transfer. Overall, these results indicate that current VLM-based autonomous driving systems share systematic cross-architecture weaknesses that architectural diversity alone does not resolve, underscoring the need for defenses and design principles that explicitly account for transferability in safety-critical deployment

Notes:

Vendor supplied data

Access Restriction:

Restricted for use by site license