Culture Clashes: LLM Support in the Engineering of Safety-Critical Systems HTW Berlin

Format:

Book

Conference/Event

Author/Creator:

Thomas, Carsten, author.

Wagner, Michael, author.

Conference Name:

WCX SAE World Congress Experience (2026-04-14 : Detroit, Michigan, United States)

Language:

English

Subjects (All):

Artificial intelligence (AI).

Risk management.

Safety critical systems.

Tools and equipment.

Computer software and hardware.

Local Subjects:

Artificial intelligence (AI).

Risk management.

Safety critical systems.

Tools and equipment.

Computer software and hardware.

Physical Description:

1 online resource

Place of Publication:

Warrendale, PA SAE International 2026

Summary:

The proven usefulness of large language models (LLMs) as tools for software development and the recent rapid increase in their capabilities have made it possible and attractive to extend their scope of application to almost all tasks in the engineering of complex and even safety-critical systems. While these tools promise substantial efficiency gains and improved engineering productivity, they remain prone to errors, and the generated artifacts may not meet the stringent quality requirements for safety-critical systems. In this paper, we systematically analyze potential applications of LLMs throughout the engineering lifecycle of safety-critical systems and identify associated risks as well as practical approaches to risk mitigation. We classify LLM-supported use cases according to LLM autonomy, impact, and artifact observability, and compare the corresponding mitigation strategies with established approaches used for traditional engineering automation. In addition, we examine the cultural and psychological aspects influencing trust in LLM-based engineering tools and the risks of both over-reliance and unwarranted rejection. Our analysis shows that LLMs can provide substantial benefits as engineering support tools, but they also represent a significant source of development risk if applied without appropriate safeguards. Based on these findings, we propose guidelines for responsibly using LLM-based tools in the engineering of safety-critical systems

Notes:

Vendor supplied data

Access Restriction:

Restricted for use by site license