awesome-safety-critical

This is a list of resources about programming practices for writing
safety-critical software.

The starting point for me to create this resource was my interest in a solid
software:

What kind of special training do engineers working on mission-critical software receive? [closed] and its followup on Reddit.

Disclaimer: Resources presented here are not necessarily authoritative or
latest documents on the topic.

Contents

Table of Contents generated with DocToc

• Friendly lists
• Resources
• Software safety standards
• Safety handbooks
• Coding guidelines
• Topics
  • Certification
  • Formal verification
  • MC/DC
• Articles
• Papers
• Reports
• Accidents
• Questions and Answers
• Books
• Videos
• Interviews
• Press
• License

Friendly lists

• resilience-engineering

Resilience engineering papers http://resiliencepapers.club

• awesome-software-quality

List of free software testing and verification resources

• awesome-provable

A curated set of links to formal methods involving provable code.

• practical-fm

A List of companies that use Formal methods in Software engineering

• awesome-static-analysis

A curated list of static analysis tools, linters and code quality checkers
for various programming languages

• Computer-Based System Safety Essential Reading List

Resources

• European Cooperation for Space Standardization

The European Cooperation for Space Standardization is an initiative established to develop a coherent, single set of user-friendly standards for use in all European space activities.

This list has a number of links from this resource.

• The International System Safety Society

The International System Safety Society is a non-profit organization dedicated to supporting the Safety Professional in the application of Systems Engineering and Systems Management to the process of hazard, safety and risk analysis. The Society is international in scope and draws members throughout the world. It is affiliated with major corporations, educational institutions and other agencies in the United States and abroad.

This list has a number of links from this resource.

• NASA Langley Formal Methods Research Program

The NASA Langley's Formal Methods Research Program of the NASA Langley Safety-Critical Avionics Systems Branch develops formal methods technology for the development of mission-critical and safety-critical digital systems of interest to NASA.

Software safety standards

• IEC 61508

  IEC 61508 is an international standard published by the International
  Electrotechnical Commission of rules applied in industry. It is titled
  Functional Safety of Electrical/Electronic/Programmable Electronic
  Safety-related Systems (E/E/PE, or E/E/PES).

  • ISO 26262

  The ISO 26262 Standard is prepared by the ISO Committee and is a derivative of the IEC 61508 standard listed above. The committee members include the major vehicle manufacturers and suppliers. It is expressly a safety standard, but includes details about Hazard Analysis and Risk Assessment and system design to detect faults and their potential failures.

  • IEC 62279

  IEC 62279 provides a specific interpretation of IEC 61508 for railway applications. It is intended to cover the development of software for railway control and protection including communications, signaling and processing systems.

  • IEC 61513

  IEC 61513 provides requirements and recommendations for the instrumentation
  and control for systems important to safety of nuclear power plants. It
  indicates the general requirements for systems that contain conventional
  hardwired equipment, computer-based equipment or a combination of both types of
  equipment.

• DO-178C

  DO-178C, Software Considerations in Airborne Systems and
  Equipment Certification is the primary document by which the certification
  authorities such as FAA, EASA and Transport Canada approve all commercial
  software-based aerospace systems. The document is published by
  RTCA, Incorporated, in a joint effort with EUROCAE, and replaces DO-178B.
  The new document is called DO-178C/ED-12C and was completed in November 2011
  and approved by the RTCA in December 2011. It became available for sale and
  use in January 2012.

  The FAA approved AC 20-115C on 19 Jul 2013, making DO-178C a recognized
  "acceptable means, but not the only means, for showing compliance with the
  applicable airworthiness regulations for the software aspects of
  airborne systems and equipment certification." (Wikipedia)

• ARINC standards

  The ARINC Standards are prepared by the Airlines Electronic Engineering Committee (AEEC) where Rockwell Collins and other aviation suppliers serve as a contributor in support of their airline customer base. (Wikipedia)

  • ARINC 653

  ARINC 653 is a standard Real Time Operating System (RTOS) interface for partitioning of computer resources in the time and space domains. The standard also specifies Application Program Interfaces (APIs) for abstraction of the application from the underlying hardware and software.

• MIL-STD-882E, System Safety

• MIL-STD-1472F, Human Engineering

• NASA-STD-8719.13B, NASA Software Safety Standard

• ECSS-E-ST-40C, Software

• ECSS-Q-ST-80C Rev.1 – Software product assurance

Safety handbooks

• ECSS-E-HB-40A – Software engineering handbook

• NASA Software Safety Guidebook

NASA's Software Safety Guidebook (pdf file). The handbook complement to the
Software Safety Standard.

• NASA Systems Engineering Handbook

• NASA Fault Management Handbook

  • The Development of NASA’s Fault Management Handbook (Slides)

• NASA Safety Culture Handbook

• Software System Safety Handbook

From the Joint Services Computer Resources Management Group, US Navy, US Army,
And US Air Force (pdf file)

• Joint Software Systems Safety Engineering Handbook

• Air Force System Safety Handbook

First chapter has an excellent introduction to system safety with a discussion
of the evolution of the DoD Standard 882 (DOD Standard Practice for
System Safety).

• European Space Agency - Technology Readiness Levels Handbook for Space Applications

Coding guidelines

• MISRA guidelines

  • (MISRA C:2012) Guidelines for the Use of the C Language in Critical Systems,
    ISBN 978-1-906400-10-1 (paperback), ISBN 978-1-906400-11-8 (PDF), March 2013.
  • (MISRA C++:2008) Guidelines for the Use of the C++ Language in Critical Systems, ISBN 978-906400-03-3 (paperback), ISBN 978-906400-04-0 (PDF), June 2008.
  • See more papers there.

• AUTOSAR C++14: Guidelines for the use of the C++14 language in critical and safety-related systems

• The Power of Ten – Rules for Developing Safety Critical Code

• JPL Institutional Coding Standard for the C Programming Language

• NASA C STYLE GUIDE

• C++ Coding Standards and Style Guide

This document is based on the "C Style Guide" (SEL-94-003). It contains recommendations for C++ implementations that build on, or in some cases replace, the style described in the C style guide.

• SEI CERT C Coding Standard

• SEI CERT C++ Coding Standard

• SEI CERT C and C++ Coding Standards are now freely available in pdf
  format:
  C++ Coding Standard,
  C Coding Standard

• JOINT STRIKE FIGHTER AIR VEHICLE C++ CODING STANDARDS, also video:
  CppCon2014: Bill Emshoff "Using C++ on Mission and Safety Critical Platforms"

Topics

Certification

• Introduction to Certification by Quentin Ochem, AdaCore

• Certification Requirements for Safety-Critical Software

• Technology Readiness Level, ESA, Technology Readiness Level, NASA

Technology Readiness Levels (TRL) are a type of measurement system used to
assess the maturity level of a particular technology. Each technology project is
evaluated against the parameters for each technology level and is then assigned
a TRL rating based on the projects progress. There are nine technology readiness
levels. TRL 1 is the lowest and TRL 9 is the highest.

Formal verification

• Verified Squared: Does Critical Software Deserve Verified Tools?

MC/DC

• A practical approach to Modified Condition/Decision Coverage

This paper provides a practical 5-step approach for assessing MC/DC for
aviation software products, and an analysis of some types of errors expected
to be caught when MC/DC is achieved.

• A Practical Tutorial on Modified Condition/Decision Coverage

This tutorial provides a practical approach to assessing modified
condition/decision coverage (MC/DC) for aviation software products that must
comply with regulatory guidance for DO-178B level A software.

• An Empirical Evaluation of the MC/DC Coverage Criterion on the HETE-2 Satellite Software

...In this paper, we present the results of an empirical study that compared
functional testing and functional testing augmented with test cases to satisfy
MC/DC coverage. The evaluation was performed during the testing of the
attitude control software for the HETE-2 (High Energy Transient Explorer)
scientific satellite...

Articles

• About Safety Critical Software

• IEEE Spectrum-Why Software Fails

• IEEE Spectrum-Lessons From a Decade of IT Failures

Papers

• White Paper on Approaches to Safety Engineering

This white paper lays out some foundational information about different
approaches to safety: how various industries differ in their approaches to
safety engineering, and a comparison of three general approaches to safety
(system safety, industrial safety engineering, and reliability engineering).
An attempt is made to lay out the properties of industries and systems that
make one approach more appropriate than another.

• Software Safety Standards: Evolution and Lessons Learned

• An Overview of Software Safety Standards

• Making reliable distributed systems in the presence of software errors

• Why Do Computers Stop and What Can Be Done About It?

• Targeting Safety-Related Errors During Software Requirem.ents Analysis

• How Complex Systems Fail

• The Qualification of Software Development Tools From the DO-178B Certification Perspective

• The Role of Software in Spacecraft Accidents

Reports

• An Assessment of Space Shuttle Flight Software Development Processes

• Licensing of safety critical software for nuclear reactors (2018)

Common position of international nuclear regulators and authorised technical support organisations

• Assessment of Safety Standards for Automotive Electronic Control Systems

Van Eikema Hommes, Q. D. (2016, June). Assessment of safety standards for
automotive electronic control systems. (Report No. DOT HS 812 285).
Washington, DC: National Highway Traffic Safety Administration.

Accidents

• Failure Knowledge Database

• Mars Climate Orbiter Mishap Investigation

• Report on the Loss of the Mars Polar Lander and Deep Space 2 Missions

• An Investigation of the Therac-25 Accidents (original paper),
  Medical Devices: The Therac-25 (updated version of the paper),
  Killed by a Machine: The Therac-25 (article)

• ESA ARIANE 5 Flight 501 Failure

• ExoMars 2016 - Schiaparelli Anomaly Inquiry (PDF at the bottom), ESA Schiaparelli Lander Crash

• A Case Study of Toyota Unintended Acceleration and Software Safety and NASA report on the Toyota Unintended Acceleration Issue

Questions and Answers

- Which languages are used for safety-critical software?

See Which languages are used for safety-critical software? [closed].

- What is the difference between mission-critical and safety-critical software?

This article contains interesting section on what is the difference between
mission-critical and safety-critical software: Military COTS-based systems: Not necessarily right off the shelf

- What kind of special training do engineers working on mission-critical software receive?

See What kind of special training do engineers working on mission-critical software receive? [closed] and its followup on Reddit. In the Reddit thread there are 2 expanded answers. The thread is also archived here.

- What are the software safety standards?

See the Software Safety Standards here in this list.

Also see on StackOverflow: Coding for high reliability/availability/security - what standards do I read?
and Software Safety Standards

- Safety-critical software and optimising compilers?

Safety-critical software and optimising compilers

- Does Rust have a chance in mission-critical software?

Does Rust have a chance in mission-critical software? (currently Ada and proven C niches)

Books

• Safety Critical Systems Handbook: A Straight forward Guide to Functional Safety, IEC 61508 (2010 EDITION) and Related Standards, Including Process IEC 61511 and Machinery IEC 62061 and ISO 13849 1st Edition

• Engineering a Safer World. Systems Thinking Applied to Safety

• Computer-Related Risks

• Building High Integrity Applications with SPARK

• Building Parallel, Embedded, and Real-Time Applications with Ada

• Writing Solid Code

• SafeScrum® – Agile Development of Safety-Critical Software

Videos

• Embedded System Safety Lecture Video Series

• The Need for a Paradigm Shift in Safety and Cyber Security

CREDC Seminar Series. Presented on November 7, 2016 by Nancy Leveson,
Professor of Aeronautics and Astronautics and Engineering Systems, MIT.
Cyber Resilient Energy Delivery Consortium (CREDC), http://cred-c.org

• Velocity 2012: Richard Cook, "How Complex Systems Fail"

Dr. Richard Cook is the Professor of Healthcare Systems Safety and Chairman of the Department of Patient Safety at the Kungliga Techniska Hogskolan (the Royal Institute of Technology) in Stockholm, Sweden. He is a practicing physician, researcher and educator.

See also paper "How Complex Systems Fail".

• 2017 EuroLLVM Developers’ Meeting: M. Beemster "Using LLVM for Safety-Critical Applications" and Using LLVM for Safety-Critical Applications. Interview with Marcel Beemster (Euro LLVM 2017).

Marcel Beemster, Solid Sands B.V. http://solidsands.nl/, http://www.LLVM.org/devmtg/2017-03/

• Formal Method for Avionics Software Verification
  • Formal Method for Avionics Software Verification pt1 (Hervé Delseny)
  • Formal Method for Avionics Software Verification pt2 (Hervé Delseny)
  • Formal Method for Avionics Software Verification pt3 (Hervé Delseny)
  • Formal Method for Avionics Software Verification pt4 (Hervé Delseny)

This talk will give examples of Airbus use of Formal Methods to verify
avionics software, and summarises the integration of Formal Methods in the
upcoming ED-12/DO-178 issue C. Firstly, examples of verification based on
theorem proving or abstract interpretation will show how Airbus has already
taken advantage of the use of Formal Methods to verify avionics software.
Secondly, we will show how Formal Method for verification has been introduced
in the upcoming issue C of ED-12/DO-178.

• Programming Languages for High-Assurance Vehicles

• Mars Code - Gerard Holzmann, JPL Laboratory for Reliable Software (2012)

Interviews

Safety in Medical Device Software: Questions and Answers

Press

They Write the Right Stuff

This software is the work of 260 women and men based in an anonymous office
building across the street from the Johnson Space Center in Clear Lake, Texas,
southeast of Houston. They work for the “on-board shuttle group,” a branch of
Lockheed Martin Corps space mission systems division, and their prowess is
world renowned: the shuttle software group is one of just four outfits in
the world to win the coveted Level 5 ranking of the federal governments
Software Engineering Institute (SEI) a measure of the sophistication and
reliability of the way they do their work. In fact, the SEI based it standards
in part from watching the on-board shuttle group do its work.

License

To the extent possible under law,

Stanislav Pankevich
has waived all copyright and related or neighboring rights to
awesome-safety-critical.

This list's repository contains a backup of all content presented in the list.
This is done to ensure availability of these resources in case if their original
sources become unavailable. Every link always points to its original source
unless it becomes unavailable in which case a resource from a backup is used
or a link to web.archive.org if possible.
awesome_bot tool is used to check the dead links.