Catastrophic risk in the context of AI refers to the potential for failures or adverse outcomes that could have severe, widespread consequences, such as significant societal disruption, economic loss, or endangerment of human safety. These risks emerge from the increasing capabilities and autonomy of AI systems, which, if not properly safeguarded, can amplify errors or be exploited for malicious purposes. The challenge lies in designing AI systems that remain safe and robust under all circumstances, which involves anticipating complex interactions and unintended uses, and establishing failsafe mechanisms. Addressing catastrophic risks is critical as AI continues to integrate into crucial sectors like healthcare, finance, and autonomous vehicles, where errors can have dire ramifications.
The term "catastrophic risk" began appearing in discussions around AI safety as early as the 1990s but gained significant traction in the academic and technical communities in the late 2000s as AI systems grew in complexity and impact potential.
Prominent figures in the development of the catastrophic risk concept include Nick Bostrom, whose work in existential risk and AI safety has been foundational, and Stuart Russell, who has contributed extensively to discussions about aligning AI systems with human values to mitigate potential harms. Additionally, organizations such as the Future of Humanity Institute and the Machine Intelligence Research Institute have advanced understanding and policy recommendations for managing these risks.