Several areas where the development and use of AI can benefit from learning unlearned lessons are outlined: improved risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety that include social and organizational dynamics as first-order engineering concerns.
Abstract
As automated decision-making and data-driven technologies pervade society and are used to manage consequential outcomes, understanding the technology's capabilities, limitations, and attendant risks in context requires analysis of full sociotechnical systems. Sociotechnical analysis of risks in highly complex systems provides clear lessons for the design and evaluation of AI systems, transcending a technical focus on reliable or"responsibly designed"components to understand risks at a systems level. Human-made catastrophes have been studied for decades because of the severity of these events: consider Chernobyl, Three Mile Island, Fukushima-Daiichi, Bhopal, the Challenger disaster. A common misconception is that these kinds of events are freak accidents, resulting from the inherently unforeseeable interactions in complex systems. Closer examination reveals that the risks and hazards were well-known beforehand but not acted upon due to social structural, political and economic factors. We outline several areas where the development and use of AI can benefit from learning these unlearned lessons: improved risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety that include social and organizational dynamics as first-order engineering concerns. For each area, we offer concrete unlearned lessons and exemplify how they led to failure in prior accidents as well as examples of how these lessons remain unlearned for modern computing systems, particularly AI.
To manage complex disaster risks, it is important to develop a response that spans multiple areas of expertise. Bridging the gap between law, social sciences, and natural sciences is an important part of any disaster risk reduction. It helps systems adapt quickly as AI technologies are constantly changing and have significantly impacted where law and the natural environment intersect, influencing legal systems and environmental policies, and how legal and environmental issues can prevent AI from having a greater impact on society and the economy. As part of a participatory assessment of production, with the assistance of legal experts, social and environmentalists, the key principles of responsible data analysis are proposed. These principles focus on security, transparency, fairness, accountability, and the ability to challenge or challenge decisions. This conversation describes how different disciplines can work together to create flexible legal systems that use AI, while drawing knowledge from the environmental and social sciences. The way environmentalists and decision-makers talk about useful and accurate information leads to differences that make the use of artificial intelligence difficult due to legal issues related to the reliability, reliability, and contentiousness of disaster management systems. If social media is useful for disaster risk reduction through AI, it's important to consider legal issues related to who is responsible and how sensitive the information used in disaster management is. Ideas for a fair and responsible process focus on the environment and encourage discussions on social and economic issues related to public participation. AI is also very important in education, as it brings together the next generation of law, social and natural sciences to jointly find solutions that bring together different disciplines in a balanced way. AI tools can be very useful in emergency management, but it's important to use them fairly and responsibly. You need to think about things like possible unfair benefits, clear explanations, and protecting people's personal data. Careful use of AI techniques, considering how AI, laws, and environmental risks interact with each other, helps create equitable and sustainable ways to collect and use data.
R. Paper, Research Supervisor Prof, Dominique Ferraro· The social science· 0 citations
Engaging CDV implementation is transformative, vastly improving the authors' collective ability to understand complex, disparate information, and fundamentally accelerate the transition from data visualization to actionable insight.
Shayna S. Solis, A. Silcott, Zoey Armstrong et al.· GeoHorizons· 0 citations
Summary As artificial intelligence (AI) is increasingly integrated into disaster risk reduction/management (DRR/M), critical scrutiny remains essential to avert unintended consequences that may compound existing challenges. This study aims to explore the use of AI in hazard, vulnerability, and risk assessments through the analytical lens of the Black Mirror conceptual framework, focusing on (1) the critical shortcomings of these integrations and (2) the potential dangers of translating faulty research outputs into DRR/DRM policy and practices. A semi-systematic review of the literature underpins these identified limitations. While observable in AI applications, the presented aspects are not confined to them. To address the identified concerns, we outline 5 rules of thumb that create a roadmap for improved AI development and integration. We contend that progress in DRR and DRM will be determined less by the sophistication of AI models and more by the intellectual and ethical rigor we apply in their development.
Artificial Intelligence (AI) is increasingly integrated into complex sociotechnical systems, including Critical National Infrastructure (CNI), where harms emerge from interactions between technical, human, and organisational elements. Yet current AI evaluation remains model-centric, offering little insight into how observed behaviours might translate into system-level risk. We propose a framework that links structured hazard analysis, component-level testing, and probabilistic system modelling to bridge this gap. By providing a traceable pathway from model behaviour to system-level outcomes, the framework enables practitioners to answer the"so what?"of AI failures, quantify their systemic impact, and move toward evidence-based and anticipatory governance of AI in complex systems. Applied to the UK's Real Time Gross Settlement (RTGS) system as an illustrative worked example, we derive AI-driven loss scenarios using Systems Theoretic Process Analysis (STPA) and examine adversarial manipulation of LLM-based trading as one such loss scenario. Component-level experiments show that simple adversarial inputs induce measurable behavioural shifts where AI recommendations are followed. Under the component-to-system mapping used here for a financial contagion model, these shifts alter system resilience, increasing bank failures and lowering the threshold at which shocks lead to cascading disruption, particularly under widespread or monopolistic AI adoption.
Paul Vautravers, Oliver Chalkley, Gabriel Downer et al.· 0 citations
To continue exposing AI-driven military violence, OSI practices must reorient their methodologies and focus more systemically and systematically on the dispersed material infrastructures and political forces underpinning AI militarism, so investigators can better reveal concealed networks of accountability linking state and corporate actors.
As AI systems become increasingly integrated into consequential domains such as healthcare, journalism, education, scientific research, organizational decision-making, and defense, effective human-AI collaboration has emerged as a critical challenge. However, the sociotechnical risks that undermine collaboration are often studied in isolation, obscuring the recurring failure mechanisms that cut across domains. This paper presents a lifecycle-oriented synthesis of human-AI collaboration risks spanning four stages: task allocation, interaction, feedback, and adoption. Drawing on evidence from diverse application domains, we identify six recurring cross-domain risk clusters: Trust Miscalibration, Cognitive Burden, Accountability Gap, Capability Erosion, Goal Misalignment, and AI Anxiety and Technostress. We further propose a conceptual interaction model that illustrates how these risks emerge from sociotechnical drivers, interact through cascading pathways, and ultimately affect team performance and human well-being. Our analysis shows that many collaboration failures stem not from isolated technical deficiencies but from interconnected sociotechnical dynamics, helping explain why piecemeal interventions frequently create unintended consequences. By synthesizing fragmented literature into a unified framework, this work provides a foundation for future empirical research, lifecycle-oriented governance, and the design of more resilient, trustworthy, and human-centered human-AI collaboration systems.
Md Foysal Ahmed, Isaac Kobby Anni, Md Main Uddin Rony· 0 citations
