FEDERAL CYBERSECURITY RESEARCH AND DEVELOPMENT STRATEGIC PLAN (2019) Challenges AI systems design should be informed by security, safety, and privacy principles. While these principles are not synonymous and in fact can be in tension, there are common characteristics upon which converged results can be achieved: reproducibility, accountability, interpretability, explainability, verifiability, transparency, and reliability. Existing cybersecurity concepts may have to be reformulated and redesigned to cope with AI models, algorithms, and the human-AI system interactions. Below are key issues at the interface of AI and cybersecurity: • AI effects on operational speed and scale. AI systems operate at speeds and scale beyond human and current technological capabilities. This is of concern in areas where malicious activities may be driven by AI, leading to an increasingly asymmetric engagement between attackers and defenders unless AI is likewise used for cyber defenses. In general, AI systems will enable more sophisticated and automated cyber defense, and if not implemented with appropriate controls, could also be wielded offensively to perform malicious activities. • Interpretability, explainability, and transparency of AI. Reasoning used in AI systems is very different from that used by humans and is not always intuitive to humans. Even though some individual algorithms used in AI can be comprehensible, their collective behavior may not always be. Thus, being able to understand, interpret, explain, and anticipate AI outputs and outcomes with transparency is difficult. Assessing biases in data or AI algorithms and guarding against these potential biases without compromising privacy will contribute to creating trustworthy AI, as will strengthening AI’s accuracy, resiliency, safety, reliability, objectivity, and security. This raises questions such as how to model and measure trust in AI systems, and what levels of certainty should guide deployment of these types of systems. This is of concern in cybersecurity areas such as situational awareness, threat and risk estimation and management, privacy risk mitigation, and resource allocation. • Vulnerability of systems with AI components. Many machine learning (ML) algorithms are subject to attacks throughout their lifecycles. The following kinds of attacks can take place at any stage: poisoning datasets to degrade model quality, creating backdoors that allow model creators access to other systems, crafting instances that induce classification errors, and inference attacks on both the model itself and on the dataset used to train it. Although the vulnerability surface of AI/ML is not yet well understood, it must be accounted for in AI/ML implementations. In this respect, the threat models for AI systems differ from traditional software- and/or hardware-related threats. • Efficacy evaluation of AI cybersecurity systems. AI is increasingly becoming part of technologies that are central to our daily lives, such as healthcare and transportation. The boundaries between the cyber, the physical, the social, and the economic are ever more blurred. While this results in efficiency gains, it also increases dependencies and increases exposures to natural, manmade, and programmatic accidents and threats. This raises the likelihood of systemic risks and cascading harms from threats in one domain spilling over into other domains. AI introduces more complexity on top of existing, insufficient understanding of the efficacy of current cybersecurity technologies, such as how much security can be gained from investing in certain controls, and which controls best reduce risk. Measuring the multidimensional causal links between AI-based security controls, levels of resulting security, and outcomes in the face of threats is a significant challenge. R&D Goals • Simulate different decision-support scenarios with respect to threat models, including attacker/defender strategies related to specific AI/ML implementations, to avoid endless attackdefense cycles perpetuated by AI/ML techniques. Expand and explore new AI-based techniques – 12 –