What the White House Executive Order means for AI development and testing: Tricentis takeaways

On October 30, 2023, President Joe Biden issued a landmark Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence (AI). The Executive Order represents a comprehensive statement of intent for AI regulation, mandating transparency from firms that use AI and establishing safety and security standards. Moreover, it compels AI developers to disclose the outcomes of safety evaluations to the U.S. government, especially if the results indicate a potential threat to national security. This Order is bound by the limitations of the executive branch and so it is not yet a law, but it is likely be adopted or to trigger future laws.

The Order’s definition of AI systems is broad: “a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments.” This covers far more than generative AI (GenAI) or systems developed on neural networks. Here, we will focus on organizations that are embedding AI into their systems, versus organizations that are training their own AI foundation models.

What does the Order stipulate?

Several critical elements of the Order will impact organizations that embed AI into their systems: the involvement of the National Institute of Standards and Technology (NIST), the establishment of Red Teams for testing, and a greater focus on invalidation in testing.

• The role of the National Institute of Standards and Technology (NIST): The Order gives NIST a leading role in developing best practices for AI safety, security, and trustworthiness, ensuring that AI developers prioritize safety and security.
• More regulatory scrutiny: The Order introduces stricter regulations for AI models that are broadly trained and applicable in various contexts, referred to as Dual Use Foundation Models.
• More AI content and labeling: The Order focuses on tracking the origin and creation of AI-generated content, including provenance tracking, synthetic content tracking, and managing AI in copyrighted works.
• More talent development: The Order aims to spur innovation by concentrating on talent training and modifying immigration policies.
• Consumer protection: The Order addresses consumer protection against fraud and discrimination within systems operated by the U.S. Department of Health and Human Services.
• Red Teams: The Order emphasizes the importance of Red Teaming as a rigorous testing process to identify vulnerabilities and flaws in AI systems. This process is aimed at ensuring AI robustness and security, distinct from traditional QA, and emphasizes the independence and impact of these teams on AI development.

What is Red Teaming and why is it important in QA and testing?

Apart from standard QA practices, Red Team AI testing refers to a structured effort to identify vulnerabilities and flaws in an AI system. Red Team AI testing originated at the White House’s first-ever GenAI competition and is based on Cold War-era battle simulations where red teams attack and blue teams defend against intrusion.

Red Team AI testing often involves the deliberate probing of an AI model to test the limits of its capabilities. This is often referred to as adversarial testing and is done in a controlled environment. It is important to uncover specific vulnerabilities and identify priorities for risk mitigation. Red Teams seek to identify harmful or discriminatory outputs from an AI system, unforeseen or undesirable system behaviors, limitations, and potential risks associated with misuse. Because a core component of the system is effectively “unknowable,” we are obligated to explore the negative space around the system’s performance.

How will the Order impact organizations using AI today?

This Order is a precursor to regulation. Its intent is not to prevent AI innovation, but to provide a framework under which innovation can safely thrive. In fact, the Order itself explicitly states: “Agencies are discouraged from imposing broad general bans or blocks on agency use of generative AI.” It shouldn’t stop you from undertaking new initiatives around AI-driven technologies, but it should shape how you plan to go about them.

For organizations that are embedding AI into their systems (versus those that are training their own foundation models), the Order has implications in six major areas. Going forward, these organizations should be prepared to:

• Expect validation times to vary independent of build times: For a long time, it’s been an industry truism that the time it takes to validate the quality of something is roughly proportional to the time it takes to build it. However, AI-based projects form such a deviation from this norm that it’s best to forget that this correlation ever existed.
• Complete comprehensive validation: Proper validation as defined by NIST calls for assessment of potential liabilities, Red Team testing of adversarial attacks and penetration attempts, security validation, data review, and identification of biases.
• Adopt a Red Team strategy in QA: We should adopt Red Team practices in QA. This includes testing the AI system and pushing its boundaries through adversarial testing. By simulating real-world attacks or problematic scenarios, Red Teaming provides a more thorough evaluation of the AI’s robustness and reliability.

Organizations must also have clear strategies to support Red Team operations, addressing their independence and impact on project timelines. Organizations should identify how Red Teams are structured and how independent they are within an organization. Are Red Teams working with developers or independently? Are Red Teams measured by different KPIs than developers?

• Adopt a specific plan for validating AI systems: The question to ask is: What is the risk of failure, and what is the impact when it occurs? Many organizations gloss over these possibilities, shipping acceptable software without taking the time to explore the ways in which the software could misbehave. From a testing standpoint it is more difficult and time-consuming to force an AI system into a harmful state, but the Order mandates this approach.
• Invest in AI skills adoption and training: For your teams to effectively explore the edges of an AI system, they need a fundamental understanding of AI. Educate your teams to understand how different AI responses are generated, how AI safety features work and the best techniques to bypass them, how embeddings are used to evaluate context retrieval, and how in-prompt learning works in content extraction.
• Provide clarity that AI is present: This Order, along with the EU IA Act, places the burden of communication squarely on the developer. If your chatbot is replying, your users should know that the replies are AI-generated. If the output of your process is content, you may be required to keep a registry that can be queried to determine whether the answers are “real.”

What are the challenges in testing an AI system?

Outside of determining the practical usefulness of an AI system, it’s important to stay aware of several common fallacies in thinking around AI.

Mistaking it for a logical entity

Given the surface-level intelligence of AI responses, it’s easy to mistake it for a logical being. Here are some traits that QA teams need to be aware of:

• Incuriosity: AI never asks questions nor seeks clarification. It often gives answers without full details.
• Placation: It immediately changes the answer whenever any concern is shown about that answer.
• Incongruence: It fails to follow its own advice or processes. For instance, it might repeat mistakes or make new ones even after acknowledging them.
• Negligence/laziness: It often gives simple answers even when a practitioner provides more details about nuances and critical ambiguities.

Mistaking it for an information repository

It’s also easy to mistake an AI system for an information repository. But an AI doesn’t look up information; it synthesizes a response based on what it has been taught. Here are some traits that result from this fact:

• Hallucination: It makes up facts from nothing, even if they contradict other facts.
• Opacity: It provides limited explanation about what it covered, its decisions, or its choices.
• Unreliability: It cannot reliably give a consistent answer to a similar question in similar circumstances.

Mistaking approximate answers as fact

AI approximates responses. In some cases it can be quite good, but in most cases there will be errors. This is best illustrated by:

• Innumeracy: It struggles with accurate calculations.
• Format inconsistency: Often it does not stick to requested formats, like providing a specific number of items.
• Code invention: It suggests or refers to non-existent code or resources, like recommending code libraries that don’t exist.

Misunderstanding how it learns

Training the system is uncommon and costly. Sometimes it seems like it’s learning because it keeps track of conversation contexts or examples, but its learning is limited. You’ll see this when information is out of context or a new conversation starts.

• Unteachability: It will not remember your teachings between conversations or over an extended context.
• Forgetfulness: It often seems to forget its earlier output. It rarely refers back to its past responses and is only aware of recent information.

Best practices: What does an AI testing strategy look like?

Guidance from NIST’s AI Risk Management Framework and the EU AI Act both begin with assessing the foundational risk of the system. This assessment will tell you whether it is legitimate for an AI to be used in this system, and how much scrutiny you should be placing on the system under test.

Risk assessments should take into account the system’s target functionality (for instance, health, lifestyle, or finance) and its target audience, especially if the audience contains children, the elderly, or other vulnerable communities. It should also consider the system’s target operating mode, whether autonomous, autonomous with human review, or true recommendations.

What is the Tricentis approach to AI testing?

We believe that an effective testing strategy should encompass seven key elements:

1. Risk assessment: Begin by assessing the potential risks associated with the AI system. Consider the system’s functionality, its target audience, and the severity of potential failures.
2. Understand the limitations of AI: Be aware of novel failure modes unique to AI, such as lack of logical reasoning, hallucinations, and information synthesis issues, including recognizing the boundaries of its learning capacity.
3. Augment AI skills and training: Educate testing teams on the intricacies of AI, including training methods, data science basics, GenAI, and classical AI. This knowledge is essential for identifying potential issues and understanding the system’s behavior.
4. Plan for Red Team testing: Adopting Red Teaming in QA is crucial for ensuring an AI systems’ safety and security, including correct and ethical functionality. QA teams can conduct tests under various scenarios, including adversarial attacks, to assess the system’s resilience and response to unexpected inputs or actions. They can also test for data privacy, security breaches, and bias in the AI system.
5. Regular review and adaptation: AI systems evolve, and so should testing strategies. Regularly review and update your testing approach to adapt to new developments in AI technology and emerging threats.
6. Compliance and documentation: Ensure that all testing procedures and results are well-documented for compliance purposes, especially in light of the new Executive Order requirements. This includes detailing the AI system’s training data and methodologies.
7. Transparency and communication: Be transparent about the AI’s capabilities and limitations with stakeholders. If your AI system is customer-facing, ensure that users are aware they are interacting with an AI.

These considerations are key in crafting AI testing strategies that align with evolving regulatory standards, but it’s important to remember that as AI technology continuously evolves, so too must our approaches to testing and QA.