016 - Guardrails
The age-old innovation vs safety dilemma.
Editors Note: This blog article is a distillation of an in-person event held in San Francisco on 2023-07-09 facilitated by
. Quotes are paraphrased from the original conversation.
Last summer (a lifetime ago in AI governance!) the Ai Salon held a discussion on “guardrails”.
👉 To jump directly to a list of takeaways and open questions, click here.
Predicting AI’s Evolution and Policy Effects
The development of effective guardrails for AI systems hinges on a fundamental question: how predictable is AI’s evolution and the effects of AI policies? This question strikes at the heart of the debate between safeguarding and empiricism in AI governance.
On one side of the debate, some participants argued that certain aspects of AI evolution, such as scaling laws and the emergence of powerful language models, were predictable based on past trends and established patterns. They suggest that we can extrapolate towards the future and anticipate the market-making and industrial choices of large corporations, and estimate the capabilities (and thus societal impact) of AI systems as a downstream consequence.
However, other participants countered that AI systems are often black boxes, and that emergent capacities can only be materialized and understood through empirical testing. They argue that while some aspects of AI evolution may be predictable, there are also many capabilities that emerge unexpectedly and can only be identified through hands-on experience with the systems.
The reality is that the capability expansion of scaled up AI systems is sometimes predictable and sometimes not. Even though some researchers argue that the surprises may be a mirage, a product of overly course measurement, the consequence for humans is thus far that we occasionally can anticipate model behaviors and occasionally can’t. This unpredictability poses significant challenges for policymakers seeking to develop effective governance frameworks, as they must balance the need for safeguards with the desire to enable innovation and realize the potential benefits of AI.
Balancing Innovation and Regulation
The unpredictability of AI's evolution raises important questions about how guardrails should be imposed, particularly when it comes to difficult-to-change regulations. One perspective is that too much regulation early on could stifle innovation, as the full potential and trajectory of AI development are not yet known. This view holds that a more empirical approach, allowing for experimentation and iteration, is necessary to uncover the true capabilities and limitations of AI systems, and that regulation and standards should only be imposed after best practices have actually been developed, empirically. This perspective is particularly salient for AI, as it is such a new technology, with immense potential benefit.
However, the risks and uncertainties associated with AI systems cannot be ignored, and appropriate safeguards and oversight may be necessary to mitigate potential harms. Essentially, uncertainty around AI's impact in the short and long term are immense, and people differ in how they respond to that uncertainty.
Participants in the discussion grappled with this tension between innovation and regulation, recognizing that finding the right balance is a critical challenge for AI governance. Some argued that the precautionary principle should be applied, with strong regulations and safeguards put in place even before the full impacts of AI systems are known. Others countered that this approach could lead to a scenario where only bad actors have access to the most powerful AI capabilities, as responsible developers are hindered by restrictive regulations.
One potential way forward that was discussed is the use of regulatory sandboxes and other mechanisms for controlled experimentation and testing. By creating spaces where AI systems can be developed and deployed in a monitored and controlled environment, it may be possible to strike a balance between enabling innovation and ensuring safety and accountability.
Transparency and Responsibility in AI Systems
Independent of how participants saw regulation, they agreed that transparency was critical. Transparency enables a more thriving open-source ecosystem of innovation, encourages public sentiment to drive more responsible and safe AI use, and is the necessary foundation of government oversight.
As AI systems become more complex and integrated into various aspects of society, ensuring that they are transparent enough to enable accountability while also protecting intellectual property and commercial interests is a significant challenge. This is particularly important given increasingly complicated dependencies between different AI systems. Foundation models are wrapped in systems with various safety measures, potentially fine-tuned, and then prompted to enable final value in a particular use case. One participant highlighted this complexity, connecting the issue with accountability:
…responsibility is a big question here, and identifying the different components in the value chain. Deciding who takes responsibility is really important for knowing what guardrails we can set, because at the end of the day, if someone violates one of those guardrails, we need to know who's responsible.
One potential solution that was discussed is the use of open source AI models. By making the underlying code and architecture of AI systems publicly available, it may be easier to identify the source of problems and assign responsibility accordingly. However, this approach also raises concerns about the potential for misuse and the difficulty of protecting commercial interests. Some key challenges that open source AI models may face include:
Ensuring that the open source code is properly maintained and updated to address emerging security vulnerabilities and other issues.
Balancing the need for transparency with the desire to protect intellectual property and maintain competitive advantages.
Developing effective mechanisms for attributing responsibility and liability when issues arise with open source AI systems.
Despite these challenges, many participants agreed that transparency was necessary to enable an accountable ecosystem. As one participant noted:
Transparency enables watchdog groups, government agencies or academia to participate. One goal could be taking difficult transparent reports and making them understandable for consumers. I want the information to exist to allow that ecosystem to develop rather than one company doing everything.
Diversity and Inclusivity in AI Governance
The discussion also highlighted the importance of diversity and inclusivity in AI governance conversations. Participants noted that current discussions around AI governance are often dominated by perspectives from the United States and Europe, potentially excluding the priorities and concerns of other regions and cultures.
This lack of diversity in AI governance conversations is particularly concerning given the global nature of AI development and deployment. The benefit/risk calculus and thus the response to the innovation/regulation conversation and which guardrails to create is highly dependent on a nation's context in the global order and the challenges they face domestically. For example, countries that are more vulnerable to the impacts of climate change may have different priorities when it comes to balancing the potential benefits and risks of AI development. Similarly, countries with less developed economies may be more willing to take risks in order to accelerate innovation and catch up with more advanced nations. Summing up the general policy conversation, one participation noted:
…guardrails conversation is very US & EU centric. Bigger question is which countries excluded from narrative?
To address this challenge, participants emphasized the need to actively seek out and include diverse perspectives in AI governance conversations. This includes not only geographic and cultural diversity, but also diversity in terms of expertise and stakeholder groups, such as developers, policymakers, civil society organizations, and affected communities. Some specific steps that could be taken to promote diversity and inclusivity in AI governance include:
Actively seeking out and inviting participants from underrepresented regions and communities to join AI governance conversations and decision-making processes.
Providing resources and support to enable participation from individuals and organizations with limited financial or technical means.
Developing AI governance frameworks that are adaptable to different cultural and regional contexts, rather than imposing a one-size-fits-all approach.
By promoting diversity and inclusivity in AI governance conversations, we can help ensure that the benefits and risks of AI development are more evenly distributed and that the priorities and concerns of all stakeholders are taken into account.
Conclusion
AI governance is not a one-time problem to be solved, but an ongoing process of negotiation and collaboration that will be a central challenge for humanity in the future. The development of effective guardrails for AI systems is a complex and multifaceted challenge that requires grappling with fundamental questions about the predictability of AI evolution, the transparency and responsibility of AI systems, the balance between innovation and regulation, and the importance of diversity and inclusivity in AI governance conversations.
While there are no easy answers to these questions, the discussion at the Ai Salon highlighted the importance of ongoing dialogue and collaboration among diverse stakeholders to navigate these challenges and ensure the responsible development and deployment of AI systems.
Notes from the conversation
Transparency in AI systems is crucial for assigning responsibility and accountability.
The value chain of AI systems is complex and not well-defined, making it difficult to apportion risk and liability.
Clinical trials could serve as a model for testing AI systems in a precautionary regime.
The pace of AI development may outstrip the ability of regulators to keep up with new capabilities and risks.
Diversity of thought and representation is essential when designing AI guardrails and risk taxonomies.
AI systems could be used to improve user interfaces and make complex systems more accessible.
The tech culture may not prioritize the "polluter pays" principle, leading to a lack of responsibility for downstream impacts.
Open source AI models may not always be at the frontier of development, but they can still pose risks.
AI systems could be used to support and enhance human decision-making rather than fully automating processes.
Balancing innovation and regulation is a key challenge in AI governance.
Reputational risk and attributional transparency could incentivize responsible AI development.
The EU AI Act is attempting to define high-risk AI applications and establish transparency requirements.
Emergent capabilities of AI systems can be difficult to predict and may require empirical testing to uncover.
The exponential growth in AI capabilities may make it difficult to control access to powerful models.
AI governance discussions are often US and EU-centric, potentially excluding perspectives from other countries and cultures.
Transparency alone may not be sufficient if the information is too complex for the general public to understand and act upon.
Defining clear risk taxonomies and categories is an ongoing challenge in AI governance.
The potential for AI systems to be used for malicious purposes by bad actors is a significant concern.
There is a lack of consensus within the AI research community about the level of understanding and potential risks of large language models.
Collaboration between AI developers, policymakers, and other stakeholders is necessary to establish effective guardrails and governance mechanisms.
Questions
How can we effectively assign responsibility and liability in complex AI value chains?
Can clinical trial-like processes be adapted for testing AI systems, given the pace of development?
How do we balance the need for transparency with concerns around intellectual property and commercial interests?
What mechanisms can be put in place to ensure that AI governance keeps pace with technological advancements?
How can we ensure that diverse perspectives, particularly from underrepresented countries and cultures, are included in AI governance discussions?
What is the right balance between using AI for decision support versus full automation?
How can we encourage a culture of responsibility and accountability within the tech industry?
What safeguards are needed to mitigate the risks posed by open source AI models?
How do we define and categorize the various risks associated with AI systems?
What level of transparency is necessary for the general public to make informed decisions about AI?
How can we effectively incentivize responsible AI development through mechanisms like reputational risk?
What are the potential unintended consequences of AI regulations like the EU AI Act?
How can we anticipate and prepare for emergent capabilities of AI systems that are difficult to predict?
What measures can be taken to prevent the misuse of powerful AI models by malicious actors?
How do we resolve disagreements within the AI research community about the level of understanding and potential risks of AI systems?
What forms of collaboration between stakeholders are most effective for establishing AI governance frameworks?
How can we ensure that AI governance mechanisms are adaptable and responsive to changing circumstances?
What is the role of governments, international organizations, and industry in shaping AI governance?
How do we balance the potential benefits of AI with the need to mitigate risks and negative impacts?
What are the long-term implications of AI development for society, and how can we prepare for them?