021 - Trustworthy AI Futures: 2024 Reflections
Looking towards 2025
Editors Note: This blog article is an AI-supported distillation of an in-person event held in San Francisco on 2024-11-17 facilitated by
and. It does not reflect the views of the facilitators, writer, or the Ai Salon - it is meant to capture the conversations at the event. Quotes are paraphrased from the original conversation.
As artificial intelligence continues its rapid evolution, a gathering of AI governance practitioners, researchers, and policy experts met in San Francisco to reflect on a transformative year in AI development and oversight. The conversation revealed a complex landscape where institutional progress collides with technical uncertainty, where open source ideals meet national security concerns, and where the gap between Silicon Valley's vision and policy realities continues to widen. Yet amidst these tensions, 2024 has seen unprecedented cooperation in establishing governance frameworks and a maturing understanding of AI's possibilities and limitations.
Top line takeaways
The AI governance landscape saw unprecedented institutional development in 2024, from AI Safety Institutes to international agreements, but these nascent structures face serious tests in 2025 amid political uncertainty and implementation challenges
Technical progress in AI appears to be reaching an inflection point, with hints of diminishing returns in traditional scaling approaches spurring new directions in interpretability, reasoning capabilities, and architecture design
We're witnessing a fundamental tension between Silicon Valley's open source, innovation-first culture and growing concerns about AI safety and national security - a tension that may come to a head in 2025 as capabilities advance and geopolitical pressures mount
👉 Jump to a longer list of takeaways and open questions
2024: A Year in Review
The participants' reflections on 2024's highs and lows painted a nuanced picture of the field's evolution:
Policy and Institutional Progress
The formation and progress of the various AI Safety Institutes was seen as a major success. Their successful creation, hiring, and convening was seen as an unexpected positive. The week after our discussion the US AISI was hosting a convening Safety Institutes, which was also seen positively. This optimism was tempered by Trump’s election, which was generally seen as a negative development for AI safety at an institutional level.
The adoption of the UN Global Digital Compact by over 190 countries was highlighted as a significant achievement, though participants noted the challenges ahead in implementation. The formation of an "IPCC-like body" for AI was seen as a promising step toward evidence-based understanding of AI safety.
While the passage of the EU AI Act was also heralded as a great success, participants also highlighted the hopeful shift in the EU towards AI innovation in addition to regulatory courage. Informed by Draghi’s report on European competetiveness, participants highlighted the EU's strategic pivot toward competitiveness in AI development as a significant positive shift.
Silicon Valley's Safety Schism: The SB 1047 Debate
One of the most contentious developments in 2024 was the debate around California's SB 1047, which highlighted deep divisions within the AI community about regulation and development. Within Silicon Valley, resistance to the bill was fierce, particularly among technical communities excited about open source developments and concerned about constraints on innovation.
The response from academic leadership was particularly noteworthy. Stanford's Fei-Fei Li's opposition to the bill, framed through appeals to "evidence-based risk perspectives" and "science-based" approaches, exemplified a concerning trend. As one participant noted, this rhetoric appeared to create a false dichotomy between immediate, measurable risks and longer-term societal concerns, using academic authority to minimize certain categories of risk.
While the bill was ultimately vetoed, participants saw the debate as revealing about the state of AI governance discourse. The intensity of opposition from certain technical communities, coupled with the alignment of academic leaders with venture capital interests, suggested that bridging the gap between Silicon Valley's build-first culture and thoughtful regulation remains a crucial challenge heading into 2025.
Breakthroughs in interpretability
Researchers in the group pointed to Anthropic's innovations in representation engineering (also pushed forward by the Center for AI Safety) as a major high point, suggesting potential pathways toward more controllable AI systems.
Industry Evolution
The growth of AI governance startups and increasing investment in safety-focused initiatives in enterprise was cited as a positive trend.
Open source developments, particularly Meta’s release of Llama 405B, were seen as immensely consequential though the participants were mixed in their response to how positive a direction this was. Participants also debated where open source will likely move with some citing recent evidence that China was building models for military use on Meta’s Llama as a reason the US may place restrictions on future open source deployment.
There was universal pessimism in how OpenAI had changed over the last year, particularly the loss of many AI safety leaders and turn towards for-profit.
2025: Looking Forward
The Hardware Inflection Point
Several key developments in 2024 point to hardware becoming a crucial battleground in 2025. With evidence that China is leveraging open source models for military applications, participants anticipated increased U.S. export controls on both AI chips and potentially model weights. The conversation suggested that while software restrictions have historically proved difficult to enforce, hardware-level controls might become an important policy lever for shaping AI development.
Recent attempts to regulate chip exports to China were seen as a preview of broader controls to come. However, participants noted that previous export controls have had mixed success, with one noting that "China can find a loophole somehow like through other countries." This skepticism was balanced against the unique characteristics of the AI chip market, where manufacturing capabilities are more concentrated and thus potentially more controllable than software distribution.
The discussion turned to innovative proposals for building safety mechanisms directly into hardware. Though discussed in communities focused on existential risk, most participants had not heard of such proposals and raised complex questions about implementation and effectiveness. The historical parallel to trusted computing initiatives of the early 2000s suggested both the potential and pitfalls of hardware-level controls.
The Open Source Question
Despite current momentum behind open source AI development, participants raised the possibility the future might see this trajectory challenged. The combination of national security concerns and the increasing capabilities of open source models could lead to new restrictions. As one participant noted,
"open source may continue until it reaches some threshold that the United States is like, no, no more."
The discussion highlighted how open source development is becoming increasingly entangled with questions of digital sovereignty and national security, bringing to mind the Tik Tok ban passed in 2024 set to go into effect in 2025
While open source and regulation are sometimes pitted against each other, several participants pushed back against this notion. They argued that effective regulation might actually be easier in an open source context where systems are more transparent and accessible to scrutiny. The challenge for 2025 will be developing frameworks that can maintain the benefits of open development while addressing legitimate security concerns.
The conversation also touched on how the open source community might evolve in response to potential restrictions. Some suggested that rather than complete restrictions, we might see a shift toward "small tailored or bespoke open source models" that are more controlled and specific in their applications. This could represent a middle ground between completely open and closed development.
Technical Trajectories
There are some hints that GPT-5 is showing diminished returns. Though rumor at this point, a shift away from the scaling laws we have come to expect would be significant. However, improvements due to pre-training are just one component of enhanced intelligence. Even if GPT-5 and other large language models show diminishing returns, participants expected continued progress through other avenues. Three particular areas of focus emerged:
Improved reasoning capabilities through computed inference
Development of smaller, more efficient models
Exploration beyond transformer architectures toward more explainable approaches
The discussion highlighted how innovation might shift from raw scaling to more sophisticated approaches to reasoning and inference. One participant described recent advances in "compute-time inference" like OpenAI o1 and developments like chain-of-thought prompting as indicating a new direction for progress that doesn't rely solely on larger models. As models become more efficient, more sophisticated inference-time strategies can be pursued like the Monte Carlo tree search employed by AlphaZero.
There was particular interest in the potential for smaller, more efficient models that could maintain current capabilities while requiring less computational resources. This "distillation" of capabilities could lead to more widespread deployment and integration of AI systems, even if the frontier of model size plateaus.
The limitations of current transformer architectures were also discussed, with several participants suggesting 2025 might see increased focus on developing fundamentally new approaches, particularly in support of safety and interpretability. This harkens back to a previous trend where model architectures were being developed to be interpretable by design. As one participant noted,
"we should move beyond the transformer model... Because now it's still unexplainable AI basically."
The Enterprise and Consumer Value Reality Check
2025 may be a crucial year for enterprise AI adoption. If concrete value and practical benefits don’t materialized there could be a decreased investment and increased appetite for regulation as risks loom larger. As one participant observed,
"if we don't as consumers and enterprises see tangible benefits... there's the possibility for consumers to be like, you know, ChatGPT was kind of fun, but I'm kind of souring on it."
The discussion revealed a significant gap between how enterprises evaluate AI systems and how the technical community approaches them. While technical researchers focus on benchmarks and formal evaluation metrics, enterprises are primarily concerned with practical utility in their specific context. Multiple participants brought up that their conversations with enterprises about AI adoption and governance never end up touching on benchmarks.
This disconnect could become more pronounced in 2025 as enterprises move from experimentation to deployment at scale. The conversation suggested that bridging this gap might require reframing safety considerations in terms that directly connect to business outcomes and specific use cases rather than seeking better safety benchmarks.
There was also discussion of how enterprise adoption might be affected by the development of more specialized, domain-specific models. Rather than relying on general-purpose AI systems, enterprises might increasingly seek out or develop models tailored to their particular needs and constraints.
Safety Institute Impact
The various AI Safety Institutes will face their first real test in 2025. Their ability to effectively evaluate models and establish meaningful safety standards while maintaining international cooperation will be crucial. However, political transitions, particularly in the U.S., create uncertainty about their future impact and authority.
Participants discussed how these institutes might need to balance competing demands: maintaining rigorous standards while not impeding innovation, fostering international cooperation while addressing national security concerns, and developing practical guidelines while engaging with longer-term safety considerations.
The success of these institutes could significantly influence the broader governance landscape. As one participant noted, they represent an attempt to develop "bureaucrats that are also technical, capable of understanding some of these very frontier foundation models to a level and extent where we can make much more informed policy."
The conversation also touched on how these institutes might help bridge the gap between technical and policy communities, potentially serving as translators and mediators between different stakeholders in the AI ecosystem.
New Data Frontiers
Rather than hitting a data ceiling, participants anticipated exploration of new data domains in 2025, including:
Enterprise data currently siloed within organizations
Multimodal and spatial data
Sensor data from an expanding Internet of Things
Novel forms of data from biological and environmental systems
The discussion highlighted how current assumptions about data exhaustion might be too narrow. One participant suggested that while traditional language data might be reaching limits, there are so many other types, including spatial data (currently being pursued by World Labs, a startup co-founded by Fei-Fei Li).
Some participants even suggested exploring entirely new forms of intelligence and data, such as "adapting or feeding in intelligence that exists within other species." This could include understanding how different organisms process information and interact with their environment.
The conversation also touched on how advances in sensor technology and the Internet of Things might provide entirely new categories of training data. As one participant noted, "we're at the very beginning of software eating the world," using a striking metaphor to illustrate the point:
"if you're thinking about the phrase 'software eats the world', let's imagine we're in a multi-course dinner - where do you think we are in the dinner?The answer: "We're at the amuse-bouche. We're not even at the beginning."
Looking around the room, they noted that even basic objects aren't yet "smart" - suggesting that the current limitations of language model training data might be overcome through these new frontiers as our world becomes increasingly instrumented and connected.
Looking Ahead
As we enter 2025, the AI landscape stands at several crucial inflection points. While institutional frameworks for AI governance have developed with surprising speed, their resilience will be tested by political transitions and practical implementation challenges. Technical progress may shift from raw scaling to more nuanced approaches - from novel architectures to sophisticated inference strategies. Yet perhaps the most profound changes lie not in the models themselves but in how they interact with our world. As sensors proliferate and new data frontiers emerge, we are reminded that we are at the very beginning of this AI revolution. Indeed, one may say we are still at the very beginning of the Information Age.
The challenge ahead is not just technical or regulatory, but finding ways to ensure this integration creates tangible value while addressing legitimate safety concerns. The conversations in 2024 suggest that bridging the gap between Silicon Valley's technical optimism and thoughtful governance will be crucial to realizing AI's potential while mitigating its risks.
Notes from the conversation
There's a growing tension between open source AI development and national security concerns, particularly regarding China's access to and use of open source models
The AI governance landscape has matured significantly in 2024, with unprecedented international cooperation and institutional development
A divide exists between Silicon Valley's technical community and policy makers regarding regulation approaches
There's emerging skepticism about whether current LLM architectures will continue scaling or if we're approaching diminishing returns
The conversation around AI safety has evolved from theoretical to practical as real-world applications emerge
Many enterprises focus more on practical utility than formal safety benchmarks when adopting AI
There's a shift from discussing what AI can do to what it should not do
The funding landscape for AI governance and safety startups is growing despite broader tech industry challenges
Multi-modal and spatial data represent untapped potential beyond current language model limitations
There's increasing recognition of the need for interdisciplinary perspectives in AI development and governance
The relationship between open source development and effective regulation isn't necessarily antagonistic
Consumer expectations of AI capabilities are rising faster than actual technological progress
Current regulatory frameworks may be inadequate for addressing agentic AI systems
There's growing attention to the role of hardware (chips) in AI safety and control
The distinction between hypothetical and evidence-based risks is becoming a key point of debate
Traditional benchmarks may be insufficient for enterprise-specific AI applications
The AI safety community is developing more sophisticated approaches to testing and validation
Environmental and energy constraints may become limiting factors in AI scaling
There's increasing focus on developing smaller, more efficient models rather than just scaling up
The gap between technical capability and governance readiness remains a significant concern
Questions
How can we balance innovation with responsible development in open source AI?
What metrics should we use to evaluate AI system safety beyond traditional benchmarks?
How will diminishing returns in current architectures affect the trajectory of AI development?
Can hardware-level controls effectively contribute to AI safety?
How do we bridge the gap between technical and policy communities?
What role should private companies play in setting AI safety standards?
How can we effectively regulate agentic AI systems?
What are the appropriate boundaries for national security restrictions on AI research?
How do we balance transparency with intellectual property protection?
Can we develop meaningful safety standards without stifling innovation?
What role should international bodies play in AI governance?
How do we address the increasing energy demands of AI systems?
What are the implications of AI systems becoming more agentic?
How do we maintain human oversight as AI systems become more autonomous?
What are the long-term implications of current architectural choices in AI development?
How do we ensure AI safety measures scale with capability advances?
What role should public opinion play in shaping AI development?
How do we balance immediate practical benefits with long-term safety concerns?
What are the appropriate metrics for measuring AI progress?
How do we ensure equitable access to AI capabilities while maintaining safety?