Discussion on Artificial Intelligence: Focus on Security, Administration, and Validation between Lex and Roman
In the rapidly evolving world of Artificial Intelligence (AI), concerns about the safety and oversight of AGI development have come to the forefront. Unlike traditional products, AI development currently lacks rigorous oversight in terms of safety, posing unique challenges due to its ability to self-modify, rewrite its code, and interact with the physical world in unpredictable ways.
The complexity of verifying self-improving AI systems is a significant concern, as highlighted by the Stanford AI safety research. This complexity, coupled with the exodus of safety researchers from major AI companies, raises red flags about the level of focus on safety concerns within these organisations.
The current state of software liability, where users agree to terms without understanding the implications, is not inspiring confidence in our ability to manage superintelligent systems. Even with mathematical proofs, which are our most rigorous form of verification, limitations are encountered as proofs become more complex.
Currently, there are no working safety mechanisms in place for AGI. Regulation alone may not solve the issue of bugs in AI systems as compute power becomes more accessible. The definition of AGI has evolved to include the concept of superintelligence, a system superior to all humans in all domains. Reaching 100% certainty in safety mechanisms for AI appears impossible due to the system's capacity to make billions of decisions per second over extended periods.
However, there is a growing recognition of the importance of AI safety. Prediction markets and tech leaders suggest AGI could be achieved by 2026. In response to these concerns and the potential for AGI to social engineer, the US government has outlined a comprehensive AI Action Plan titled Winning the Race: America’s AI Action Plan, released in July 2025.
This plan emphasises several key strategies. Firstly, it focuses on cybersecurity and critical infrastructure protection, developing multi-pronged strategies to protect critical U.S. infrastructure against AI-enabled cyber threats. It also promotes secure-by-design AI systems, ensuring AI technologies used in critical national security and infrastructure contexts comply with rigorous standards.
The plan also addresses AI risk management and governance, updating and expanding frameworks to incorporate AI interpretability, control, and robustness. It encourages international AI diplomacy and security, exporting a “full-stack” American AI technology package to allies to prevent reliance on adversarial foreign AI.
Furthermore, the plan accelerates federal permitting for data centers and builds high-security data centers for defence and intelligence purposes. It establishes an AI-specific incident response ecosystem across government agencies to rapidly address emerging AI threats and vulnerabilities.
These combined technical, governance, and diplomatic strategies reflect a broad understanding of AI safety challenges beyond traditional programming risks, acknowledging AI’s potential roles in social engineering and manipulation. They emphasise coordination, robust design, threat intelligence, and international cooperation.
Despite these efforts, some individuals are actively trying to accelerate AGI timelines, viewing the current pace as too slow. The most pressing concern about AGI is its potential for social engineering, not its direct physical capabilities. As we continue to navigate this exciting and challenging field, it is crucial that we build only what can be genuinely controlled and understood, requiring tech leaders to seriously consider the risks of developing systems beyond human control.
[1] White House Office of Science and Technology Policy. (2025). Winning the Race: America’s AI Action Plan. Retrieved from https://www.whitehouse.gov/ai/ [2] National Institute of Standards and Technology. (2025). AI Risk Management Framework (AI RMF). Retrieved from https://www.nist.gov/ai/ai-risk-management-framework-aimrf [3] Department of Homeland Security. (2025). AI Information Sharing and Analysis Center (AI-ISAC). Retrieved from https://www.dhs.gov/ai-isac [4] Cybersecurity and Infrastructure Security Agency. (2025). AI-specific Incident Response Structures. Retrieved from https://www.cisa.gov/ai-incident-response [5] Federal Permitting Improvement Steering Council. (2025). Accelerating Federal Permitting for Data Centers. Retrieved from https://www.fpisc.gov/data-centers/
Artificial Intelligence (AI) plays a crucial role in education and self-development, particularly in the realm of personal growth, asadvanced AI systems can provide personalized learning experiences and support for individuals seeking self-improvement. However, the safety and oversight of AGI development, including its potential for social engineering and manipulation, remain significant concerns, as highlighted by the US government's AI Action Plan.
The exodus of safety researchers from major AI companies and the complexity of verifying self-improving AI systems contribute to the fears regarding the safety of AI, and it is essential that tech leaders consider these risks when developing systems beyond human control. This balancing act between embracing AI's potential benefits and being mindful of its risks is crucial for the future of both technology and personal growth.
