Meet The Entrepreneur Leading The Charge For Formal Methods in Mechanistic Interpretability: Neel Somani
As the artificial intelligence (AI) technology rush continues apace, showing no signs of slowing, the individuals and organizations leading the charge are forced to find answers to critical questions. New technology promises to change every industry it touches, and few technological innovations promise to do so as much as modern AI, generative or otherwise. However, the internal workings of neural networks, large language models, and AI are often poorly understood at the mechanistic level, which naturally raises further questions about its potential, optimal use cases, and future innovations. This problem is the root of a field of research known as mechanistic interpretability (MI).
Mechanistic interpretability may be the key to confirming the safety and comprehensibility of modern artificial intelligence tools, but it’s not well-understood or applied yet. That’s something that entrepreneur, CEO, and AI researcher Neel Somani is looking to change. With a proven track record within machine learning, quantitative research, and blockchain development spaces, Somani has made a name for himself solving complex problems in mathematics and global markets alike. He’s contributed to research in type systems, differential privacy, and scalable machine learning frameworks, and is eagerly applying that expertise toward AI tools and mechanistic interpretability.
“I’m excited about the development of the field of mechanistic interpretability—explaining why the models do what they do,” Somani says. “I am broadly interested in applying a series of techniques called ‘formal methods’. By applying these methods, we can achieve better safety and interpretability guarantees.”
One of the driving goals of this work is to make the underlying behavior of modern AI systems understandable, verifiable, and secure. Learning how to assess AI models and ensure they keep to their objective without compromising other goals or ethical limitations is incredibly important. Through research into mechanistic interpretability via formal methods, Neel Somani hopes to change the way companies approach and leverage AI tools moving forward.
Researcher, Entrepreneur, Mathematician
Neel Somani has been passionate about technology, business, and the intersection of the two for much of his life, a passion that is exemplified in his rigorous educational background. Somani holds a triple major in computer science, mathematics, and business administration from UC Berkeley, an institution to which he contributed a large subset of his previously mentioned research. He credits his work in Professor Dawn Song’s research lab at the university for inspiring his later work with machine learning and security research.
“At UC Berkeley, I contributed to a project called Duet, which is a formal verifier that can automatically prove that code is privacy-preserving (according to a formal definition of privacy called ‘differential privacy’),” he recalls. “That was the first serious project that I worked on in formal methods.”
After graduating, Somani worked for Citadel LLC—an investment firm, hedge fund, and financial services company with over $65 billion in managed assets—as a quantitative researcher in the commodities group for two years, where his analytical expertise was key for solving problems across global markets. Later, seeing opportunity in new markets, he founded a company that became Ethereum’s fastest Layer 2 platform powered by the Solana Virtual Machine, and gained significant Series A funding. Today, he’s taking all of the lessons and experiences he’s gained over the years and aiming them at machine learning verification, mechanistic interpretability, and driving AI technology forward.
“In an ideal world, for high stakes or mission critical ML systems (health care, financial applications), the entire workflow would be formally specified at all levels,” he says. “As machine learning models write a larger percentage of our code, we might expect more out of them, such as requiring that these formal semantics are defined for the purpose of verification.”
Interpretability And Formal Methods
Many of today’s modern AI users have little to no idea how the technology actually works, including major corporate executives and business decision-makers. The full depth of the technology’s complexity makes it incredibly difficult to verify that models are doing what they’re supposed to be doing, without crossing any ethical lines or sabotaging other processes. Mechanistic interpretability as a field exists to build a complete understanding of these models’ functions, both for future innovations and to identify potential risks and safety concerns. Unfortunately, most companies and users either aren’t willing or aren’t capable of addressing this problem.
“Most businesses won’t touch machine learning model internals,” Somani explains. “They might use machine learning models to build agentic workflows. A few businesses at the bleeding-edge might dive into fine tuning and other forms of customization for business-critical workflows. My work really becomes relevant for the foundational model itself.”
Neel Somani’s research seeks to do this through the application of formal methods, a series of techniques and proofs that are well-established in security and privacy fields, but not quite so in machine learning. A driving principle of formal methods is that by applying appropriate mathematical analysis and examination, a given software’s reliability and robustness can be ascertained. Put simply, the idea is that software and hardware design can be as rigorously tested and understood as a design from any other engineering discipline. The result of these formal methods is a formal description and/or specification of the system—a playbook that outlines how things work, why, and what output to expect for a given input.
When put that way, the application—and value—of formal methods in AI and machine learning becomes clear. The technology is new and volatile, and the lack of widespread understanding of its function makes it incredibly difficult to make any guarantees of safety, robustness, and stability. These tools operate with continuous inputs, making tracking reliable outputs (and thus robustness) nearly impossible without formal methods. This is only one of multiple problems that Neel Somani seeks to solve with his research in mechanistic interpretability. The other, larger problem, is one of safety.
“A concrete problem in safety is that many claims are difficult to falsify with current methodologies,” says Somani. “For example, when someone analyzes a model, they might have a guess as to what it’s doing under-the-hood. But there’s no way to really prove or disprove that hypothesis.”
Somani’s research and professional work is meant to provide a defensible and technically grounded North Star for AI safety research groups to build toward, which is something that’ll only grow more necessary as regulatory bodies seek and implement standards for the technology. With regulation an unavoidable inevitability, the path forward for AI and machine learning will demand some reasonable standard that provides certifiable guarantees without crippling the models. Somani believes that mechanistic interpretability research and formal methods may form the backbone of the industry’s future.
You May Also Like:
