LLM-enabled viruses are fiction, for now. Don't let them become fact.
Vibe coded viruses? No. LLM-assisted synthesis of a pandemic virus? Uncomfortably close
Yesterday, LLM-generated viruses headlined the New York Times Science section.
Two weeks ago, LLM-assisted viral creation was front-page on WSJ.com.
Is the threat real, or just AI hype?
In my view, LLM-assisted threats warrant concern. The SARS2 pandemic was devastating. A replay of 2020-2021 would be clamatous. I think, in opposition to AI hype, there’s often a reflexive dismissal of this quite real threat. Synthesis of viruses is more tractable than many appreciate. LLMs could soon make it easier. We should make very modest investments to secure our biotechnology infrastructure against synthetic viral PPP or bioweapon acquisition.
I am not a virologist. But, I am a biotech founder with extensive hands-on experience in molecular biology, protein engineering, chemical synthesis, and drug discovery and development. To my surprise, the present LLM-enabled bioweapon discussion appears dominated by perspectives that largely lack the relevant laboratory experience. Given the bottleneck to synthetic viral creation is implementation in a molecular biology laboratory, that seems important. Therefore, I think this conversation could thoughts from a different perspective could be quite informative.
Vibe Coding Viruses?
Let me be clear: we are not yet at the point where anyone can vibe-code a novel, stealthy high-risk Potential Pandemic Pathogen (PPP).1 The headline AI-assisted synthetic viruses are innocuous phages, and in fact quite similar by sequence to the parental virus. In the future, novel high-risk pathogens might be a prompt away. For now, and in my estimation for the near future, it’s not. Perhaps it’s never possible.
But, LLM assisted synthesis of a known PPP is emergent. The baseline risk of a pandemic from a synthetic virus is already high. Many of the discussed barriers are weaker than appreciated. And, unlike other prospective bioterror pathogens such as anthrax, scientists and nefarious actors can synthesize the entire genome of a PPP virus and “boot-up” the pathogen in cell culture.2 These procedures typically require substantial experience, but LLM-assistance might soon enable a single skilled technician, perhaps an early-stage PhD student, to acquire a synthetic PPP virus. The synthetic tractability and transmissibility should make viral synthesis a major terrorism concern.
Thankfully, some key physical bottlenecks - sequence access, DNA synthesis screening & logging - can be hardened at virtually no cost with effectively no impact to existing work. Yet, these simple controls would significantly mitigate the risk of synthetic virus attacks independent of AI progress or open LLM proliferation. Long-term, we should establish the legal and institutional framework to support Know-Your-Customer (KYC) infrastructure across all key viral creation inputs.
Synthetic viruses present a unique threat
Why should we care about viral threats?3
Viral threats self-amplify, unlike nuclear or chemical weapons
Viral threats can transmit human-to-human; one introduction can transmit throughout the world
Modern DNA synthesis and viral reverse genetics can create high-risk viruses
The viral assembly and start-up manuals are public
The cost and skill required to produce synthetic viruses is modest and declining
Institutions lack the capability and will to snuff out emerging pandemic threats
LLMs won’t create magic, novel viruses from single prompts. But, continued improvements in LLM-assisted DNA design, DNA synthesis, and recombinant DNA technology, make the remaining bottlenecks more critical. Therefore, we must take immediate action by hardening our biotechnology infrastructure to block construction of high-risk viruses. We can mitigate the majority of the emerging synthetic viral threats with nearly no impact to nearly all biotechnology work.
The Minimum Viable Path
We should focus our effort on avoiding the accidental or intentional release of high-risk viral pathogens. A useful concept is the Minimum Viable Path (“MVP”).
What is the cheapest, fastest, lowest-lift way to create a high-risk virus from sequence alone? This path is already shorter than many people think.4 Public discussion tends to oversell most barriers barriers. Large expert teams are not required. The dollars required here are shockingly low. I hesitate to say more because detailing these specific paths publicly creates an info hazard.
Yes, experience with recombinant DNA technology and cell culture are still required. That is a barrier. But, not a big one. The inputs are accessible. I can buy high-quality, synthetic DNA online cheaply. DNA writing costs continue to decline. Multiple commercial vendors offer whole plasmid synthesis service at modest cost. Getting access to cell culture and incubator space is achievable. A lot of start-ups, mine included, source quality equipment cheaply from eBay and lab liquidators. And, I’ve never had anyone ask me why I needed an incubator. Again, there are more examples, but I intentionally avoid linking to specific details here.
Further, some threat vectors have all the infrastructure to execute high-risk viral construction.5 For context, in early 2020, before SARS2 viral isolates could be shared, some academic labs and companies synthesized the whole viral genome, including synthesizing the modular reverse genetic system itself.
Furthermore, many grad students in virology labs have access to the requisite infrastructure. A trainee could run an undisclosed, parallel project without anyone realizing. Many already do. Often, it’s an exciting side-project. But, in some labs, that exciting side project might be an interesting, but poorly scoped, viral mutagenesis experiment that incidentally creates an ePPP. So, if someone who hasn’t set up a lab before - let alone cultured cells - starts waxing about the difficulty of and barriers to viral construction, be skeptical!
One major barrier has been the skill to scope, design, verify, and plan viral creation and weaponization projects. Typically, this is a post-doc or senior PhD student. Often, a lab head (Principal Investigator, or PI) spearheads this effort. And, this is where LLMs will soon have conceptual capabilities comparable to a post-doc or lab head.6
LLM advisors may soon - if not already - help transform a public viral creation protocol into a detailed viral synthesis cookbook. And LLMs may help terrorists acquire the requisite parts! Yes, the information is already out there. That is why we must be concerned. Integrating it, at present, is a significant barrier for a human that may soon evaporate. Additionally, at present, designing, verifying, and engineering viral assembly constructs requires experience. That process could soon be automated by strong LLMs with access to common molecular biology tools.
If LLMs may soon have the capability to design, partition, and verify all synthetic DNA inputs to a complete viral reverse genetic system (easy to error manually), a motivated terrorist with the recombinant DNA and cell culture skills of a second-year PhD may soon be up-skilled to the cusp of synthetic viral genome assembly and viral creation. If open weight and local LLMs develop project planning and design-verification capabilities before we secure key bottlenecks, Pandora’s box has been opened.7
Bottlenecks must become Chokepoints
So what remains? Key bottlenecks.
Access to DNA sequence databases (e.g. GenBank)8
Access to high-risk viral construction methods (e.g. journal articles, protocols)
Synthesis of long and short DNA fragments (e.g. Twist, IDT, Genscript)
Good news: they are cheap and easy to secure.
Right now, anyone can go onto GenBank and download the whole genome sequence of the 1918 pandemic influenza. Curious about how to build that genome? Or how to activate it? Which cell lines to use? It’s all accessible to the public… Including LLMs.
We have more barriers to opening a commercial bank account than accessing whole genome sequences of pandemic viruses
How can we choke off access to high risk information? Nearly overnight we can:
Mandate viral databases remove access to high-risk PPP sequences
Block public access to viral reverse genetics protocols
Move articles describing high-risk reverse genetics to classified journals
High-risk viral sequences and select viral genetics protocols should be treated as highly classified national security material. Going forward, access must be kept on a strict need-to-know basis. No individual (or LLM) needs unfettered access to PPP DNA sequences. For scientists and public health officials working on PPPs, that means basic KYC with gated access. For everyone else, nothing changes.
These interventions are well established. Over time, our banking system has developed significant anti-fraud and anti-money laundering infrastructure. People can still freely transfer funds for legitimate business purposes. Similarly, virtually no work will be impacted by basic access controls. Presently, we have more barriers to opening a commercial checking account than accessing whole genome sequences of pandemic viruses. That has to change.
DNA synthesis is the other key bottleneck. Major AI labs and DNA synthesis providers recognize we need stronger regulation. They argue:
Support for screening does not depend on any particular view of AI; the biosecurity case has been recognized by scientists and governments for decades. Screening is also one of the best understood and least disruptive biosecurity measures available. It asks providers of synthesized DNA and manufacturers of synthesis machines to check synthesis requests for sequences of concern and to verify customer legitimacy before shipping orders. Providers should also record synthesis orders and sequence data to support legitimate biosecurity investigations, so that any threat that might evade initial screening can be traced back to its source — including when individual sequences would not raise concern in isolation. Awareness of traceability itself deters misuse. [emphasis added]
Right now, the US lacks comprehensive, statutory pathogen screening of synthetic DNA. Screening is largely voluntary! Responsible companies do basic screening. But, there is no legal obligation for vendors to verify you aren’t ordering PPP constructs. There more, stronger statutory regulations on access to Sudafed - yes, the nasal decongestant - than PPP/ePPP DNA constructs.9 This is a major gap.
To mitigate PPP acquisition through synthetic DNA, we must soon:
Mandate KYC for all DNA synthesis vendors - there can’t be an order shipped without verification of a legitimate business or institution
Screen & log sequences - basic checks of sequences against a federal database of pathogen sequences10
Centralized DNA synthesis database - all providers should report all orders for each client. This is not mentioned in the letter, but it is essential for tracking clandestine creation of reverse genetic systems across vendors
Together, these controls will harden bottlenecks into chokepoints. Universal screening, logging, and KYC will block acquisition of the key DNA inputs for nearly all existing high-risk PPPs. Removing public access to high-risk viral sequences blocks terrorists and LLMs from accessing known PPPs. Further, the KYC on-ramp rate limits rogue actors (LLM assisted or not) from ordering inputs to create high-risk pathogen DNA sequences. Screening and logging verify the absence of pathogen content and deter nefarious actors. Alone, these two interventions can snuff out most attempts at bioweapon acquisition. But, sophisticated rogue actors can partition orders across vendors.11 With LLM assistance it becomes trivial. Therefore, we need a centralized database to log and monitor on the back-end to ensure that no institution or individual is systematically evading first and second level controls. This second layer screening will further reinforce security across vendors deterring acquisition.
Secure physical infrastructure
In pandemic prevention, basic security is the low hanging fruit. PPP sequences and high-risk publications can be removed nearly overnight. Similarly, KYC, screening, and logging DNA synthesis orders can be operational within a month. Within a quarter, we can establish a centralized DNA synthesis database and create KYC on-ramps for high-risk viral sequences and reverse genetics protocols. Nearly all routine research would proceed largely unaffected.
Longer term, we need to set up stronger institutional barriers to high-risk viral pathogen creation and dissemination. Some suggestions I favor:
Mandate liability insurance for entities pursuing work with high-risk viruses
Institute a mandatory licensing regime for DNA synthesis providers, technology
License benchtop DNA synthesizers (often not economical)
KYC controls on key nucleic acid synthesis reagents
Prohibit public and criminalize private funding of ePPP GoF research
Legislate criminal liability for mishandling high risk PPP sequences
Centralize testing of PPPs and prophylaxis away from cities (a la Plum Island)
Establish IAEA-like agency to harmonize DNA synthesis regulation globally
Develop rapid (<1 min) viral/pathogen screening at borders
While LLMs have gotten a lot of attention, filters on LLM inputs/outputs are low yield and fragile. Legislating LLM filters is worth considering. But, if open weight models become capable and dispersed, then any LLM-level intervention is moot.
In contrast, physical bottlenecks are robust independent of LLM capabilities. The AI labs and DNA synthesis companies recognize the opportunity to turn these bottlenecks into hardened chokepoints. In chemical synthesis, many common reagents used in chemical synthesis (e.g. piperidine for fentanyl, methylamine in methamphetamine) are restricted, tracked, and controlled. This regulation is much more restrictive than my proposed KYC access and screening regime for high-risk viral sequences. And, yet, chemical innovation continues. Not every lab activity needs to be licensed, tracked, and reported. We could secure the PPP acquisition bottlenecks points without most scientists noticing any changes to their work. Some scientists pursuing synthesis and mutagenesis of high-risk viruses may not like the idea controls slowing their pace of work. Some of these labs continue to search for high-risk PPPs at BSL2/BSL2+ (common in labs) instead of BSL3+ largely because it is faster. The speed-safety tradeoff is real. However, I believe the safety of the global public should be prioritized over easier access to PPP viruses.
If anyone builds it, many may die
Some people will argue that these inexpensive and modest access gates are not worth the hassle. In return, I ask everyone to consider what would happen if another novel, highly-transmissible respiratory pathogen appears tomorrow.
Last time, it was devastating large scale lockdowns. Millions of lives lost and trillions of dollars spent to adapt to the microscopic threat. That was for a virus with what appeared to be a ~0.1% case fatality rate. Early in a pandemic, the lethality is unknown. Release of a novel or known high-risk virus could kill 0.01-30% of people infected. We don’t know if the population has partial protection until people start getting infected. The uncertainty mandates a strong, life disruptive response. Arguing about how lethal a PPP/ePPP might be misses the point. If spread is uncontrolled and lethality is uncertain, governments need to take action with limited information.
Even if we have “rapid” defensive responses, production, validation, and distribution will take months, not days. The specific nature of a viral threat matters. These defenses are not on the shelf. We may face a novel threat for which we cannot plan. Uptake is not guaranteed.12 Given the erosion of public trust and institutional competence following the 2020 SARS2 response, the social capacity to execute a strong response to a PPP is weaker now than then. If a new respiratory virus appears tomorrow that kills 0.5% of people infected, will everyone immediately put on an N95 and line up two months later for an unvalidated mRNA shot? We can’t rely solely on eroding barriers, unvalidated countermeasures, and detection to halt a synthetic viral pandemic.
The threat of synthetic viral pandemics, LLM-assisted or not, looms. More capable LLMs will magnify those risks. Securing our biotechnology infrastructure to detect, deter, and prevent pandemic threats is straightforward and inexpensive. If we act now, we can eliminate major paths towards another pandemic - independent of one’s view on AI progress.
In this discussion, PPP is limited to viruses. Of course, non-viral pathogens can be PPPs and can be bioterror threats.
There are other barriers to using bacterial and fungal pathogens. I won’t elaborate publicly. But, both remain worthy of consideration.
This list is non-exhaustive. Some have been left off this list intentionally.
I focus on acquisition of a PPP/ePPP. These are known viruses and derivatives of known viruses. I specifically avoid describing parts of acquisition, threat generation, and release. This is not because they aren’t fair questions. Rather, these are info hazards with answers I do not want shared publicly. From my perspective, if someone is sufficiently motivated and capable to acquire a PPP/ePPP, then generation and release are are more functions of sustained effort than difficulty. To a semi-sophisticated (or lucky) attacker, knowing what to engineer is not a barrier. Nihilists and lone-actors don’t necessarily value self preservation.
We must consider this acquisition path seriously. Insider attacks might be a larger threat than lone-actor terrorists or nihilist groups. Small changes in project oversight (e.g. project based procurement, independent audits) could mitigate much of this risk. For incidental introductions via near-PPP lab work, stronger biosafety regulations and independent oversight are required.
A skeptical non-scientist might ask: what good is a lab head?
Enhanced-PPP (ePPP) Gain of Function (GoF) pioneers Fouchier and Kawaoka didn’t pipet an ePPP into existence. Yet, we credit them for the existence of strains of enhanced H5N1s. They conceived the idea, obtained funding, and oversaw its reduction to practice. We’re not far off from LLMs with access to public-facing databases and literature serving as high-quality advisors for viral construction.
Realistically, we can’t prevent strong local LLMs from accessing information hazards. Sequence camouflage with viral LLMs is also a threat, but it is constrained by the same checkpoints and if that capability does emerge - which it never should because nobody sane should ever train a viral foundation model with high risk pathogens - it’s still subject to physical implementation
Copies of high-risk viral genomes exist outside of databases. There’s still value to gating public repositories to make viral construction more difficult. High-risk sequences should be classified and treated as such - including removal from any AI/LLM training datasets.
What would Alanis Morissette think about a world where the law deters acquisition of effective nasal decongestants but not a virus that engenders their demand?
This, of course, is an area where LLMs could help detect threats from masked or intentionally obscured sequences, including identifying complex threat assembly strategies that can become costly to detect at scale
It’s far more difficult to address acquisition of high-risk PPPs from nation-state level actors. State-sponsored bioweapons programs can exempt themselves from many barriers to acquisition. Therefore, they should remain major concerns. However, state-sponsored bioweapons programs face tradeoffs that nihilist terrorists can ignore (e.g. self-infection, target scope, uncontrolled exposure) on an MVP acquisition route.
There might not be atheists in foxholes, but I have no doubt there will be anti-vaxxers in the next global pandemic