Links

Note: This is someone else's document, not mine.

Stephen Casper

This doc might help with the following:

• Helping you know what I'm thinking about.
• If you're working on similar things, helping you coordinate with me so we don't scoop each other.
• Helping you know what I might be able to collaborate on.
• Potentially getting you interested in these things too. I am making no attempt to "dibs" any of these ideas.
• Getting errors or gaps in my thinking corrected.

Choose your own adventure and feel free to forward it to others. Comments are welcome.

---

Making unlearning better at conferring tamper resistance

(see this doc and this paper for more details)

• The goal of this project would be to make machine unlearning algorithms that are good at making models resist re-learning via fine-tuning. The SOTA is for these methods to resist a few hundred steps of fine-tuning. I'd love to do 5 or 10x better if possible. I would see this as a first step toward developing models that can resist being bio/cyber/chem-hazardous for a billion fine-tuning tokens or more.
• Things that I think would make unlearning algorithms better:
  • Running them for longer, interspersed with recovery training
  • Fine-tuning the full model, not a LoRA
  • Making better "forget" datasets that trim/compress away irrelevant things
  • Using techniques that promote "neuroplasticity."
  • Improving the forget losses so that they can't be optimized for by approximating simple transformations (unlike essentially every unlearning algorithm introduced to date)
  • Adversarial training under diverse mechanistic tampering threats

Effectively implanting harmless synthetic beliefs into models

(see this doc and this paper for more details)

• The goal of this project would be to introduce an effective way of implanting incorrect, benign information into models about harmful domains.
• The main idea that I envision would be to (1) create/obtain a "textbook" on a topic, (2) rewrite it to change facts in a way that is consistent and coherent, and (3) use that textbook to rewrite a "forget" dataset to create a dual "retain" dataset of incorrect information. Then we could use an unlearning algorithm with those forget/retain datasets.
• I think this is so important because right now, a small amount of anti-refusal fine-tuning + giving an AI agent internet access seems to be a very effective way of defeating current state-of-the-art safeguards for open-weight AI models. I think that implanting harmless synthetic beliefs into models is the best chance we have at patching this very large hole in current methods for making open models safer.

Megapaper on best technical practices for mitigating open-weight AI model risks

• This paper is not ready to be written yet, but it could be written in a year or two after some more research has been done on pre-training, post-training, and synthetic document training methods for making open weight models safer and harder to misuse.
• Once the toolkit gets decent for open model safety, it will be very useful to write a mega paper with a lot of authors on that toolkit in a way that is accessible to lawyers and policymakers.

Legal and ethical methods to study image/video models in the wild for NCII/CSAM risks

• When I talk to policy people about managing risks from non-consensual AI deepfakes, they often bring up the challenge of legally/ethically evaluating whether a model may have been trained on CSAM or trained to specialize in generating CSAM. This project would try to introduce a method for this.
• Idea: take ethically obtained (or generated) datasets of porn depicting older adults (e.g., 30s) vs younger adults (e.g., early 20s). Then use those datasets to calculate a model's loss (or the output of membership inference algorithms) on each dataset. The hypothesis to test is that the loss difference between these two datasets can be a proxy for how "good" a model is at making CSAM.
• Update: this paper may have partially scooped this topic a bit, but there might be some opportunities for more work in the space.

The AI people, politics, and product positivity index

• AI is a social technology, and AI systems are becoming critical media in the marketplace of ideas. Ideally, AI developers would always work to make their AI systems neutral, but they don't. This index could track this.
• The idea would be to curate a list of models and a list of public figures, political ideologies, and companies/products to evaluate how positively vs harshly AI systems discuss them.
• After the initial paper, it could be fun and easy to set up a website and a backend for it that auto-evaluates the AI systems again, like once every month. It could be really fun to track results over time, especially over election years.

Nonconsensual AI deepfakes as a case study in how to fail at AI risk management

• I think that nonconsensual deepfakes are the first domain in which we have AI harms that are truly widespread and truly terrible. And I think that the case law and accountability frameworks that we develop in response to it will set the stage for handling other AI harms in the future.
• The proliferation of nonconsensual AI deepfake technology suggests that even if early developers of a technology make their versions safe (e.g., DALLE-2 from OpenAI), the harmful capabilities will eventually proliferate via the open-weight ecosystem or irresponsible closed-weight model developers. The lessons to take away seem to be that:
  • We almost certainly won't make AI safe by making safe AI.
  • AI safety is not a model property; it's an ecosystem property.
  • Harmful AI capabilities proliferate reliably, sometimes via open models, and sometimes via companies that don't care about risks
  • Any agenda for AI safety that does not account for proliferation and open models should be considered unserious.
  • Instead of hoping for safety guarantees, we should adopt a harm mitigation paradigm. Like we already do with CSAM, digital piracy, crime, etc.
• Context: YouTube video
• More context: Google Drive
• Even more context: Google Doc

You can't make AI safe by making safe AI

• This could be a position paper that could talk about how making safe AI systems is an empirically overwhelmingly ineffective way of making AI safe overall.
• It could engage a lot with deepfakes (see above) as an example of how safe systems like DALLE, Sora, and Veo haven't made the image/video ecosystem safe overall.
• It could emphasize the ecosystemic nature of the AI safety problem, critique the approach of major AI companies, and talk about how safety research is much less valuable (and potentially safetywashing moat-building BS) if it stays under wraps without being able to influence the broader understanding of best practices.
• This project could probably be merged with the above one. The project idea above could be turned into a section of this one.

Policy levers for mitigating AI-enabled terrorism risks

• I think that a paper like this could be cool. Its research questions could revolve around what current evidence says about the types of developers, choices, models, safeguards, deployment strategies etc. are most likely to be implicated in AI terrorism.

Technocrats always win: against pluralistic alignment and pluralism washing in AI

• The main thesis of this paper is related to this one. But a ton of stuff to write about has happened in the last 6 years.
• The idea would be to write a position paper about how pluralism isn't a property of model behaviors and that it can only be a property of how a system is created and integrated into society. It would warn against pluralism washing while also being clear that it is useful and important for AI developers to try to balance competing human viewpoints when they develop systems.

How is the AI industry working to influence UK AISI, US CAISI, and the EU AI Office?

• I want to study how mechanisms of regulatory capture may be at play, including: revolving doors for personnel, industry-favorable contracts, and capitulation to industry.
• This project is currently on hold because of FOI/FOIA delays that I have in the US, UK, and EU.
• …China? It might be hard to study though because the Chinese AI governance apparatus is large and nebulous.

An audit of American frontier AI laws proposed in [insert year] for challenges, ambiguities, and loopholes

• Straightforward idea: we define some criteria for selecting "important" AI laws that have been passed in the USA and then write about their issues.
• Remember that ambiguities in law are not inherently bad.
• This kind of paper could become a yearly thing.
• The first one could cover 2025-2026 and include SB 53, SB 315, RAISE, Obernolte/Trahah, TRUMP AMERICA AI ACT, and maybe others.

The biggest AI Incidents of [insert year]: How could they have been prevented?

• I think this could make for a great yearly paper.
• This one probably shouldn't be started until the fall.
• I think writing this in a pretty modular way could be good. The main goal would be to just assemble the right team and assign people to sections corresponding to incidents. Every section could be like a page and have a listed set of section authors.
• I think the hardest thing about this paper is figuring out a way to somewhat objectively select for incidents that are "big". But I imagine doing so by using three possible qualifying criteria: death, money, and clicks.
• In addition to putting a large focus on prominent incidents, it might also be interesting to have a section where we scrape the OECD AI Database or the AI Incident Database and do some investigation and annotation of all incidents to see if there exist any at all that could not have been prevented using known best practices.

Some sort of projects studying AI regulatory patchworks. Are they really that bad?

• Maybe it would be possible to do an AI-assisted analysis of all state bills passed in the last few years and sort them by whether they merely mention AI versus whether they impose a regulatory burden, and of what kind. This could be useful for debunking claims about how 1000 or 2000 bills result in impossible regulatory burdens.
• It could also be useful to study certain patchworks to see how complicated the union of all regulatory requirements actually is.

Suicide Squad: Evaluating production AI models for responses to long-horizon mental health crises using personified agents

• Right now, one of the topics with the greatest degree of public and legislative concern in AI is AI systems that facilitate suicide, psychosis, and other mental health crises in users across very long horizons. For example, the Raine vs. OpenAI complaint has quotes from long transcripts between Adam Raine and ChatGPT. However, and please correct me if I'm wrong, to the best of my knowledge, there does not currently exist a benchmark to measure this kind of harmful stuff from AI systems.
• This project could revolve around engineering a few dozen suicidal or self-harming personas for automated red-teaming AI agents to adopt, possibly based on anonymized real people. The personas could be designed to discuss multiple topics, some benign, some psychologically concerning, before eventually leading to discussing suicidal or self-harm ideations after a series of long interactions. In designing personas and instructions, care should be taken to emulate some of the aspects of publicly available transcripts from people like Adam Raine. Dozens of production AI models (and maybe AI companion products) could then be evaluated by judges based on how they interact in simulated chats with these personas. One of the top-line results of this kind of paper would be findings about which AI products responded the most poorly.