Frontier alignment researchAdvancedLesson 3237 min read

Choosing an alignment research problem

Good research starts with a good problem. Learn how to judge taste, tractability and feedback loops, find and read papers, work with little compute, collaborate, and think about the safety of publishing.

✦ Ask the tutor🧪 Lab · 60 min📝 Lesson test
Lesson in motion

In 60 seconds

Choosing an alignment research problem

Good research starts with a good problem. Learn how to judge taste, tractability and feedback loops, find and read papers, work with little compute, collaborate, and think about…

1/5
In simple words
If you want to learn cricket, you do not start by trying to bowl in a World Cup final. You start in the nets, with one skill, and a coach who tells you quickly what went wrong. Good first research problems are like the nets.
You now know the main methods: oversight, rule-based training, debate, weak-to-strong, evaluations and model organisms. The next step is to choose a problem to work on. This choice often matters more than how hard you work.

Four questions to ask about any problem

QuestionWhat it meansWarning sign
Is it important?If you solved it, would it change what people build or believe about safety?You cannot explain why it matters in two sentences.
Is it tractable?Can you make real progress with your time, skills and compute?The first step needs a huge model or years of work.
Is the feedback loop fast?Can you learn whether you are on track in days, not months?You would only know if it worked at the very end.
Is it neglected?Are few people already working on this exact question?Many strong teams are already doing the same thing.
Research taste is the skill of picking problems that score well on these questions. It grows with practice and with feedback from more experienced people. Nobody starts with it.

Kinds of project that suit a first attempt

  • Replicate a small result. Re-run a published experiment on a small open model. Does it hold? This is how many researchers start. See Reproducing a result in the How to do AI research track.
  • Extend to a new setting. Test a known method in Indian languages, on a new task, or with a different model size.
  • Build an evaluation. A careful, well-documented test set for a specific behaviour, such as sycophancy in Hindi or Tamil.
  • Stress-test a claim. Find conditions where a published method stops working, and explain why.
  • Conceptual writing. A clear explanation of a disagreement, or a careful argument about when a method should or should not work.

Finding and reading papers

Most alignment research is shared as preprints on arXiv, a free archive of research papers. Useful categories include computer science artificial intelligence (cs.AI), machine learning (cs.LG) and computation and language (cs.CL). Many labs and research groups also publish on their own blogs and research pages.
  1. 1

    Start from a survey or overview

    Search arXiv for an overview paper on your topic, such as scalable oversight or AI evaluations.
  2. 2

    Follow the references

    Read the related-work section and pick the papers cited most.
  3. 3

    Look forward too

    Search for newer papers that cite the key paper.
  4. 4

    Keep a reading log

    For each paper: question, method, main result, limits, and one idea for follow-up work.
  5. 5

    Build your own reading list

    Ten well-understood papers beat fifty skimmed ones.
Note
Preprints on arXiv are not always peer reviewed. Read results critically: check sample sizes, baselines and whether the code is available.

Doing it on a small compute budget

  • Use small open-weight models from Hugging Face that run on a laptop, or free notebook services.
  • Many evaluation and behaviour studies need only API calls to a hosted model and careful design.
  • Toy models, like the Python example in lesson 4, can test ideas in minutes.
  • Start with the smallest version of your experiment that could possibly give a signal. Scale only after it works.

Working with others

  • Find a mentor or a reading group. Even one short conversation a month with someone more experienced saves weeks.
  • Share early drafts. Asking "what is wrong with this plan?" before you start is cheaper than after.
  • Online alignment communities, university groups and research fellowships exist in many countries. Look for current ones, since programmes open and close often.
  • Credit everyone who helped, and agree on authorship early.

The safety of publishing

Most alignment research is safe and helpful to publish. But some results need thought first. Ask before you publish:
  • Does this show a new, reliable way to remove safety training, hide behaviour, or get serious harmful help from models?
  • Would this mainly help people build more capable systems faster, without a matching safety benefit?
  • If a model developer is affected, have I told them first and given time to fix it?
Researchers disagree on how cautious to be. Openness helps others check and build on work; too much caution slows the field. Too little can hand harmful tools to others. When unsure, ask experienced colleagues before posting.
A first project that often stalls
  • "Solve deceptive alignment"
  • Needs a frontier model you cannot access
  • No result for six months
  • Working alone with no feedback
A first project that often works
  • "Does a known sycophancy test give the same results in Hindi as in English?"
  • Runs on small open models or a free API tier
  • First signal in a week
  • A mentor or reading group checks the plan
Try this. Write down three problems you are curious about. Score each from 1 to 5 on importance, tractability, feedback speed and neglectedness. Which one wins, and does the answer surprise you?

Watch and read more

arXiv: free preprints in cs.AI, cs.LG and cs.CLarXiv · reading

Lab

A one-page research proposal for a small alignment project, plus a starter reading list.

~60 min

The problem

Write a one-page research proposal. Use these headings:

  1. Question: one sentence, specific and testable.
  2. Why it matters: two or three sentences, including your theory of change.
  3. Method: models, data, measurements and a control condition. Which family from this track does it belong to?
  4. Prediction: what you expect, and what result would prove you wrong.
  5. Resources: compute, time and cost. Show that it fits a laptop or free tools.
  6. First milestone: what you will have after one week.
  7. Risks: what could go wrong in the research, and any safety concern about publishing.
  8. Reading list: 5 papers found on arXiv, each with one line on why it is relevant.

Then score your proposal from 1 to 5 on importance, tractability, feedback speed and neglectedness, and ask one other person to criticise it.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your proposal scores high on tractability and feedback speed but low on importance. Should you still do it?Reveal

Often yes for a first project, if it builds skills you need for more important work later and still adds something real. But check whether a small change could raise importance, for example by testing a method people actually rely on, or choosing a setting such as Indian languages where results would change practice. The goal is to avoid a project that is easy only because it does not matter.

Q2Two respected researchers give you opposite advice on your proposal. How should you decide?Reveal

Find out why they disagree. Often they hold different views about which methods transfer to future systems, or about what matters most. Ask each for the strongest argument against the other's view. Then see whether a cheap first experiment could tell them apart. If not, choose the direction whose assumptions you understand best and state those assumptions clearly in your write-up. Disagreement is normal in this field.

Please sign in to continue.

Questions people ask

Should I work on the most important problem or the most tractable one?

For a first project, lean toward tractable. You learn research skills by finishing projects and getting feedback. Once you have those skills, you can take on harder and more important problems with a better chance of success.

Can I do alignment research from India without joining a big lab?

Yes. Papers, open models and many tools are free. You can replicate results, build evaluations, especially for Indian languages where fewer people are working, and share your work publicly. Online communities and fellowships accept people from many countries; check their current terms.

How do I know if my idea has already been done?

Search arXiv and general search engines with several different phrasings. Ask in a research community. If you find a close match, that is good news: you can replicate it, extend it, or test it in a new setting.

What if my research could help make models more capable?

Many methods help both capability and safety. Think about the balance: does the safety value clearly outweigh the capability boost? Discuss it with experienced people, and consider how and when you share details.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress