FAQs and General Advice on AI Policy Careers

Abstract

AI policy is a young, fast-changing field in which no single credential qualifies you and written output is the main currency. Rather than planning for a specific dream job, build general-purpose skills and a visible portfolio, get experience in whichever sector you can, keep hands-on familiarity with the state of the technology, and expect a lot of failure along the way.

Note: originally published on Medium in April 2024; republished on Substack in October 2024. The author frames the whole piece as personal opinion informed by his time at OpenAI, ASU, Oxford, the US Department of Energy and various think tanks, and defaults to focusing on the AI policy research side.

What “AI policy” means and what the work looks like

• The author’s working definition: the theory and practice of authoritative decision-making about AI by public and private actors.
• It spans three “modes” that people, teams and organisations specialise in to differing degrees:
  • AI policy research
  • AI policy implementation
  • AI policy engagement
• Some roles are clear-cut (e.g. purely academic research on international AI institutions, or stakeholder engagement on behalf of a company); others blur into AI ethics and safety. A trust-and-safety team at an AI company sits at the intersection — mostly implementation with some research and engagement mixed in.
• His own day-to-day: meetings on research and deployment topics, following internal developments on Slack, external meetings, reading work from his team and other organisations, reading material outside AI policy, writing, and giving feedback on writing.
• As a manager, more time goes to meetings and internal memos; before management, more time went to going deep on specific topics and running projects end to end (typically months, sometimes weeks or years).

Backgrounds and candidate profiles

• There is no single profile. His team includes people from political science, law, computer science, science and technology studies, tech policy, economics and philosophy.
• Educational credentials are treated as signals of grit and skill — not the only such signals — and a degree usually implies reasonable writing ability.
• No formal credential guarantees value-add, so hiring processes interview for things CVs don’t capture, such as experience with and attitudes toward collaboration.
• Written output “is in some sense the currency of AI policy and of AI policy careers.” Even technically focused people must communicate results clearly and connect them to wider considerations.

On being “non-technical”

• The author dislikes the term: it’s ill-defined, often applied in biased or inaccurate ways to people from underrepresented backgrounds, and implies a static view of skills.
• “Technicalness” is better treated as a cluster of sub-skills, some of which nominally technical people are also weak at:
  • Intuitive understanding of the state of the art from lots of hands-on experimentation with AI systems (he flags this as most important)
  • Fluency in relevant jargon (a matter of degree)
  • Math skills
  • General coding skills — at minimum, comfort using the terminal to install something or call an API, plus some Python familiarity
  • Familiarity with the “lay of the land” of AI as a field: who works on what, what’s easy or hard and at what engineering cost
  • Understanding of the machine learning literature
  • Understanding of machine learning practice
• There’s no fixed blueprint for how much to invest in each, but a diminishing-returns mentality applies: if you’re at zero in one, learning the basics yields a lot.
• The skills needed to accomplish any given task are changing rapidly because of AI itself, which is part of why he emphasises experimentation.

Sector trade-offs

The post groups industry + government together, and academia + civil society together.

Industry and government

• Best if you want to engage with cutting-edge capabilities and day-to-day decision-making, and accept the trade-offs.
• There’s a presumption you will engage with the cutting edge — a new model in industry, a new executive order or piece of legislation in government — since non-model-specific work could in theory happen anywhere.
• In practice, some research empirically won’t be done outside certain labs, because proximity and proprietary information make issues salient sooner.
• Less autonomy over what you work on than in academia: your value-add is framed as helping the organisation decide and act better, not adding to public knowledge.
• Harder to publish findings and personal views, since output may be read as the organisation’s official position; expectation to focus on developments at your organisation rather than the wider world.

Academia and civil society

• Best if you want to understand, comment on and critique developments while being (perceived as) a relatively impartial voice. Industry people may be seen as informed but not neutral.
• Civil society suits those who want to track public opinion and stakeholder interests while shaping public debate in a way often seen as more legitimate than industry advocacy; it’s also critical for holding industry and governments accountable.
• Academia is hard to beat if your theory of change rests on deep scholarly understanding and you aren’t fussed about implementation details — and is the clear winner for university teaching.
• Main drawbacks: resourcing (salary, compute) and access to information about state-of-the-art technical developments.
• Substantial information asymmetry persists: companies know more about the technology’s frontier; governments know more about what’s politically and practically feasible, and aggregate information “from a firehose” across society.

Caveats

• The distinctions aren’t clear-cut — you can be more or less proximate to industry from within academia or civil society.
• He generally encourages getting experience in multiple sectors, but notes some people reasonably become “lifers” in one (e.g. academia for publishing autonomy, civil society if your theory of change involves activism).

What the author is thinking about

• A recurring tension between:
  1. Making the most of the current moment — learning about the real impact of current models and the risks/opportunities of the next, improving public discourse and the implementation of industry norms and public policy.
  2. Thinking ahead to how the constraints and affordances of AI policy could or should change substantially.
• He glosses these as “What’s next?” and “What’s missing?”, and as a manager thinks about the right portfolio across the two.

The job market

• Anecdotal impression (he notes he isn’t familiar with hard data): an under-supply of senior AI policy researchers who can pursue impactful research autonomously and mentor others, and an over-supply of junior researchers needing supervision.
• “Oversupply” here means many applicants per job, not that the world needs fewer people — he thinks the field should be bigger; the question is how to organise and fund it.
• His explanation: the field is young in its current shape, experience takes time to accrue, so there hasn’t been time to grow senior talent. Hence many senior listings, and a growing ecosystem of new think tanks and startups absorbing that talent. Government hiring may create junior opportunities but doesn’t help the senior bottleneck.
• Practical implication: get experience where you can. An AI engineering job, product management in another sector, or research on a different topic all leave you better off — and you can probably publish on AI policy even if your current job isn’t in AI policy. Building mentorship and management experience helps address the senior bottleneck.

General career advice

Don’t over-anchor on specific roles

• The field is highly dynamic — OpenAI didn’t exist when he started out, Anthropic is only a few years old, many non-profits and startups are months old, and AI-policy-specific government roles are recent.
• Planning for a very specific “dream job” is impractical (unlikely to be open when you need it, or to go to you) and unwise (you may miss better opportunities).
• Better: stay flexible, build general-purpose skills, establish a solid portfolio, take experience where you can.

Have as much fun as you can

• Be grateful to be alive while fundamental aspects of humanity’s relationship with technology are being debated and steered.
• If the work feels boring, it may be the wrong career — or you may be doing it too generically and not pushing the Overton window enough. Wide exploration and doing things that haven’t been done tend to be both higher-impact and more fun.

Expect a lot of failure

• Failure happens to everyone: in recent months he had a paper rejected, missed a deadline he cared about, had misunderstandings with coworkers, and abandoned projects that weren’t going well.
• Things he now does reliably were things he failed at repeatedly for a while. If you’re not failing a lot, you may not be ambitious enough.
• You may also fail through no fault of your own. Don’t be discouraged; do learn from it. People aren’t gossiping about your failures — they’re focused on their own work.

Timing matters

• Work can be too early (too abstract to inform any actor’s decision) or too late (specific, but past the window of opportunity). A paper in February may be less useful than the same paper in January, or than a similar publication that preceded it.
• This is part of why access advantages matter — they help you see windows of opportunity and looming challenges before they get attention.
• Related to punctuated equilibrium in policy change and Collingridge’s dilemma.
• Reinforces the case for keeping abreast of the technology by trying it directly: even without privileged information, you can spend more time than others on what’s possible, what issues arise in which domains, and how safety mitigations evolve across products.

Think carefully about the purpose of your writing

• Valid purposes include: gaining experience, name recognition, building references for job applications, signalling a career pivot, synthesising or explaining information better than has been done, introducing a new idea, normalising an existing one, or fleshing out a vague one.
• These trade off against one another, so have an explicit understanding of what a given piece is meant to achieve.

Write multiple kinds of things

• Formality is a spectrum: tweets → threads → blog posts → workshop papers → conference and journal papers.
• Don’t put all your time at the fully formal end. More outputs means faster iteration on feedback and better odds of hitting the right timing, since less lead time is needed.
• Plenty of worthwhile writing requires no big new insight: book reviews, conference summaries (even from virtual attendance), comments on a recent paper or news story.

Just apply

• If you’re interested in a role, apply — the only real downside is time, and you get faster with practice.
• Anecdotally, people from underrepresented backgrounds sell themselves short on what they’re “qualified” for. Don’t read listed requirements too literally — competitors won’t, and expectations may be illustrative or subject to change.
• Don’t spend so long on an application that you miss the window; being late definitely doesn’t help.

Don’t be too generic

• Explore widely in reading and project types — the most impactful ideas often come from analogies between distant literatures and disciplines. Read and work on the same things as everyone else and you’ll say the same things they do.
• There’s a risk of exploring too much and becoming a dilettante, but most people under-explore.
• Have a niche — this isn’t inconsistent with wide exploration. Aim to be known as someone with broad interests who knows (or is deliberately learning) the basics, while carving a unique path on a topic they were first or best to tackle.

Get practice with public speaking

• Helps with presentations and meetings, which matter for communicating ideas to decision-makers, getting feedback and demonstrating value to employers — even if you never plan to give public talks.
• “Public” can mean a small group, a one-person Zoom presentation, or Toastmasters.

Play to the strengths of your current sector

• Do the things that are impossible elsewhere while you can. Building foundations far from your discipline is hard once you have a full-time job, so explore many disciplines while in school.
• If you have industry access, use it by talking to lots of people on the technical and product sides — while doing your homework beforehand out of respect for their time.

Optimise for experience and impact, not credit

• Policy differs from academia in that fair credit assignment can’t be assumed and is uncommon — many outputs have no listed authors, or are credited to a policymaker who drew on unnamed staff and stakeholders.
• Many important developments never appear in papers and may not look like “real research” from an academic vantage point.
• Accomplishments may be invisible publicly and partial in nature (e.g. influencing one part of a much larger document).
• Employers are aware of this and look beyond public profiles, but it’s still worth building a personal “brand” outside official work and keeping track of things you have fingerprints on even when unattributed.

Expertise, confidence and taste come from practice

• “Excellence is mundane.” Expect gradual improvement across writing, public speaking, research taste and technical skills; practise where you’re weak and keep investing where you’re strong.
• Accumulated wins — public and private — build confidence over time. Avoid overconfidence, except perhaps in the sense of being willing to make bets and psych yourself into believing in them so you can find out whether they work.
• Written wisdom (he cites Tom Kalil’s piece on policy entrepreneurship at the White House) is worth reading but is no substitute for experience. Knowing tips is different from knowing when to apply or ignore them.
• It’s OK to feel like you don’t know what you’re doing — everyone does to some extent, if honest.

Solicit feedback and sit on it before responding

• People don’t give feedback by default. When they do and aren’t being rude, be grateful and take time to process before responding.
• Don’t take it personally; look for the truth in it even when it feels wrong.
• Don’t over-anchor on any single piece either. A writing adage: negative feedback usually has a point, though specific suggested fixes are rarely correct. He’s found feedback that seemed wrong often had more truth than he initially recognised.

PhDs aren’t all or nothing

• If you have time and are somewhat seriously interested, apply. You can always drop out later, and you’ll be better off in skills and credentials as a PhD student or candidate than as neither.

Use social media, with a grain of salt

• Much AI policy debate and idea discovery happens on social media, and it’s a good way to be discovered and to get feedback.
• But it isn’t representative: it’s easier to tweet about discrete papers and announcements than continuous projects, and some work isn’t publicly discussable.
• Social media also overstates polarisation relative to the ground — there’s something to the “AI ethics vs. AI safety” divide, but it’s exaggerated.
• If it seems toxic, that doesn’t mean you should avoid the field or that you can safely ignore what’s discussed there.

Always be learning

• You can learn from everything you read, do or hear, though the returns from different activities shift over time.
• Direct knowledge gains from reading AI policy will decline as repetition sets in — but value remains at a higher level of abstraction: who knows what, how issues are framed across disciplines, why others find a publication more interesting than you do.
• Public output should show constant learning. Have a niche at any given time, but let it evolve, grow and sharpen; avoid stagnation. Employers want someone who adds value now and will add more over time. Ideally, outputs simultaneously demonstrate growth, build research and writing experience, and generate information about how people react.

Acknowledgments (from the original)

• Thanks to Larissa Schiavo and Girish Sastry for feedback; remaining mistakes are the author’s own.