Episodes

  • INTERVIEW: Scaling Democracy w/ (Dr.) Igor Krawczuk

    INTERVIEW: Scaling Democracy w/ (Dr.) Igor Krawczuk
    Jun 3 2024
    The almost Dr. Igor Krawczuk joins me for what is the equivalent of 4 of my previous episodes. We get into all the classics: eugenics, capitalism, philosophical toads... Need I say more?If you're interested in connecting with Igor, head on over to his website, or check out placeholder for thesis (it isn't published yet).Because the full show notes have a whopping 115 additional links, I'll highlight some that I think are particularly worthwhile here:The best article you'll ever read on Open Source AIThe best article you'll ever read on emergence in MLKate Crawford's Atlas of AI (Wikipedia)On the Measure of IntelligenceThomas Piketty's Capital in the Twenty-First Century (Wikipedia)Yurii Nesterov's Introductory Lectures on Convex OptimizationChapters(02:32) - Introducing Igor (10:11) - Aside on EY, LW, EA, etc., a.k.a. lettersoup (18:30) - Igor on AI alignment (33:06) - "Open Source" in AI (41:20) - The story of infinite riches and suffering (59:11) - On AI threat models (01:09:25) - Representation in AI (01:15:00) - Hazard fishing (01:18:52) - Intelligence and eugenics (01:34:38) - Emergence (01:48:19) - Considering externalities (01:53:33) - The shape of an argument (02:01:39) - More eugenics (02:06:09) - I'm convinced, what now? (02:18:03) - AIxBio (round ??) (02:29:09) - On open release of models (02:40:28) - Data and copyright (02:44:09) - Scientific accessibility and bullshit (02:53:04) - Igor's point of view (02:57:20) - OutroLinksLinks to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance. All references, including those only mentioned in the extended version of this episode, are included.Suspicious Machines Methodology, referred to as the "Rotterdam Lighthouse Report" in the episodeLIONS Lab at EPFLThe meme that Igor referencesOn the Hardness of Learning Under SymmetriesCourse on the concept of equivariant deep learningAside on EY/EA/etc.Sources on Eliezer YudkowskiScholarly Community EncyclopediaTIME100 AIYudkowski's personal websiteEY WikipediaA Very Literary Wiki -TIME article: Pausing AI Developments Isn’t Enough. We Need to Shut it All Down documenting EY's ruminations of bombing datacenters; this comes up later in the episode but is included here because it about EY.LessWrongLW WikipediaMIRICoverage on Nick Bostrom (being a racist)The Guardian article: ‘Eugenics on steroids’: the toxic and contested legacy of Oxford’s Future of Humanity InstituteThe Guardian article: Oxford shuts down institute run by Elon Musk-backed philosopherInvestigative piece on Émile TorresOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜NY Times article: We Teach A.I. Systems Everything, Including Our BiasesNY Times article: Google Researcher Says She Was Fired Over Paper Highlighting Bias in A.I.Timnit Gebru's WikipediaThe TESCREAL Bundle: Eugenics and the Promise of Utopia through Artificial General IntelligenceSources on the environmental impact of LLMsThe Environmental Impact of LLMsThe Cost of Inference: Running the ModelsEnergy and Policy Considerations for Deep Learning in NLPThe Carbon Impact of AI vs Search EnginesFilling Gaps in Trustworthy Development of AI (Igor is an author on this one)A Computational Turn in Policy Process Studies: Coevolving Network Dynamics of Policy ChangeThe Smoothed Possibility of Social Choice, an intro in social choice theory and how it overlaps with MLRelating to Dan HendrycksNatural Selection Favors AIs over Humans"One easy-to-digest source to highlight what he gets wrong [is] Social and Biopolitical Dimensions of Evolutionary Thinking" -IgorIntroduction to AI Safety, Ethics, and Society, recently published textbook"Source to the section [of this paper] that makes Dan one of my favs from that crowd." -IgorTwitter post referenced in the episode<...
    Show More Show Less
    2 hrs and 59 mins
  • INTERVIEW: StakeOut.AI w/ Dr. Peter Park (3)

    INTERVIEW: StakeOut.AI w/ Dr. Peter Park (3)
    Mar 25 2024
    As always, the best things come in 3s: dimensions, musketeers, pyramids, and... 3 installments of my interview with Dr. Peter Park, an AI Existential Safety Post-doctoral Fellow working with Dr. Max Tegmark at MIT.As you may have ascertained from the previous two segments of the interview, Dr. Park cofounded StakeOut.AI along with Harry Luk and one other cofounder whose name has been removed due to requirements of her current position. The non-profit had a simple but important mission: make the adoption of AI technology go well, for humanity, but unfortunately, StakeOut.AI had to dissolve in late February of 2024 because no granter would fund them. Although it certainly is disappointing that the organization is no longer functioning, all three cofounders continue to contribute positively towards improving our world in their current roles.If you would like to investigate further into Dr. Park's work, view his website, Google Scholar, or follow him on Twitter00:00:54 ❙ Intro00:02:41 ❙ Rapid development00:08:25 ❙ Provable safety, safety factors, & CSAM00:18:50 ❙ Litigation00:23:06 ❙ Open/Closed Source00:38:52 ❙ AIxBio00:47:50 ❙ Scientific rigor in AI00:56:22 ❙ AI deception01:02:45 ❙ No takesies-backsies01:08:22 ❙ StakeOut.AI's start01:12:53 ❙ Sustainability & Agency01:18:21 ❙ "I'm sold, next steps?" -you01:23:53 ❙ Lessons from the amazing Spiderman01:33:15 ❙ "I'm ready to switch careers, next steps?" -you01:40:00 ❙ The most important question01:41:11 ❙ OutroLinks to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance.StakeOut.AIPause AIAI Governance Scorecard (go to Pg. 3)CIVITAIArticle on CIVITAI and CSAMSenate Hearing: Protecting Children OnlinePBS Newshour CoverageThe Times Sues OpenAI and Microsoft Over A.I. Use of Copyrighted WorkOpen Source/Weights/Release/InterpretationOpen Source InitiativeHistory of the OSIMeta’s LLaMa 2 license is not Open SourceIs Llama 2 open source? No – and perhaps we need a new definition of open…Apache License, Version 2.03Blue1Brown: Neural NetworksOpening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorsThe online tableSignalBloomz model on HuggingFaceMistral websiteNASA TragediesChallenger disaster on WikipediaColumbia disaster on WikipediaAIxBio RiskDual use of artificial-intelligence-powered drug discoveryCan large language models democratize access to dual-use biotechnology?Open-Sourcing Highly Capable Foundation Models (sadly, I can't rename the article...)Propaganda or Science: Open Source AI and Bioterrorism RiskExaggerating the risks (Part 15: Biorisk from LLMs)Will releasing the weights of future large language models grant widespread access to pandemic agents?On the Societal Impact of Open Foundation ModelsPolicy briefApart ResearchScienceCiceroHuman-level play in the game of Diplomacy by combining language models with strategic reasoningCicero webpageAI Deception: A Survey of Examples, Risks, and Potential SolutionsOpen Sourcing the AI Revolution: Framing the debate on open source, artificial intelligence and regulationAI Safety CampInto AI Safety Patreon
    Show More Show Less
    1 hr and 42 mins
  • INTERVIEW: StakeOut.AI w/ Dr. Peter Park (2)

    INTERVIEW: StakeOut.AI w/ Dr. Peter Park (2)
    Mar 18 2024
    Join me for round 2 with Dr. Peter Park, an AI Existential Safety Postdoctoral Fellow working with Dr. Max Tegmark at MIT. Dr. Park was a cofounder of StakeOut.AI, a non-profit focused on making AI go well for humans, along with Harry Luk and one other individual, whose name has been removed due to requirements of her current position.In addition to the normal links, I wanted to include the links to the petitions that Dr. Park mentions during the podcast. Note that the nonprofit which began these petitions, StakeOut.AI, has been dissolved.Right AI Laws, to Right Our Future: Support Artificial Intelligence Safety Regulations NowIs Deepfake Illegal? Not Yet! Ban Deepfakes to Protect Your Family & Demand Deepfake LawsBan Superintelligence: Stop AI-Driven Human Extinction Risk 00:00:54 - Intro00:02:34 - Battleground 1: Copyright00:06:28 - Battleground 2: Moral Critique of AI Collaborationists00:08:15 - Rich Sutton00:20:41 - OpenAI Drama00:34:28 - Battleground 3: Contract Negotiations for AI Ban Clauses00:37:57 - Tesla, Autopilot, and FSD00:40:02 - Recycling00:47:40 - Battleground 4: New Laws and Policies00:50:00 - Battleground 5: Whistleblower Protections00:53:07 - Whistleblowing on Microsoft00:54:43 - Andrej Karpathy & Exercises in Empathy01:05:57 - OutroLinks to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance.StakeOut.AIThe Times Sues OpenAI and Microsoft Over A.I. Use of Copyrighted WorkSusman Godfrey LLPRich SuttonReinforcement Learning: An Introduction (textbook)AI Succession (presentation by Rich Sutton)The Alberta Plan for AI ResearchMoore's LawThe Future of Integrated Electronics (original paper)Computer History Museum's entry on Moore's LawStochastic gradient descent (SGD) on WikipediaOpenAI DramaMax Read's Substack postZvi Mowshowitz's Substack series, in order of postingOpenAI: Facts from a WeekendOpenAI: The Battle of the BoardOpenAI: Altman ReturnsOpenAI: Leaks Confirm the Story ← best singular post in the seriesOpenAI: The Board ExpandsOfficial OpenAI announcementWGA on WikipediaSAG-AFTRA on WikipediaTesla's False AdvertisingTesla's response to the DMV's false-advertising allegations: What took so long?Tesla Tells California DMV that FSD Is Not Capable of Autonomous DrivingWhat to Call Full Self-Driving When It Isn't Full Self-Driving?Tesla fired an employee after he posted driverless tech reviews on YouTubeTesla's page on Autopilot and Full Self-DrivingRecyclingBoulder County Recycling Center Stockpiles Accurately Sorted Recyclable MaterialsOut of sight, out of mindBoulder Eco-Cycle Recycling GuidelinesDivide-and-Conquer Dynamics in AI-Driven DisempowermentMicrosoft WhistleblowerWhistleblowers call out AI's flawsShane's LinkedIn postLetters sent by JonesKarpathy announces departure from OpenAI
    Show More Show Less
    1 hr and 6 mins
  • MINISODE: Restructure Vol. 2

    MINISODE: Restructure Vol. 2
    Mar 11 2024

    UPDATE: Contrary to what I say in this episode, I won't be removing any episodes that are already published from the podcast RSS feed.

    After getting some advice and reflecting more on my own personal goals, I have decided to shift the direction of the podcast towards accessible content regarding "AI" instead of the show's original focus. I will still be releasing what I am calling research ride-along content to my Patreon, but the show's feed will consist only of content that I aim to make as accessible as possible.

    00:35 - TL;DL
    01:12 - Advice from Pete
    03:10 - My personal goal
    05:39 - Reflection on refining my goal
    09:08 - Looking forward (logistics
    Show More Show Less
    13 mins
  • INTERVIEW: StakeOut.AI w/ Dr. Peter Park (1)

    INTERVIEW: StakeOut.AI w/ Dr. Peter Park (1)
    Mar 4 2024

    Dr. Peter Park is an AI Existential Safety Postdoctoral Fellow working with Dr. Max Tegmark at MIT. In conjunction with Harry Luk and one other cofounder, he founded ⁠StakeOut.AI, a non-profit focused on making AI go well for humans.

    00:54 - Intro
    03:15 - Dr. Park, x-risk, and AGI
    08:55 - StakeOut.AI
    12:05 - Governance scorecard
    19:34 - Hollywood webinar
    22:02 - Regulations.gov comments
    23:48 - Open letters
    26:15 - EU AI Act
    35:07 - Effective accelerationism
    40:50 - Divide and conquer dynamics
    45:40 - AI "art"
    53:09 - Outro

    Links to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance.

    • StakeOut.AI
    • AI Governance Scorecard (go to Pg. 3)
    • Pause AI
    • Regulations.gov
      • USCO StakeOut.AI Comment
      • OMB StakeOut.AI Comment
    • AI Treaty open letter
    • TAISC
    • Alpaca: A Strong, Replicable Instruction-Following Model
    • References on EU AI Act and Cedric O
      • Tweet from Cedric O
      • EU policymakers enter the last mile for Artificial Intelligence rulebook
      • AI Act: EU Parliament’s legal office gives damning opinion on high-risk classification ‘filters’
      • EU’s AI Act negotiations hit the brakes over foundation models
      • The EU AI Act needs Foundation Model Regulation
      • BigTech’s Efforts to Derail the AI Act
    • Open Sourcing the AI Revolution: Framing the debate on open source, artificial intelligence and regulation
    • Divide-and-Conquer Dynamics in AI-Driven Disempowerment
    Show More Show Less
    54 mins
  • MINISODE: "LLMs, a Survey"

    MINISODE: "LLMs, a Survey"
    Feb 26 2024

    Take a trip with me through the paper Large Language Models, A Survey, published on February 9th of 2024. All figures and tables mentioned throughout the episode can be found on the Into AI Safety podcast website.

    00:36 - Intro and authors
    01:50 - My takes and paper structure
    04:40 - Getting to LLMs
    07:27 - Defining LLMs & emergence
    12:12 - Overview of PLMs
    15:00 - How LLMs are built
    18:52 - Limitations if LLMs
    23:06 - Uses of LLMs
    25:16 - Evaluations and Benchmarks
    28:11 - Challenges and future directions
    29:21 - Recap & outro

    Links to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance.

    • Large Language Models, A Survey
    • Meysam's LinkedIn Post
    • Claude E. Shannon
      • A symbolic analysis of relay and switching circuits (Master's Thesis)
      • Communication theory of secrecy systems
      • A mathematical theory of communication
      • Prediction and entropy of printed English
    • Future ML Systems Will Be Qualitatively Different
    • More Is Different
    • Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
    • Are Emergent Abilities of Large Language Models a Mirage?
    • Are Emergent Abilities of Large Language Models just In-Context Learning?
    • Attention is all you need
    • Direct Preference Optimization: Your Language Model is Secretly a Reward Model
    • KTO: Model Alignment as Prospect Theoretic Optimization
    • Optimization by Simulated Annealing
    • Memory and new controls for ChatGPT
    • Hallucinations and related concepts—their conceptual background
    Show More Show Less
    31 mins
  • FEEDBACK: Applying for Funding w/ Esben Kran

    FEEDBACK: Applying for Funding w/ Esben Kran
    Feb 19 2024

    Esben reviews an application that I would soon submit for Open Philanthropy's Career Transitition Funding opportunity. Although I didn't end up receiving the funding, I do think that this episode can be a valuable resource for both others and myself when applying for funding in the future.

    Head over to Apart Research's website to check out their work, or the Alignment Jam website for information on upcoming hackathons.

    A doc-capsule of the application at the time of this recording can be found at this link.

    01:38 - Interview starts
    05:41 - Proposal
    11:00 - Personal statement
    14:00 - Budget
    21:12 - CV
    22:45 - Application questions
    34:06 - Funding questions
    44:25 - Outro

    Links to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance.

    • AI governance talent profiles we’d like to see
    • The AI Governance Research Sprint
    • Reasoning Transparency
    • Places to look for funding
      • Open Philanthropy's Career development and transition funding
      • Long-Term Future Fund
      • Manifund
    Show More Show Less
    45 mins
  • MINISODE: Reading a Research Paper

    MINISODE: Reading a Research Paper
    Feb 12 2024

    Before I begin with the paper-distillation based minisodes, I figured we would go over best practices for reading research papers. I go through the anatomy of typical papers, and some generally applicable advice.

    00:56 - Anatomy of a paper
    02:38 - Most common advice
    05:24 - Reading sparsity and path
    07:30 - Notes and motivation

    Links to all articles/papers which are mentioned throughout the episode can be found below, in order of their appearance.

    • Ten simple rules for reading a scientific paper
    • Best sources I found
      • Let's get critical: Reading academic articles
      • #GradHacks: A guide to reading research papers
      • How to read a scientific paper (presentation)
    • Some more sources
      • How to read a scientific article
      • How to read a research paper
      • Reading a scientific article
    Show More Show Less
    9 mins