Hello, and welcome back to the Cognitive Revolution!

Today my guest is Linus Lee, AI product leader at Notion and AI explorer extraordinaire.

I have followed Linus online for a couple of years now, fascinated by his many groundbreaking projects and unique way of thinking about AI systems. From creating novel interfaces that visualize and manipulate generative models in their latent space, to developing techniques for semantically editing text and images, Linus has been a pioneering tinkerer.

In this wide-ranging conversation, we dig into the details of how Linus goes about his explorations. He shares his toolkit, from PyTorch as a foundation to the custom tools he's built over time for data visualization, model evaluation, and rapid experimentation. We discuss the importance of spending time with data and failure cases, and the value of building your own tools to deeply understand the problems you're trying to solve.

Linus also offers his perspective on the current capabilities of language models, and where he sees the biggest opportunities for improvement. Beyond hallucination reduction, instruction following, and cost efficiency, he emphasizes the need for better general reasoning as a longer-term goal.

We speculate about the future, considering scenarios like models communicating with one another via high-dimensional embeddings, techniques to connect latent spaces across modalities, and the societal implications as AI capabilities continue to advance. Linus articulates a vision centered on amplifying rather than replacing human intelligence, a principle he believes should guide the development and deployment of AI-powered products.

Throughout this episode, Linus demonstrates the curiosity, resourcefulness, and thoughtfulness that have made him such an influential figure in the AI community. His work inspires us to engage more deeply with these systems, to build the tools we need to understand them, and to steer their development in a direction that deepens rather than diminishes human creativity and agency.

As always, if you find value in the show, please share it with others who might appreciate it. Particularly right now as we are establishing the new feed, a tweet or a comment on Youtube would be especially valuable.

And of course, we invite you to reach out with feedback or suggestions on our website, cognitiverevolution.ai

Now, please enjoy this deep dive into the art and science of exploring AI systems with Linus Lee.

In this conversation, we cover 6 papers in detail. They are:

• RT-2 – which shows how internet-scale vision-language allow robots to understand and manipulate objects they've never seen in training
• RT-X – a collaboration with academic labs across the country that demonstrates how a single model can be trained to control a diverse range of robot embodiments, achieving performance that often surpasses specialist models trained on individual robots.
• RT-Trajectory – a project that shows how robots can learn new skills, in context, from a single human demonstration, as represented by a simple line drawings
• Auto-RT – a system that scales human oversight of robots, even in unseen environments, by using large language models and a "robot constitution" to power first-line ethical and safety checks on robot behavior.
• Learning to Learn Faster – an approach that enables robots to learn more efficiently from human verbal feedback,
• Pivot - another project that shows how vision-language models can be used to guide robots – no special fine-tuning required.

While progress in robotics is still trailing behind the advances in language & vision, there are still challenges to be overcome before robotics models will have the scale of data and/or the sample efficiency needed to achieve reliable general-purpose capabilities, and the study of robot safety and alignment is still in its infancy, ultimately I see this rapid-fire series of papers as strong evidence that the same core architectures and scaling techniques that have worked so well in other contexts will ultimately succeed in robotics as well.

The work being done at Google DeepMind Robotics is pushing the boundaries of what's possible, investment in a new generation of robotics startups is heating up, and the pace of progress shows no signs of slowing down.

As always, if you're finding value in the show, please take a moment to share it with friends. This one would be perfect for anyone who has ever day-dreamed of having a robot that could fold their laundry or pick up their kids toys.

And especially as we are just building the new feed, a review on Apple Podcasts, Spotify, or a comment on Youtube would be much appreciated.

Now, here's my conversation with Keerthana Gopalakrishnan and Ted Xiao of Google Deepmind Robotics.

Like many of us, I've been using Gmail for roughly 20 years now, and while I've tried several alternative email clients over the years, for me, none have really stuck.

Shortwave, however, has truly impressed me, and I have continued to use it past the point of curiosity and into the realm of forming a genuinely new, and likely lasting habit.

My favorite feature, by far, is the "AI assistant", which presents in the increasingly familiar form factor of the natural language sidebar chatbot. It can help you search through and configure your inbox, check your availability and schedule meetings, and refer to similar emails you've sent in the past so that it can imitate your style when drafting responses for you.

As someone who has long since given up on inbox zero and really just wants an AI assistant to help me navigate my overloaded inbox, I can definitely say I've had a few magical experiences with this product – the time saved in searching for things I know exist but can't quite remember the keywords for, unto itself, has been delightful.

Andrew, who previously founded Firebase and has already been acquired by Google once, was extremely open about the technology underlying Shortwave, reflecting the fact that this is no thin wrapper, and it was a ton of fun to get so deep into the details.

We covered:

• Shortwave's RAG stack, which is powered by a full download and re-indexing of your entire inbox, a process that takes hours and costs Shortwave real money, but which creates a remarkably responsive experience downstream
• How the AI assistant works from user message input to AI response, including tool selection, query reformulation, feature extraction, retrieval, re-ranking, and answer generation.
• Which models Shortwave is using, which includes Mistral, fine-tuned GPT-3.5, and GPT-4-turbo – a list which is always subject to change, and right now even more than usual since Claude-3 launched just after we recorded, and long-context Gemini 1.5 is on the horizon as well.

Finally, we discussed how Andrew thinks about building and timing product launches in such a fast-moving space, his vision for Shortwave, how he expects email to evolve and how we will deal with the inevitable rise in high-quality spam, and deep AI integration will help manage knowledge on a company-wide basis.

If you're building with LLMs, this conversation has a ton of great nuggets which you won't want to miss.

And if you're an email user, and I'm pretty sure all of you are, I definitely recommend checking out Shortwave.

Note that this is not a paid promotion. Andrew was kind enough to give me a free year of Shortwave, but that's it. I am genuinely just super enthusiastic about this product, and you have my commitment that I will always be transparent about any sponsorship deals that we do in the future.

As always, we appreciate it when folks share The Cognitive Revolution with friends. And right now, in particular, as we are ramping up the new feed, a social media share would be especially valuable.

And of course, your feedback is always welcome – you can leave a message on our new website at CognitiveRevolution.ai, or DM me on the social media platform of your choice.

Now, let's dive deep into the AI technology powering the future of email with Andrew Lee and Shortwave.