Beyond LLMs: Building magic

Large language models reshaped AI. Though they were invented in 2017, LLMs entered the mainstream with the November 2022 launch of ChatGPT, the fastest-growing consumer application of all time. Since then, there has been an explosion of innovation. LLMs are likely the most transformative technology since the web browser, changing everyday activities like internet search, image creation, and many forms of writing. Within the enterprise, LLMs have the potential to redefine most white-collar jobs and critical business functions. 

In 2022, it felt like the race was over and OpenAI had won. Today, however, the race is back on, with a number of viable alternatives to the company’s ChatGPT and other GPT models. In 2023 alone, nearly 150 LLMs were released with models from Meta, Amazon, Google, Microsoft, Anthropic, and Mistral, all vying to supplant GPT-4. Overall, performance gaps have narrowed, and model prices are decreasing, with inference costs dropping by more than half in the past year. 

Every CEO, whether they make computer chips or potato chips, has announced an AI strategy, and markets are anticipating profit improvements across the board as a result of AI. At the same time, enterprise deployment of LLM-based applications remains nascent. LLMs have plenty of shortcomings, including hallucinations, bias, and data exfiltration. There are also a host of adoption challenges, including the fact that most company data needs significant preprocessing before it can be used by a model, and use cases that worked in demos quickly break down when scaled up to production.

As founders begin to deploy LLM-based applications at scale, they should look to leverage three innovations:

1. Multimodal models are moving beyond text, opening up new use cases. 
2. Multi-agent systems transform our ability to automate complex tasks.
3. New model architectures address the limitations of transformers. 

Here we’ll explore how all three will drive transformation across diverse sectors.

Multimodal models are moving beyond text

Multimodal AI means that the inputs and/or outputs can be text, images, audio, and video. OpenAI recently rolled out its GPT-4o model which does end-to-end processing of multimodal data within the same model, leading to significant advances in performance. One day after the OpenAI announcement, Google rolled out its own multimodal model, and Anthropic did the same last week.

All of this moves AI closer to human-like cognition and brings the physical world closer to the digital one. Think of multimodal models as a supercharged Siri or Alexa, where the model can incorporate context clues from audio (background noise, emotion, or tone) along with visual information (things like facial expressions and gestures). By integrating multimodal data, these models are generating richer insights, understanding not only a human’s words, but also those words’ nuanced meanings—and then using that understanding to respond to complex scenarios.

Multimodal capabilities open up a new realm of industry use cases. Customer service chatbots get a boost from faster and more accurate voice agents, resulting in more natural-seeming interactions. In healthcare, where diagnoses typically rely on a mixture of data types, new applications can have a huge impact. Color Health is working with OpenAI on a copilot app that uses GPT-4o in cancer patient treatment plans. The copilot extracts and normalizes patient information found in disparate formats, such as PDFs or clinical notes, identifying missing diagnostics and generating a personalized cancer screening plan. Beyond healthcare, multimodal AI will shake up fields that regularly deal with diverse data from both the physical and digital worlds, areas like robotics, e-commerce, and manufacturing.

The possibilities for content analysis and generation use cases are also extensive. Multimodal models can intake multiple content formats (videos, podcasts, presentations) and create outputs that combine audio, text, and visual elements. This could be useful in: 

• Content repurposing: Marketing teams might quickly remix existing content into new formats or campaigns.
• Multilingual adaptation: Converting existing content to multiple languages for global distribution.
• Enhanced training: Creating immersive role-playing scenarios and interactive simulations for employee onboarding and skill development.

Multi-agent systems

While multimodal models expand the input and output capabilities of AI, combining multimodal models with agentic systems transforms our ability to automate complex tasks and processes. 

So what is a multi-agent system? Such a system takes an overall goal, breaks it down into a series of steps, executes that sequence transferring output from one step to the next, and ultimately combines outputs to achieve an outcome. The system has agency because it plans and initiates actions independently based on its assessment of a given situation. An agent (or several) will make the decision to call multiple interacting systems and possibly even check in with a human if required. 

Where an AI model generates output in a single pass, an agent is different in that it follows a process akin to humans approaching challenging goals. Agents gather information, draft initial solutions, self-reflect to identify weaknesses, and revise for optimal output. While model performance has improved exponentially, models by themselves are still akin to Q&A bots; human inputs a question, model answers a question (though the questions are increasingly difficult). Agents are a new paradigm; they resemble human beings, in that they take on projects with end-to-end accountability. Agents transform AI from a passive tool into an active team member. 

Agents are a new paradigm. They transform AI from a passive tool into an active team member.

Recent research has demonstrated the power of collaborative agentic systems. Microsoft’s AutoGen project has shown that multi-agent systems can significantly outperform individual agents on complex tasks. In a coding challenge, Andrew Ng’s team tested an LLM alone vs an agentic system on the HumanEval benchmark (common for evaluating the functional correctness of code). With prompting, GPT-3.5 logged a 48% accuracy rate. When the LLM was wrapped in an “agent loop”—where agents used a number of techniques, including reflection (the LLM examines and improves its own work) and planning (the LLM devises and executes a multistep plan)—GPT-3.5 was 95% accurate. The dramatic improvement underscores the potential of multi-agent systems to tackle complex problems that stymie single-model, or even single-agent, approaches.