Why AutoGen Is So Important

What I'm playing with right now...

Oct 23, 2023

Since there’s no AI news video this week, I wanted to take the time to talk about what I’ve been experimenting with lately. I’ve mainly been playing with two main projects: AutoGen and MemGPT. I’ve already published multiple videos about AutoGen, with more to come, and just this morning, I posted a video about MemGPT. What excites me is the potential for bringing the tech behind these two projects together.

AutoGen

This is the project I’m most obsessed with right now. For those who haven’t followed my YouTube channel closely, AutoGen is a new project from Microsoft Research that allows developers to build multi-agent teams to accomplish almost any task. These AI agents can write and execute code, critique each other, plan, embody different roles and personas, and more.

Automated Iteration

But why is having multiple agents work together so important? Can’t I do any of that with ChatGPT directly? Yes, you can. However, with just a little testing, you’ll quickly realize that working with a single AI produces hallucinations, incorrect code, and not following directions properly. When that happens, the typical next step is for the human user to provide feedback to the AI, which allows it to understand its mistake and correct it. That iterative process is crucial but implemented by a human. However, having the human be the only feedback provided is not completely necessary. With AutoGen or any multi-agent framework, you can enable another AI to give feedback, shortening the feedback loop and enabling less work from the human user.

This feedback loop is especially prevalent when using AI to write code. The first attempt at writing code frequently contains errors or bugs, but executing the code is the only way to find those issues. While this was traditionally done by the human user, now, with AutoGen, another AI agent can execute that code on behalf of the human user. When errors arise, the AI agent can feed that error back into the original AI and ask it to update the code to address the issue. Now, that entire feedback process is entirely automated and fast.

Code Execution

One of the most essential elements of AutoGen is its ability to execute code in a containerized environment. This is essentially what Code Interpreter from ChatGPT is, but this is entirely free and open source. Also, with multiple AI agents working together, they can use tools like Python libraries.

Let’s say you give AI agents the task of creating a TSLA 1.49%↑ stock price chart. The agents will attempt to do so using Python, and they will quickly realize they need a charting library, which they will then install into an isolated environment and then use to write the chart! It all happens seamlessly and in an automated way, with the user only giving feedback when necessary.

It can also reuse code it has successfully run, with multiple options for accomplishing that. AutoGen has built-in caching, which helps reduce the cost of using ChatGPT to power AutoGen. But of course, you can also use open-source models, which I’ll touch on below.

AutoGen also has dockerization built-in, all you need to do is install the Docker Python module, and it’s turned on by default. They recommend you use Docker, which is understandable since you’re allowing AI to write and execute code on your server otherwise!

Drop-In Replacement For ChatGPT API

One highly overlooked (at least by me) feature of AutoGen is its “drop-in replacement” ability for the ChatGPT API. I didn’t understand why this was so valuable at first, but now it makes a lot of sense. It’s also odd because it doesn’t seem related to AI at all, but they included it as a significant feature. I guess that they realized after building the agents portion of AutoGen that they just happened to build a lot of fantastic functionality on top of the ChatGPT API.

Even if you don’t plan to use AutoGen to build AI agents or agent teams, the wrapper around ChatGPTs API is worth using alone. For example, it has built-in retry logic, caching, model fallback, multi-model support, goal setting, budgeting, and more. And, you don’t only have to use OpenAI models, you can easily spin up a local service running an open-source model and insert that into the daisy chain of models in your AutoGen application, more on that in the next section.

Open-Source Models

One of the nicest features of AutoGen is the ability to use open-source models. You also don’t have to choose between open-source and closed-source, you can instead assign different agents to use different models, giving the power of GPT4 to the most complex use cases and less sophisticated models to other use cases.

As much as I love open-source support, AutoGen can be a challenging project for open-source models to power. This is due to AutoGen’s extreme complexity and leveraging of function calls, which tend to work best using GPT4 (not even GPT3.5 works well with function calls).

However, I have confidence that open-source models will continue to get better, and specific fine-tuned models will arise that are excellent for narrow use cases, which would fit perfectly with AutoGen’s ability to assign models to specific agents to particular tasks.