ChatGPT Plugins: Early Observations and Learnings
A new distribution channel is coming to software and how building ChatGPT Plugins has provided me some obvious and not so obvious insights.
Since its release, ChatGPT has taken the world by storm. I’m sure that pretty much anyone reading this has at least played around with the tool somewhat. There’s probably even a decent portion of you that use it now to do a number of tasks for your job1.
The more you use the tool, however, the more you start to bump up against some of its limitations. The largest one being that it’s not really connected to anything (the internet, your documents, your email, etc.). Of course, it was only a matter of time before that changed. ChatGPT is now starting to introduce ChatGPT Plugins to do exactly that, connect external programs to ChatGPT.
Plugins not only unlock a number of interesting use cases that will be enabled through ChatGPT (check out this demo video from its CTO on how you can use some of these plugins), but they also present an interesting business and market opportunity.
Anyone who’s tried to build a business knows (or eventually figures out the hard way) that distribution is massively important. The saying people like to use about this is that the “build it and they will come” attitude just doesn’t work. You can build the best product in the world, and unless people know how to find it or how you can get it in front of people’s faces, no one’s going to use that product. This also helps explain why some of the most popular products in the world can seem, for lack of a better way to put it, low quality. Because they have great distribution, along with other factors like high switching costs, network effects, economies of scale, etc., there’s less of a need to focus much on product quality2.
Anyways, the reason for me saying all of this is that great distribution channels don’t come around very often. The reason for this is because they suffer from the classic chicken and egg problem: you need a large audience to distribute to before you can get developers to create apps to distribute. So often these distribution channels are built off the backs of products that themselves have built a large audience. In software, some of the best examples of these distribution channels are the iOS and Android app stores, Microsoft’s Windows developer platform, and Shopify’s App Store. But there are other examples that can be of high value too, depending on your target audience3.
So that brings us back to ChatGPT’s Plugins. You can already see how this could be another massive distribution channel for software products. By doing some quick back of the napkin math, if you take the 100 million users that ChatGPT has gotten since its launch (and this could already be much larger), even if you think there’s only 10% of those users that are actively using ChatGPT, and even if you think 1% of those users will even care or find your product useful, you’re still talking about 100,000 people. Given that it’s also early days in the ChatGPT Plugin history (most people don’t even have access to them today, since they are invite only), that distribution channel is likely to have low competition initially, just because the number of plugins will be low in the beginning4.
So, with all this in mind, I’ve been pretty intrigued by what you could potentially do with plugins. Since I’ve gotten early access to develop plugins, I’ve begun to play around with what you might be able to do with them, just to spur some ideas. From this, I’ve had a few observations so far, some are obvious, and some were a bit surprising, or at least to me, a new way to think about things.
The first is the obvious stuff. There are going to be a whole host of plugins that come that are basically going to be a flavor of: “connect my personal/business/third-party data to ChatGPT”. In fact, I’d be shocked if this wasn’t the vast majority of plugins, especially in the early days. OpenAI even created an open source repository which provides you the tools to connect any set of files to ChatGPT (check it out here if you’re curious).
With this in mind, and coming from the Shopify ecosystem myself, I was curious how easy it would be to hook up to Shopify stores and see what you could do with it. Turns out, pretty easy. You can see the demo video below and I’ve also open sourced the code to do this if you have a Shopify store yourself here.
While it’s nothing to write home about, I think it demonstrates some of the things we’ll see be possible in ChatGPT very soon. In particular, I think it’s the combination of plugins that provides most of the magic to these plugins. In the demo video, I tried to show this by connecting Wolfram Alpha’s plugin to the Shopify plugin to show how you can use the two together. Ask Shopify for your store’s recent orders and run a few calculations on them using Wolfram Alpha. You could also then craft a customer email with ChatGPT’s assistance and send it to them using Zapier’s plugin. And that’s just a basic demo, a lot more is possible here.
The less obvious stuff I discovered when I tried building a plugin for a software product that I created a few years ago for myself called Decision Journal. Without going too much into the details on what Decision Journal is (I could nerd out about decision making for many hours if you wanted), the high-level summary is that it is a way to record and review the decisions you make in an effort to improve your decision making over the long run5.
The ChatGPT plugin idea was to connect your decision journal to ChatGPT so that you could have it record and review your decisions, all without having to leave ChatGPT. Another advantage of it would be that you could use ChatGPT as a helpful guide in making decisions, all while also recording them into your journal for later review. And like the Shopify plugin, you could also use it alongside other plugins (for example, if you were deciding to buy a particular stock, you could use Wolfram Alpha’s plugin to pull market data for you).
And while I finally got it closer to where I had envisioned it (check out the demo video below), there were a few things that I hit upon that I thought were interesting. If you don’t like technical discussions, feel free to skip to the end.
The first observation was that ChatGPT makes a lot of assumptions with how the plugin’s API works. I’m not familiar with how they trained the model or how it works exactly behind the scenes but I noticed a few different things. The first was that even if I provided descriptions for all API fields for the plugin in the docs, the plugin basically completely ignored these descriptions and instead made some assumptions about the fields based on their names.
For example, there’s a feature in Decision Journal that allows you to estimate the role of skill vs luck in a decision. In the API, you can set this information using the field “skillLuckWeight”. In my description of this field in the API docs, I make note that it should be a number between 0 and 1, with 0 being all skill and 1 being all luck. It’s kind of nuanced so I thought having an explanation would help guide the model.
Well, turns out it basically ignores the description and instead tries to make assumptions with how the field works based on its name alone. This is fine and a bit clever to be honest (most of the time it just works as expected), except that it sometimes thinks it should be 0 to 100 and reverses the weights with 0 being all luck and 1 being all skill. That could be problematic if you think it is the opposite. And this could happen on any number of fields. I’m sure it will get better over time, but it is one potential thing that may require rethinking how you name things in your APIs6.
This leads to another observation. When the bot goes a bit beyond what the API is capable of, the API returns an error. And with so many assumptions being made about how the API works, this happens a decent amount. But much to OpenAI’s credit, it can correct itself if it knows how. And that’s the key. Having instructions for a natural language model in your API errors, goes pretty far.
Take the example above, where ChatGPT sends back a score of 90 for the skillLuckWeight rating. You could handle this error in a lot of different ways. However, through iteration, what I came to realize was that you could also use these error messages to your advantage and help guide ChatGPT to the way your API expects things. So, in this example, instead of sending back something like “Invalid skillLuckWeight, should be a number between 0 and 1” (which would be logical and help it correct itself), if I returned “Invalid skillLuckWeight, should be a number between 0 and 1, 0 means the outcome is all skill and 1 means all luck” ChatGPT learns how to use the field in the intended way.
And that leads to the last observation that I’ll share, which is that I think we’re moving to a world away from structured and overly technical APIs to one where most APIs are essentially just natural language inputs and outputs. The errors example above gives one way this could play out, but if I wanted to avoid this error in the first place, I really shouldn’t even have a field called “skillLuckWeight”, since that might confuse a natural language AI and make it unclear about how it works. Instead, I should just have it send the information in a more natural language way and have my own system parse it and store it how it needs to be.
So, those are just some of the thoughts and observations that I’ve had since I’ve begun to play around with ChatGPT plugins. If anyone reading this has any other observations that you’ve made or insights that you’ve gained, especially if they are different from my own, I’d be really curious to hear them. And if you have access to plugins right now and want to try any of the above, let me know, happy to share them with others.
Wrap up
We seem to be entering a period that likely doesn’t happen that often: a new uncrowded distribution channel with a huge audience. That means a ton of opportunity and potential for software developers and businesses alike. It also means there will be lots to learn and iterate on.
Looking forward to seeing what it all brings about.
If you’re not, you’re probably not using it to its fullest capabilities.
Yes, this does hurt me deep down as a builder and product person, but it’s reality.
One of these from personal experience is Google’s Workspace Marketplace. Though not often realized unless you’re in the business of selling software to businesses that use Google’s Workspace products, the Workspace Marketplace is a huge opportunity to get your software in front of businesses. I know this first hand because I built and sold a product that was exclusively built for Google Sheets and distributed through the Marketplace. And there are a few other distribution channels that are similar, like Zoom’s Marketplace or Salesforce’s AppExchange.
To put it more concretely, there are 12 plugins available today. Even if there’s a huge backlog of plugins that have been built or are being built, I can’t imagine there will be more than a few 1,000 at most early on. Compare that to something like the iOS app store, which has over 2 million apps in its app store.
If you’re interested, you can read more about decision journals and their benefits here: https://fs.blog/decision-journal/. You don’t have to take it from me :)
Good naming is surprisingly difficult, which now seems to be exacerbated with natural language understanding AIs, if only I had paid attention in 7th grade English class!