View All Posts
Want to keep up to date with the latest posts and videos? Subscribe to the newsletter
HELP SUPPORT MY WORK: If you're feeling flush then please stop by Patreon Or you can make a one off donation via ko-fi

I’ve been creating ChatGPT plugins and started thinking - wouldn’t it be fun to connect this to a Raspberry Pi and control some hardware? Turns out, it’s very easy to do.

You can watch a video of this article here:

ChatGPT plugins are surprisingly simple. They are just standard Web APIs - no different from any other API you might write or call. My particular plugin is really basic. It has just two endpoints.

One returns a list of lights. The other toggles a light on and off.

I’ve set up my little demo with 5 lights:

  • Kitchen
  • Bedroom
  • Dining table lamp
  • Bathroom
  • Lounge

It’s worth watching the video as it’s pretty cool. We just say to ChatGPT, “let there be light,” and it switches on all the lights connected to my Raspberry Pi.

All the lights turn on

What’s really impressive is that the only thing ChatGPT has to work with are a few hints describing the plugin in free text and the OpenAPI yaml file. From that, it figures out that it needs to first get the list of lights, and then it needs to loop through the lights and toggle them on.

ChatGPT Sequence Diagram

Now, I say “figured out” but I’m not going to claim that it’s thinking or reasoning about the world - that would be a big stretch. But I am going to claim that it’s pretty incredible.

We can even run through a really simple home automation scenario.

  • I’ve just woken up - it turns on the bedroom light
  • Time to shower - it turns on the bathroom light and turns off the bedroom light
  • Time for breakfast - it turns on the kitchen light and turns off the bathroom light
  • I’m off to work - it turns off all the lights
  • I’m back from work with takeout - it turns on the dining table light
  • Time to relax - it turns on the lounge light
  • Time for bed - it turns on the bedroom light

I then tried something a bit different. I pretended to be another device in the bedroom that could detect sounds and told ChatGPT that I could hear snoring. It then turned off the bedroom light and wished me goodnight.

Now obviously, this is a really straightforward demonstration, but to me, the possibilities are quite mind-boggling.

I could see large language models being an interesting way to connect a whole bunch of different systems and services together. There are a ton of poorly designed APIs out there, and integrating them is a nightmare.

Wouldn’t it be funny if we ended up with computers talking to each other using human language instead of APIs?

Brave New World

One of the things that I really enjoyed about this project (apart from the fact that it worked!) was that I didn’t have to write much code at all - I was able to get ChatGPT to generate all the code.

This is the prompt I used:

Write me a Python Flask API to run on a Raspberry Pi and control some lights attached to GPIO pins.

I have the following lights:

Kitchen: pin 6
Bedroom: pin 13
Dining table: pin 19
Bathroom: pin 26
Lounge: pin 5

I want the following endpoints:

get lights - returns the list of lights along with their current state
post toggle_light - switches a light on or off

I asked it to make a Flask API - I used this as I’m most familiar with it so I could check what ChatGPT outputs. As always with these Large Language Models, it’s definitely a case of “Trust, but verify”.

I told it that it was running on a Raspberry Pi and that there were lights connected to GPIO pins.

And I gave it the list of lights along with the pin numbers.

I then gave it a very vague spec for the endpoints I wanted.

With that basic spec, I was able to get ChatGPT to generate the code for me. I added on a bunch of bells and whistles afterwards, but the core of the API was generated by ChatGPT.

from flask import Flask, request, jsonify
import RPi.GPIO as GPIO

app = Flask(__name__)

# GPIO setup

# Define your lights
lights = {
    "Kitchen": {"pin": 6, "state": GPIO.LOW},
    "Bedroom": {"pin": 13, "state": GPIO.LOW},
    "Dining table": {"pin": 19, "state": GPIO.LOW},
    "Bathroom": {"pin": 26, "state": GPIO.LOW},
    "Lounge": {"pin": 5, "state": GPIO.LOW},

# Setup pins
for light in lights.values():
    GPIO.setup(light['pin'], GPIO.OUT)
    GPIO.output(light['pin'], GPIO.LOW)

@app.route('/lights', methods=['GET'])
def get_lights():
    return jsonify(lights)

@app.route('/toggle_light', methods=['POST'])
def toggle_light():
    data = request.get_json()
    light = data.get('light')
    if light not in lights:
        return {"error": "Light not found"}, 404

    # Toggle light
    lights[light]['state'] = GPIO.HIGH if lights[light]['state'] == GPIO.LOW else GPIO.LOW
    GPIO.output(lights[light]['pin'], lights[light]['state'])

    return {"success": f"{light} toggled"}, 200

if __name__ == "__main__":'', port=5000)

The other thing that is needed for a ChatGPT plugin is an OpenAPI or Swagger file - these are tedious to write - and it has been pointed out to me that if I’d used FastAPI, I wouldn’t have needed to write one at all. But one of the really nice things you can do with ChatGPT is paste in your Flask code, and it will generate an OpenAPI file for you.

I’m just running this plugin locally, I don’t have any plans to publish it as that would require a lot more plumbing. For developing local plugins, you need to have a server running on localhost. Fortunately, we can use the excellent VSCode support for remote development. I’m connecting through to the Pi from my desktop machine, and when I launch the server, VSCode tunnels through to the Raspberry Pi and exposes the server on my local machine.

ChatGPT looks at the /.well-known/ai-plugin.json and uses that to get the name, description, logo, and OpenAPI file. That’s all it has to work with to figure out what to do with the API.

We can get some insight into how it works under the hood. We can just ask ChatGPT to show us its thinking. If we paste in the OpenAPI yaml file and ask it to explain how to turn lights on and off, we get this:

ChatGPT Explain

We can even get it to generate a sequence diagram for us:

ChatGPT Sequence Diagram

It’s pretty amazing. Now go and watch the video:


Related Posts

Do you need a ChatGPT plugin? - We've seen two major shifts in technology trends with websites and mobile apps - now there's a third one rearing its head. OpenAI's ChatGPT with plugins is on the cards and you better be ready for it. In the midst of fumbling for answers to whether we need these plugins or not, let me reassure you that it's not too complex. Far from requiring a squad of specialist developers, all you need to know is how to make an API to create a plugin for ChatGPT. Yes, there are potential pitfalls around security and data protection, but with the right precautions, you will be fine. So, dear developer, explore, experiment and gear up for this exciting phase!
Using ChatGPT As a Co-Founder - In the quest to explore the capabilities of ChatGPT, I decided to utilized it as a startup brainstorming partner. From product description to building the product on AWS and GCP, crafting an elevator pitch, highlighting the ideal customer profile, sketching a business plan, and even generating a logo idea, ChatGPT has been surprisingly helpful and creative. We even explored potential team structures that can bring the business to life! Turns out, ChatGPT might just be the co-founder you never thought you needed.
Cocktail Bot Using Official ChatGPT API - In this post, I cover the migration of our cocktail bot to use the newly available ChatGPT API. Prompt engineering is performed to accurately guide the bot's versatility in cocktail recommendations. A Python application is provided via GitHub for testing the chatbot. To ensure appropriateness in user inputs, a moderation system from OpenAI’s API has also been integrated. However, readers are reminded to consume the AI's cocktail suggestions responsibly and remember that it's just a machine- its advice could potentially lead to less than satisfactory results.
It's Plausible, But Is It True? - Here's a wild ride with ChatGPT and other large language models! They're ace at cooking up plausible-sounding text, but they're not always the best when it comes to spitting out the truth - they've got a funky relationship with facts. One research paper showed they can come out with believable but totally fake answers to seemingly straightforward facts. But when I messed around with various models, there were a few discrepancies. Some got it right or plausibly wrong, but we humans are pretty gullible and tend to believe plausible-sounding info. So when it comes to using ChatGPT, make sure you fact-check, stay away from complex reasoning tasks, and don't try and solve maths problems - seriously, just stop. But it's a cracking tool for generating marketing copy, code (with a fact-check), finding bugs, and getting the creative juices flowing. Stay tuned though - tech's ever-evolving and these intelligent library computers aren't going away anytime soon!
Why does ChatGPT make mistakes - a layman's explanation - In this enlightening blog post, I dive into the tantalizing world of ChatGPT and Large Language Models. Clarifying its operation, I unlock this enigma by comparing its mechanisms to a simple language model. However, Challenges arise due to the explosion of possible token combinations, leading to an inherent 'lossy' compression of our world's vast information. Surprisingly, even with such compression, these models can mimic human language in a compelling manner. I also investigate possible strategies to optimize this amazing technology - including zero-shot learning, one-shot learning, few-shot learning, and fine-tuning. Entering the era of prompt engineering and larger models, we're stepping into a thrilling future, so buckle up, folks!

Related Videos

AI Powered Raspberry Pi Home Automation - Is this the future? - Witness the power of ChatGPT controlling home automation lights through a Raspberry Pi, making life easier with plugins. Delve into the fascinating world of large language models, redefining interactions with APIs.
Unlocking the Power of ChatGPT: Effortlessly Generate Arduino Code for Your Projects! - Witness ChatGPT's impressive potential for generating working Arduino code, as demonstrated in a step-by-step ESP32-based project utilizing a potentiometer and dot star LED.
Automating Blog Improvements with AI: Summaries, Tags, and Related Articles - Learn how to use ChatGPT to enhance your blog's homepage, create summaries and tags, find related articles, and generate post images with ease, leveraging AI to save valuable time and effort.
ChatGPT vs Stockfish: Can an AI Plugin Improve its Chess Game? - Watch as chat GPT takes on Stockfish, a world-class chess engine, in a thrilling match! See how GPT utilizes a chess plugin to improve its gameplay and compete against the best.
Magic LEDs: Self-Organizing with ESP32 CAM & Simple Image Processing! - Unleash your creativity and transform chaos into order using an ESP32 camera board, image processing, and a string of WS2811 RGB LEDs. Find out how wiring, level shifting, and a simple web interface bring this mesmerizing project to life.
HELP SUPPORT MY WORK: If you're feeling flush then please stop by Patreon Or you can make a one off donation via ko-fi
Want to keep up to date with the latest posts and videos? Subscribe to the newsletter
Blog Logo

Chris Greening


> Image


A collection of slightly mad projects, instructive/educational videos, and generally interesting stuff. Building projects around the Arduino and ESP32 platforms - we'll be exploring AI, Computer Vision, Audio, 3D Printing - it may get a bit eclectic...

View All Posts