View All Posts
Want to keep up to date with the latest posts and videos? Subscribe to the newsletter
HELP SUPPORT MY WORK: If you're feeling flush then please stop by Patreon Or you can make a one off donation via ko-fi

Lots of LEDs? It’s not Christmas yet!

I had a big bundle of addressable WS2811 LED strings and an ESP-CAM board (an ESP32 dev board with a camera). There’s only one possible project that you can do with these components. Turn the disorganised chaos of lights into something a bit more organised.

As an added bonus I’ve ended up duplicating the image processing code in JavaScript so you don’t even need a camera on your ESP32 board - you can just use a plain dev board to drive the LEDs.

You can see the results of my efforts in the video below and I’ll run through a bit more detail of the code in the following text.

The full sample code can be found here:

If you want to do this yourself then you will need an ESP32 dev board of some kind and of course you’ll need some kind of addressable LEDs. I’m using the FastLED library for driving the LEDs so with some small code changes you can probably support pretty much any addressable LEDs.

ESP32 and ESP-CAM Boards

Our challenge comes down to a very basic problem, given access to a stream of images from a camera identity the approximate locations of each LED in 2D space. Once you’ve done that it’s a simple problem to map from each LED’s x and y location onto a frame buffer containing the pattern you want to show.

There’s a bunch of boiler plate code to initialise the ESP-CAM - I took inspiration from the sample code here and copied the bits I needed to get the camera up and running.

An important change I’ve made is only capture greyscale images at the lowest framesize:

config.pixel_format = PIXFORMAT_GRAYSCALE;
config.frame_size = FRAMESIZE_QQVGA;

And then to grab a frame from the camera we simply do:

    camera_fb_t *fb = esp_camera_fb_get();
    Frame *frame = new Frame(fb);

With our Frame class grabbing a copy of the pixels along with the width and the height of the image.

pixels = (uint8_t *)malloc(fb->height * fb->width);
memcpy(pixels, fb->buf, fb->height * fb->width);
width = fb->width;
height = fb->height;
length = fb->width * fb->height;

The image below shows a frame grabbed from the ESP-CAM sensor.

Grabbed Frame

For the JavaScript version of this code it’s a bit more complicated. One of the biggest problems is that we need to be running over HTTPS to have access to the camera - more on this later….

const stream = await navigator.mediaDevices.getUserMedia({
  video: { facingMode: "environment" },
  audio: false,
const canPlayListener = () => {
  // the video is loaded and we can grab frames
  video.removeEventListener("canplay", canPlayListener);
video.addEventListener("canplay", canPlayListener);
video.srcObject = stream;;

Once we have a video stream coming from the camera we can grab a frame by drawing the video to a canvas context and then getting the imageData from it.

function getVideoFrame(video: HTMLVideoElement, canvas: HTMLCanvasElement) {
  const width = video.videoWidth;
  const height = video.videoHeight;
  const context = canvas.getContext("2d");
  // draw the video to the canvas
  context!.drawImage(video, 0, 0, width, height);
  // get the raw image bytes
  const imageData = context!.getImageData(0, 0, width, height);
  // convert to greyscale
  const bytes = new Uint8Array(width * height);
  for (let y = 0; y < height; y++) {
    for (let x = 0; x < width; x++) {
      const r =[(y * width + x) * 4];
      const g =[(y * width + x) * 4 + 1];
      const b =[(y * width + x) * 4 + 2];
      const grey = Math.min(255, 0.299 * r + 0.587 * g + 0.114 * b);
      bytes[y * width + x] = grey;
  return bytes;

Phone Interface

Now we can grab frames we just need to grab a frame with no LEDs lit, light one LED, grab another frame and then compare the two. The difference should tell us where the LED is. To avoid noise or small movements of the camera having a bit impact we apply a guassian blur to the captured frames before taking the difference.

This is a of course a very naive and simple algorithm and could easily be improved on.

In the C++ code of the ESP32 we do all this directly in code. In the JavaScript version we call API functions on the web interface of the ESP32 to turn LEDs on and off and once we’ve finished the processing send up the calculated positions of the LEDs to the board.

In our ESP32 code we create a framebuffer and draw patterns into it. We then use the locations of each LED to work out what color it should be.


To solve the issue of needing HTTPS to access the camera and also needing the API calls to be HTTPS as well (we can’t mix content nowadays!) we need a way of serving both the UI and the API from the ESP32 web server over HTTPS. There are web servers that support HTTPS and self signed certificates available for the ESP32 but this leads to other problems and would require a rewrite of the device code. An easy workaround to this problem is to use a service such as ngrok to provide a secure URL from the cloud through our computer to the ESP32 device. Slightly convoluted, but it works!

This in only needed if you are not using an ESP-CAM and have to use your phone’s camera for calibrating the LEDs. Sign up for a free acount with ngrok and then find the IP address of your ESP32 board:

ping espcam.local
PING espcam.local ( 56 data bytes
64 bytes from icmp_seq=0 ttl=255 time=14.343 ms
64 bytes from icmp_seq=1 ttl=255 time=6.493 ms

Take the IP-Address and ask ngrok to start proxying requests for us:

ngrok http -inspect=false

You’ll need steady hands - there’s quite a lot of latency going on so the space between turning an LED on and off and grabbing a frame can be quite large. I slightly moved as I was taking the frame to show the locations so the positions are slightly off.

LED locations

Checkout the video to see how well it works - surprising for such a simple algorithm.


Related Posts

The PCBs are in production - what have I messed up? - After some stress and trepidation, I finally took the plunge and sent my PCB design off for manufacturing. My design centers around building a large seven-segment clock with LED filaments. Jumping hurdles such as voltages, pin usage, and limiting the load on my power supply, I've settled on the ESP32 as the system's heart and come up with a final circuit design. While doing this, I've quickly realized I could improve my layout and fixed a small mistake. Also, I've prepared for either types of LED filaments - the high-voltage ones or the larger, 3v ones. However, I did bungle up a couple of things on the enable line of the shift registers and board layout. But hey, this is a learning curve, right? Can't wait to get the boards and see what other exciting errors surface!
Decoding AVI Files for Fun and... - After some quality time with my ESP32 microcontroller, I've developed a version of the TinyTV and learned a lot about video and audio streaming along the way. Using Python and Wi-Fi technology, I was able to set up the streaming server with audio data, video frames, and metadata. I've can also explored the picture quality challenges of uncompressed image data and learned about MJPEG frames. Together with JPEGDEC for depth decoding, I've managed to effectively use ESP32's dual cores to achieve an inspiring 28 frames per second. Discussing audio sync, storage options and the intricacies of container file formats for video storage led me to the AVI format. The process of reading and processing AVI file headers and the listing subtype 'movi' allowed me to make significant headway in my project. All in all, I'm pretty chuffed with my portable battery powered video player. You can check out my code over on Github!
Minimalist Microcontroller: Building a Bare-Bones Dev Board - In a thrilling DIY endeavour, I attempted to build the most minimalist ESP32 dev board possible. Diving deep into the schematic of the ESP32 S3 WROOM module, I chopped out the non-essentials and whittled our needs down to bare bones. The experiment saw me juggling USB data lines and voltage regulators, waving goodbye to an array of capacitors and connectors and boldly embracing the simplicity of direct connections. Despite a few hitches, the miniature Frankenboard came alive, proving that sometimes less is least in the world of microcontrollers.
A Faster ESP32 JPEG Decoder? - An intriguing issue appeared in the esp32-tv project that deals with speeding up JPEG file decoding using SIMD (Single Instruction Multiple Data) instructions, showing immense performance boost. However, there were some notable differences in speed when it comes to drawing the images versus simply decoding them. The problem was found to be with the DMA drawing mechanism and the way the new fast library decodes the image all at once. But despite this hiccup, by overlapped decoding and displaying process, a high frame rate can still be achieved. Joined me in this dissecting process and my initial tests showing approximately 40 frames per second display rate, on our journey to find the most efficient way to get images on screens.
ESP32-S3 Hardware SPI on the Adafruit ST7789 - I've had some commenters point out the issue with the slow display updates in my recent Arduino Nano ESP32 video. It turns out, the software SPI of the Adafruit_ST7789 library was the culprit. Lo and behold, the solution is simple - using the hardware SPI constructor of the library. Apparently, this isn't well documented, so I wrote some code to serve as reference for myself and others who might run into the same snags. Trust me, the difference in speed is absolutely bonkers. Check out the video to see the magic in action.

Related Videos

Magic LEDs: Self-Organizing with ESP32 CAM & Simple Image Processing! - Unleash your creativity and transform chaos into order using an ESP32 camera board, image processing, and a string of WS2811 RGB LEDs. Find out how wiring, level shifting, and a simple web interface bring this mesmerizing project to life.
Musical Light Show: Make Your Christmas Tree Dance to the Beat! - Transform your Christmas tree into an interactive light show using a string of 200 WS2811 addressable LEDs, self-organizing LEDs project, and an ESP32 to drive them. Learn how to map images onto LED locations and create amazing visual effects.
Custom ESP32 PCB Soldering - Watch the process of soldering and testing a custom ESP32 circuit board for a Moon Lamp project, featuring various components like a linear regulator, output drivers, and indicator LEDs.
ESP32 HTTP Web Server With Content Served from SPIFFS Filesystem - Learn how to create a simple ESP32 web server and user interface using basic HTML and JavaScript, and control an LED via an HTTP endpoint. This tutorial demonstrates the ease of setting up a web server on ESP32 devices and serving compressed content from SPIFFS.
Unlocking the Power of ChatGPT: Effortlessly Generate Arduino Code for Your Projects! - Witness ChatGPT's impressive potential for generating working Arduino code, as demonstrated in a step-by-step ESP32-based project utilizing a potentiometer and dot star LED.
HELP SUPPORT MY WORK: If you're feeling flush then please stop by Patreon Or you can make a one off donation via ko-fi
Want to keep up to date with the latest posts and videos? Subscribe to the newsletter
Blog Logo

Chris Greening


> Image


A collection of slightly mad projects, instructive/educational videos, and generally interesting stuff. Building projects around the Arduino and ESP32 platforms - we'll be exploring AI, Computer Vision, Audio, 3D Printing - it may get a bit eclectic...

View All Posts