Client Side Tool Calling with the OpenAI WebRTC Realtime API

OpenAI recently released the WebRTC interface for their realtime API. It supports tool calling, which means you can use it to do client side JavaScript function calling. This opens up a whole new set of use cases.

In this video, Craig shows off how he used Cloudflare Worker to relay the WebRTC call securely. Then he uses function calling to control the style of a web page. And even a Bluetooth enabled robot hand 🤖🖐️!

Create an account on Cloudflare today for free: https://dash.cloudflare.com/sign-up

Tools mentioned:
Repo: https://github.com/craigsdennis/talk-...
https://developers.cloudflare.com
https://developers.cloudflare.com/wor...

Chapters:
00:00 Intro
00:27 Tool Calling Explanation
01:07 Exploring the WebRTC Relay code
01:24 Demo: Change the colors of a web page with your voice
02:26 Console.logging the magic
02:42 Demo: Retrieve HTML from the current webpage
03:12 Discuss Web Bluetooth BLE
03:33 Demo: Control the robot hand over Web Bluetooth
04:11 Demo: Play a game allowing the AI to use the Robot Hand to answer
04:58 Browse the code
06:18 Wrap and what's next

#cloudflare #cloudflareworkers #developer