Do you want an AI agent to do something to an iPhone? You could try to jailbreak the device and hack heavy software, or… you could grab a microcontroller off the shelf and let it do the hard work. [Henry Yuan] explains how to achieve this with an ESP32. The first step is to get an AI agent to access the current state of the phone screen, which [Henry] It achieves this with AirPlay and UxPlay screen mirroring. A Gstreamer pipeline is then used to send the screen as a JPEG sequence to the AI agent of your choice. Next comes control, which is achieved using an ESP32-S3 that acts as a USB-HID device, emulating both the USB mouse and keyboard. With iOS AssistiveTouch enabled, this gives the agent control over inputs to the device. From there, it is up to the AI agent to determine the actions to take to achieve the desired behavior, which is obviously very specific to the use case. It may seem a little complicated, but this approach has benefits. You do not need to have an Apple, XCode, or Macintosh developer account. You can have your agent work over the phone without those annoying overheads. Lately we have also featured other great iPhone tricks. Video after the break.
