Tap things by name
Say “tap Settings” and it finds the word on screen and taps it. No coordinates needed.
Give your AI a real iPhone.
TapLyne lets an AI agent see, tap, scroll and type on your iPhone. Start with a USB cable and Bluetooth, with nothing installed on the phone. Add the optional companion and encrypted remote setup for text and voice away from your Mac.
BluetoothTaps and keys to the iPhone
USBLive screen to the Mac
Agent
TapLyne MCP
YouOpen Settings and show me General.
list_phonesiPhone · online · unlockeddescribe_screenframe f_3a9c · 9 labelstap_label "Settings"expect “General”verified · “General” on screentap_label "General"expect “About”verified · “About” on screen
General is open on your iPhone.
You can stop it any time.
- USB carries the screen.Captured locally on your Mac.
- Bluetooth carries the taps.AssistiveTouch turns them into touches.
- No phone app in USB mode.No jailbreak required.
- You stay in charge.Pause or take over any time.
In one sentence
You tell your AI what you want done on your iPhone. It looks at the screen, taps, types and scrolls until it is done. You watch, and you can stop it any time.
It seesa live picture of your phone screen
It thinksusing the AI you already pay for
It actswith real taps, like a finger would
It checksthat each step actually worked
How it works
Start with the cable.
USB mode needs no iPhone app or jailbreak. Follow the setup guide for permissions, pairing and your first request.
-
1
Plug in your iPhone
Use a normal USB cable. Your Mac now sees the phone screen live.
-
2
Pair over Bluetooth
Your Mac acts like a mouse and keyboard for the phone. The setup card walks you through it.
-
3
Ask in plain English
Type or speak a request. Your AI does it on the real phone while you watch.
Hands
The touches your AI needs.
Taps, drags, scrolling and typing, plus shortcuts for common tasks.
Tap
Double tap
Triple tap
Press and hold
Swipe and scroll
Drag
Hold and drag
Type text
Scroll until found
It scrolls a list until the item shows up, then stops. It never scrolls forever.
Fill in forms
It works through fields one by one and checks their values when readback is available. Review the result before asking the agent to submit.
Go anywhere
Home, Back, the app switcher, notifications and Control Center are one command each.
Open an app by name
It finds an app on the Home Screen or in search, then opens a clear, unique match.
Wait for the right moment
It can wait for a word to appear, like “Sent” or “Loaded”, before moving on.
Eyes
It reads the screen, right on your Mac.
Actions return a fresh picture and the text the screen reader recognizes, with positions for each label.
- Aims like a person. It watches the pointer and corrects its aim before tapping.
- Finds app icons. A detected icon grid connects app captions to their tap targets.
- Never acts on an old picture. If the screen moved, it looks again first.
- English and Hebrew. English uses Apple’s built‑in reading. Hebrew is an optional download.
Honest results
It tells you when it’s not sure.
Ask for an expected result and TapLyne checks the screen. Each result carries one of three statuses.
It worked.
The thing you expected is on screen. Example: tap “Settings”, expect “General”. It’s there.
Maybe. Look first.
The tap went through, but the result can’t be proven yet. It checks before going on.
It didn’t work.
It says so, and tells you whether the tap reached the phone at all.
Check before retrying: results report whether input reached the device. Inspect an uncertain send, purchase or submission before asking the agent to try again.
You’re in charge
Three buttons. Always there.
The AI never has the last word on your phone. You do.
PauseStops all automatic taps right now.
Take overCancels everything queued. The phone is yours.
ResumeThe AI looks again, checks what changed, then carries on.
You can resume from the Mac or paired companion. The AI’s phone tools can pause or stop, but cannot resume control.
Brains New
Use the AI you already have.
Chat right inside the app. Pick who does the thinking.
Your OpenAI key
Paste a key once. It’s checked, stored in your Mac’s Keychain, and you can remove it any time.
Sign in with ChatGPT
Experimental in 0.3.0. Live sign-in still needs validation. Start with an OpenAI key or an existing Claude Code login for the verified setup path.
Claude Code
Use your existing Claude Code login. The built-in agent receives tools for the selected phone.
No shared keys are bundled. Your requests go only to the provider you choose.
Voice New
Just say it.
Talk to your phone’s new assistant like you would to a person.
- Dictation. Speak instead of typing. Works with any AI.
- Spoken replies. It tells you out loud when it’s done.
- Live AI voice. A real back‑and‑forth conversation, powered by OpenAI.
- Interrupt any time. Start talking and it stops, and the phone pauses too.
Live voice requires your own OpenAI API key. Spoken commands and background replies passed on a physical iPad over a cellular hotspot.
Memory New
Do it once. Run it forever.
Save the jobs you repeat. Fill in the blanks next time.
Saved workflows
Turn any request into a reusable recipe with blanks to fill in. A recent‑runs log shows how each one went.
A conversation per phone
Each phone keeps its own chat history, encrypted on your Mac.
Steer it mid‑task
Change your mind halfway? Send a new message and it adjusts while it works.
Never replays by surprise
If a run gets cut off, it waits for you. It won’t redo it on its own.
Anywhere New
Leave the cable. Keep the control.
Optional companion and remote control in 0.3.0. Your Mac stays online while you use the phone elsewhere.
Companion app for iPhone and iPad
Chat with your Mac’s AI by text or voice. Pair with its QR code or pairing link. Each pairing can reach only the device you picked.
Build and install the companion with Xcode. There is no App Store download in 0.3.0. Companion setup.
Remote mode, even on cellular
After advanced setup, a signed runner keeps screen capture and control connected through your encrypted relay. Unplugged control and background voice passed on a physical iPad using an iPhone’s cellular hotspot.
Requires Xcode, Apple development signing, Developer Mode, and a self-hosted WSS relay. Keep the device unlocked and Mac online. Reconnect locally if the runner is killed or the device reboots. Remote setup guide.
For builders
Plug in any AI agent.
TapLyne runs a small server on your Mac. Anything that speaks MCP or plain HTTP can drive the phone.
- MCP. One command connects Claude Code, Cursor or any MCP client.
- REST API. Simple routes with async jobs for long actions.
- Live dashboard. Watch the phone in your browser.
- Works beside Argent for simulator and app testing.
- Agent skill included. Teaches your agent the look, act, check loop.
- Connection guide
$ claude mcp add --transport http taplyne \
http://127.0.0.1:7788/mcp \
--header "X-API-Key: ••••••"
✓ Added taplyne
tools: list_phones screenshot describe_screen
tap_label scroll_to_item fill_form
open_app navigate type_text +14 morePrivacy
Your phone stays yours.
Reads on your MacScreen text is recognized locally. Remote device traffic is encrypted.
No trackingNo analytics. No TapLyne account.
Locked awayKeys live in Keychain. Chats are encrypted on disk.
Encrypted remoteRelay traffic is end‑to‑end encrypted with replay protection.
Your clipboardRestored after pasting, and never overwrites a newer copy.
Local by defaultThe server only listens on your Mac unless you allow your network.
Scoped accessCompanion keys reach one device only. New keys revoke old ones.
Open sourceEvery line is public under AGPL‑3.0. Read it, build it.
Your selected AI provider receives requests and screen evidence. Live voice sends audio to OpenAI.
Ideas
What people use it for.
Test your own iPhone app on a real device
Walk through a signup flow and report bugs
Change a setting you can never find
Check an order or delivery status
Fill in long forms, then let you review
Help a less techy relative from your Mac
Take screenshots of every screen in an app
Give an AI agent a phone for apps with no website
Get started
Your first request, step by step.
Start with USB mode. The getting started guide covers the first connection, AI setup and optional remote mode.
- Download TapLyne and drag it to Applications.
- Install the USB helper with Homebrew and Terminal:
brew install uv && uv tool install pymobiledevice3 - Plug in your iPhone, unlock it and tap Trust.
- Follow the setup card: allow screen capture, pair Bluetooth, turn on AssistiveTouch, calibrate.
- Choose your AI in Settings > Agent, then ask it to describe the screen. Use your OpenAI API key or an existing Claude Code login.
You need
- A Mac on macOS 15 or later (Apple silicon or Intel)
- An iPhone and a USB data cable
- Your OpenAI API key or an existing Claude Code login
- The phone unlocked while it works
TapLyne uses the “camera” permission only to receive the phone’s screen. It never turns on your Mac’s camera.
Questions
Good questions.
Is it really free?
Yes. TapLyne is free and open source. AI usage and optional relay hosting may cost extra.
Do I need to install anything on my iPhone?
USB mode uses the cable and Bluetooth, with no phone app. Optional remote control requires a signed runner. The text and voice companion is a separate Xcode build for iPhone and iPad.
Does it work on an Android phone?
No. TapLyne is built for iPhone and iPad.
Can the AI buy things or send messages without me?
An agent can tap Send, Buy or Submit when instructed. TapLyne does not add a universal approval screen. Review consequential actions and use Pause or Take over to stop further input; an action already delivered cannot be undone.
Why Bluetooth?
Your Mac pretends to be a mouse and keyboard. iPhone’s AssistiveTouch turns those into real taps. That is why nothing needs to be installed on the phone.
Can it type in other languages and emoji?
In USB mode, Hebrew, emoji and long text use Universal Clipboard and remain experimental. Remote mode supports exact field readback; Hebrew and emoji entry passed on a physical iPad. Fields that cannot be verified report that limitation.
Does my phone need to be unlocked?
Yes. Leave it unlocked while the AI works. TapLyne never handles your passcode or Face ID.
Is this made by Apple?
No. TapLyne is an independent project and is not affiliated with Apple.
Give your AI a real iPhone.
Version 0.3.0. Free and open source. Start with the guided USB setup.