Media Streaming over BiDi
mobile: startMediaStream sends the screen to the
client as WebDriver BiDi events instead of serving it over a
host port. It supports both jpeg frames (an MJPEG-like stream) and H.264/HEVC video with optional AAC audio,
needs no extra host tools, and works on Android 10 (API level 29) and newer.
- Start a session with BiDi enabled (
webSocketUrl: true) and subscribe to theappium:mediaStreammodule. - Call
mobile: startMediaStreamwithkindset tojpegorvideo. - Handle the
appium:mediaStream.unitsevents. JPEG units are complete images. Video units are Annex-B and start with the codec configuration at every keyframe, so decoding can begin at any keyframe. - Call
mobile: stopMediaStreamwhen done. Streams are also stopped when the session ends.
Since every frame is base64-encoded into a JSON message, this costs more CPU and bandwidth than a plain
MJPEG server. Prefer video for long or high resolution streams. A client that needs an HTTP MJPEG URL
(for example a browser or ffmpeg) can run a small local bridge that unwraps the jpeg units and serves them.
Example client¶
This script is a complete Node.js client that unwraps the events of a stream and sends the data to either of:
mjpeg: a local HTTP server that serves thejpegunits asmultipart/x-mixed-replace(MJPEG). Openhttp://127.0.0.1:8093in a browser, or use it as a source forffmpegand other tools. A new viewer gets the last frame right away, and a slow viewer skips frames instead of making the script buffer them.mp4: an.mp4file written byffmpeg, which only remuxes thevideo(and optionallyaudio) units without re-encoding them. The codec configuration is part of every keyframe, soffmpegcan parse the stream from the first unit.
npm install ws tsx # a WebSocket client and a TypeScript runner
npx tsx media-stream-client.mts mjpeg 8093 # MJPEG at http://127.0.0.1:8093
npx tsx media-stream-client.mts mp4 screen.mp4 --audio
It needs a running Appium server with a device, and ffmpeg on the PATH for the mp4 mode. Press Ctrl+C
to stop the stream and finish the file. The file uses the .mts extension so that it is an ES module, which
the top-level await needs. Node.js 22.18 or newer can also run it without tsx: node media-stream-client.mts ....
Notes:
- The client subscribes first and only then starts the stream, since events are only emitted while a client is subscribed.
- Appium closes the connection with code
1013if the client cannot keep up with the stream. This script then stops; a real client could reconnect, subscribe again and restart the stream. - If an
ffmpeginput cannot keep up, the script skips units for it instead of buffering the recording in memory; video resumes at the next keyframe. Ifffmpegfails or exits, the script stops the stream, ends the session and exits with a nonzero code. - A raw H.264/HEVC stream has no timestamps, so
ffmpegtimestamps the units by the time they arrive. The result is accurate to about the batching window (batchMs), and the audio is not aligned to the video. If you need exact timing, use thetsof the units (microseconds, one clock for all the tracks). - The script was tested on macOS. It uses no platform-specific code, but the audio goes to
ffmpegthrough an extra pipe (pipe:3), which may not work on Windows. If it does not, record without--audio. - To record HEVC, pass
codec: 'hevc'tomobile: startMediaStreamand usehevcinstead ofh264as the input format offfmpeg.