A New RTSP Source Element Written in Rust - HedgeDoc

<style> .red { color: #df0000 !important; } a, .blue { color: #678dc6 !important; } .force-left { display: block !important; text-align: left !important; } </style>  # A New RTSP Source Element Written in Rust <h2 class="red">rtspsrc2</h2> #### GStreamer Conference 2024 #### Nirbheek Chauhan https://nirbheek.in ---  # Who Am I? ## Platform Stuff, Rust, WebRTC,<br>RTSP, Meson, Cerbero, CI ## Whatever needs doing Everywhere and also Nowhere ---  # What's wrong with rtspsrc? <ul> <li>Created in 2005, simple design</li> <ul> <li>At the time, neither live nor VOD</li> <li>DESCRIBE SETUP PLAY PAUSE TEARDOWN</li> <li>READY→PAUSED ⇒ DESCRIBE+SETUP+PLAY aka <code>open()</code></li>  <li>PAUSED→READY ⇒ PAUSE</li> <li>pad deactivation ⇒ TEARDOWN aka <code>close()</code></li> </ul> </ul> Note: Sounds fine. Actually a fatal mistake. Broken from the get-go. What is the fatal mistake? ---  # What's wrong with rtspsrc? <ul> <li>Ties server state to client (pipeline!) state</li> <ul> <li>Multicast playback</li> <li>PAUSE is sent on error, immediately followed by a TEARDOWN</li> <li>Flushing seeks glitch & <b>deadlock</b></li> <li>Incorrect communication of intent</li> </ul> <li>Some of these have workarounds that every app must implement, others are unfixable</li> </ul> Note: A client leaving pauses the stream (or VOD) for everyone, PAUSE on errors. ---  # What's wrong with rtspsrc? * 2011: open/pause/play/close are async but racy * Fixed with a "cmd loop queue" * Only one pending command * 2018: New commands SET/GET_PARAMETER are added * Both Client→Server and Server→Client * Collision ⇒  **messages are dropped silently** * 2021: Fix requires large code changes * Not tenable in rtspsrc * Workaround: add some sleeps in the application 🫠 Note: cmd loop probably makes sense for a particular use-case, but incorrect. SET/GET use the same cmd loop. Didn't want to be stuck maintaining it. ---  # What's wrong with rtspsrc? <ul> <li>Some re-organising <i>was</i> done</li> <ul> <li><code>gstreamer-rtsp-1.0</code> & <code>gstreamer-sdp-1.0</code></li> </ul> <li>No major changes, only incremental</li> <li>RTSP commands &amp SDP from the network parsed in C</li> <li>RTSP 2.0 is impossible</li> <li>End-Of-Life</li> </ul> Note: Likely an absolute goldmine for vulnerabilities. No one is looking (fuzzing), yet. People kept piling hacks on top of hacks, and somehow it actually works well enough. All code rots in 20 years, but this code has reached end-of-life ---  # So we wrote something new <div style="display: flex; align-items: start; text-align: left; gap: 10px;"> <div style="flex: 1;"> <p> Sovereign Tech Fund </p> <p> <a href="https://indico.freedesktop.org/event/9/contributions/351/">“GStreamer RTP sessions in Rust”</a> by Matthew Waters </p> <ul> <li>RTP send and recv bins</li> <li>rtpjitterbuffer</li> <li>payloaders, depayloaders</li> <li>New RTSP source element</li> </ul> </div> <img style="max-height: 400px; height: auto; align-self: center;" src="https://notes.centricular.com/uploads/10eaf14a-6fde-4c27-afd2-f6ac720ef0a5.png"/> </div> Note: We submitted a funding proposal to the Sovereign Tech Fund to rewrite the GStreamer RTP stack in Rust. ---  # rtspsrc2: Features * Live streaming N audio and N video, with RTCP-based A/V sync * Lower transports: TCP, UDP, UDP-Multicast * RTP, RTCP SR, RTCP RR * OPTIONS DESCRIBE SETUP PLAY TEARDOWN * Supports both the old `rtpbin` and the new `rtpbin2` rust rewrite - Set `USE_RTPBIN2=1` to use `rtpbin2` * Lower transport selection **and** priority (NEW!) - Even supports different transport for each SETUP call - F.ex., KLV could be sent over TCP and the rest over UDP * `port-start` instead of `port-range` ---  # rtspsrc2: Characteristics * `rtsp-types` and `sdp-types` crates * `rtpbin2` ⇒ all reading, parsing, writing, etc is done in Rust - Except codec parsing and demuxing * Async Rust, multi-threaded Tokio - Simpler and decoupled, easy to follow * Network I/O - Does not use `udpsrc`, `udpsink` - Can be made pluggable or even I/O-less Note: Sebastian wrote those crates ---  # rtspsrc2: Characteristics * appsink + appsrc + ghostpads - Simplified event handling, startup/shutdown - Expose pads early, sparse streams * Performance on-par with `rtspsrc` * Tested against live555 and gst-rtsp-server * Most features have not been implemented yet * Plan is to replace rtspsrc Note: instead of directly ghosting rtpbin pads ---  # rtspsrc2: Low-Hanging Fruit * Credentials support  * TCP over TLS support  * NAT hole punching  * `SET_PARAMETER` & `GET_PARAMETER` support * Option to reconnect TCP control on disconnect * Missing configuration properties: - `latency` & `do-rtx`  - `do-rtcp`  - `iface`  - `user-agent`  ---  # rtspsrc2: Giraffe Food * Stream Collection API - Live streams first, VOD later * SRTP support, incl. all key exchange mechanisms - Parsing more SDP attributes * Control and media flow tunnelling over HTTP * HTTP and SOCKS proxy support * Make TCP connections optional when using UDP transports * Explicit media clock sync support, such as RFC7273 Note: parse SDP attrs like key-mgmt ---  # rtspsrc2: Re-Forestation * RTSP 2.0 support - PLAY_NOTIFY, Media-Properties, End-Of-Stream, Scale-Change, etc - Needs an accompanying RTSP 2.0 server implementation * Video-On-Demand support, both RTSP 1.0 and 2.0 - PAUSE, Seeking, Looping * ONVIF backchannel support * ONVIF trick modes support * ... ---  <div style="text-align: left;"> # rtspsrc2: Help Needed The main selling-point of the old rtspsrc is the IP camera compatibility that has been built-up over the years. Matching that is absolutely a necessity. <b>Please give me access to your IP cameras</b> <span>either using a VPN (tailscale is a good option), or you can ship to me IP cameras that you want to get rid of.</span> You will thank me in 3-5 years when regulatory bodies force everyone to move to memory-safe languages for government contracting. 🙃 It's a fun project for me, and everyone benefits. Please get in touch! nirbheek@centricular.com Thanks!! </div>

{"type":"slide","slideOptions":{"transition":"none","width":1536,"height":864,"center":false,"controlsLayout":"edges"}}