macOS and iOS Support - HedgeDoc

<style> .red { color: #df0000 !important; } a, .blue { color: #678dc6 !important; } .transparent { color: transparent !important; } .force-left { display: block !important; text-align: left !important; } </style>  # macOS and iOS Support <h2 class="red">History and Future Goals</h2> ### GStreamer Conference 2024 #### Nirbheek Chauhan https://nirbheek.in ---  # Who Am I? ## Platform Stuff, Rust, WebRTC,<br>RTSP, Meson, Cerbero, CI ## Whatever needs doing Everywhere and also Nowhere Note: Windows, UWP, macOS ---  # Index <br> * Today's Feature List * Ancient History * Noteworthy Events * Status Quo * Future Goals ---  ## Audio Features * Capturing and Rendering Audio - <code class="red">osxaudiosrc osxaudiosink osxaudiodeviceprovider</code> * CoreAudio-based * Also work on iOS, should be renamed - <code class="blue">sckitaudiosrc</code> * New API for capturing arbitrary windows * Pending MR, blocked on Swift⇔Meson integration * Encoding & Decoding Audio - <code class="red">atenc atdec</code> + AAC encode & AAC, MP3, MP1 decode + AudioToolbox supports more ---  # Video Features * Rendering Video - <code class="red">osxvideosink glimagesink caopengllayersink qmlglsink</code> * Capturing Video (Camera & Screen) - <code class="red">avfvideosrc</code> * Processing Video - <code class="red">glcolorconvert gloverlaycompositor glvideomixer</code> - <code class="red">vulkancolorconvert vulkanoverlaycompositor</code> * Encoding & Decoding Video - <code class="red">vtdec, vtenc_h264 vtenc_h265 vtenc_prores</code> - Might be possible to get Vulkan Video decoders working Note: There's also avsamplebufferlayersink. Vulkan things are via MoltenVK. SCKit-based video source is under development by Piotr. ---  # A Note on Video * CGL was deprecated on all Apple platforms a decade ago * Apple have a proprietary graphics API called Metal * The current direction for Metal support in GStreamer is via MoltenVK - Official Khronos translation layer from Vulkan to Metal * GStreamer has decent support for Vulkan, but integrations are lacking - No blockers, just needs someone to work on it - Missing video mixer implementation - Need GPU-GPU copy support with GL Note: other than audio and video, there's also asset source elements, but I won't talk about those here. Let's move on to the history of audio and video support. ---  # A Brief History of macOS & iOS Support ---  <h2>A Brief History of <span class="red">osxaudio</span></h2> | Years | "Maintainers" | Affiliation | |-------|---------------|-------------| | 2004 | Zaheer Merali | Songbird | | 2008-2009 | Michael Smith | Songbird, Xiph | | 2012-2013,2017 | Josep Torra | Fluendo, Oblong | | 2013 | Andoni Morales Alastruey | Fluendo | | 2014-2015 | Arun Raghavan | Collabora | | 2014-2015 | Sebastian Dröge | Centricular | | 2015 | Ilya Konstantinov | | | 2016,2022-2024 | Nirbheek Chauhan | Centricular | | 2023-2024 | Jan Schmidt | Centricular | Note: My categorization of maintainership is definitely questionable, I tried to pick people who made at least 2-3 major commits, and I allowed for a year of no-commits before the maintainership timed out. But the most interesting thing to me is that the very first macOS support in GStreamer was added by Songbird! I was a big fan back in the day, so that nugget of info is special to me. Maintainership has changed hands a lot, and this is a recurring theme in all the elements. ---  <h2> A Brief History of <span class="red">osxvideo</span></h2> | Years | "Maintainers" | Affiliation | |-------|---------------|-------------| | 2004 | Zaheer Merali | Songbird | | 2007 | Edward Hervey | Fluendo | | 2008-2009 | Michael Smith | Songbird, Xiph | | 2009 | Andy Wingo | Fluendo | | 2010, 2013 | Alessandro Decina | Collabora | | 2012-2014 | Andoni Morales Alastruey | Fluendo | | 2014-2015 | Sebastian Dröge | Centricular | Deprecated and effectively unmaintained in a decade. Use <span class="red">glimagesink</span> instead. ---  <h2>A Brief History of <span class="red">qtkitvideosrc</span></h2> | Years | "Maintainers" | Affiliation | |-------|---------------|-------------| | 2010-2013 | Ole André Vadla Ravnås | Cisco | | 2010-2012 | Alessandro Decina | Collabora | | 2013-2014 | Andoni Morales Alastruey | Fluendo | | 2015-2016 | Alessandro Decina | Collabora | Removed in 2017 in favor of <span class="red">avfvideosrc</span> ---  <h2>A Brief History of <span class="red">avfvideosrc</span></h2> | Years | "Maintainers" | Affiliation | |-------|---------------|-------------| | 2010-2012 | Ole André Vadla Ravnås | Cisco | | 2013 | Andoni Morales Alastruey | Fluendo | | 2013-2014 | Matthieu Bouron | Collabora | | 2014-2016 | Alessandro Decina | Collabora | | 2015 | Ilya Konstantinov | | | 2022-2024 | Nirbheek Chauhan | Centricular | Apple has started to deprecate these APIs in favour of ScreenCaptureKit ---  <h2>A Brief History of <span class="red">vtenc</span> and <span class="red">vtdec</span></h2> | Years | "Maintainers" | Affiliation | |-------|---------------|-------------| | 2010,2013 | Ole André Vadla Ravnås | Cisco | | 2013,2023 | Andoni Morales Alastruey | Fluendo | | 2013-2014 | Matthieu Bouron | Collabora | | 2013-2016 | Alessandro Decina | Collabora | | 2014-2017 | Sebastian Dröge | Centricular | | 2015-2022 | Matthew Waters | Centricular | | 2015 | Ilya Konstantinov | | | 2021-2024 | Nirbheek Chauhan | Centricular | | 2022-2024 | Piotr Brzeziński | Centricular | ---  <h2>A Brief History of <span class="red">CoreAnimation</span></h2> | Years | "Maintainers" | Affiliation | |-------|---------------|-------------| | 2015-2019,2024 | Matthew Waters | Centricular | <br> <ul class="force-left" style="margin-left: 20%;"> <li><code>caopengllayersink</code></li> <li><code>avsamplebuffersink</code></li> </ul> ---  # Noteworthy Things * Approximately zero maintainers from 2015 to 2020 - Most likely reason: barrier to entry * Autotools era: ad-hoc build setup with gst-uninstalled * Port to Meson in 2018, dramatic improvement in ergonomics * MoltenVK-based zerocopy video decode added * CoreAudio support has been drastically improved - Achievable capture/render latency from 20ms/80ms to 1ms/2ms - Device provider: updates, unique ID, default device * Qt/QML plugin reliability improvements - Should build on all setups: qmake, pkg-config, homebrew, etc Note: There was a certain amount of bitrot between 2015 and 2021, which didn't go unnoticed. So if you had issues with macOS support during that time, please give it another go. Only a handful of people knew how to build on macOS with Autotools. With Meson now, you can Just Build it after `pip install meson ninja`. CoreAudio numbers are with an RME MADIface USB. With the built-in audio card, lowest achievable is 2.5ms/5ms capture/render. ---  # Noteworthy Things * Cerbero Framework has been steadily been improving - Universal support with relocation - MoltenVK integration - RPATH support — took quite some back and forth to get right - Can consume both as a Framework and using pkg-config * Extensive CI support - Cerbero macOS universal, iOS universal, iOS examples - Moved from Intel to ARM64 runners - GStreamer monorepo * Some challenges around running unsigned code locally Note: Getting RPATHs right is surprisingly hard, many small things can go wrong and cause an app store validation error. Have to work around Apple ld and dyld bugs constantly. Shoutout to Amy, who has been a great help on the build side of things. Universal support is also non-trivial. Framework linkage requires custom includes to work correctly, because Frameworks are designed to ship a single namespace, not multiple like we need to. Andoni, Matt, and Amy have done a stellar job here. Monorepo build is stable now, excepy if you have MacPorts or Homebrew. ---  # Noteworthy Things * GStreamer is fully-functional in Homebrew now * Was broken: patched GLib for running NSRunLoop on the main thread - Without the patch, video playback would just be stuck - Apps can call `gst_macos_main()` for implicit integration * Separate prefixes meant most plugins wouldn't load * Switch to monorepo, also made it easier to ship Rust plugins * But the monorepo build had circular deps, which broke webrtc support * Fixed now with a custom patch to libnice * Homebrew for external deps breaks too often * macOS also trolls us by shipping broken deps: ncurses, jpeg, etc * Suggestions welcome Note: NSRunLoop change happened in 1.22, thanks to Piotr. macOS has been harder than Windows in terms of platform support ---  # What comprises platform support? <ul> <li>People can <b>use</b> GStreamer on it</li> <ul style="list-style-type: '⇒ '"> <li>Official GStreamer binary releases</li> </ul> <li>Project <b>investigates</b> issues related to it</li> <ul style="list-style-type: '⇒ '"> <li>Your issue won't be closed as WONTFIX</li> </ul> <li>People can <b>build</b> GStreamer for it</li> <ul style="list-style-type: '⇒ '"> <li>Cerbero can target it to build releases and has CI for it</li> </ul> </ul> Note: ARM64 support in the CI was a big milestone, as was monorepo CI added by Matt. Before that, wraps added by people for other platforms like Windows would break the build every couple of weeks. Productivity hit when developing on macOS. ---  <h1>How does platform support <span><span class="red">flourish</span>?</span></h1> ## Streamlined Developer Workflow <ul style="list-style-type: '⇒ '"> <li>Build GStreamer <b class="red">on</b> macOS <span class="transparent">✅</span></li> <li>Run GStreamer <b class="red">on</b> macOS <span class="transparent">✅</span></li> <li>Iterate quickly <b class="red">on</b> macOS <span class="transparent">✅</span></li> <li>Create merge requests <b class="red">on</b> macOS <span class="transparent">🫠</span></li> </ul> Note: as we saw earlier, maintainers come and go, it is really important that people who want to be maintainers can be ---  <h1>How does platform support <span><span class="red">flourish</span>?</span></h1> ## Streamlined Developer Workflow <ul style="list-style-type: '⇒ '"> <li>Build GStreamer <b class="red">on</b> macOS ✅</li> <li>Run GStreamer <b class="red">on</b> macOS ✅</li> <li>Iterate quickly <b class="red">on</b> macOS ✅</li> <li>Create merge requests <b class="red">on</b> macOS 🫠</li> </ul> ---  # Future Goals * Ship codesigned and notarized frameworks for macOS and iOS * Switch iOS from a Framework to an XCFramework - ARM64 Simulator support - tvOS and watchOS build support * CMake module for static linking * Make monorepo build and devenv even more streamlined and reliable - indent, linters, build docs, run tests * More testing on iOS and other non-macOS * Move from Core OpenGL to Vulkan - Metal is supported on macOS 10.11 and newer, same as us * More app developer documentation Note: without an xcframework, you cannot ship an iOS app from Apple Silicon, because you can't build or run your app for iPhoneSimulator ARM64. Our plugin docs are incomplete, because we can't build docs on macOS. For example, capopengllayersink is undocumented because it is dynamically registered. vtenc features are not dynamically registered, but av1 will be, and that will also be undocumented. Piotr and Amy aren't here, so no one can stop me from promising that they will do ALL this work. ---  # Thank you! ## Questions?

{"type":"slide","slideOptions":{"transition":"none","width":1536,"height":864,"center":false,"controlsLayout":"edges"}}