Sync with livekit's m125 (#42)

* Update to m125. (#119)

Use M125 as the latest version and migrate historical patches to m125

Patches Group:

## 1. Update README.md
webrtc-sdk/webrtc@b6c65fc
* Add Apache-2.0 license and some note to README.md. (#9)
* Updated readme detailing changes from original (#42)
* Adding membrane framework (#51)
* Updated readme (#83)

## 2. Audio Device Optimization
webrtc-sdk/webrtc@7454824
* allow listen-only mode in AudioUnit, adjust when category changes
(webrtc-sdk/webrtc#2)
* release mic when category changes
(webrtc-sdk/webrtc#5)
* Change defaults to iOS defaults
(webrtc-sdk/webrtc#7)
* Sync audio session config
(webrtc-sdk/webrtc#8)
* feat: support bypass voice processing for iOS.
(webrtc-sdk/webrtc#15)
* Remove MacBookPro audio pan right code
(webrtc-sdk/webrtc#22)
* fix: Fix can't open mic alone when built-in AEC is enabled.
(webrtc-sdk/webrtc#29)
* feat: add audio device changes detect for windows.
(webrtc-sdk/webrtc#41)
* fix Linux compile (webrtc-sdk/webrtc#47)
* AudioUnit: Don't rely on category switch for mic indicator to turn off
(webrtc-sdk/webrtc#52)
* Stop recording on mute (turn off mic indicator)
(webrtc-sdk/webrtc#55)
* Cherry pick audio selection from m97 release
(webrtc-sdk/webrtc#35)
* [Mac] Allow audio device selection
(webrtc-sdk/webrtc#21)
* RTCAudioDeviceModule.outputDevice / inputDevice getter and setter
(webrtc-sdk/webrtc#80)
* Allow custom audio processing by exposing AudioProcessingModule
(webrtc-sdk/webrtc#85)
* Expose audio sample buffers for Android
(webrtc-sdk/webrtc#89)
* feat: add external audio processor for android.
(webrtc-sdk/webrtc#103)
* android: make audio output attributes modifiable
(webrtc-sdk/webrtc#118)
* Fix external audio processor sample rate calculation
(webrtc-sdk/webrtc#108)
* Expose remote audio sample buffers on RTCAudioTrack
(webrtc-sdk/webrtc#84)
* Fix memory leak when creating audio CMSampleBuffer
webrtc-sdk/webrtc#86

## 3. Simulcast/SVC support for iOS/Android.
webrtc-sdk/webrtc@b0b9fe9

- Simulcast support for iOS SDK (#4)
- Support for simulcast in Android SDK (#3)
- include simulcast headers for mac also (#10)
- Fix simulcast using hardware encoder on Android (#48)
- Add scalabilityMode support for AV1/VP9. (#90)

## 4. Android improvements.
webrtc-sdk/webrtc@9aaaab5
- Start/Stop receiving stream method for VideoTrack (#25)
- Properly remove observer upon deconstruction (#26)
- feat: Expose setCodecPreferences/getCapabilities for android. (#61)
- fix: add WrappedVideoDecoderFactory.java. (#74)

## 5. Darwin improvements
webrtc-sdk/webrtc@a13ea17
- [Mac/iOS] feat: Add RTCYUVHelper for darwin. (#28)
- Cross-platform `RTCMTLVideoView` for both iOS / macOS (#40)
- rotationOverride should not be assign (#44)
- [ObjC] Expose properties / methods required for AV1 codec support
(#60)
- Workaround: Render PixelBuffer in RTCMTLVideoView (#58)
- Improve iOS/macOS H264 encoder (#70)
- fix: fix video encoder not resuming correctly upon foregrounding
(#75).
- add PrivacyInfo.xcprivacy to darwin frameworks. (#112)
- Add NSPrivacyCollectedDataTypes key to xcprivacy file (#114)
- Thread-safe `RTCInitFieldTrialDictionary` (#116)
- Set RTCCameraVideoCapturer initial zoom factor (#121)
- Unlock configuration before starting capture session (#122)

## 6. Desktop Capture for macOS.
webrtc-sdk/webrtc@841d78f
- [Mac] feat: Support screen capture for macOS. (#24) (#36)
- fix: Get thumbnails asynchronously. (#37)
- fix: Use CVPixelBuffer to build DesktopCapture Frame, fix the crash
caused by non-CVPixelBuffer frame in RTCVideoEncoderH264 that cannot be
cropped. (#63)
- Fix the crash when setting the fps of the virtual camera. (#62)

## 7. Frame Cryptor Support.
webrtc-sdk/webrtc@fc08745
- feat: Frame Cryptor (aes gcm/cbc). (#54)
- feat: key ratchet/derive. (#66)
- fix: skip invalid key when decryption failed. (#81)
- Improve e2ee, add setSharedKey to KeyProvider. (#88)
- add failure tolerance for framecryptor. (#91)
- fix h264 freeze. (#93)
- Fix/send frame cryptor events from signaling thread (#95)
- more improvements for E2EE. (#96)
- remove too verbose logs (#107)
- Add key ring size to keyProviderOptions. (#109)

## 8. Other improvements.
webrtc-sdk/webrtc@eed6c8a
- Added yuv_helper (#57)
- ABGRToI420, ARGBToI420 & ARGBToRGB24 (#65)
- more yuv wrappers (#87)
- Fix naming for yuv helper (#113)
- Fix missing `RTC_OBJC_TYPE` macros (#100)

---------

Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: David Zhao <[email protected]>
Co-authored-by: davidliu <[email protected]>
Co-authored-by: Angelika Serwa <[email protected]>
Co-authored-by: Théo Monnom <[email protected]>
# Conflicts:
#	README.md
#	media/engine/webrtc_video_engine.cc
#	media/engine/webrtc_video_engine.h
#	modules/audio_device/audio_device_impl.cc
#	sdk/BUILD.gn
#	sdk/android/BUILD.gn
#	sdk/android/api/org/webrtc/RtpParameters.java
#	sdk/android/api/org/webrtc/SimulcastVideoEncoder.java
#	sdk/android/api/org/webrtc/SimulcastVideoEncoderFactory.java
#	sdk/android/api/org/webrtc/VideoCodecInfo.java
#	sdk/android/src/jni/pc/rtp_parameters.cc
#	sdk/android/src/jni/simulcast_video_encoder.cc
#	sdk/android/src/jni/simulcast_video_encoder.h
#	sdk/android/src/jni/video_codec_info.cc
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule+Private.h
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule.h
#	sdk/objc/api/peerconnection/RTCAudioDeviceModule.mm
#	sdk/objc/api/peerconnection/RTCAudioTrack.mm
#	sdk/objc/api/peerconnection/RTCIODevice+Private.h
#	sdk/objc/api/peerconnection/RTCIODevice.mm
#	sdk/objc/api/peerconnection/RTCPeerConnectionFactory.h
#	sdk/objc/api/peerconnection/RTCPeerConnectionFactory.mm
#	sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.h
#	sdk/objc/api/video_codec/RTCVideoEncoderSimulcast.mm
#	sdk/objc/base/RTCAudioRenderer.h
#	sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.h
#	sdk/objc/components/video_codec/RTCVideoEncoderFactorySimulcast.mm

* fix: duplicate simulcast entries

* remove duplicate declaration

* remove duplicate audioDeviceModule

* fix: removed livekit's external audio processor

* fix: add back simulcast factories

* Fix missing RTC_OBJC_TYPE macros

* Fix missing headers and Metal linking

# Conflicts:
#	sdk/BUILD.gn

* Fix Mac Catalyst `RTCCameraVideoCapturer` rotation (#126)

* Fix set frame transformer (#125)

* Fix webrtc_voice_engine not notifying mute change (#128)

Looks like this line was missed during the m125 update.

webrtc-sdk/webrtc@272127d#diff-56f5e0c459b287281ef3b0431d3f4129e8e4be4c6955d845bcb22210f08b7ba5R2289

Adding it back in so that mic is properly released when muted.
# Conflicts:
#	media/engine/webrtc_voice_engine.cc

* android: Allow for skipping checking the audio playstate if needed (#129)

Pausing/stopping the audio track can lead to a race condition against
the AudioTrackThread due to this assert. Normally this is fine since
directly pausing/stopping isn't possible, but user is using reflection
to workaround another audio issue (muted participants still have a
sending audio stream which keeps the audio alive, affecting global sound
if in the background).

Not a full fix, as would like to manually control the audio track
directly (needs a bigger fix to handle proper synchronization before
allowing public access), but this will work through reflection (user
takes responsibility for usage).

* Allow to pass in capture session to RTCCameraVideoCapturer (#132)

Expose initializers to pass in capture session to RTCCameraVideoCapturer
so we can use AVCaptureMultiCamSession etc to capture front and back
simultaneously for iOS.

* Fix NetworkMonitor race condition when dispatching native observers (#135)

There is a race condition in NetworkMonitor where native observers may
be removed concurrently with a notification being dispatched, leading to
a dangling pointer dereference (trying to dispatch an observer that was
already removed and destroyed), and from there a crash with access
violation.

By ensuring dispatching to native observers is done within the
synchronization lock that guards additions/removals of native observers
protects against this race condition. Since native observers callbacks
are posted to the networking thread in the C++ side anyway, there should
be no risk of deadlock/starvation due to long-running observers.

Bug: webrtc:15837
Change-Id: Id2b788f102dbd25de76ceed434c4cd68aa9a569e
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/338643
Reviewed-by: Taylor Brandstetter <[email protected]>
Commit-Queue: Harald Alvestrand <[email protected]>
Reviewed-by: Harald Alvestrand <[email protected]>
Cr-Commit-Position: refs/heads/main@{#42256}

Co-authored-by: Guy Hershenbaum <[email protected]>

* Support for Vision Pro (#131)

TODO:
- [x]  fix compile for RTCCameraVideoCapturer
- [ ]  fix RTCMTLRenderer ?

---------

Co-authored-by: Hiroshi Horie <[email protected]>

* Multicam support (#137)

TODO:
- [x] Return `.systemPreferredCamera` for devices (visionOS only).
- [x] Use `AVCaptureMultiCamSession` only if `isMultiCamSupported` is
true.
- [x] Silence statusBarOrientation warning.

---------

Co-authored-by: [email protected] <[email protected]>

* tvOS support (#139)

17.0+ only atm

---------

Co-authored-by: cloudwebrtc <[email protected]>

* Add isDisposed to MediaStreamTrack (#140)

* chore: handle invalid cipher from key size. (#142)

* Allow software AEC for Simulator (#143)

~Allow to use "googEchoCancellation" constraint for software AEC.
For devices "googEchoCancellation" should be false to use
VoiceProcessingIO.~

* Fix AudioRenderer crash & expose AVAudioPCMBuffer (#144)

* fix: Fix bug for bypass voice processing. (#147)

* chore: remove aes cbc for framecryptor. (#145)

* Change audio renderer output format (#149)

Instead of converting to Float, output original Int data without
conversion.
Output the raw format and convert when required.

* Fixed issue with missing network interfaces on iOS (#151)

Related issue: webrtc-sdk/webrtc#148
Cherry-pick :
https://webrtc.googlesource.com/src/+/fea60ef8e72fb17b4f8a5363aff7e63ab8027b4f

Fixed issue with network interfaces due to a missing return value in the
"nw_path_enumerate_interfaces(...)" block. Exposed in iOS 18,
RTCNetworkMonitor::initWithObserver will only enumerate the first
interface, instead of all device interfaces

Bug: webrtc:359245764
Change-Id: Ifb9f28c33306c0096476a4afb0cdb4d734e87b2c
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/359541
Auto-Submit: Corby <[email protected]>
Commit-Queue: Jonas Oreland <[email protected]>
Reviewed-by: Kári Helgason <[email protected]>
Reviewed-by: Jonas Oreland <[email protected]>
Cr-Commit-Position: refs/heads/main@{#42818}

Co-authored-by: Corby Hoback <[email protected]>

* Custom audio input for Android (#154)

# Conflicts:
#	sdk/android/api/org/webrtc/audio/JavaAudioDeviceModule.java
#	sdk/android/src/java/org/webrtc/audio/WebRtcAudioRecord.java

---------

Co-authored-by: CloudWebRTC <[email protected]>
Co-authored-by: Hiroshi Horie <[email protected]>
Co-authored-by: davidliu <[email protected]>
Co-authored-by: Guy Hershenbaum <[email protected]>
Co-authored-by: Corby Hoback <[email protected]>

Author: 6 people

api/peer_connection_interface.h

@@ -700,6 +700,15 @@ class RTC_EXPORT PeerConnectionInterface : public webrtc::RefCountInterface {
     // of having to allocate additional candidates.
     bool enable_any_address_ports = false;
 
+    // When this flag is set, ports not bound to any specific network interface
+    // will be used, in addition to normal ports bound to the enumerated
+    // interfaces. Without this flag, these "any address" ports would only be
+    // used when network enumeration fails or is disabled. But under certain
+    // conditions, these ports may succeed where others fail, so they may allow
+    // the application to work in a wider variety of environments, at the expense
+    // of having to allocate additional candidates.
+    bool enable_any_address_ports = false;
+
     //
     // Don't forget to update operator== if adding something.
     //

api/video/BUILD.gn

@@ -92,6 +92,19 @@ rtc_library("yuv_helper") {
   ]
 }
 
+rtc_library("yuv_helper") {
+  visibility = [ "*" ]
+  sources = [
+    "yuv_helper.cc",
+    "yuv_helper.h",
+  ]
+
+  deps = [
+    "../../rtc_base/system:rtc_export",
+    "//third_party/libyuv",
+  ]
+}
+
 if (is_android) {
   java_cpp_enum("video_frame_enums") {
     sources = [ "video_frame_buffer.h" ]

audio/audio_state.cc

@@ -159,7 +159,9 @@ void AudioState::RemoveSendingStream(webrtc::AudioSendStream* stream) {
   RTC_DCHECK_EQ(1, count);
   UpdateAudioTransportWithSendingStreams();
 
-  if (!ShouldRecord()) {
+  bool should_record = ShouldRecord();
+  RTC_LOG(LS_INFO) << "RemoveSendingStream: should_record = " << should_record;
+  if (!should_record) {
     config_.audio_device_module->StopRecording();
   }
 }
@@ -221,6 +223,7 @@ void AudioState::OnMuteStreamChanged() {
   auto* adm = config_.audio_device_module.get();
   bool should_record = ShouldRecord();
 
+  RTC_LOG(LS_INFO) << "OnMuteStreamChanged: should_record = " << should_record;
   if (should_record && !adm->Recording()) {
     if (adm->InitRecording() == 0) {
       adm->StartRecording();
@@ -231,8 +234,10 @@ void AudioState::OnMuteStreamChanged() {
 }
 
 bool AudioState::ShouldRecord() {
+  RTC_LOG(LS_INFO) << "ShouldRecord";
   // no streams to send
   if (sending_streams_.empty()) {
+    RTC_LOG(LS_INFO) << "ShouldRecord: send stream = empty";
     return false;
   }
 
@@ -245,6 +250,7 @@ bool AudioState::ShouldRecord() {
     }
   }
 
+  RTC_LOG(LS_INFO) << "ShouldRecord: " << muted_count << " muted, " << stream_count << " sending";
   return muted_count != stream_count;
 }
 

media/engine/webrtc_video_engine.cc

@@ -2779,24 +2779,6 @@ WebRtcVideoReceiveChannel::~WebRtcVideoReceiveChannel() {
     delete kv.second;
 }
 
-void WebRtcVideoReceiveChannel::StartReceive(uint32_t ssrc) {
-  RTC_DCHECK_RUN_ON(&thread_checker_);
-  WebRtcVideoReceiveStream* stream = FindReceiveStream(ssrc);
-  if(!stream) {
-    return;
-  }
-  stream->StartStream();
-}
-
-void WebRtcVideoReceiveChannel::StopReceive(uint32_t ssrc) {
-  RTC_DCHECK_RUN_ON(&thread_checker_);
-  WebRtcVideoReceiveStream* stream = FindReceiveStream(ssrc);
-  if(!stream) {
-    return;
-  }
-  stream->StopStream();
-}
-
 void WebRtcVideoReceiveChannel::SetReceiverFeedbackParameters(
     bool lntf_enabled,
     bool nack_enabled,

sdk/android/BUILD.gn

@@ -920,23 +920,6 @@ if (current_os == "linux" || is_android) {
     ]
   }
 
-  rtc_library("simulcast_jni") {
-    visibility = [ "*" ]
-    allow_poison = [ "software_video_codecs" ]
-    sources = [
-      "src/jni/simulcast_video_encoder.cc",
-      "src/jni/simulcast_video_encoder.h",
-      "src/jni/simulcast_video_encoder_factory.cc",
-      "src/jni/simulcast_video_encoder_factory.h"
-    ]
-    deps = [
-      ":base_jni",
-      ":video_jni",
-      ":native_api_codecs",
-      "../../media:rtc_simulcast_encoder_adapter"
-    ]
-  }
-
   rtc_library("libaom_av1_encoder_jni") {
     visibility = [ "*" ]
     allow_poison = [ "software_video_codecs" ]
@@ -967,7 +950,9 @@ if (current_os == "linux" || is_android) {
     allow_poison = [ "software_video_codecs" ]
     sources = [
       "src/jni/simulcast_video_encoder.cc",
-      "src/jni/simulcast_video_encoder.h"
+      "src/jni/simulcast_video_encoder.h",
+      "src/jni/simulcast_video_encoder_factory.cc",
+      "src/jni/simulcast_video_encoder_factory.h"
     ]
     deps = [
       ":base_jni",
@@ -989,7 +974,6 @@ if (current_os == "linux" || is_android) {
       ":generated_swcodecs_jni",
       ":libvpx_vp8_jni",
       ":libvpx_vp9_jni",
-      ":simulcast_jni",
       ":native_api_jni",
       ":simulcast_jni",
       ":video_jni",

sdk/android/src/java/org/webrtc/audio/WebRtcAudioRecord.java

@@ -115,7 +115,6 @@ class WebRtcAudioRecord {
 
   private final @Nullable AudioRecordErrorCallback errorCallback;
   private final @Nullable AudioRecordStateCallback stateCallback;
-  private final @Nullable AudioRecordDataCallback audioRecordDataCallback;
   private final @Nullable SamplesReadyCallback audioSamplesReadyCallback;
   private final @Nullable AudioBufferCallback audioBufferCallback;
   private final boolean isAcousticEchoCancelerSupported;

sdk/objc/api/peerconnection/RTCRtpEncodingParameters.mm

@@ -102,6 +102,9 @@ - (instancetype)initWithNativeParameters:
   if (_scalabilityMode != nil) {
     parameters.scalability_mode = std::optional<std::string>(std::string([_scalabilityMode UTF8String]));
   }
+  if (_scalabilityMode != nil) {
+    parameters.scalability_mode = absl::optional<std::string>(std::string([_scalabilityMode UTF8String]));
+  }
   parameters.bitrate_priority = _bitratePriority;
   parameters.network_priority = [RTC_OBJC_TYPE(RTCRtpEncodingParameters)
       nativePriorityFromPriority:_networkPriority];

sdk/objc/api/peerconnection/RTCRtpTransceiver.h

@@ -16,6 +16,8 @@
 
 @class RTC_OBJC_TYPE(RTCRtpCodecCapability);
 
+@class RTC_OBJC_TYPE(RTCRtpCodecCapability);
+
 NS_ASSUME_NONNULL_BEGIN
 
 extern NSString *const RTC_CONSTANT_TYPE(RTCRtpTransceiverErrorDomain);
@@ -125,6 +127,8 @@ RTC_OBJC_EXPORT
     NSArray<RTC_OBJC_TYPE(RTCRtpHeaderExtensionCapability) *>
         *negotiatedHeaderExtensions;
 
+@property(nonatomic, copy) NSArray<RTC_OBJC_TYPE(RTCRtpCodecCapability) *> *codecPreferences;
+
 /** The currentDirection attribute indicates the current direction negotiated
  *  for this transceiver. If this transceiver has never been represented in an
  *  offer/answer exchange, or if the transceiver is stopped, the value is not

sdk/objc/base/RTCVideoCodecInfo.h

@@ -32,6 +32,11 @@ RTC_OBJC_EXPORT
              scalabilityModes:(nullable NSArray<NSString *> *)scalabilityModes
     NS_DESIGNATED_INITIALIZER;
 
+- (instancetype)initWithName:(NSString *)name
+                  parameters:(nullable NSDictionary<NSString *, NSString *> *)parameters
+             scalabilityModes:(nullable NSArray<NSString *> *)scalabilityModes
+    NS_DESIGNATED_INITIALIZER;
+
 - (BOOL)isEqualToCodecInfo:(RTC_OBJC_TYPE(RTCVideoCodecInfo) *)info;
 
 @property(nonatomic, readonly) NSString *name;

sdk/objc/components/audio/RTCAudioSession+Configuration.mm

@@ -97,6 +97,33 @@ - (BOOL)setConfiguration:
     }
   }
 
+  if (self.mode != configuration.mode) {
+    NSError *modeError = nil;
+    if (![self setMode:configuration.mode error:&modeError]) {
+      RTCLogError(@"Failed to set mode to %@: %@",
+                  self.mode,
+                  modeError.localizedDescription);
+      error = modeError;
+    } else {
+      RTCLog(@"Set mode to: %@", configuration.mode);
+    }
+  }
+
+  // Sometimes category options don't stick after setting mode.
+  if (self.categoryOptions != configuration.categoryOptions) {
+    NSError *categoryError = nil;
+    if (![self setCategory:configuration.category
+               withOptions:configuration.categoryOptions
+                     error:&categoryError]) {
+      RTCLogError(@"Failed to set category options: %@",
+                  categoryError.localizedDescription);
+      error = categoryError;
+    } else {
+      RTCLog(@"Set category options to: %ld",
+             (long)configuration.categoryOptions);
+    }
+  }
+
   if (self.preferredSampleRate != configuration.sampleRate) {
     NSError *sampleRateError = nil;
     if (![self setPreferredSampleRate:configuration.sampleRate