[camera] add video stabilization #7108

ruicraveiro · 2024-07-12T14:54:49Z

Adds support for video stabilization to camera_platform_interface, camera_avfoundation, camera_android_camerax and camera packages.

The video stabilization modes are defined in the new VideoStabilizationMode enum defined in camera_platform_interface:

/// The possible video stabilization modes that can be capturing video.
enum VideoStabilizationMode {
  /// Video stabilization is disabled.
  off,

  /// Basic video stabilization is enabled.
  /// Maps to CONTROL_VIDEO_STABILIZATION_MODE_ON on Android
  /// and throws CameraException on iOS.
  on,

  /// Standard video stabilization is enabled.
  /// Maps to CONTROL_VIDEO_STABILIZATION_MODE_PREVIEW_STABILIZATION on Android
  /// (camera_android_camerax) and to AVCaptureVideoStabilizationModeStandard
  /// on iOS.
  standard,

  /// Cinematic video stabilization is enabled.
  /// Maps to CONTROL_VIDEO_STABILIZATION_MODE_PREVIEW_STABILIZATION on Android
  /// (camera_android_camerax) and to AVCaptureVideoStabilizationModeCinematic
  /// on iOS.
  cinematic,

  /// Extended cinematic video stabilization is enabled.
  /// Maps to AVCaptureVideoStabilizationModeCinematicExtended on iOS and
  /// throws CameraException on Android.
  cinematicExtended,
}

There is some subjectivity on the way with which I mapped the modes to both platforms, and here's a document that compares the several modes: https://docs.google.com/spreadsheets/d/1TLOLZHR5AcyPlr-y75aN-DbR0ssZLJjpV_OAJkRC1FI/edit?usp=sharing, which you can comment on.

List which issues are fixed by this PR. You must list at least one issue.
Partially implements flutter/flutter#89525

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

google-cla · 2024-07-12T14:54:53Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

hellohuanlin

only reviewed iOS part

...amera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/CameraPlugin.m

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

.../camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/messages.g.m

ruicraveiro · 2024-07-16T11:07:56Z

Hi @hellohuanlin,

I have improved camera_avfoundation's implementation based on your comments. Instead of adding a new commit, I forced pushed a squash commit of those changes and another commit on top of it that adds the dependency_overrides to the pubspecs.

Assuming there will be more improvments I need to make, is it OK I keep adding commits until everything is fixed, and only then do a final squash merge, excluding the dependency_overrides commit, and force that squash commit?

Thanks!

stuartmorgan-g · 2024-12-03T21:19:24Z

I would prefer to get it approved as it is

The current version conflates Exceptions and Errors as discussed in the long comment thread above, so it would not be approved as is; the version without fallback would still require changes.

ruicraveiro · 2024-12-03T21:34:33Z

I would prefer to get it approved as it is

The current version conflates Exceptions and Errors as discussed in the long comment thread above, so it would not be approved as is; the version without fallback would still require changes.

Sure. It will be easier for me to find the time if the required changes are limited to fixing the confusion between exceptions and errors, which will increase the odds of being able to do so this year instead of later in 2025.

ruicraveiro · 2024-12-20T19:37:55Z

Hi @stuartmorgan,

I think next week I'll find enough time to finish this off. But, before doing so, I want to get a full agreement on the specification of setTargetVideoStabilizationMode, so I can make the best use of my time.

Error Handling

First, concerning error handling:

For platforms without an implementation, we throw UnimplementedError.
For platforms with implementation, but without any capability, right now I am returning an empty array. I am not sure if this is the ideal behaviour, or if we should always return at least [ VideoStabilizationMode.off ]. What do you think? I will assume we'll keep the current implementation for the rest of this comment.
Methods already implemented in CameraController check to see if the camera is already initialized before doing their thing by calling _throwIfNotInitialized, which throws a CameraException if they are called before initialization. While I would argue that this is also a programming error, so it would make sense to throw an error instead of an exception, I see a stronger reason to follow the pattern and call _throwIfNotInitialized just like every other method is doing. This is what is currently implemented.

Fallback behaviour

Concerning the implementation with a fallback, here's a table of what setTargetVideoStabilizationMode should do:

requested mode	no supported mode ([])	only off supported	max level1 supported	max level2 supported	max level3 supported
off	throw ArgumentError()	off	off	off	off
level1	throw ArgumentError()	throw ArgumentError()	level1	level1	level1
level2	throw ArgumentError()	throw ArgumentError()	level1	level2	level2
level3	throw ArgumentError()	throw ArgumentError()	level1	level2	level 3

The implementation of this logic would be in the camera package, not in the platform implementations to ensure it is always consistent. I don't see a reason to place it in the platform implementation packages.

Extra

Because of the time distance between the last time we discussed about this and now, I had to go over the entire discussion from scratch and while doing so it struck me that we could have it both ways. You wanted to make it easier for developers that want to just have the best available mode up to X and I wanted a more strict approach to cater to developers who, like me, prefer less nuanced behaviour. So, it occurred to me that the default behaviour could be the fallback behaviour, but we could add an optional bool named argument, for example, strict, which would revert the method to, as it says, a strict behaviour. So, the method's final signature would be Future<VideoStabilizationMode> setTargetVideoStabilizationMode(VideoStabilizationMode mode, {bool strict = false}), and when strict would be true, the behaviour would be:

requested mode	no supported mode ([])	only off supported	max level1 supported	max level2 supported	max level3 supported
off	throw ArgumentError()	off	off	off	off
level1	throw ArgumentError()	throw ArgumentError()	level1	level1	level1
level2	throw ArgumentError()	throw ArgumentError()	throw ArgumentError()	level2	level2
level3	throw ArgumentError()	throw ArgumentError()	throw ArgumentError()	throw ArgumentError()	level3

I don't want to spend much more energy discussing this, so if you don't agree with this extra (strict), it's OK and I'll just implement without it, but if you do, I think we would be striking a really good balance here. The only reason I see against this is that it will require an effort from me to document this in such a way that it is very easy and clear to understand.

Let me know what you think.

Thanks!

Rui

hellohuanlin · 2024-12-23T19:35:00Z

...amera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/CameraPlugin.m

+  __weak typeof(self) weakSelf = self;
+
+  dispatch_async(self.captureSessionQueue, ^{
+    bool isSupported = [weakSelf.camera isVideoStabilizationModeSupported:mode];


nit: BOOL

hellohuanlin · 2024-12-23T19:35:32Z

...a/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/CameraProperties.m

+
+    case FCPPlatformVideoStabilizationModeCinematic:
+      return AVCaptureVideoStabilizationModeCinematic;
+


nit: inconsistent blank lines in this func

hellohuanlin

iOS part LGTM

ruicraveiro · 2025-01-02T15:18:42Z

iOS part LGTM

Thanks!!

stuartmorgan-g · 2025-01-13T18:36:11Z

I think next week I'll find enough time to finish this off.

Sorry I was not able to respond before then; I was out of office for several weeks starting shortly before that comment.

For platforms with implementation, but without any capability, right now I am returning an empty array. I am not sure if this is the ideal behaviour, or if we should always return at least [ VideoStabilizationMode.off ]. What do you think?

I agree with empty array; I think not including off in the set of things that method can return will make it much easier for clients to use.

So, it occurred to me that the default behaviour could be the fallback behaviour, but we could add an optional bool named argument, for example, strict, which would revert the method to, as it says, a strict behaviour. [...] The only reason I see against this is that it will require an effort from me to document this in such a way that it is very easy and clear to understand.

That sounds like a reasonable API, so adding it to better support your use case sounds good to me. I don't think it should be too hard to explain in the doc comment what it means; my suggestion would be to reverse its meaning and make it allowFallback = true, since I think that's more self-explanatory in code, and will thus be easier to document.

gmr-renato · 2025-03-10T18:01:55Z

Hi guys, any progress on this one? I'm working on a sport's recording camera and video stabilization is a huge feature.
@ruicraveiro can I help you to progress on this?

ruicraveiro · 2025-03-10T18:29:59Z

Hi guys, any progress on this one? I'm working on a sport's recording camera and video stabilization is a huge feature. @ruicraveiro can I help you to progress on this?

Hi Renato,

No, not at this point. Maybe during the next couple of weeks I'll have time to finish this off.

Rui

…lizationModes

ruicraveiro · 2025-03-13T00:02:18Z

Hi @stuartmorgan, I have just pushed an update to this branch with the following changes:

At the platform levels, camera_android_camerax and camera_avfoundation, if setVideoStabilizationMode() is called with a mode that is unavailable, then that method with throw an ArgumentError. I also figured a cleaner way to map between VideoStabilizationMode and the platform's specific representation of the mode. Now I am mapping both ways using a single method, which in both platforms is called _getSupportedVideoStabilizationModeMap(). I also fixed a couple of nits picked up by @hellohuanlin.
Because it is a common feature with a common logic, I implemented the fallback logic in CameraController, in the camera package, for setVideoStabilizationMode() method, as specified in my message on the 20th December. When allowFallback is true, then CameraController will only call the platform's implementation with a supported video stabilization mode and will throw ArgumentException if none is available (or if more than off is asked for and only off is available). When allowFallback is false, then CameraController passes the mode as is to the platform's implementation.

I have some broken tests, but I don't think they're related to anything I did, except merging the main branch onto this one. On my machine, all Dart unit tests related to this package are OK. Native tests are indeed failing, but they're also failing when I run them on the latest version of the main branch (plus I'm getting failed Dart tests for other packages on both this branch and main). Except for the very tiny nits (bool -> BOOL and a couple of empty lines removed), all changes were made to Dart code, not native code.

leooruiz · 2025-03-28T18:43:40Z

Hello @ruicraveiro,

Thank you for your work on this stabilization PR. I tested the update on my sports recording app and encountered a compilation error on iOS. Below is the error message from the console.

Error log:

Launching lib/main.dart on iPhone 11 in release mode...
Automatically signing iOS for device deployment using specified development team in Xcode project: H9UAZA7PV4
Xcode build done.                                           59.5s
Failed to build iOS app
Could not build the precompiled application for the device.
ARC Semantic Issue (Xcode): No visible @interface for 'NSObject<FLTCaptureDeviceFormat>' declares the selector 'isVideoStabilizationModeSupported:'
/Users/leo_ruiz/repositories/xports/flutter_packages/packages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m:1246:36

ARC Semantic Issue (Xcode): No visible @interface for 'NSObject<FLTCaptureDeviceFormat>' declares the selector 'isVideoStabilizationModeSupported:'
/Users/leo_ruiz/repositories/xports/flutter_packages/packages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m:1262:38
2

Error running application on iPhone 11.

Exited (1).

Before compiling, I ran flutter clean and flutter pub get.

Environment details:

Flutter: 3.27.1 (stable channel)
Dart: 3.6.0
macOS: Version 15.2 (M1 Pro)
Xcode: 16.2
Device: iPhone 11 running iOS 18.3.2

Flutter doctor output:

❯ flutter doctor -v
[✓] Flutter (Channel stable, 3.27.1, on macOS 15.2 24C101 darwin-arm64, locale en-BR)
    • Flutter version 3.27.1 on channel stable at /Users/leo_ruiz/.asdf/installs/flutter/3.27.1
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 17025dd882 (3 months ago), 2024-12-17 03:23:09 +0900
    • Engine revision cb4b5fff73
    • Dart version 3.6.0
    • DevTools version 2.40.2

[✓] Android toolchain - develop for Android devices (Android SDK version 35.0.0)
    • Android SDK at /Users/leo_ruiz/Library/Android/sdk
    • Platform android-35, build-tools 35.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 16.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 16C5032a
    • CocoaPods version 1.16.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694)

[✓] VS Code (version 1.98.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.106.0

[✓] Connected device (5 available)
    • SM A055M (mobile)               • R9XX60289MX               • android-arm64  • Android 14 (API 34)
    • iPhone 11 (mobile)     • 00008030-0014059A1ED0402E • ios            • iOS 18.3.2 22D82
    • macOS (desktop)                 • macos                     • darwin-arm64   • macOS 15.2 24C101 darwin-arm64
    • Mac Designed for iPad (desktop) • mac-designed-for-ipad     • darwin         • macOS 15.2 24C101 darwin-arm64
    • Chrome (web)                    • chrome                    • web-javascript • Google Chrome 134.0.6998.166
    ! Error: Browsing on the local area network for iPad. Ensure the device is unlocked and attached with a cable or associated with the same local area network as this Mac.
      The device must be opted into Developer Mode to connect wirelessly. (code -27)

[✓] Network resources
    • All expected network resources are available.

• No issues found!

If you need any further information or additional testing on my side, please let me know. Thanks again for your support and the excellent work on this PR!

stuartmorgan-g · 2025-04-01T16:38:48Z

I have some broken tests, but I don't think they're related to anything I did, except merging the main branch onto this one. On my machine, all Dart unit tests related to this package are OK. Native tests are indeed failing, but they're also failing when I run them on the latest version of the main branch

I'm not sure why you would be getting failures on main; we run the tests continuously in CI. But the compile failures in the native tests here are related to the changes in the PR; likely the PR just needs to be updated to add the methods you are calling to the wrappers that were recently added as part of refactoring of the camera implementation to increase testability and allow for a migration to Swift (where we can't use OCMock).

stuartmorgan-g

The app-level fallback logic looks good (modulo questions about how off is managed in those APIs), mostly just minor comments before this can be split into sub-PRs.

stuartmorgan-g · 2025-04-01T16:41:26Z