Skip to content

MediaStreamConstraints for getUserMedia #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
May 7, 2023
Merged

MediaStreamConstraints for getUserMedia #270

merged 4 commits into from
May 7, 2023

Conversation

orottier
Copy link
Owner

@orottier orottier commented May 7, 2023

Seems to work on my device, however some caveats

  1. microphone input is not buffered so some frames are lost (about a single chunk of 128 frames every second on my device). This introduces some crackling. You can see this when you run RUST_LOG=debug cargo run --release --example microphone
  2. our cubeb input implementation only supports stereo microphone input. Not sure how I can fix that.. The whole cubeb implementation is experimental anyway cubeb audio backend improvements #187
  3. when selecting a non-default sink-id, closing the media track does not actually suspend the microphone. I suspect this is a cpal bug Stream dropped but audio callback still being called RustAudio/cpal#771

1 and 2 are not related to this PR by the way.
In any case I think we can merge this despite the caveats.
Let me know how it works for you!

@orottier orottier requested a review from b-ma May 7, 2023 08:53
orottier
orottier commented May 7, 2023
View reviewed changes
src/media_devices/mod.rs Outdated
@@ -89,6 +89,51 @@ impl MediaDeviceInfo {
}
}

/// Dictionary used to instruct what sort of tracks to include in the [`MediaStream`] returned by
/// [`get_user_media`]
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this to be the most friendly way of modeling the constraints, but let me know if you think otherwise

@github-actions
Copy link

github-actions bot commented May 7, 2023 

Benchmark result:


bench_ctor
  Instructions:             5165068 (+0.000019%)
  L1 Accesses:              7759204 (+0.000077%)
  L2 Accesses:                54126 (-0.001848%)
  RAM Accesses:               61572 (-0.006496%)
  Estimated Cycles:        10184854 (-0.001365%)

bench_sine
  Instructions:            78597680 (+0.000001%)
  L1 Accesses:            115799288 (+0.001567%)
  L2 Accesses:               273944 (-0.656022%)
  RAM Accesses:               62655 (-0.007980%)
  Estimated Cycles:       119361933 (-0.006203%)

bench_sine_gain
  Instructions:            83572805 (+0.000001%)
  L1 Accesses:            123330044 (-0.001539%)
  L2 Accesses:               305622 (+0.626566%)
  RAM Accesses:               62809 (-0.006368%)
  Estimated Cycles:       127056469 (+0.005885%)

bench_sine_gain_delay
  Instructions:           156007353 (+0.000001%)
  L1 Accesses:            222972563 (-0.014567%)
  L2 Accesses:               764135 (+4.440821%)
  RAM Accesses:               64486 (-0.006203%)
  Estimated Cycles:       229050248 (+0.056714%)

bench_buffer_src
  Instructions:            19000610 (+0.000074%)
  L1 Accesses:             27986571 (-0.001633%)
  L2 Accesses:                93331 (+0.522370%)
  RAM Accesses:               96109 (-0.013524%)
  Estimated Cycles:        31817041 (+0.004756%)

bench_buffer_src_delay
  Instructions:            90569695 (+0.000021%)
  L1 Accesses:            126610908 (-0.016099%)
  L2 Accesses:               229051 (+9.788142%)
  RAM Accesses:               96293 (-0.013499%)
  Estimated Cycles:       131126418 (+0.062012%)

bench_buffer_src_iir
  Instructions:            51244268 (+0.000096%)
  L1 Accesses:             71550727 (-0.008325%)
  L2 Accesses:               113810 (+5.585913%)
  RAM Accesses:               96203 (-0.015590%)
  Estimated Cycles:        75486882 (+0.031304%)

bench_buffer_src_biquad
  Instructions:            40052723 (-0.000027%)
  L1 Accesses:             56969492 (-0.014290%)
  L2 Accesses:               184566 (+4.615611%)
  RAM Accesses:               96294 (-0.016613%)
  Estimated Cycles:        61262612 (+0.052283%)

bench_stereo_positional
  Instructions:            48505429 (+0.000058%)
  L1 Accesses:             72409511 (+0.006371%)
  L2 Accesses:               748830 (-0.606583%)
  RAM Accesses:               96476 (-0.013473%)
  Estimated Cycles:        79530321 (-0.023497%)

bench_stereo_panning_automation
  Instructions:            33188830 (+0.000003%)
  L1 Accesses:             49848042 (-0.028645%)
  L2 Accesses:               203104 (+7.572853%)
  RAM Accesses:               96227 (-0.014547%)
  Estimated Cycles:        54231507 (+0.104693%)

bench_analyser_node
  Instructions:            39533019 (+0.000010%)
  L1 Accesses:             55847655 (-0.027968%)
  L2 Accesses:               221894 (+7.585491%)
  RAM Accesses:               96810 (-0.020655%)
  Estimated Cycles:        60345475 (+0.102683%)

@b-ma
Copy link
Collaborator

b-ma commented May 7, 2023

Cool, the API looks ok to me

Just tested on my side and, a bit hard to tell, but point 1 doesn't seem to be an issue on my machine. Points 2 and 3 behave the same.

One last think I'm just thinking about, maybe we could just rename get_user_media to get_user_media_sync (just as start_rendering_sync and decode_audio_data_sync)?

@github-actions
Copy link

github-actions bot commented May 7, 2023 

Benchmark result:


bench_ctor
  Instructions:             5165068 (+0.000019%)
  L1 Accesses:              7759204 (+0.000090%)
  L2 Accesses:                54125 (-0.003695%)
  RAM Accesses:               61573 (-0.006496%)
  Estimated Cycles:        10184884 (-0.001404%)

bench_sine
  Instructions:            78597680 (+0.000001%)
  L1 Accesses:            115799150 (+0.001568%)
  L2 Accesses:               274081 (-0.656056%)
  RAM Accesses:               62656 (-0.007979%)
  Estimated Cycles:       119362515 (-0.006207%)

bench_sine_gain
  Instructions:            83572805 (+0.000001%)
  L1 Accesses:            123329206 (-0.002767%)
  L2 Accesses:               306457 (+1.127574%)
  RAM Accesses:               62812 (-0.006368%)
  Estimated Cycles:       127059911 (+0.010652%)

bench_sine_gain_delay
  Instructions:           156007353 (+0.000001%)
  L1 Accesses:            222963007 (-0.013770%)
  L2 Accesses:               773690 (+4.133635%)
  RAM Accesses:               64487 (-0.006202%)
  Estimated Cycles:       229088502 (+0.053595%)

bench_buffer_src
  Instructions:            19000610 (-0.000021%)
  L1 Accesses:             27986383 (-0.001701%)
  L2 Accesses:                93519 (+0.521315%)
  RAM Accesses:               96109 (-0.015605%)
  Estimated Cycles:        31817793 (+0.004476%)

bench_buffer_src_delay
  Instructions:            90569616 (-0.000018%)
  L1 Accesses:            126612235 (-0.013367%)
  L2 Accesses:               227640 (+8.030638%)
  RAM Accesses:               96295 (-0.013498%)
  Estimated Cycles:       131120760 (+0.051298%)

bench_buffer_src_iir
  Instructions:            51244212 (-0.000008%)
  L1 Accesses:             71552124 (-0.008168%)
  L2 Accesses:               112352 (+5.494836%)
  RAM Accesses:               96206 (-0.013511%)
  Estimated Cycles:        75481094 (+0.030427%)

bench_buffer_src_biquad
  Instructions:            40052749 (+0.000037%)
  L1 Accesses:             56975483 (-0.015753%)
  L2 Accesses:               178602 (+5.310267%)
  RAM Accesses:               96298 (-0.013498%)
  Estimated Cycles:        61238923 (+0.058164%)

bench_stereo_positional
  Instructions:            48505431 (+0.000031%)
  L1 Accesses:             72401079 (+0.014628%)
  L2 Accesses:               757263 (-1.375060%)
  RAM Accesses:               96477 (-0.015545%)
  Estimated Cycles:        79564089 (-0.053671%)

bench_stereo_panning_automation
  Instructions:            33188833 (+0.000012%)
  L1 Accesses:             49852899 (-0.018920%)
  L2 Accesses:               198250 (+5.006965%)
  RAM Accesses:               96227 (-0.015586%)
  Estimated Cycles:        54212094 (+0.068862%)

bench_analyser_node
  Instructions:            39533056 (+0.000134%)
  L1 Accesses:             55863501 (-0.002732%)
  L2 Accesses:               206087 (+0.783924%)
  RAM Accesses:               96809 (-0.020655%)
  Estimated Cycles:        60282251 (+0.009604%)

@orottier orottier merged commit 9344b14 into main May 7, 2023
@orottier orottier deleted the feature/MediaStreamConstraints branch June 2, 2023 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@b-ma b-ma Awaiting requested review from b-ma

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@orottier @b-ma