What is the feature detection story for MediaStreamTrack support?

[beaufortfrancois]

As a web developer, I wish there was a way to feature-detect before calling start(audioTrack) following #118 changes. This is not possible for now, even after calling it.

May I suggest some ideas?

1. Rename start(MediaStreamTrack audioTrack) - bikeshedding is welcome

     undefined start();
-    undefined start(MediaStreamTrack audioTrack);
+    undefined startWithMediaStreamTrack(MediaStreamTrack audioTrack);

if ('startWithMediaStreamTrack' in webkitSpeechRecognition.prototype) {
  // MediaStreamTrack is supported by the Web Speech API.
}

2. Tie MediaStreamTrack support to other properties

It may be appropriate to assume some of the changes to the Web Speech API will come with MediaStreamTrack support. I'm not sure though which ones yet. Maybe SpeechRecognitionMode mode from #122?

if ('mode' in webkitSpeechRecognition.prototype) {
  // MediaStreamTrack is supported by the Web Speech API.
}

FYI @evanbliu

[padenot]

setSource(MediaStreamTrack), then start() would also allow detection.

[evanbliu]

#2 sounds like a pretty straightforward option that doesn't require any changes to the shape of the API. In Chromium, on-device speech recognition support and MediaStreamTrack support are both behind the same feature flag, so they will launch at the same time. This feature association isn't necessarily guaranteed by other browsers in the future though.

[beaufortfrancois]

This feature association isn't necessarily guaranteed by other browsers in the future though.

This worries me indeed.

setSource(MediaStreamTrack), then start() would also allow detection.

If we go down this road, would it make sense to have a source attribute reflected as well?

Moreover, the spec should say whether setSource(MediaStreamTrack) can be called after start() or not. I believe it should not be possible to call it successfully after speech recognition has started.

[evanbliu]

We could have a source attribute, but then it would make the setSource(MediaStreamTrack) redundant as it would only be needed for feature detection purposes. What do you all think about renaming start(MediaStreamTrack) to startWithMediaStreamTrack(MediaStreamTrack)?

[beaufortfrancois]

I'm fine with either Option 1 startWithMediaStreamTrack(MediaStreamTrack) modulo name, or a source attribute only as described below.

3. Add a readonly attribute source

partial interface SpeechRecognition : EventTarget {
    readonly attribute (MediaStreamTrack or undefined) source;
}

if ('source' in webkitSpeechRecognition.prototype) {
  // MediaStreamTrack is supported by the Web Speech API.
  
  const recognition = new webkitSpeechRecognition();
  recognition.start(myAudioTrack);
  console.log(recognition.source); // MediaStreamTrack myAudioTrack
}

Note that if we expect more sources support, we'll have the same issue next time to detect whether "newSource" is supported or not.

[padenot]

Can we not use the standard isConfigSupported pattern ?

[beaufortfrancois]

Can we not use the standard isConfigSupported pattern ?

The following seems a bit too much to me but why not?

4. Add static isSourceSupported(source)

dictionary SpeechRecognitionSourceSupport {
  boolean supported;
  SpeechRecognitionSource source;
};

typedef (MediaStreamTrack or undefined) SpeechRecognitionSource;

partial interface SpeechRecognition : EventTarget {
  static Promise<SpeechRecognitionSourceSupport> isSourceSupported(SpeechRecognitionSource source)
};

if ('isSourceSupported' in webkitSpeechRecognition.prototype) {
  const {supported} = await webkitSpeechRecognition?.isSourceSupported(myAudioTrack);
  if (supported) {
    // MediaStreamTrack is supported by the Web Speech API.
  }
}

[beaufortfrancois]

gentle ping

[padenot]

This is extensible, potentially providing an avenue to offer offline transcription ("faster-than-realtime") in the future, by adding a third SpeechRecognitionSource, that could be a ReadableStream.

MediaStreamTrack are always tied to a clock domain, be it of an audio device or system clock, and cannot work faster than real-time.

Considering the speed at which my voice-to-text transcription experiments (with various software and hardware) have been running, and the ongoing rate of progress of software and hardware, I think this is a good feature to consider.

[beaufortfrancois]

One thing I just realized is that isSourceSupported(myReadableStream) would throw a TypeError right away now IIUC because of typedef (MediaStreamTrack or undefined) SpeechRecognitionSource. Is that what we want or is it going to fail properly and reject with supported = false?

[beaufortfrancois]

I prototyped this in Chromium to see how this would work with https://chromium-review.googlesource.com/c/chromium/src/+/6179415/1 and it looks like Blink bindings are catching this type error before we can actually return a promise with supported: false:

await webkitSpeechRecognition.isSourceSupported(undefined);
// > {source: undefined, supported: true}
await webkitSpeechRecognition.isSourceSupported(1);
// > Uncaught TypeError: Failed to execute 'isSourceSupported' on 'SpeechRecognition': 
// > The provided value is not of type '(MediaStreamTrack or undefined)'.

Is this pattern still good enough?

try {
  const { supported } = await webkitSpeechRecognition.isSourceSupported(myAudioTrack);
  if (supported) {
    // MediaStreamTrack is supported by the Web Speech API.
  } else {
    throw Error('"¯\_(ツ)_/¯"');
  }
} catch(error) {
  // MediaStreamTrack is NOT supported by the Web Speech API.
}

[padenot]

Try to model this after Web Codecs, this works: https://w3c.github.io/webcodecs/#audiodecoder-interface.

[beaufortfrancois]

This model uses dictionary strings instead of "typedef (MediaStreamTrack or undefined)".
Would it be better this way?

dictionary SpeechRecognitionSource {
  required DOMString source; // "mediastreamtrack" or "microphone"
};

dictionary SpeechRecognitionSourceSupport {
  boolean supported;
  SpeechRecognitionSource source;
};

partial interface SpeechRecognition : EventTarget {
  static Promise<SpeechRecognitionSourceSupport> isSourceSupported(SpeechRecognitionSource source)
};

if ('isSourceSupported' in webkitSpeechRecognition.prototype) {
  const {supported} = await webkitSpeechRecognition?.isSourceSupported({source: "mediastreamtrack"});
  if (supported) {
    // MediaStreamTrack is supported by the Web Speech API.
  }
}

[beaufortfrancois]

@padenot @evanbliu What do you think?

[padenot]

This is in line with what I see in API "around" our API here, such as media decoding and encoding, and WebRTC, thanks a lot for taking the time to revise the proposal multiple times. Here are some simple change to this interface: lower-case strings is indeed the recommended way to do "enums" on the Web, but we can make it explicit, e.g. we'd have:

enum SpeechRecognitionSource {
  "microphone",
  "mediastreamtrack",
  "streams" // future extension w/ WhatWG Streams
}

dictionary SpeechRecognitionOptions {
  // Do we need more members here?
  required SpeechRecognitionSource source;
};

dictionary SpeechRecognitionSourceSupport {
  boolean supported;
  SpeechRecognitionOptions options;
};
partial interface SpeechRecognition : EventTarget {
  static Promise<SpeechRecognitionSourceSupport> isSourceSupported(SpeechRecognitionOptions options)
};

if we foresee more options to be added (otherwise, we can cut a layer). I'm thinking of a language identifier, or a model size, etc. Do we also want to add provision besides supported, e.g. "supported but we'd have to download a big model" ? I'm not entirely sure if this is warranted.

I understand it's a bit verbose, but I think the verbosity isn't unwarranted in this instance:

• asynchronicity is necessary, because it is possible to need e.g. cross-process calls or checking the presence of a model on disk before answering
• using an enum allows extensibility in the future