test262 tests

[domenic]

One of the requirements for stage 4 is test262 tests. Besides being a process requirement, it's also great for interoperability.

Are there people that are willing to help work on this? I know @caitp has been working on an implementation in V8, which comes with its own tests presumably in mjsunit format; apparently it's now possible to write test262 tests directly in V8 for later upstreaming, which would be one route.

Alternately, the community and/or myself could help port @caitp's tests and write our own. @benjamingr seems to have volunteered :)

One valuable thing we can do besides even writing tests is come up with a test plan. That is, go through the spec and write down all the testable things and edge case behaviors that we want to be sure to hit when writing tests. Just getting that done alone will help the test writers a lot.

[littledan]

cc @leobalter @rwaldron

[benjamingr]

I'd love to help with this but I'm not really sure how to get started.

[gskachkov]

+1 I would like to participate also.

[domenic]

I'd love to help with this but I'm not really sure how to get started.

I think a test plan would be the most valuable thing to get started. If you really want to dive in and start writing tests, you can use e.g. https://github.com/tc39/test262/pull/479/files as a guide.

[caitp]

Here's a starter test plan informed by my experience prototyping. There's plenty of stuff missing, though.

• Symbol.asyncIterator
  1. test exposed on Symbol with correct property descriptor
• AsyncGeneratorFunction builtin
  1. not exposed on global
  2. exposed via .constructor with correct property descriptor, etc
  3. has the right length value
  4. tests relating to CreateDynamicFunction
  5. tests relating to Function.prototype.toString() (similar to other tests upstream in test262 --- I really don't want to be the one to write these ones :))
• For-Await-OF / GetIterator
  1. Test Symbol.asyncIterator is invoked if present on an object during for-await-of (only caller of GetIterator(iterable, async)
  2. Test restriction that Symbol.asyncIterator must return an Object else throw a TypeError
  3. Test that, in the absence of a a Symbol.asyncIterator property (property is undefined or null), Symbol.iterator is invoked instead (and again, if absent, a TypeError is thrown).
  4. Test that in the absence of an asyncIterator symbol, the iterator returned is always an instance of Async-from-Sync Iterator, with [[Prototype]] == %AsyncFromSyncIteratorPrototype%, regardless of what iterator type is returned (see Async-from-Sync iterator section)
• Async-from-Sync Iterator
  1. Unfortunately, short of for-await-of, there isn't a way to observe access to the Async-from-Sync iterator. If no method for programmatically creating an Async-from-Sync iterator is introduced, we need to expose VM hooks for doing it (similar to $.detachArrayBuffer (Use $.detachArrayBuffer API instead of throwing test262#795)).
  2. Test that .next(), .throw() and .return() return Promise instances, which are eventually resolved with expected values.
  3. Test corner cases. In normal Iterators, .return() and .throw() are optional, and for-of loops invoke them conditionally if present. Async-from-Sync iterators always expose them, so we want to check behaviours of when the sync iterator provides them, doesn't provide them, or provides misbehaving versions of them. I don't have a complete list of these corner cases in mind, but I am happy to help get someone started (or finish it myself), just ping me on mozilla IRC in #jslang, or in #v8 on freenode.
  4. Test that, If the value of a sync iterator is a Promise, the resolved value of the Promise becomes the value of the unwrapped iterator result
  5. Similarly, test that if the promise is rejected, the returned Promise from the async-from-sync iterator method is also rejected (Ping @domenic: I think currently Async-from-Sync Iterators just eat these rejections and aren't specified to reject the returned Promise (https://tc39.github.io/proposal-async-iteration/#sec-async-iterator-value-unwrap-functions), but that seems like an oversight. Edit: this seems to work as expected, currently, although I have no idea why it works for rejected Promises, but not resolved Promises. Edit2: I think either A) the spec was not actually eating rejections here and the default promise rejection handler does the right thing, or B) there's a bug in v8 re the default Promise rejection handler, that happens to make things work slightly better/faster :) But we'll look at that elsewhere)
• Await Expressions in Async Generators
  1. Test that the Generator is not resumed when an Await is in progress (Await must block resumption, otherwise resuming from the Await can have the Generator at an unexpected continuation, or even crash if the VM has "invalid jump table index" assertions like V8 does!)
  2. Test that invalid optimizations (such as https://bugs.chromium.org/p/v8/issues/detail?id=5691, https://bugs.chromium.org/p/v8/issues/detail?id=5694#c3) are not present when evaluating AwaitExpressions.
• Async Generators
  1. Test that it's possible to call .next(), .resume() and .throw() during execution of an async generator, and those requests will be honoured in order (order of returned Promise resolution for those requests should match the order the requests were made)
  2. Test that once the async generator is closed, it is not possible to resume it any longer

There's a lot of stuff I've left out of this list:

• yield* in async generators
• Prototype inheritance of builtin AsyncIterators
• For-await-of being allowed in "ordinary" Async Functions
• More in-depth testing of Async Generator resume mechanics (similar to the Sync Iterator tests)

[benjamingr]

Awesome, thanks! @domenic @caitp I'll take a look this Friday and I'll ensure I know what I'm doing that I'll let you know if I get stuck.

[domenic]

Unfortunately, short of for-await-of, there isn't a way to observe access to the Async-from-Sync iterator. If no method for programmatically creating an Async-from-Sync iterator is introduced, we need to expose VM hooks for doing it (similar to $.detachArrayBuffer

I don't think we want to add this kind of direct test to test262. Instead, we only want to test the aspects that are observable from the language, i.e. for-await-of and yield*. If some implementation were able to implement those without every creating a concrete Async-from-Sync iterator class, then that's great, and should be allowed! We shouldn't write test262 tests that they would start failing.

Maybe we should even mention in the spec the aspects of Async-from-Sync iterator that are unobservable. It sounds like there are a lot.

[benjamingr]

Ok, I have a few volunteers and we'll meet and discuss this this Friday.

[caitp]

I think it's worth having tests upstream wrt interactions between the async iterator and sync iterator (and whoever is calling the iterator methods), because those are observable. It's awkward to only test it in for-await-of, especially if eventually there is another user of GetIterator(async) user.

[domenic]

It may be awkward, but I think it's worth writing awkward tests to allow implementations to optimize fully.

[caitp]

well, by "VM Hooks" I'm not necessarily saying "expose Async-from-Sync Iterator to normal JS", I'm saying "allow VM test harnesses to provide a way to perform this thing, even if normal JS code can't"

I'm absolutely in favour of minimizing the cost of the runtime Async-from-Sync iterator proxy, and if subclassing it or manually creating it, or modifying its prototype chain is out of the question, that works for me

[domenic]

Yeah, but my point is that some implementations might not even be able to expose it, if they've optimized it away super-extensively, and so we shouldn't write tests that those implementations are unable to pass (since they are unable to provide $.createAsyncFromSyncIterator at all).

[thefourtheye]

Could someone please change @caitp's test plan, a check-list? We can tick them whenever tests land.

[benjamingr]

Here's a link to @thefourtheye 's work which we'll build on on Friday (if he won't finish it by then :P)

#71