1527875 - [Encrypted Media] Firefox Only failing Web Platform Tests

Status: NEW

(Core :: Audio/Video: Playback, defect, P3)

Description

[David Burns :automatedtester]

The following are Firefox only failing web platform tests and are passing in Safari and Chrome.

/encrypted-media/drm-check-encryption-scheme.https.html [1 subtests]
▶ /encrypted-media/drm-check-status-for-hdcp.https.html [1 subtests]
▶ /encrypted-media/drm-events-session-closed-event.https.html [1 subtests]
▶ /encrypted-media/drm-events.https.html [1 subtests]
▶ /encrypted-media/drm-expiration.https.html [1 subtests]
▶ /encrypted-media/drm-generate-request-disallowed-input.https.html [1 subtests]
▶ /encrypted-media/drm-invalid-license.https.html [1 subtests]
▶ /encrypted-media/drm-keystatuses-multiple-sessions.https.html [1 subtests]
▶ /encrypted-media/drm-keystatuses.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-destroy-persistent-license.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-persistent-license-events.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-persistent-license.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-persistent-usage-record-events.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-persistent-usage-record.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-retrieve-destroy-persistent-license.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-retrieve-persistent-license.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-retrieve-persistent-usage-record.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-clear-encrypted.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-encrypted-clear-sources.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-encrypted-clear.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-events.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-expired.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-multikey-sequential.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-multikey.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-multisession.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-playduration-keystatus.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-playduration.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-src.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-setMediaKeys-immediately.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-setMediaKeys-onencrypted.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-two-videos.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-waitingforkey.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-requestmediakeysystemaccess.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-reset-src-after-setmediakeys.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-again-after-playback.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-again-after-resetting-src.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-at-same-time.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-multiple-times-with-different-mediakeys.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-multiple-times-with-the-same-mediakeys.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-to-multiple-video-elements.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-syntax-mediakeys.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-syntax-mediakeysession.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-syntax-mediakeysystemaccess.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-unique-origin.https.html [1 subtests]
▶ /encrypted-media/drm-not-callable-after-createsession.https.html [1 subtests]
▶ /encrypted-media/drm-temporary-license-type.https.html [1 subtests]

See https://jgraham.github.io/wptdash/?bugComponent=core%3A%3Aaudio%2Fvideo%3A+playback

Comment 1

[Paul Adenot (:padenot)]

Some of those are expected, but I'd expect us to be more compliant on most of those.

jya, bryce, is this something that we'd expect based on the current implementation?

Comment 2

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Agreed that I'd expect us to work for more of those. Bug 1522547 may be complicating matters, which I'm currently working on.

If I follow the live link for some of those tests I see a couple of things:

• Block autoplay seems to result in us failing the test.
• Chrome has the same issue but is marked as passing.

E.g. encrypted-media/drm-mp4-playback-temporary.https.html (note I've modified the link to be https compared to the original on the page in comment #0)

So perhaps some of these are block autoplay related?

I'm going to hold the NI to remind me to take another look at this once I'm done with that other bug.

Comment 3

[David Burns :automatedtester]

It would appear that we have some unstable tests looking at the data today. We have

/encrypted-media/drm-events-session-closed-event.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-persistent-license.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-persistent-usage-record-events.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-persistent-usage-record.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-encrypted-clear.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-expired.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-multikey-sequential-readyState.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-setMediaKeys-after-update.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary-setMediaKeys-immediately.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-playback-temporary.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-at-same-time.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-multiple-times-with-different-mediakeys.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-setmediakeys-multiple-times-with-the-same-mediakeys.https.html [1 subtests]
▶ /encrypted-media/drm-mp4-syntax-mediakeys.https.html [1 subtests]
▶ /encrypted-media/drm-not-callable-after-createsession.https.html [1 subtests]

from the same link in comment 0. Could someone see if there was a way to make these tests more stable?

Comment 4

[Jean-Yves Avenard [:jya]]

Leaving :bryce to investigate

Comment 5

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Bug 1522547 is fixed, which helps a bit here. But I see a number of other things causing us to have failures here (non-exhaustive):

• We're timing out a lot, I notice we're having issues with some tests where we should be fetching the CDM and then running the test, but we just time out.
  • We're timing out on harness status for a lot of tests.
• I don't believe we implement some of the APIs, like persistent licenses.

I'd like to look at the encrypted cases in particular, but it's a matter of getting some cycles to do so. Going to hold the NI for now to remind me to take a look if I can.

Comment 6

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

I had dug into some of these, but never had time to get to the bottom of all the tests. Passing NI.