1500754 - Sky Go broken is broken in Firefox 64 (MEDIA KEYMESSERR LICENSER ERROR)

Status: RESOLVED INVALID

(Core :: Audio/Video: Playback, defect, P1)

Description

[Sören Hentzschel]

Sky Go is broken in Firefox 64. Sky is one of the most important streaming services in Germany (if not the most important because Germany is a soccer nation and Sky broadcasts most of the socccer matches ;-) ). So this is a serious regression. I attached a screenshot with the error message.

---

Firefox Nightly (64.0a1, 2018-10-20):
Widewine version: 4.10.1146.0
error message: MEDIA KEYMESSERR LICENSER ERROR / missing error message

It's broken since a few weeks at least, it's not a very recent regression.

Firefox Developer Edition (64.0b2):
Widewine version: 4.10.1146.0
error message: MEDIA KEYMESSERR LICENSER ERROR / missing error message

Firefox Beta (63 RC):
Widewine version: 1.4.9.1088
it works.

Firefox Stable (62.0.3):
Widewine version: 1.4.9.1088
it works.

Comment 1

[Sören Hentzschel]

Attachment: screenshot

Comment 2

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Thanks for the report. I'm not able to debug this directly due to geoblocking, but am reaching out to Sky to see what can be done.

Sören, would you be able to help us by collecting some logs of the issue taking place? If you could set the following env vars: MOZ_LOG=EME:5,GMP:5,sync MOZ_LOG_FILE=Bug1500754.txt (or similar) and attach the generated logs (there should be a number of files generated), that will hopefully help us get a better idea on what's going on.

Comment 3

[Sören Hentzschel]

Attachment: Archive.zip

(In reply to Bryce Seager van Dyk (:bryce) from comment #2) 
> Sören, would you be able to help us by collecting some logs of the issue
> taking place? If you could set the following env vars:
> MOZ_LOG=EME:5,GMP:5,sync MOZ_LOG_FILE=Bug1500754.txt (or similar) and attach
> the generated logs (there should be a number of files generated), that will
> hopefully help us get a better idea on what's going on.

I attached the logs.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

I assume this is fallout from the Widevine 4.10.1146.0 update that landed in 64.

Comment 5

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Thanks, Sören. Could you produce logs via the same steps, but for an older version of Firefox where the issue is not taking place? Are you able to check if the issue exists in Chrome?

:RyanVM, it looks that way.

The logs show some the CDM giving us a rather short response to `generateRequest()`. We're getting 2 bytes back in these logs. Relevant log line:
>> [Child 62197: Main Thread]: D/EME MediaKeySession[0x130a792c0,'A4381AFDB056F087FD1C965AB684B242'] DispatchKeyMessage() type=license-request message='0804'

Where `message='0804'` is the message we're going to pass back to Sky to go fetch its license. I can't test with Sky, but looking at logs from some local runs on other sites I see several hundred to more than a thousand bytes for this.

I do not believe how we handle setup of licenses changed with the most recent update. Another factor making this trickier is that Firefox does not control a lot of the data being exchanged here. I don't see any rejected calls by the Widevine CDM, and aside from the oddly short response from the CDM, it looks like a normal exchange to me.

I'm still waiting to hear from Sky and don't have any indication of when/if they'll respond. I'm going to reach out to Widevine to see if they have any insight into the CDM giving us such a short response.

Comment 6

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Discussion with Widevine suggests that the bytes being returned by the CDM indicate that Sky Go have not setServerCertificate on MediaKeys. This should either be done in response to the above bytes being returned, or done before attempting the license request. The logs don't show an attempt to do so -- though if such a request fails very early, it won't be logged.

This should need to be done for both the new and old CDM, so it's odd we'd see different behaviour unless Sky are serving different JS for new and old Firefoxes.

Sören, could you please let me know if you see anything in the browser console when the load fails? This will help disambiguate if Sky is trying to set the cert but failing very early.

Comment 7

[Sören Hentzschel]

(In reply to Bryce Seager van Dyk (:bryce) from comment #5)
> Thanks, Sören. Could you produce logs via the same steps, but for an older
> version of Firefox where the issue is not taking place?

I will attach logs in the next comment.

(In reply to Bryce Seager van Dyk (:bryce) from comment #5)
> Are you able to check if the issue exists in Chrome?

I can't test in Chrome because in Chrome Sky redirects to an app download instead.

(In reply to Bryce Seager van Dyk (:bryce) from comment #6) 
> Sören, could you please let me know if you see anything in the browser
> console when the load fails? This will help disambiguate if Sky is trying to
> set the cert but failing very early.

Using Encrypted Media Extensions at http://www.skygo.sky.de/sport/fussball-bundesliga/2.-bundesliga/fussball--2--bundesliga-msv-duisburg---fc-st--pauli--10--spieltag/asset/sportsection/820974.html on an insecure (i.e. non-HTTPS) context is deprecated and will soon be removed. You should consider switching to a secure origin such as HTTPS. html5player.min.js:48:11882 

navigator.requestMediaKeySystemAccess promise rejected 0x80530009 'Key system is unsupported'

MediaKeySystemAccess::GetKeySystemStatus(com.widevine.alpha) result=available msg=''

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://wvguard.sky.de/WidevineLicenser/WidevineLicenser. (Reason: CORS request did not succeed).

Invalid URI. Load of media resource  failed.

AbortError: The fetching process for the media resource was aborted by the user agent at the user's request.

AbortError: The operation was aborted.

video CANPLAY event never fired! Restarting playback

Comment 8

[Sören Hentzschel]

Attachment: Archive-Fx62.zip

I attached logs from a working Firefox.

For reference: The working Firefox also shows the following errors and warnings in the console:

Using Encrypted Media Extensions at http://www.skygo.sky.de/sport/fussball-bundesliga/2.-bundesliga/fussball--2--bundesliga-msv-duisburg---fc-st--pauli--10--spieltag/asset/sportsection/820974.html on an insecure (i.e. non-HTTPS) context is deprecated and will soon be removed. You should consider switching to a secure origin such as HTTPS. html5player.min.js:48:11882

navigator.requestMediaKeySystemAccess promise rejected 0x80530009 'Key system is unsupported'

MediaKeySystemAccess::GetKeySystemStatus(com.widevine.alpha) result=available msg=''

But these errors and warnings don't appear in the working Firefox, only in Firefox 64:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://wvguard.sky.de/WidevineLicenser/WidevineLicenser. (Reason: CORS request did not succeed).

Invalid URI. Load of media resource  failed.

AbortError: The fetching process for the media resource was aborted by the user agent at the user's request.

AbortError: The operation was aborted.

video CANPLAY event never fired! Restarting playback

Comment 9

[Sören Hentzschel]

> Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://wvguard.sky.de/WidevineLicenser/WidevineLicenser. (Reason: CORS request did not succeed).

It seems to be an certificate issue. I opened https://wvguard.sky.de/WidevineLicenser/WidevineLicenser in a new tab and there was an error page (MOZILLA_PKIX_ERROR_ADDITIONAL_POLICY_CONSTRAINT_FAILED). After pressing the button for accepting the risk Sky Go works again (until the next browser restart, then I have to repeat this).

Comment 10

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Thanks for the detailed responses. This doesn't look to be related to the recent CDM update we did after all -- removing this as a blocker.

What appears to be happening is that Sky have a certificate issued by 'thawte, Inc.' for use on `https://wvguard.sky.de/WidevineLicenser/WidevineLicenser`. Given issues with certs from this authority[0] (and several others), we've been rolling out mitigations. These mitigations (blocking the site unless an exception is added) are being rolled out in steps, and that's what's causing the difference between the versions here. In newer versions when the Sky player tries to reach the Licenser domain it will be blocked.

This is something Sky will need to fix on their end. I'm not sure of an appropriate bug category to track this, and I don't think it's webcompat (other vendors are moving to block these certs), so I'm going to resolve this. That said, I will continue to try and get in touch with Sky to resolve this on their end.

[0]: https://wiki.mozilla.org/CA:Symantec_Issues

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Has anything improved on Sky's end at this point?

Comment 12

[Sören Hentzschel]

I would say 'yes' because https://wvguard.sky.de/WidevineLicenser/WidevineLicenser now has a valid DigiCert certificate and no longer uses a certificate issued by Thawte. But like for Chrome users Sky now forces the use of their app for Firefox users, too, and no longer allows me to use the web version at all.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

Ugh :(. Calling this fixed, for some definition of "fixed" anyway...