1132594 - Gradual Memory consumption leading to crashes (see testcase)

Status: RESOLVED WORKSFORME

(Core :: Audio/Video: Playback, defect, P1)

Description

[Caspy7]

Attachment: Just before a crash (report: bp-5589a91e-8edd-4e37-99f1-71b582150212)

I've had memory and crash problems for some time now and decided to create a reduced testcase to document some hopefully-helpful data.

My issues have often seem connected to images and recently HTML5 video has seemed a worse culprit.  I suspect my GPU is somehow involved and have been told it's a known problematic graphics card (but I can't do anything about that).

I'm running Windows 7, Firefox 37.0a2.  
I created a fresh profile, set the homepage to six tabs of the same URL: https://www.youtube.com/watch?v=LizAjaOXi4U (defaults to 480p)
Let the videos play for at least 30 seconds and closed them all leaving a blank tab and then waited and checked Firefox's memory usage in my process browser.  I then open the six tabs again. 
The "resting" memory climbs upward each time I open the six tabs.

In my recent round of testing I got three crashes:
bp-f2d768d2-8aae-49d3-9fb2-dde382150212
bp-55acc24a-22e8-45dd-9876-2c6062150212
bp-5589a91e-8edd-4e37-99f1-71b582150212

The last of which I generated a memory report soon before crashing (which I'm attaching).  It was about 794MB at the time of the report. I then opened the youtube tabs one last time before it crashed at 1.07 GB

Since fixing bug 1131768 it's definitely doing *better* than it was. The baseline memory expansion is not as severe, but I was able to continually run it up from the one blank tab at 244MB to one blank tab at 794MB as well as incur multiple crashes.

On my normal profile, with multiple addons and unrestored tabs, Firefox often goes past 2 GB before having problems and then crashing.

Comment 1

[Matt Woodrow (:mattwoodrow)]

This one is interesting, we don't appear to be leaking d3d9 textures here, which is the case for the others we've looked at.

Comment 2

[Matt Woodrow (:mattwoodrow)]

Can you please attach the contents of about:support so we know which GPU this is.

Comment 3

[Matt Woodrow (:mattwoodrow)]

Anthony, can you reproduce this on your machine?

Comment 4

[Caspy7]

Attachment: Troubleshooting Information

Comment 5

[Caspy7]

Attachment: Memory Leak Test using only images (Flickr, Yahoo Image Search)

Because in the past I have associated my issues with images and then with the use of HTML5 video, they grew quickly worse, I decided to run a similar test, but rather than loading videos I loaded long pages of Flickr favorite images and then periodically close the window, leaving one tab, and then minimized memory.
I eventually included yahoo images searches.

I was able to get it up to 720MB for resting/minimized memory (again, starting from about 244MB).  Oddly after a few more rounds the size went down slightly getting to about 700MB. All these readings are according to Windows "Private Bytes" reading.  I'm attaching the memory report taken directly after that.

I figure if this memory bloat is related, it may show up in the memory report.

Comment 6

[Matt Woodrow (:mattwoodrow)]

Does the leak still occur with media.windows-media-foundation.use-dxva=false?

Comment 7

[(Away)]

Can you try getting the memory as high as possible, then force a crash with this extension: http://ted.mielczarek.org/mozilla/crashme.html

The resulting crash dump may offer some clues about where that memory is coming from.

Comment 8

[Matt Woodrow (:mattwoodrow)]

David, the crashes linked from the first post were done with when watching video.

Is that sufficient, or does it need to be a forced crash?

Comment 9

[Caspy7]

(In reply to Matt Woodrow (:mattwoodrow) from comment #6)
> Does the leak still occur with media.windows-media-foundation.use-dxva=false?

It got up to 450MB resting and then on the 3rd or 4th trial it crash - which seemed a little early for normal testing.

Comment 10

[Caspy7]

(In reply to David Major [:dmajor] (UTC+13) from comment #7)
> Can you try getting the memory as high as possible, then force a crash with
> this extension: http://ted.mielczarek.org/mozilla/crashme.html
> 
> The resulting crash dump may offer some clues about where that memory is
> coming from.

Were you meaning for the images test I commented about? The video testing was basically at peak memory for all three crashes I originally listed.  
I did try and abuse the images testing, but was unable to trigger a spontaneous crash.

Comment 11

[(Away)]

The first three crashes didn't seem to be that high on memory usage (still 2GB+ address space available in all of them) and the crash signatures did not appear to be OOM related. At those levels it's less obvious what's leaking versus legitimate memory usage. If the leak can be made more extreme then it stands out better in the reports. If it's not possible then we can try to make do with what we have.

Comment 12

[Marcia Knous [:marcia]]

Caspy7: Can you please check the most recent Aurora build and see if the issue still persists?  Thanks.

Comment 13

[Matt Woodrow (:mattwoodrow)]

We might need to wait until bug 1128170 gets uplifted to Aurora, that one more closes matches this STR.

Comment 14

[(Away)]

(In reply to Matt Woodrow (:mattwoodrow) from comment #13)
> We might need to wait until bug 1128170 gets uplifted to Aurora, that one
> more closes matches this STR.

That bug is now fixed on Aurora. Caspy7, can you give the latest build a try?

Comment 15

[Caspy7]

I ran the same test as previously using a recent build: 37.0a2 (2015-02-21).  Sorry I didn't use today's but I got the impression that yesterday's should have had the appropriate fix.

Starting with a blank tab: 222 MB
After multiple tests, the "resting" memory (after minimization) was 453 MB
Had a peak use of 1.13 GB while playing videos

It did not crash on its own (and I took no extraordinary video playback means to do so), so I induced a crash using the Crash Me Now addon (while resting with just a blank tab open). 
I'm attaching the memory report taken directly before the crash and the crash report was bp-6e808aed-2a6f-4be8-baa0-f01342150223

The last 3-4 tests all were not too far from 450MB for resting memory.

Comment 16

[Caspy7]

Attachment: Testing after the Fix

Comment 17

[(Away)]

> The last 3-4 tests all were not too far from 450MB for resting memory.
Do I understand correctly, that subsequent rounds of your test do not increase memory usage (as reported by your process monitor)? If so, that's good news!

However, I'm still worried about the numbers in attachment 8567782 [details] and bp-6e808aed-2a6f-4be8-baa0-f01342150223:
102.85 MB (100.0%) ++ explicit
  233.00 MB ── heap-mapped
  461.34 MB ── private
  450.42 MB ── resident
1,234.18 MB ── vsize

Does your 'vsize' continue to grow after many rounds of testing?

Comment 18

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Does setting media.decoder.heuristic.dormant.enabled=true in about:config make a difference?

Comment 19

[Caspy7]

Attachment: March 9 testing Memory Report

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #18)
> Does setting media.decoder.heuristic.dormant.enabled=true in about:config
> make a difference?

This setting was already true.

(In reply to David Major [:dmajor] (UTC+13) from comment #17)
> Does your 'vsize' continue to grow after many rounds of testing?

I got around to testing again, here are my results.
I ran the test case as before, letting the videos play between 30 seconds and 1.5 minutes. Recording "Private bytes" and "Virtual Size" (as reported by my process manager) after each test. 
Here are the first eight runs: (all numbers are in MB unless otherwise noted)

Private bytes, Virtual Bytes
baseline: 227.88, 592.32
367.17, 915.73
407.00, 1009.84
435.64, 1020.32
453.11, 1.01 GB
445.76, 1.02 GB
463.83, 1.04 GB
466.68, 1.04 GB
456.54, 1.07 GB

Because I apparently have no life, I then ran the videos for a full 10 minutes before closing the window (just shy of the videos ending).
473.55, 1.18 GB
487.49, 1.24 GB
491.24, 1.33 GB 

I then saved a memory report (which I'm uploading) and induced a crash with the signature bp-19138af6-5d1f-4ccb-ba1a-f2ace2150309

This was done in Firefox 38.0a2 (2015-03-09).
It looks like the answer to the question of continued VSize growth is "yes."

Comment 20

[Ralph Giles (:rillian)]

Hi Caspy. Thanks for following up. Reducing priority since it sounds like this isn't causing so many crashes now. Which _is_ good news.

If you have time to try again, a fresh profile with a 37 beta would be helpful for comparison. We'd like to get data on how it's going with the default dormant and other pref settings.

Comment 21

[(Away)]

At the risk of adding distraction: A memory report from a nightly 39 could also be useful, as bug 1134030 has some reporting for these mystery memory regions (where vsize is high but explicit is low).

Comment 22

[(Away)]

(In reply to David Major [:dmajor] (UTC+13) from comment #21)
> At the risk of adding distraction: A memory report from a nightly 39 could
> also be useful, as bug 1134030 has some reporting for these mystery memory
> regions (where vsize is high but explicit is low).

That reporting has been uplifted, so any 37 or 38 built after today would also work.

Comment 23

[Caspy7]

Attachment: memory-report-Mar12-A.json.gz

Everything I said in my previous comment holds true except this testing was done in the most recent Beta (as of today) 37.0.

baseline: 218.39, 578.30
327.53, 867.06
367.40, 917.18
388.76, 940.84
394.91, 942.37
415.07, 960.16
415.48, 977.70
432.25, 984.49
440.40, 999.05
====
457.09, 1.1 GB
465.16, 1.11 GB
473.36, 1.11 GB 

Attached the memory report taken after testing.
Induced a crash after testing: bp-70e871cb-ed64-49cb-8399-7fb362150312

Comment 24

[Jean-Yves Avenard [:jya]]

Is this still an issue?

not an MSE bug anyway