Closed Bug 1125557 Opened 10 years ago Closed 8 years ago

Very high memory use on startup with many unloaded tabs

Categories

(Firefox :: General, defect)

48 Branch
x86_64
Windows 7
defect
Not set
normal

Tracking

()

RESOLVED DUPLICATE of bug 906076

People

(Reporter: tyaremco, Unassigned, NeedInfo)

References

Details

(Whiteboard: [MemShrink:P3])

Attachments

(1 file, 2 obsolete files)

When I have many tabs in my session store (about 1000) Firefox Nightly and the plugin container report very high memory usage (nearly 1.5GB combined) on browser startup and thereafter. Because I've just started the browser and all my tabs (except the focused tab) are in an _unloaded_ state I would not expect Firefox to use that much memory. If I use the same profile but replace the sessionstore.sqlite with one having a single tab, the memory use is only about 200MB.
What about a clean profile with only that session store file?
Flags: needinfo?(tyaremco)
All my testing was done in a clean profile created by Nightly.
Flags: needinfo?(tyaremco)
Do you use e10s ?
See Also: → 1125578
If I disable e10s, memory lowers to approx. 1GB on startup with about:preferences the only tab loaded. About the same in Firefox 37.
Whiteboard: [MemShrink]
I looked closely at the memory reports. In short, even an unloaded tab has a small cost, and when you have 1,100 of them those small costs add up. There are lots of ways to use a browser and I don't bat an eyelid when I hear of people having hundreds of tabs open, but there are only a tiny fraction of users who exceed 1,000 tabs -- you're the second person I've heard of doing so. It's inevitable it's going to stress the browser, particularly if you're using 32-bit builds (as is standard on Windows). Having said that, there were a couple of surprising things. Details below. ------ PARENT ------ Totals: > 777.46 MB ── private > 754.90 MB ── resident > 1,186.34 MB ── vsize Window objects: ├──118.88 MB (19.75%) -- window-objects │ ├───92.29 MB (15.33%) -- top(<anonymized-19>, id=19) │ │ ├──92.10 MB (15.30%) -- active │ │ │ ├──92.10 MB (15.30%) -- window(<anonymized-39>) │ │ │ │ ├──91.01 MB (15.12%) -- dom │ │ │ │ │ ├──53.73 MB (08.92%) ── element-nodes │ │ │ │ │ ├──26.18 MB (04.35%) ── text-nodes │ │ │ │ │ ├──11.10 MB (01.84%) ── other │ │ │ │ │ └───0.00 MB (00.00%) ++ (2 tiny) │ │ │ │ └───1.09 MB (00.18%) ++ (4 tiny) │ │ │ └───0.00 MB (00.00%) ++ window(<anonymized-19>)/dom │ │ └───0.19 MB (00.03%) ++ js-zone(0xa29e400) This must be a chrome page such as one of the about: pages. My guess is that it is about:memory itself. ├──131.67 MB (21.87%) -- workers/workers(chrome) │ ├──127.82 MB (21.23%) -- worker(resource:///modules/sessionstore/SessionWorker.js, 0x76bfc00) │ │ ├──126.24 MB (20.97%) -- zone(0x133ca000) │ │ │ ├───83.93 MB (13.94%) ++ strings │ │ │ ├───42.22 MB (07.01%) -- compartment(web-worker) │ │ │ │ ├──42.15 MB (07.00%) -- classes │ │ │ │ │ ├──41.97 MB (06.97%) ++ class(ArrayBuffer) │ │ │ │ │ └───0.18 MB (00.03%) ++ (3 tiny) │ │ │ │ └───0.07 MB (00.01%) ++ (3 tiny) │ │ │ └────0.09 MB (00.01%) ++ (4 tiny) │ │ └────1.58 MB (00.26%) ++ (3 tiny) │ └────3.85 MB (00.64%) ++ (2 tiny) That's a big SessionWorker! I guess it's because there are so many tabs. ├──197.98 MB (32.88%) -- js-non-window │ ├──182.76 MB (30.36%) -- zones │ │ ├──180.49 MB (29.98%) -- zone(0x76b6800) │ │ │ ├──144.69 MB (24.03%) -- strings │ │ │ │ ├──142.74 MB (23.71%) -- malloc-heap │ │ │ │ │ ├──132.24 MB (21.96%) ── two-byte │ │ │ │ │ └───10.50 MB (01.74%) ── latin1 │ │ │ │ └────1.95 MB (00.32%) ++ gc-heap More strings. My guess is it's related to the session store stuff again, e.g. maybe a lot of the strings in the SessionWorker get copied to the main thread. ----- CHILD ----- Totals: > 926.39 MB ── private > 859.98 MB ── resident > 1,160.36 MB ── vsize The tabs are here: ├──240.58 MB (33.92%) -- window-objects Each unloaded tab currently uses about 0.22 MiB, which explains most of that. It used to be more like 1 MiB until we reduced it greatly a couple of years ago. Getting it lower still would be harder. More surprising is this: ├──325.75 MB (45.93%) -- js-non-window │ ├──305.74 MB (43.10%) -- zones │ │ ├──304.84 MB (42.98%) -- zone(0x6704000) │ │ │ ├──229.53 MB (32.36%) -- compartment([System Principal], outOfProcessTabChildGlobal) │ │ │ │ ├──153.28 MB (21.61%) -- classes │ │ │ │ │ ├───64.12 MB (09.04%) -- class(Function) │ │ │ │ │ │ ├──55.44 MB (07.82%) -- objects │ │ │ │ │ │ │ ├──50.83 MB (07.17%) ── gc-heap [1144] │ │ │ │ │ │ │ └───4.61 MB (00.65%) ── malloc-heap/slots [1144] │ │ │ │ │ │ └───8.68 MB (01.22%) -- shapes │ │ │ │ │ │ ├──8.08 MB (01.14%) ++ gc-heap │ │ │ │ │ │ └──0.59 MB (00.08%) ── malloc-heap/tree-kids [1144] │ │ │ │ │ ├───58.65 MB (08.27%) -- class(<non-notable classes>) │ │ │ │ │ │ ├──34.27 MB (04.83%) -- shapes │ │ │ │ │ │ │ ├──29.33 MB (04.13%) -- gc-heap │ │ │ │ │ │ │ │ ├──22.15 MB (03.12%) ── tree [1144] │ │ │ │ │ │ │ │ └───7.17 MB (01.01%) ++ (2 tiny) │ │ │ │ │ │ │ └───4.94 MB (00.70%) ++ malloc-heap │ │ │ │ │ │ └──24.39 MB (03.44%) -- objects │ │ │ │ │ │ ├──12.31 MB (01.74%) -- malloc-heap │ │ │ │ │ │ │ ├──12.11 MB (01.71%) ── slots [1144] │ │ │ │ │ │ │ └───0.20 MB (00.03%) ++ (2 tiny) │ │ │ │ │ │ └──12.07 MB (01.70%) ── gc-heap [1144] │ │ │ │ │ └───30.51 MB (04.30%) -- class(Object) │ │ │ │ │ ├──20.07 MB (02.83%) -- shapes │ │ │ │ │ │ ├──15.88 MB (02.24%) -- gc-heap │ │ │ │ │ │ │ ├──13.70 MB (01.93%) ── tree [1144] │ │ │ │ │ │ │ └───2.17 MB (00.31%) ++ (2 tiny) │ │ │ │ │ │ └───4.20 MB (00.59%) ++ malloc-heap │ │ │ │ │ └──10.43 MB (01.47%) ++ objects │ │ │ │ ├───43.21 MB (06.09%) ── scripts/gc-heap [1144] │ │ │ │ ├───21.23 MB (02.99%) ── compartment-tables [1144] │ │ │ │ └───11.82 MB (01.67%) ── sundries/malloc-heap [1144] The [1144] indicates duplicates, so there must be one outOfProcessTabChildGlobal system compartment per window, in addition to the standard compartment per window. That's unfortunate, and would explain a lot of the difference between e10s and non-e10s runs. I wonder if it's avoidable. billm, any thoughts about this?
Flags: needinfo?(wmccloskey)
Thanks for looking at this! > The tabs are here: > > ├──240.58 MB (33.92%) -- window-objects > > Each unloaded tab currently uses about 0.22 MiB, which explains most of > that. It used to be more like 1 MiB until we reduced it greatly a couple of > years ago. Getting it lower still would be harder. > If you're saying the unloaded tabs themselves accounted for 240MB then I don't find that necessarily unreasonable. With an empty session store, Nightly uses under 200MB on my machine at startup. I would be downright impressed if I could start Firefox with ~1000 unloaded tabs and have it use just 540MB! Instead, it uses nearly 1GB (with e10s disabled). What's accounting for the remainder? I gather from your comment above you think a lot of it is SessionWorker related? Could you elaborate for me, please?
There is an out-of-process TabChild global per tab in e10s. However, there's an nsInProcessTabChildGlobal per tab in non-e10s. So that's not really any different. Session store data is unfortunately duplicated between the chrome-level session store code and the content script (which is probably why the outOfProcessTabChildGlobals are so big). However, this also happens in non-e10s. This is at least something we should be able to fix though. I'm having trouble thinking what could cause a 500MB difference with e10s. It would be useful to have the memory report for non-e10s for comparison. Would you mind providing that tyaremco? The SessionWorker is responsible for writing session data out to disk. I can see why it would have a large heap, but the data should be very short-lived. Our GC heuristics for workers have always been pretty bad. I wonder if there's something we could fix here.
Flags: needinfo?(wmccloskey) → needinfo?(tyaremco)
David, can you think of any reason why SessionWorker.js should keep a lot of data alive?
Flags: needinfo?(dteller)
I tested both e10s enabled and disabled one after the other and attached the reports. The actual tab count is 701. In both cases, the only tab loaded is about:memory. Private Working Set -- e10s enabled -- 1,031,116K plugincontainer.exe 453,668K firefox.exe -- e10s disabled -- 1,209,788K firefox.exe So, e10s enabled/disabled are bit closer than my approximations above. However, even with e10s disabled, the 701 unloaded tabs results in a full gigabyte more memory used than when the sessionstore is empty!
Attachment #8554208 - Attachment is obsolete: true
Flags: needinfo?(tyaremco)
One thing that I notice is that the empty windows in e10s are 0.2 MB. In non-e10s 0.1 MB. With so many tabs that adds up. It looks like the difference is in layout information. Non-e10s doesn't include any while in e10s layout takes 0.08 MB, mostly for style sheets. That accounts for 137 MB of difference. We also seem to use somewhat more space for the out of process TabChild globals than for the in-process ones. I don't know why that would be. It could be a fluke I guess.
Sorry, correction: actual tab count is 1497. Memory (Private) 99,464 KB firefox.exe (empty session store, e10s disabled) 345,338 KB unloaded tabs (1497 * .22MB) -------- = 444,802 KB firefox.exe expected memory use 1,209,788 KB firefox.exe actual memory use -------- 764,986 KB actual - expected Where is this extra 764MB coming from?
(In reply to Bill McCloskey (:billm) from comment #8) > David, can you think of any reason why SessionWorker.js should keep a lot of > data alive? Nothing in the code should keep such amounts of data active.
Flags: needinfo?(dteller)
Whiteboard: [MemShrink] → [MemShrink:P3]
(In reply to tyaremco from comment #12) > Sorry, correction: actual tab count is 1497. > > Memory (Private) > 99,464 KB firefox.exe (empty session store, e10s disabled) > 345,338 KB unloaded tabs (1497 * .22MB) > -------- > = 444,802 KB firefox.exe expected memory use > > 1,209,788 KB firefox.exe actual memory use > -------- > 764,986 KB actual - expected > > > Where is this extra 764MB coming from? Looking at the the non-e10s memory report we can see the situation isn't quite as bad as in bug 1054660 (it looks like roughly 300KiB per tab if we add up window-object and js-zone). As in that bug it would be useful to get a memory report before and after restoring the tabs so that we can do a proper diff. Some points of interest: 829.56 MB (100.0%) -- explicit ... ├──110.17 MB (13.28%) -- heap-overhead │ ├──100.67 MB (12.13%) ── bin-unused │ └────9.50 MB (01.15%) -- (3 tiny) ... ├───88.91 MB (10.72%) ── heap-unclassified 57.22 MB ── gpu-committed 23.75 MB ── gpu-dedicated 47.29 MB ── gpu-shared ... 320.18 MB ── heap-allocated 329.69 MB ── heap-committed 537.00 MB ── heap-mapped 1,186.88 MB ── resident Lets break that down: #1 - There's 100MiB of memory hanging around in bin-unused (12% of explicit). This is memory that will be used eventually, so it's somewhat ok. #2 - heap-unclassified is 89MiB, even if this is *all* related to unrestored tabs (which it certainly is not) that's only ~50KiB/tab #3 - we have 829MiB of explicit, but only 320MiB of heap-allocated, indicating ~500MiB of non-heap memory. That's actually interesting, I'm not sure what to say about that yet. #4 - We have ~125MiB of gpu stuff going on, that's windows being windows. It probably accounts for a portion of the non-heap allocations #5 - There's 537MiB of heap-mapped vs 329MiB of heap-committed. That gives us about 207MiB of mapped but not committed. While that does affect virtual memory exhaustion, that doesn't seem to be a problem in your case. So really this is bumping the resident value up by ~207 MiB. That's a lot to think about, but we can make a small conclusion. The *real* (in the stuff that can't be used and is our fault sense) memory use is probably more like: mem_usage = resident - (heap-mapped - heap-committed) - gpu_stuff - bin-unused = ~755MiB Which, drumroll please, about sums up the difference you're seeing. All of that said, it does seem a bit excessive to have 1 compartment per unrestored tab. I'd also argue it's excessive to even have 1 window-object per unrestored tab, but that's a fight we're not going to win.
(In reply to Eric Rahm [:erahm] from comment #14) > I'd also argue it's excessive to even have 1 window-object > per unrestored tab, but that's a fight we're not going to win. ttaubert and I have repeatedly had conversations about lazily creating browsers (and thereby the windows for these tabs). IIRC it's not actually super-hard to get the basics in place - the difficulty is making sure nobody hits the getter and creates the window (for compat reasons it's essentially impossible to break the basic tab.linkedBrowser property). We've not prioritized putting time into this because we think the number of users who benefit from this is relatively small, also if/when add-ons are likely to break the optimization (and may be slow to catch up to such a "new world"). If the MemShrink team thinks this should be a priority, then we could have that discussion and consider working on it, I guess - or did your comment essentially mean to summarize something like the above (if perhaps more abstract) ? :-)
Flags: needinfo?(erahm)
(In reply to :Gijs Kruitbosch from comment #17) > (In reply to Eric Rahm [:erahm] from comment #16) > > My understanding from previous discussions was that it was too hard to be > > worth the effort to change how unrestored tabs are loaded, but it sounds > > like that might not be the case! It seems indeed quite hard doing that without changing a lot of things and breaking a lot of expectations. > > I brought this up at the memshrink meeting today and there was general > > consensus that improving this situation would be a good thing. Well, we also think it would be a good thing, it's just that the effort doesn't seem justified so far :) > Tim, can you remind me where your WIP stuff was, and/or can we look at > getting this prio'd for the early 39 iterations? There's some highly experimental work going on in bug 906076, I hadn't had enough time to keep an eye on it. David has been giving feedback there mostly and before we land any of that we should request review from more browser peers (esp. for the tabbrowser.xml changes). At FOSDEM I also talked to Florian about it. He had a nice idea, playing around with CSS/XUL bindings that might enable a more elegant solution. It would be worth investigating that too before signing off any very intrusive changes.
Flags: needinfo?(ttaubert)
(In reply to Tim Taubert [:ttaubert] from comment #18) > > > I brought this up at the memshrink meeting today and there was general > > > consensus that improving this situation would be a good thing. > > Well, we also think it would be a good thing, it's just that the effort > doesn't seem justified so far :) It seems worth iterating that this really only improves the situation for the most pathological cases. Heavy users of Tab Groups will probably benefit the most. For the average user, even with a few windows, the win here is probably negligible.
> For the average user, even with a few windows, the win here is > probably negligible. Yes, this is definitely geared towards tab hoarders who have 100s or even 1000+ tabs. Which is a small cohort, but probably an influential one. It's also worth mentioning that this is a use case that Chrome simply doesn't handle -- I've heard countless time that Chrome basically becomes flaky and unusable once you get to about 60 or 70 tabs. Depending on your point of view, this fact could make this bug less desirable (we're not going to lose these users to Chrome any time soon) or more desirable (we can improve an existing point of differentiation, use it for bragging purposes, etc).
(In reply to Nicholas Nethercote [:njn] from comment #20) > Yes, this is definitely geared towards tab hoarders who have 100s or even > 1000+ tabs. Which is a small cohort, but probably an influential one. Agreed. > It's also worth mentioning that this is a use case that Chrome simply > doesn't handle -- I've heard countless time that Chrome basically becomes > flaky and unusable once you get to about 60 or 70 tabs. Depending on your > point of view, this fact could make this bug less desirable (we're not going > to lose these users to Chrome any time soon) or more desirable (we can > improve an existing point of differentiation, use it for bragging purposes, > etc). I do think we want to handle this case well. The proposed solutions so far unfortunately would probably lead to lots of add-ons breaking as certain expectations are invalidated. Breaking add-ons would certainly hit the same groups of users harder than the average user so we should try to find a more elegant solution than simply setting a property here or there. Florian's suggestion that I hinted to earlier could come very close to that. At the same time, the whole problem we're hitting here seems like a flaw in Tab Groups itself. We should not create tabs for hidden groups as long as they aren't made the active group. This however breaks a few other things like switch-to-tab etc.
(In reply to Tim Taubert [:ttaubert] from comment #18) > At FOSDEM I also talked to Florian about it. He had a nice idea, playing > around with CSS/XUL bindings that might enable a more elegant solution. It > would be worth investigating that too before signing off any very intrusive > changes. I played a little bit with that idea and CSS wasn't enough to prevent <browser> tags from loading about:blank. I haven't completely given up yet, but I now think a little bit of C++ somewhere would be needed.
Mistakenly filed against Firefox 38 and should be instead 38 Branch. Sorry for the spam. dkl
Version: Firefox 38 → 38 Branch
- Firefox Nightly 41 (May 30) - clean profile generated by Nightly - sessiontore.js is replaced with one having 1496 tabs, about:blank is the focused tab - e10s disabled (enabling e10s makes a negligable difference) - attached memlogs for single tab and many tab states for comparison To compare the single tab and many tab states, I did the following: 1) replace sessionstore.js with a large one 2) open Firefox; the default homepage is be shown (single tab) 3) record memory from task manager and then generate memlogs 4) Click "Restore Previous Session" on homepage (many tab) 5) Wait for CPU to subside and repeat step 3) Below is the recorded memory use (private). > one tab state (1 tab loaded, 0 tabs unloaded) A = 135.53 MB > many tab state (1 tab loaded, 1495 tabs unloaded) B = 1264.85 MB > estimated memory use for scenario B assuming .22MB per unloaded tab (comment 5) > A + (.22MB * 1495 tabs) C = 464.43 MB > change in memory from A to B D = +1129.32 MB > discrepency in B compared to C E = +800.42 MB -- Summary -- Having 1495 _unloaded_ tabs in a firefox session is responsible for an additional 1129MB of memory. This is 800MB larger than would be estimated if we assume unloaded tabs use .22MB. In fact, unloaded tabs seem to be (indirectly) responsible for about .75MB each.
Attachment #8556309 - Attachment is obsolete: true
Version: 38 Branch → 41 Branch
I did a quick test on the latest nightly to see if Bug 988266 affected this issue. Unfortunately, I found the many tab state memory to have increased by ~40MB compared to the May 30 test above. So no wins yet...
See Also: → 1105634
One year later, I will test the per-unloaded-tab memory penalty again. - Firefox Nightly 48 (April 6) - clean profile generated by Nightly - sessionstore.js is replaced with one having 1496 tabs, about:blank is the focused tab To compare the single tab and many tab states, I did the following: 1) replace sessionstore.js with a large one 2) open Firefox; the default homepage is be shown (single tab state) 3) record memory from task manager and then generate memlogs 4) Click "Restore Previous Session" on homepage (many tab state) 5) Wait for CPU to subside and repeat step 3) Below is the recorded memory use (private working set). ED = Electrolysis disabled EE = Electrolysis enabled (includes sub-processes) > one tab state (1 tab loaded, 0 tabs unloaded) ED = 148.44 MB EE = 195.31 MB > many tab state (1 tab loaded, 1495 tabs unloaded) ED = 1210.45 MB EE = 1293.58 MB > delta from one tab to many tab ED = + 1062.02 MB (+ 715%) EE = + 1098.27 MB (+ 562%) > delta per unloaded tab ED = 0.71 MB EE = 0.73 MB -- Summary -- 1495 unloaded tabs in a Firefox session are responsible for an additional 1062MB of memory. Therefore, each unloaded tab increases memory use by 0.71MB. Enabling Electrolysis increases this value slightly. Unfortunately, not much has changed since last year on this front.
Version: 41 Branch → 48 Branch
It looks like bug 906076 might help a lot with this bug...
See Also: → lazytabs
(In reply to Nicholas Nethercote [:njn] from comment #27) > It looks like bug 906076 might help a lot with this bug... That's for sure; I've been following it with bated breath for months! Tim mentioned it back in comment 18.
Depends on: lazytabs
See Also: lazytabs
(In reply to tyaremco from comment #26) > One year later, I will test the per-unloaded-tab memory penalty again. > > - Firefox Nightly 48 (April 6) > - clean profile generated by Nightly > - sessionstore.js is replaced with one having 1496 tabs, about:blank is the > focused tab > > > To compare the single tab and many tab states, I did the following: > 1) replace sessionstore.js with a large one > 2) open Firefox; the default homepage is be shown (single tab state) > 3) record memory from task manager and then generate memlogs > 4) Click "Restore Previous Session" on homepage (many tab state) > 5) Wait for CPU to subside and repeat step 3) > Below is the recorded memory use (private working set). > > > ED = Electrolysis disabled > EE = Electrolysis enabled (includes sub-processes) > > > > one tab state (1 tab loaded, 0 tabs unloaded) > ED = 148.44 MB > EE = 195.31 MB > > > > many tab state (1 tab loaded, 1495 tabs unloaded) > ED = 1210.45 MB > EE = 1293.58 MB > > > > delta from one tab to many tab > ED = + 1062.02 MB (+ 715%) > EE = + 1098.27 MB (+ 562%) > > > > delta per unloaded tab > ED = 0.71 MB > EE = 0.73 MB > > > > -- Summary -- > 1495 unloaded tabs in a Firefox session are responsible for an additional > 1062MB of memory. Therefore, each unloaded tab increases memory use by > 0.71MB. Enabling Electrolysis increases this value slightly. > Unfortunately, not much has changed since last year on this front. It would be interesting to see your test results right about now on 54/55/56.
Flags: needinfo?(tyaremco)
so long as no addons are causing unloaded tabs to instantiate, this is fixed.
(In reply to Danial Horton from comment #30) > so long as no addons are causing unloaded tabs to instantiate, this is fixed. As of what version?
Blocks: 1358283
Status: UNCONFIRMED → RESOLVED
Closed: 8 years ago
No longer depends on: lazytabs
Resolution: --- → DUPLICATE
In case tyaremco is motivated to continue this work, I too would be interested in measurements for Firefox 55 or 56 as well as Quantum. With e10s neither about:memory nor Task Manager offer a useful total.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: