46707 - investigate xpt file loading performance/caching

Status: RESOLVED INCOMPLETE

(Core :: XPCOM, defect, P3)

Description

[Warren Harris]

From the footprint meeting 7/26/00:

6) .xpt file loading
  Description: Currently we're loading more type library information than we 
necessarily use because of the way xpt files are 
  created as part of the module build process. Better factoring of the xpt files 
could result in less loading at startup time and less 
  memory usage. Also, the type library info needs to be a memory flush listener.
  Module owner: [e-mail], [e-mail]
  Task owner: [e-mail]
  Status: JBand to investigate either better factoring of typelibs in xpt files 
or the idea of putting all of the individual small xpt files 
  into a single zip file.

Comment 1

[Warren Harris]

Adding dependency on jar file packaging. 

Comment 2

[Warren Harris]

Sounds like this is going to be a significant win for beta3. Marking nsbeta3+

Comment 3

[John Bandhauer]

It looks like work I did for bug 
http://bugzilla.mozilla.org/show_bug.cgi?id=30753 already got the bigger 
part of the space win. This part is to take better advantage of that win.

Here are some rough mem usage numbers I got from running taskman on NT.
This was a release build with symbols. 
I just started mozilla and let it go to the mozilla.org site.
Numbers are in bytes and show average size after a couple of runs

1) What we ship now with a few big xpt files:           18534
2) What developers currently run (97 .xpt files):       18190
3) Those 99 files zipped to one zip:                    18270
4) All .xpt files if we don't run xpt_link (479 files): 18252
5) Those 479 files xtp_link'd to one .xpt file:         18550
6) Those 479 files zipped:                              18280

So, I had thought that the best tradeoff of loading speed and 
footprint would be '6'. But, '3' is near enough the same not
too matter. It looks like the added space of the additional
zip entry names for all the file closely offsets the addition 
bits of the unused xpt stuff we actually load when we don't
keep the .xpt file to the smallest granularity. this is good.
It means that the additional build change of *not* running
xpt_link for each module would not save us anything.

So running zip rather than xpt_link at package time is the minimal
build/package change to get us the best win.

This gets us about 250K gain over what we currently ship.

I still need to do some caching of the zip files if we want to not
loose on speed. I'll hack this and make sure the added space cost
doesn't whack the entire gain here.

Comment 4

[Phil Peterson]

Not holding PR3 for this. Marking nsbeta3-. Please nominate for rtm if there's
something we really need to do for Seamonkey RTM.

Comment 5

[John Bandhauer]

I have not touched this for months. I'd be happy if someone else took it over. 
The changes to support xpt files in zips got implemented and worked when last I 
tried it. The work that is yet to be done is to change the packaging perl 
scripts to use zip.exe on the xpt files that it currently uses on xpt_link.exe 
on. This would give us browser.zip, mail.zip, etc in place of browser.xpt, 
mail.xpt, etc.

I started in on that back then and ran out of time and clues. I'm not seeing 
myself pick this up again anytime soon. If someone with a clue on these 
packaging scripts wants to dig in then I can certainly answer questions and fix 
bugs in xpti if they are uncovered.

I'll attach the stuff I had at the time. 

IIRC, it was sort of working on Windows, untested on Linux, and expected to not 
work yet on Mac. I was puzzled by the differences in the packaging between the 
mozilla distribution and the netscape stuff. The strategy was to copy xptlink.pl 
to a new file called xptzip.pl and hack it to do the zipping. The build.pl 
script would then call xptzip.pl rather than xptlink.pl and all would be well. 
Again. I think it would be great if someone would pick this up and go with it!

Comment 6

[John Bandhauer]

Attachment: unfinished xptzip.pl

Comment 7

[John Bandhauer]

Attachment: a diff of the hacked windows build.pl to use xptzip.pl

Comment 8

[John Bandhauer]

mass reassign of bugs to [e-mail]

Comment 9

[Patrick C. Beard]

Is 256K memory savings worth escalating this for?

Comment 10

[John Bandhauer]

I never managed to get anyone interested enough in this likely 256K of footprint 
savings for them to actually do the work. The reamining work is just changes in 
the packaging scripts. This ought to be fairly easy for the packaging script 
gurus. SEE ABOVE

Vidur suggested an innovative solution: just reassign the bug until someone 
pushy enough talks the right person into doing the work. Let's see how pushy 
cathleen is :)

Comment 11

[Brendan Eich [:brendan]]

(shaver pushes himself around all the time! :-)

Comment 12

[Mike Shaver (:shaver emeritus)]

I hate you, Brendan Eich.

Comment 13

[Daniel Veditz [:dveditz] out until Nov 10]

I don't think shaver is looking at this, so I'll take it.

Comment 14

[Syd Logan]

only nsbeta1+ bugs can have milestones, resetting to ---

Comment 15

[scottputterman]

Moving Netscape owned 0.9.9 and 1.0 bugs that don't have an nsbeta1, nsbeta1+,
topembed, topembed+, Mozilla0.9.9+ or Mozilla1.0+ keyword.  Please send any
questions or feedback about this to [e-mail].  You can search for
"Moving bugs not scheduled for a project" to quickly delete this bugmail.

Comment 16

[David G King]

Changing target milestone as 1.2a has been and gone. Mind you, is this bug still
an issue?

Comment 17

[Alfred Kayser]

It seems that this one is no longer valid, as the current
Mozilla, Firefox and TB releases have now only a very few
XPT's included.
So either close this one, or keep it open to see if something
can be done in the loading of one XPT file can be done.

Comment 18

[Daniel Veditz [:dveditz] out until Nov 10]

That's precisely the problem -- all the little interfaces are combined into a
very few giant files with an .xpt extension, which are completely loaded at
startup. This adds to startup time and memory bloat since loading an .xpt file
is all-or-nothing.

The perf and footprint gain would come from analyzing which interfaces are
actually used at startup or other performance critical tasks and put only those
into a few .xpt files. The infrequently used interfaces coul be put into .zip
archives from which individual interfaces could be read as needed without having
to load all of them (but with a bit more overhead than raw mini-.xpt files).

Comment 19

[Dietrich Ayala (:dietrich)]

per bug 510309, xpt files are linked on all tier 1 platforms now.

Comment 20

[Andrew McCreight [:mccr8]]

XPT stuff is more of an XPCOM thing. But think is old enough I think I can just close it.

Comment 21

[Worcester12345]

I'd probably say it should go to "WONTFIX", since it is a known item that won't be getting fixed.