1476194 - 856000 - 964325% compiler warnings (windows2012-32, windows2012-64) regression on push 896f75ea76345497797778f66b7ad9386052229a (Mon Jul 16 2018)

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Ionuț Goldan [:igoldan]]

We have detected a build metrics regression from push:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?changeset=896f75ea76345497797778f66b7ad9386052229a

As author of one of the patches included in that push, we need your help to address this regression.

Regressions:

964325%  compiler warnings windows2012-64 debug      1.00 -> 9,644.25
964250%  compiler warnings windows2012-64 debug plain1.00 -> 9,643.50
957925%  compiler warnings windows2012-32 debug      1.00 -> 9,580.25
861382%  compiler warnings windows2012-64 opt rusttests1.00 -> 8,614.82
860700%  compiler warnings windows2012-64 opt        1.00 -> 8,608.00
856000%  compiler warnings windows2012-32 opt        1.00 -> 8,561.00


You can find links to graphs and comparison views for each of the above tests at: https://treeherder.mozilla.org/perf.html#/alerts?id=14362

On the page above you can see an alert for each affected platform as well as a link to a graph showing the history of scores for this test. There is also a link to a treeherder page showing the jobs in a pushlog format.

To learn more about the regressing test(s), please see: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Automated_Performance_Testing_and_Sheriffing/Build_Metrics

Comment 1

[Masatoshi Kimura [:emk]]

This is not a real regression. These warnings were present even before bug 1476000, but clang-cl builds could not catch them. Bug 1476000 just made warnings detectable. So this "regression" is totally expected.

Comment 2

[(Away)]

Before clang-cl broke the warnings script, our last data point from MSVC was 119 warnings, so this regression is "only" 7723% :-)

This is still not ideal, of course. Bug 1090497 comment 5 has a (slightly dated) breakdown of our warnings.

A large fraction of these warnings are coming from security/pkix and security/certverifier setting -Wall, but then pulling in Gecko headers, which are far from -Wall clean. We should try to see if that code's owners will let us remove -Wall on clang-cl.

Another large contributor is the -Wpragma-pack warnings in NSS. The code deliberately uses unbalanced pragma pack in its header files, so the warning just needs to be silenced.

After that there's a very long tail of warnings, but taking out those first three groups would at least bring the numbers down to a non-ludicrous level.

Comment 3

[Nathan Froyd [:froydnj]]

The pragma pack warning is bug 1472254.

Comment 4

[Ionuț Goldan [:igoldan]]

I see bug 1476486 and bug 1476948 fixed some of the warnings on July 19:

== Change summary for alert #14567 (as of Wed, 25 Jul 2018 08:34:36 GMT) ==

Improvements:

 31%  compiler warnings windows2012-32 opt rusttests     8,458.00 -> 5,863.00
 31%  compiler warnings windows2012-32 pgo               8,458.04 -> 5,863.00
 31%  compiler warnings windows2012-32 opt               8,458.00 -> 5,863.00
 31%  compiler warnings windows2012-64 pgo               8,505.08 -> 5,910.00
 31%  compiler warnings windows2012-64 opt               8,505.00 -> 5,910.00
 30%  compiler warnings windows2012-64 opt rusttests     8,505.00 -> 5,931.50
 28%  compiler warnings windows2012-32-noopt debug       9,431.00 -> 6,836.00
 28%  compiler warnings windows2012-32 debug             9,431.08 -> 6,836.00
 27%  compiler warnings windows2012-64-noopt debug       9,495.04 -> 6,900.00
 27%  compiler warnings windows2012-64 debug             9,495.00 -> 6,900.00

( For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14567 )

 17%  compiler warnings windows2012-32 debug static-analysis     1,293.25 -> 1,079.00
 16%  compiler warnings windows2012-32 opt static-analysis       1,299.25 -> 1,085.00
 16%  compiler warnings windows2012-64 debug static-analysis     1,320.25 -> 1,106.00
 16%  compiler warnings windows2012-64 opt static-analysis       1,326.25 -> 1,112.00

( For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14568 )

Comment 5

[Ionuț Goldan [:igoldan]]

And there's another huge fix on July 23, thanks to bug 1476475: 

== Change summary for alert #14569 (as of Wed, 25 Jul 2018 08:56:17 GMT) ==

Improvements:

 84%  compiler warnings windows2012-64 asan debug     6,856.96 -> 1,070.50
 84%  compiler warnings windows2012-32-noopt debug    6,823.96 -> 1,065.50
 84%  compiler warnings windows2012-32 debug          6,823.96 -> 1,065.50
 84%  compiler warnings windows2012-64 debug plain    6,885.96 -> 1,095.50
 84%  compiler warnings windows2012-64-noopt debug    6,887.96 -> 1,097.50
 84%  compiler warnings windows2012-64 debug          6,887.96 -> 1,097.50
 82%  compiler warnings windows2012-32 opt rusttests  5,850.96 -> 1,073.50
 82%  compiler warnings windows2012-32 pgo            5,850.96 -> 1,073.50
 82%  compiler warnings windows2012-32 opt            5,850.96 -> 1,073.50
 82%  compiler warnings windows2012-64 asan opt       5,867.96 -> 1,079.42
 81%  compiler warnings windows2012-64 opt plain      5,897.96 -> 1,105.42
 81%  compiler warnings windows2012-64 opt rusttests  5,900.29 -> 1,105.75
 81%  compiler warnings windows2012-64 opt            5,897.96 -> 1,105.50
 81%  compiler warnings windows2012-64 pgo            5,897.96 -> 1,105.58

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14569

Comment 6

[Ionuț Goldan [:igoldan]]

:dmajor are there other ongoing bugs that tackle these compiler warnings?

Comment 7

[(Away)]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriffing from comment #6)
> :dmajor are there other ongoing bugs that tackle these compiler warnings?

Yes, I filed a few yesterday as blockers of bug 1476474.

Comment 8

[Ionuț Goldan [:igoldan]]

Another round of resolved compiler warnings, thanks to bug 1478014 and bug 1478008:

== Change summary for alert #14579 (as of Thu, 26 Jul 2018 05:27:19 GMT) ==

Improvements:

 13%  compiler warnings windows2012-32-noopt debug      1,065.00 -> 923.00
 13%  compiler warnings windows2012-32 debug            1,065.00 -> 923.00
 13%  compiler warnings windows2012-64 asan debug       1,070.00 -> 928.00
 13%  compiler warnings windows2012-32 opt rusttests    1,073.00 -> 931.00
 13%  compiler warnings windows2012-32 pgo              1,073.00 -> 931.00
 13%  compiler warnings windows2012-32 opt              1,073.00 -> 931.00
 13%  compiler warnings windows2012-32 debug static-analysis1,075.00 -> 933.00
 13%  compiler warnings windows2012-64 asan opt         1,079.00 -> 937.00
 13%  compiler warnings windows2012-32 opt static-analysis1,081.00 -> 939.00
 13%  compiler warnings windows2012-64 debug plain      1,095.00 -> 953.00
 13%  compiler warnings windows2012-64-noopt debug      1,097.00 -> 955.00
 13%  compiler warnings windows2012-64 debug            1,097.00 -> 955.00
 13%  compiler warnings windows2012-64 debug static-analysis1,099.00 -> 957.00
 13%  compiler warnings windows2012-64 pgo              1,105.08 -> 963.00
 13%  compiler warnings windows2012-64 opt static-analysis1,105.00 -> 963.00
 13%  compiler warnings windows2012-64 opt plain        1,105.00 -> 963.00
 13%  compiler warnings windows2012-64 opt rusttests    1,105.00 -> 963.00
 13%  compiler warnings windows2012-64 opt              1,105.00 -> 963.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14579

Comment 9

[Ionuț Goldan [:igoldan]]

Some other compiler warnings have been resolved:

== Change summary for alert #14743 (as of Fri, 03 Aug 2018 09:14:30 GMT) ==

Improvements:

  1%  compiler warnings windows2012-32 pgo      935.92 -> 922.25
  1%  compiler warnings windows2012-32 debug    927.83 -> 914.25
  1%  compiler warnings windows2012-32 debug static-analysis938.83 -> 925.25
  1%  compiler warnings windows2012-32 opt rusttests935.83 -> 922.25
  1%  compiler warnings windows2012-32 opt      935.83 -> 922.25
  1%  compiler warnings windows2012-32 opt static-analysis944.83 -> 931.25
  1%  compiler warnings windows2012-64 asan debug935.83 -> 923.25
  1%  compiler warnings windows2012-64 asan opt 944.83 -> 932.25
  1%  compiler warnings windows2012-64 debug plain960.83 -> 948.25
  1%  compiler warnings windows2012-64 debug    962.83 -> 950.25
  1%  compiler warnings windows2012-64 debug static-analysis964.83 -> 952.25
  1%  compiler warnings windows2012-64 opt static-analysis970.83 -> 958.25
  1%  compiler warnings windows2012-64 opt plain970.83 -> 958.25
  1%  compiler warnings windows2012-64 opt rusttests970.83 -> 958.25
  1%  compiler warnings windows2012-64 pgo      970.83 -> 958.25
  1%  compiler warnings windows2012-64 opt      970.83 -> 958.25

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14743

Comment 10

[Ionuț Goldan [:igoldan]]

Updates here:

== Change summary for alert #14782 (as of Tue, 07 Aug 2018 05:12:33 GMT) ==

Improvements:

  2%  compiler warnings windows2012-32 debug      924.92 -> 909.00
  2%  compiler warnings windows2012-32 opt rusttests932.92 -> 917.00
  2%  compiler warnings windows2012-32 pgo        932.96 -> 917.00
  2%  compiler warnings windows2012-32 opt        932.92 -> 917.00
  2%  compiler warnings windows2012-32 debug static-analysis935.92 -> 920.00
  2%  compiler warnings windows2012-32 opt static-analysis941.92 -> 926.00
  2%  compiler warnings windows2012-64 asan debug 933.21 -> 918.00
  2%  compiler warnings windows2012-64 asan opt   942.21 -> 927.00
  2%  compiler warnings windows2012-64 debug plain958.21 -> 943.00
  2%  compiler warnings windows2012-64 debug      960.21 -> 945.00
  2%  compiler warnings windows2012-64 debug static-analysis962.21 -> 947.00
  2%  compiler warnings windows2012-64 opt static-analysis968.21 -> 953.00
  2%  compiler warnings windows2012-64 pgo        968.21 -> 953.00
  2%  compiler warnings windows2012-64 opt plain  968.21 -> 953.00
  2%  compiler warnings windows2012-64 opt rusttests968.21 -> 953.00
  2%  compiler warnings windows2012-64 opt        968.21 -> 953.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=14782

Comment 11

[Ionuț Goldan [:igoldan]]

Bug 1478668 also helped a bit [1].

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1478668#c13

Comment 12

[Ionuț Goldan [:igoldan]]

Updates:

== Change summary for alert #15154 (as of Mon, 20 Aug 2018 09:12:23 GMT) ==

Improvements:

 46%  compiler warnings windows2012-32-noopt debug      999.00 -> 536.00
 46%  compiler warnings windows2012-32 debug            999.00 -> 536.00
 46%  compiler warnings windows2012-32 debug static-analysis1,008.00 -> 545.00
 46%  compiler warnings windows2012-32 opt rusttests    1,007.00 -> 545.00
 46%  compiler warnings windows2012-32 pgo              1,007.00 -> 545.00
 46%  compiler warnings windows2012-32 opt              1,007.00 -> 545.00
 46%  compiler warnings windows2012-32 opt static-analysis1,014.00 -> 552.00
 43%  compiler warnings windows2012-64 asan debug       1,052.00 -> 595.00
 43%  compiler warnings windows2012-64 asan opt         1,061.00 -> 605.00
 42%  compiler warnings windows2012-64 debug plain      1,077.00 -> 620.00
 42%  compiler warnings windows2012-64-noopt debug      1,079.00 -> 622.00
 42%  compiler warnings windows2012-64 debug            1,079.00 -> 622.00
 42%  compiler warnings windows2012-64 debug static-analysis1,081.00 -> 624.00
 42%  compiler warnings windows2012-64 opt static-analysis1,087.00 -> 631.00
 42%  compiler warnings windows2012-64 opt plain        1,087.00 -> 631.00
 42%  compiler warnings windows2012-64 opt rusttests    1,087.00 -> 631.00
 42%  compiler warnings windows2012-64 pgo              1,087.00 -> 631.00
 42%  compiler warnings windows2012-64 opt              1,087.00 -> 631.00

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=15154

Comment 13

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

previously we were at ~1 compiler warning, today we are at ~300-400;  :dmajor is there more work to do here?

Comment 14

[(Away)]

Note that the value of 1 was a measurement error per comment 1 and comment 2.

As far as I'm aware, nobody is signed up to make further improvements as part of their day job. I'm sure we'll see more patches but it's more of a best-effort spare-time project for people.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

marking this as fixed as there is a tracking bug for the work and given the fact that we are now measuring the right things, this seems like the right thing to do.