1351012 - Remove use of the b2g-related worker types

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task, P5)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This workerType was at once a convenient "miscellaneous" workerType, but now we should use desktop-test-xlarge or the like instead.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1351012: replace b2gtest with gecko-t-linux-xlarge;

b2gtest uses *.xlarge instances, so it seems a reasonable substitution.

Review commit: https://reviewboard.mozilla.org/r/123968/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/123968/

Comment 2

[Greg Arndt [:garndt]]

Comment on attachment 8851738 [details]
Bug 1351012: replace b2gtest with gecko-t-linux-xlarge;

https://reviewboard.mozilla.org/r/123968/#review126506

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/60377a780a5d
replace b2gtest with gecko-t-linux-xlarge; r=garndt

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I'll delete this workerType in a few months.

Comment 5

[Greg Arndt [:garndt]]

Before we delete, I can see what has created tasks for it to make sure it's 0.

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/60377a780a5d

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Greg, can you do a check for use of the workerType?

Comment 8

[Greg Arndt [:garndt]]

It appears these changes still need to be uplifted elsewhere to understand what is left using that worker type.  In the last month I see tasks using that worker type from:
 jamun
 mozilla-aurora
 mozilla-beta
 mozilla-esr45
 mozilla-esr52
 mozilla-release
 try

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

I'll wait a year..

Comment 10

[Greg Arndt [:garndt]]

Yea, the one we need to wait for is esr52.  the rest should die off soon enough.

Comment 11

[Chris Cooper [:coop] (he/him)]

I'm still seeing occasional esr52 jobs (as expected), as well as the jobs referenced in bug 1437645.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Still seeing symbol-server fetches:
  https://tools.taskcluster.net/groups/Tbs_MxN7Txm1SggUDpHBqw/tasks/Dr--IYFRTXGnDC2tCZ0TCg/details
  https://tools.taskcluster.net/groups/GA8p2PpGSsalcqCCd9QZ-A/tasks/GA8p2PpGSsalcqCCd9QZ-A/details
  https://tools.taskcluster.net/groups/Tbs_MxN7Txm1SggUDpHBqw/tasks/Tbs_MxN7Txm1SggUDpHBqw/details

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

*same

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Honestly, b2gtest isn't hurting anything.

Comment 15

[Chris Cooper [:coop] (he/him)]

We end up with a trivial amount of b2g-related spend in the AWS reporting every month due to this nomenclature. I could easily just assign this spend to another category, but we really shouldn't still be using something named "b2gtest" in 2019.

In the examples above, it just seems to be Ted performing various symbols tests, e.g. https://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/file/tip/fetch-task.json#l3

However, we still seem to reference the worker type ourselves, even in the monorepo:

https://github.com/taskcluster/taskcluster/search?q=b2gtest&type=Code
https://github.com/taskcluster/taskcluster/blob/master/services/auth/test/clients.json

I've not too worried about breaking Ted. He seems to have fixed up his github repos that reference b2gtest already, and his hg-based tests haven't run since August. We should absolutely fix up our part.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

There's nothing to worry about in our part -- it's just a snapshot of all extant clients at some point, used as input to the client tests. Go ahead and delete it :)

Comment 17

[Chris Cooper [:coop] (he/him)]

The full list of workerTypes I'd like to remove is:

b2gbuild
b2gtest
flame-kk
gaia
gaia-cache
gaia-decision

Source: https://taskcluster-web.netlify.com/provisioners/aws-provisioner-v1

Comment 18

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] pronoun: he from comment #17)

The full list of workerTypes I'd like to remove is:

b2gbuild
b2gtest
flame-kk
gaia
gaia-cache
gaia-decision

I've removed these worker types from the aws-provisioner. I have backups of the worker type definitions if we need them.

I'm working on a patch to the taskcluster monorepo to remove those worker types from our references and tests.

Comment 19

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] pronoun: he from comment #18)

I'm working on a patch to the taskcluster monorepo to remove those worker types from our references and tests.

https://github.com/taskcluster/taskcluster/pull/609

Comment 20

[Chris Cooper [:coop] (he/him)]

I removed the b2g-related worker types from the docker-worker deployment lists: https://github.com/taskcluster/docker-worker/commit/eb77f9ec64c7d77ffc4e7738290db92eb8c881c0

As a final cleanup step, I've removed the following EBS volumes from AWS:

b2gtest
gaia-cache

I think the only place where this might bite us is the build-minidump-stackwalk and fetch-windows-symbols tasks referenced in our auth tests (https://github.com/taskcluster/taskcluster/blob/master/services/auth/test/clients.json), and only if they still run occasionally.

I can't find a repo for build-minidump-stackwalk that references tasks, but the repo for fetch-windows-symbols is here: https://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/

Adding a NI to Kim in case she knows about continuity here before I close out this bug.

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

I think the only place where this might bite us is the build-minidump-stackwalk and fetch-windows-symbols tasks referenced in our auth tests (https://github.com/taskcluster/taskcluster/blob/master/services/auth/test/clients.json), and only if they still run occasionally.

This was just a snapshot of the clients/roles that existed at that time, to check that auth's role expansion could operate with "real-world" data. The scopes aren't actually used for anything.

Comment 22

[Kim Moir [:kmoir] ET]

I don't have any context on this. Gsvelto might have some insight?

Comment 23

[Gabriele Svelto [:gsvelto]]

AFAIK the fetch-win32-symbols task is still in use but I doubt the worker type is important. It's possible that Ted just copy-pasted it from some other task. :willkg might know more so I'm CC'ing him.

Comment 24

[Chris Cooper [:coop] (he/him)]

(In reply to Gabriele Svelto [:gsvelto] from comment #23)

AFAIK the fetch-win32-symbols task is still in use but I doubt the worker type is important. It's possible that Ted just copy-pasted it from some other task. :willkg might know more so I'm CC'ing him.

That was my guess, too. Because I do see a very small amount of usage for the b2gtest worker type, I went ahead and patched ted's repo to use a current worker type: https://hg.mozilla.org/users/tmielczarek_mozilla.com/fetch-win32-symbols/rev/989bf36a2d57

Comment 25

[Will Kahn-Greene [:willkg] ET needinfo? me]

I don't know anything about the fetch-win32-symbols task, but I really should. Guessing based on other tasks I've seen Ted do, I agree that it's likely the worker type was unimportant and that Ted copy-and-pasted from another task or that's how he happened to throw the task together that day.