671647 - add nagios checks for w64-ix-slaves once they are all cloned

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Armen [:armenzg]]

The checks needed are:
* PING
* disk - C
* disk - E
* buildbot-start

Thanks!

Comment 1

[Armen [:armenzg]]

Adding the other slaves as well 10,12,17 & 19.

Comment 2

[Amy Rich [:arr] [:arich]]

This needs to wait till the machines are on the new network.

Comment 3

[Armen [:armenzg]]

Bug 668521 got resolved. What is the next step? Shall I disable few w64 slaves, ask to be moved to vlan76 and then try the nagios check?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

IMHO we should wait until the w64 systems are all up and all moved to the winbuild VLAN before monitoring them - anything more will be unnecessary nagios config churn.

Comment 5

[Armen [:armenzg]]

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> IMHO we should wait until the w64 systems are all up and all moved to the
> winbuild VLAN before monitoring them - anything more will be unnecessary
> nagios config churn.

Fair enough. Adjusting dependencies. We will wait for all of them to be cloned.

Comment 6

[Armen [:armenzg]]

95% of the slaves have been reimaged.
Most of the slaves are 

Is this something that can be started now? or will deal a little later?

Comment 7

[Amy Rich [:arr] [:arich]]

I've added in the checks but downtimed the hosts for now (till next Wed) since I want to make sure we iron out any issues before they start alerting.  Over to Matt to verify status of some hosts:

w64-ix-slave02
w64-ix-slave17
w64-ix-slave19
w64-ix-slave41 

There's a mix of host down to host responding to NRPE checks but not ping checks.  I'm wondering if the ones not responding to ping are those that need to be reimaged.  Matt, please take a look at the ngios GUI.

Comment 8

[Amy Rich [:arr] [:arich]]

all of the downtimes for ping and disk have been remoted (buildbot_start remains deactivated until idelizer works).  armen fixed the firewall rules on w64-ix-slave17 and w64-ix-slave19. w64-ix-slave02 and w64-ix-slave41 are acked and destined for a trip to iX.