1072495 - Setup loop-push's testing cluster

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Ben Bangert [:benbangert]]

We need a way to easily and repeatedly setup a testing simplepush cluster configured for Loop, similar to the existing automated deployment for simplepush.

There are several differences to be accounted for:
- an etcd cluster will need to be setup
- a single etcd cluster (3 machines) will be fine for both testing/stage
- the etcd cluster needs to be running before the push cluster can be launched

Let me know what else is needed for work on this to proceed.

Comment 1

[Karl Thiessen [:kthiessen, he/him]]

What I mean by the 'blocks' statement above is that this bug blocks testing/verification of bug 1055139.

Comment 2

[Jeremy Orem [:oremj]]

I have a stage environment just about up and running, but it is delayed a few days, so that I can hook it in to the new jenkins delivery environment.

Comment 3

[Jeremy Orem [:oremj]]

To finish this up, I need a config template. There are a bunch of new options available and I'm not sure how this instance should be set up.

Comment 4

[Lina Butler (ex-Mozilla)]

:oremj, you should receive the config template from :jrconlin shortly. For posterity, the interesting options are:

* `default.websocket.addr`: The WebSocket server address and port. I think `push.services.mozilla.com:443` currently resolves to this in production. `cert_file` and `key_file` can be set for the `default.websocket`, `default.endpoint`, and `router.listener` sections, too.

* `default.endpoint.addr`: The HTTP server address and port; handles app server updates and health checks. Looks like `updates.push.services.mozilla.com:443` resolves to this now.

* `router.listener.addr`: The address and port for the internal routing listener. Should be identical for all Simple Push servers in the cluster.

* `discovery.servers`: etcd nodes used for discovery.

* `metrics.statsd_server`: The statsd server (or Heka `StatsdInput` listener) address.

Comment 5

[Jeremy Orem [:oremj]]

Thanks, I'm waiting on Amazon to increase our cloudformation stack limit. They say it may take 2 business days. We are trying to get them to expedite the ticket.

Comment 6

[Jeremy Orem [:oremj]]

1.4rc is deployed to: updates-push-loop.stage.mozaws.net and push-loop.stage.mozaws.net

Comment 7

[Jeremy Orem [:oremj]]

"testing" instances are deployed at:

push-loop-dev.stage.mozaws.net
updates-push-loop-dev.stage.mozaws.net

They should update on every commit, but this is a relatively new thing, so let me know if it does not seem up to date.

The stage instance currently has 3 m3.mediums and the dev instance has 2 m3.mediums.

Comment 8

[Lina Butler (ex-Mozilla)]

Thanks, Jeremy!

Comment 9

[Ben Bangert [:benbangert]]

Every new commit to what? Can that be restricted to only update when commits to the 'dev' branch hit?

Comment 10

[JR Conlin [:jrconlin,:jconlin]]

Are (updates-,}push-loop-dev.stage* the stage boxes or the dev boxes (or both)?
Is heka the only means to pull logging information from these machines or will there be raw access in case there's an error that prevents heka from properly reading data?

Comment 11

[Ben Bangert [:benbangert]]

There's a set of scripts that oremj linked me to that setup a testing cluster, as we'll need to do this a lot, we'll need to either run the scripts directly ourselves or have an API to hit the jenkins box that spins up the cluster for a given changeset.

Comment 12

[Jeremy Orem [:oremj]]

(In reply to Ben Bangert [:benbangert] from comment #9)
> Every new commit to what? Can that be restricted to only update when commits
> to the 'dev' branch hit?

Yes, right now it is following the "dev" branch.

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #10)
> Are (updates-,}push-loop-dev.stage* the stage boxes or the dev boxes (or
> both)?
> Is heka the only means to pull logging information from these machines or
> will there be raw access in case there's an error that prevents heka from
> properly reading data?

"-dev" update on commit to "dev". Right now heka is the method of data access, but we can discuss logins.

Comment 13

[Karl Thiessen [:kthiessen, he/him]]

The "dev" auto-updated machines now live at:

push-loop.dev.mozaws.net
updates-push-loop.dev.mozaws.net

and not at the DNS names listed in comment 7.

Comment 14

[Richard Pappalardo [:rpapa][:rpappalardo]]

This has been deployed and verified:
see: Bug 1101480 - Update loop-stage configuration to use Loop Push staging environment