1082188 - Loop-Push — Please deploy pushgo 1.4 to Stage

Status: VERIFIED FIXED

(Cloud Services :: Operations: Deployment Requests - DEPRECATED, task)

Description

[Richard Pappalardo [:rpapa][:rpappalardo]]

Please deploy the 1.4 pre-release to the staging environment.
https://github.com/mozilla-services/pushgo/releases/tag/1.4rc

Comment 1

[Jeremy Orem [:oremj]]

I've deployed 1.4rc, but need to fire up an etcd cluster and switch the discovery type before calling this FIXED. I should be able to finish that up by tomorrow afternoon.

Comment 2

[Richard Pappalardo [:rpapa][:rpappalardo]]

Thanks, Jeremy!

Comment 3

[Jeremy Orem [:oremj]]

This is blocked on Amazon raising our stack limit. They say it may take up to two days, but we are trying to expedite the process.

Comment 4

[Jeremy Orem [:oremj]]

Stage has been updated to 1.4rc

https://push.stage.mozaws.net
https://updates-push.stage.mozaws.net/

Comment 5

[Richard Pappalardo [:rpapa][:rpappalardo]]

I'm working on getting smoke/load tests to run without too much duct tape and will be noting issues in this ticket as I come across them.

Here's the smoke/load tests I'm aware of.  Please let me know if:
1. there are other tests not listed here that QA should be aware of
2. if the desire is to run them all, or pick a few best candidates.

--------------
Smoke Tests 
--------------

1. https://github.com/jrconlin/simplepush_test/blob/master/smoke_test.py 
* per JR, good smoke test candidate
2. https://github.com/mozilla-services/pushgo/

--------------
Load Tests 
--------------

1. https://github.com/bbangert/push-tester/ (Haskell test)
* uses loads, v2?
2. https://github.com/mozilla-services/simplepush-testpod (oremj/mostlygeek)
3. https://github.com/kitcambridge/push-tester-go
4. https://github.com/edmoz/load-test/tree/master/_loads/pushgo

Comment 6

[JR Conlin [:jrconlin,:jconlin]]

Crap, I mispoke to rpapa earlier and confused push against fmd. 

The go test does test the interfaces sufficiently. The smoke_test.py is an interaction test only.

Comment 7

[Richard Pappalardo [:rpapa][:rpappalardo]]

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #6)
> Crap, I mispoke to rpapa earlier and confused push against fmd. 
> 
> The go test does test the interfaces sufficiently. The smoke_test.py is an
> interaction test only.

JR, thanks for the clarification.  I still need to look through the tests more thoroughly to understand better what each does, but I may circle back to you later.

In regards to the pushgo test, I'm seeing a dep error when I do a 'make simplepush'
Should I be installing memcached first?


https://github.com/mozilla-services/pushgo/ 

1. Installed latest go version go1.3.3 darwin/amd64
2. ran make
** GOPATH wasn't set by Makefile (Kit to fix)
** JR helped me set GOPATH
3. $ make simplepush

rm -f simplepush
Building simplepush
go build -o simplepush github.com/mozilla-services/pushgo
# github.com/ianoshen/gomc
.godeps/src/github.com/ianoshen/gomc/const.go:4:10: fatal error: 'libmemcached/memcached.h' file not found
#include <libmemcached/memcached.h>
         ^
1 error generated.
make: *** [simplepush] Error 2

Comment 8

[JR Conlin [:jrconlin,:jconlin]]

I've not tried it on a mac, but you should be able to run

make libmemcached-1.0.18 

to build a compatible version of libmemcached

Comment 9

[JR Conlin [:jrconlin,:jconlin]]

(In reply to JR Conlin [:jrconlin,:jconlin] from comment #8)
> I've not tried it on a mac, but you should be able to run
> 
> make libmemcached-1.0.18 
> 
> to build a compatible version of libmemcached

correction;
make memcached will install the version to your system, but will require sudo to do so. If possible, that would probably work better.

Comment 10

[Richard Pappalardo [:rpapa][:rpappalardo]]

Thanks, JR.  I was able to install memcached with make libmemcached-1.0.18, but it looks like the header files aren't being installed where they need to be.  Is there something I need to do to get memcached to install into the .godeps/src dir?

$ make simplepush
rm -f simplepush
Building simplepush
go build -o simplepush github.com/mozilla-services/pushgo
# github.com/ianoshen/gomc
.godeps/src/github.com/ianoshen/gomc/const.go:4:10: fatal error: 'libmemcached/memcached.h' file not found
#include <libmemcached/memcached.h>
         ^
1 error generated.
make: *** [simplepush] Error 2

Comment 11

[Lina Butler (ex-Mozilla)]

Richard, you might need to set some environment variables for Cgo to find libmemcached:

export CGO_CFLAGS="-I/usr/local/include"
export CGO_CPPFLAGS="-I/usr/local/include"
export CGO_CXXFLAGS="-I/usr/local/include"
export CGO_LDFLAGS="-L/usr/local/lib"

Does the build still fail if those are set?

Comment 12

[Richard Pappalardo [:rpapa][:rpappalardo]]

when doing make libmemcached-1.0.18, memcached.h gets installed into local dir: 
/libmemcached-1.0.18/libmemcached
however, per :spindrift, you can remove dependency by renaming src/github.com/mozilla-services/pushgo/simplepush/emcee_store.go to emcee_store.no
Once this is renamed, pushgo successfully compiles and it's possible to run a test:

$ make test
go test github.com/mozilla-services/pushgo/id github.com/mozilla-services/pushgo/simplepush
ok  	github.com/mozilla-services/pushgo/id	0.006s
ok  	github.com/mozilla-services/pushgo/simplepush	1.371s

Comment 13

[JR Conlin [:jrconlin,:jconlin]]

heh, in one of my forks, I have a make rule that excludes emcee_store.go and libmemcache because of just that sort of reason.

Eventually, this will be addressed.

Comment 14

[Richard Pappalardo [:rpapa][:rpappalardo]]

Awesome.  Kit said he's going to merge that change/rule into the pushgo test and will also schedule time w/ QA next week to review load test (should be ready by mid-week).  

After meeting today, we decided the following tests would be sufficient for verifying a STAGE build:
SMOKE: https://github.com/mozilla-services/pushgo/
LOAD: https://github.com/bbangert/push-tester/

Still need to consider best way to test deployment on PROD.

Comment 15

[Richard Pappalardo [:rpapa][:rpappalardo]]

We still need to determine a prod config.
sounds like we need the following:
1. loads,v2 up and running
2. gather loadtest data for stage (from cloudwatch?)
3. deploy single cluster to prod for 34, and maybe 35 
Currently blocked by: Bug 1075654 

[12:35]  benbanger: rpapa: the staging/prod environment for loop-push
[12:36]  benbangert: there's a ticket for the prod, but its unknown what config to run it with
[12:36]  benbangert: since we don't have data from load-testing yet
[12:36]  benbangert: oremj: is there any monitoring or data for stage?
[12:38]  oremj: benbangert: cloudwatch should have some data
[12:38]  benbangert: rpapa: my thinking is, we make sure prod loop-push can survive 34, and maybe 35 with a single cluster
[12:39]  benbangert: or at least, we know where the death point is, so we don't feature-flag too many
[12:39]  benbangert: unfortunately given the way the wss caches the DNS name, we can't move ppl until client implements redirect, and we have redirect working.

Comment 16

[Richard Pappalardo [:rpapa][:rpappalardo]]

no loadtesting on this release.

soft-release of 1.4 rc2 planned for next Mon.,11/17
so closing this ticket