940025 - Assertion failure: ret == (-3), at jsutil.cpp

Status: RESOLVED WORKSFORME

(Core :: JavaScript Engine, defect)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Attachment: stack

We hit OOM on ARM when we set the following parameters:

resource.setrlimit(resource.RLIMIT_CORE, (500000000, -1))
resource.setrlimit(resource.RLIMIT_AS, (400000000, -1))

and try to run a testcase that tries to allocate a bunch of memory followed by more stuff. Sometimes the following assertion gets hit:

Assertion failure: ret == (-3), at jsutil.cpp

Jesse helped me out here. Is the assert at http://hg.mozilla.org/mozilla-central/annotate/f2adb62d07eb/js/src/jsutil.cpp#l64 - " JS_ASSERT(ret == Z_DATA_ERROR); " valid?

Compressor::~Compressor()
{
    int ret = deflateEnd(&zs);
    if (ret != Z_OK) {
        // If we finished early, we can get a Z_DATA_ERROR.
        JS_ASSERT(ret == Z_DATA_ERROR);
        JS_ASSERT(uInt(zs.next_in - inp) < inplen || !zs.avail_out);
    }
}

We probably can get around this particular testcase by bumping resource.RLIMIT_AS to 500000000 instead, but I'm not sure if this is really fixing the problem.

Comment 1

[:Benjamin Peterson]

I believe the assert is valid if zlib is following its documentation. :P Can you possibly get a debugger in there and see what the value of |ret| actually is?

Comment 2

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

No, apparently the value of ret is optimized out and this seems to require --enable-optimize --enable-debug.

Comment 3

[:Benjamin Peterson]

You can't |printf| or something?

Comment 4

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

> You can't |printf| or something?

It shows <optimized out> in gdb, unless you mean recompiling the shell to printf the ret value just before the assert is hit?

Comment 5

[:Benjamin Peterson]

(In reply to Gary Kwong [:gkw] [:nth10sd] (yes, still catching up on bugmail) from comment #4)
> > You can't |printf| or something?
> 
> It shows <optimized out> in gdb, unless you mean recompiling the shell to
> printf the ret value just before the assert is hit?

That is what I meant.

Comment 6

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Hmmm, I tried going back to the same rev and reproducing this, but am unable to. I guess I'll ponder again when I hit this the next time.