Re: [Tarantool-patches] [PATCH luajit] memprof: report stack resizing as internal event

[Sergey Kaplun via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

Hi, Sergos!

Thanks for the review!

On 10.03.21, Sergey Ostanevich wrote:
> Hi!
> 
> Thanks for the patch!
> 
> > On 9 Mar 2021, at 20:54, Sergey Kaplun <skaplun@tarantool.org> wrote:
> > 
> > Resizing of the Lua stack is not reported as internal allocation.
> > Moreover, it may lead to crash inside Lua or FF frames.
> > 
> > Profiler performs reallocation first and after reports corresponding
> > event. When the stack is resized for local function arguments, the link
> > to previous frame is invalid in the cause of reallocation. Therefore,
> 				  ^^^^^^^^ case
> > assertion in `debug_framepc()` failes, because of invalid function
> 				fails.   ^^^ is it just dubbing the link 
> 					     is invalid? Then just remove.

Fixed and force-pushed to the branch. The commit message now is the
following:

===================================================================
memprof: report stack resizing as internal event

Resizing of the Lua stack is not reported as internal allocation.
Moreover, it may lead to crash inside Lua or FF frames.

Profiler performs reallocation first and after reports corresponding
event. When the stack is resized for local function arguments, the link
to previous frame is invalid in case of reallocation. Therefore,
assertion in `debug_framepc()` fails.

Resolves tarantool/tarantool#5842
Follows up tarantool/tarantool#5442
===================================================================

> 
> > reference at previous frame.
> > 
> > Resolves tarantool/tarantool#5842
> > Follows up tarantool/tarantool#5442
> > ---
> > Branch: https://github.com/tarantool/luajit/tree/skaplun/gh-5842-memprof-core-on-resizestack
> > Tarantool branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5842-memprof-core-on-resizestack
> > Issue: https://github.com/tarantool/tarantool/issues/5842
> > 
> > 
> > src/lj_state.c                                 |  6 ++++++
> > .../misclib-memprof-lapi.test.lua              | 18 ++++++++++++++++++
> > 2 files changed, 24 insertions(+)
> > 
> > diff --git a/src/lj_state.c b/src/lj_state.c
> > index 1ed79a5..ea9abd4 100644
> > --- a/src/lj_state.c
> > +++ b/src/lj_state.c
> > @@ -64,7 +64,11 @@ static void resizestack(lua_State *L, MSize n)
> >   MSize oldsize = L->stacksize;
> >   MSize realsize = n + 1 + LJ_STACK_EXTRA;
> >   GCobj *up;
> > +  int32_t old_vmstate = G(L)->vmstate;
> > +
> >   lua_assert((MSize)(tvref(L->maxstack)-oldst)==L->stacksize-LJ_STACK_EXTRA-1);
> > +
> > +  setvmstate(G(L), INTERP);
> 
> This looks like a hack. But even so - why not to return it back right after the
> realloc, where you supposedly use lua_getinfo? 

The Lua state L is inconsistent during stack resizing, so it can't be
interpreted in any way, until reallocation and corresponding converting
are finished.

> 
> The righteous way would be to fix the debug machinery anyhow.

Don't get your point here. Lua state is invalid, debug machinery works
correct, as I understand.

> 
> >   st = (TValue *)lj_mem_realloc(L, tvref(L->stack),
> > 				(MSize)(oldsize*sizeof(TValue)),
> > 				(MSize)(realsize*sizeof(TValue)));
> > @@ -80,6 +84,8 @@ static void resizestack(lua_State *L, MSize n)
> >   L->top = (TValue *)((char *)L->top + delta);
> >   for (up = gcref(L->openupval); up != NULL; up = gcnext(up))
> >     setmref(gco2uv(up)->v, (TValue *)((char *)uvval(gco2uv(up)) + delta));
> > +
> > +  G(L)->vmstate = old_vmstate;
> > }
> > 
> > /* Relimit stack after error, in case the limit was overdrawn. */
> > diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua
> > index 1c36c8a..93cc348 100644
> > --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua
> > +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua
> > @@ -125,5 +125,23 @@ test:ok(check_alloc_report(alloc, 25, 18, 100))
> > -- Collect all previous allocated objects.
> > test:ok(free.INTERNAL.num == 102)
> > 
> > +-- Test for https://github.com/tarantool/tarantool/issues/5842.
> > +-- We do not interested in report itself.
>      We ->			     ourselves?

I've removed confusing "itself". See the iterative patch below,
branch is force-pushed.

===================================================================
diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua
index 93cc348..9b9fb76 100644
--- a/test/tarantool-tests/misclib-memprof-lapi.test.lua
+++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua
@@ -126,7 +126,7 @@ test:ok(check_alloc_report(alloc, 25, 18, 100))
 test:ok(free.INTERNAL.num == 102)
 
 -- Test for https://github.com/tarantool/tarantool/issues/5842.
--- We do not interested in report itself.
+-- We do not interested in this report.
 misc.memprof.start("/dev/null")
 -- We need to cause stack resize for local variables at function
 -- call. Let's create a new coroutine (all slots are free).
===================================================================

> > +misc.memprof.start("/dev/null")
> > +-- We need to cause stack resize for local variables at function
> > +-- call. Let's create a new coroutine (all slots are free).
> > +-- It has 1 slot for dummy frame + 39 free slots + 5 extra slots
> > +-- (so-called red zone) + 2 * LJ_FR2 slots. So 50 local variables
> > +-- is enough.
> > +local payload_str = ""
> > +for i = 1, 50 do
> > +  payload_str = payload_str..("local v%d = %d\n"):format(i, i)
> > +end
> > +local f, errmsg = loadstring(payload_str)
> > +assert(f, errmsg)
> > +local co = coroutine.create(f)
> > +coroutine.resume(co)
> > +misc.memprof.stop()
> > +
> > jit.on()
> > os.exit(test:check() and 0 or 1)
> > — 
> > 2.28.0
> > 
> 
> 
> Regards,
> Sergos
> 
> 

-- 
Best regards,
Sergey Kaplun