[Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Sergey Kaplun]

Panic at fiber.yield() occures inside any active hook.

This is the regression from 96dbc49d097a96af5273cce2b5663db5917f4ea9
('lua: prohibit fiber yield when GC hook is active').

This patch fixes false positive panic in cases when VM is not running
a GC hook.

Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>
Follows up #4727
Closes #5649
---

Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5649-fix-for-debug-hook
Issue: https://github.com/tarantool/tarantool/issues/5649
CI: https://gitlab.com/tarantool/tarantool/-/pipelines/233249505

 src/lua/utils.c                               |  2 +-
 .../gh-5649-yield-in-debug-hook.test.lua      | 25 +++++++++++++++++++
 2 files changed, 26 insertions(+), 1 deletion(-)
 create mode 100755 test/app-tap/gh-5649-yield-in-debug-hook.test.lua

diff --git a/src/lua/utils.c b/src/lua/utils.c
index 23fbdd4ad..b5a6ca5b7 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -1375,7 +1375,7 @@ void cord_on_yield(void)
 	 * earlier. As a result fiber switch is prohibited when
 	 * GC hook is active and the platform is forced to stop.
 	 */
-	if (unlikely(g->hookmask & (HOOK_ACTIVE|HOOK_GC))) {
+	if (unlikely(g->hookmask & HOOK_GC)) {
 		struct lua_State *L = fiber()->storage.lua.stack;
 		assert(L != NULL);
 		lua_pushfstring(L, "fiber %d is switched while running GC"
diff --git a/test/app-tap/gh-5649-yield-in-debug-hook.test.lua b/test/app-tap/gh-5649-yield-in-debug-hook.test.lua
new file mode 100755
index 000000000..9c1cca51f
--- /dev/null
+++ b/test/app-tap/gh-5649-yield-in-debug-hook.test.lua
@@ -0,0 +1,25 @@
+#!/usr/bin/env tarantool
+
+local fiber = require('fiber')
+local tap = require('tap')
+local test = tap.test('yield-in-debug-hook')
+
+test:plan(2)
+
+-- Test that HOOK_ACTIVE is not enough to panic and
+-- fiber still can use general hooks at switches.
+fiber.create(function()
+  local old_hook, mask, count = debug.gethook()
+  debug.sethook(function()
+    fiber.yield()
+  end, 'c')
+  -- All ok if panic doesn't occure.
+  -- Yield before hook is set back.
+  debug.sethook(old_hook, mask, count)
+  test:ok(true)
+end)
+-- Return to second fiber.
+fiber.yield()
+test:ok(true)
+
+os.exit(test:check() and 0 or 1)
-- 
2.28.0

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Igor Munkin]

Sergey,

Thanks for the patch! LGTM except a couple of nits below.

Minor: Strictly saying there is no panic if GC hook is not active
(literally any Lua code except the finalizers), but panic occurs when
the active hook is not HOOK_GC. So I propose to tune the commit subject
the following way:
| lua: avoid panic if HOOK_GC is not an active hook

On 22.12.20, Sergey Kaplun wrote:
> Panic at fiber.yield() occures inside any active hook.

Minor: I rearrange this line a bit, feel free to ignore.
| Platform panic occurs when fiber.yield() is used within any active
| (i.e. being executed now) hook.

> 
> This is the regression from 96dbc49d097a96af5273cce2b5663db5917f4ea9
> ('lua: prohibit fiber yield when GC hook is active').
> 
> This patch fixes false positive panic in cases when VM is not running
> a GC hook.
> 
> Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>
> Follows up #4727

Minor: I guess this patch doesn't follow #4727 but rather #4518
(considering the commit message for the patch you mentioned above). So I
propose to use either "Relates to #4518" or "Follows up #4518" instead.
I personally prefer the first one, since it occured #4518 relates to
totally another issue.

> Closes #5649
> ---
> 
> Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5649-fix-for-debug-hook
> Issue: https://github.com/tarantool/tarantool/issues/5649
> CI: https://gitlab.com/tarantool/tarantool/-/pipelines/233249505
> 
>  src/lua/utils.c                               |  2 +-
>  .../gh-5649-yield-in-debug-hook.test.lua      | 25 +++++++++++++++++++
>  2 files changed, 26 insertions(+), 1 deletion(-)
>  create mode 100755 test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> 

<snipped>

> diff --git a/test/app-tap/gh-5649-yield-in-debug-hook.test.lua b/test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> new file mode 100755
> index 000000000..9c1cca51f
> --- /dev/null
> +++ b/test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> @@ -0,0 +1,25 @@
> +#!/usr/bin/env tarantool
> +
> +local fiber = require('fiber')
> +local tap = require('tap')
> +local test = tap.test('yield-in-debug-hook')
> +
> +test:plan(2)
> +
> +-- Test that HOOK_ACTIVE is not enough to panic and
> +-- fiber still can use general hooks at switches.
> +fiber.create(function()
> +  local old_hook, mask, count = debug.gethook()
> +  debug.sethook(function()
> +    fiber.yield()
> +  end, 'c')
> +  -- All ok if panic doesn't occure.
> +  -- Yield before hook is set back.
> +  debug.sethook(old_hook, mask, count)
> +  test:ok(true)
> +end)
> +-- Return to second fiber.

Minor: s/to second/to the second/.

> +fiber.yield()
> +test:ok(true)
> +
> +os.exit(test:check() and 0 or 1)
> -- 
> 2.28.0
> 

-- 
Best regards,
IM

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Igor Munkin]

<snipped>

> > 
> > Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>
> > Follows up #4727

One more nit: Please move the misc-by tags below the GitHub ones.

> 

<snipped>

-- 
Best regards,
IM

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Sergey Ostanevich]

Hi!

Thanks for the patch!

Sergos

> On 22 Dec 2020, at 16:08, Sergey Kaplun <skaplun@tarantool.org> wrote:
> 
> Panic at fiber.yield() occures inside any active hook.
> 
> This is the regression from 96dbc49d097a96af5273cce2b5663db5917f4ea9
  ^^^^^^^^^^ It is a     ^^^ caused by

> ('lua: prohibit fiber yield when GC hook is active').
> 
> This patch fixes false positive panic in cases when VM is not running
> a GC hook.
> 
> Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>
> Follows up #4727
> Closes #5649
> ---
> 
> Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5649-fix-for-debug-hook
> Issue: https://github.com/tarantool/tarantool/issues/5649
> CI: https://gitlab.com/tarantool/tarantool/-/pipelines/233249505
> 
> src/lua/utils.c                               |  2 +-
> .../gh-5649-yield-in-debug-hook.test.lua      | 25 +++++++++++++++++++
> 2 files changed, 26 insertions(+), 1 deletion(-)
> create mode 100755 test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> 
> diff --git a/src/lua/utils.c b/src/lua/utils.c
> index 23fbdd4ad..b5a6ca5b7 100644
> --- a/src/lua/utils.c
> +++ b/src/lua/utils.c
> @@ -1375,7 +1375,7 @@ void cord_on_yield(void)
> 	 * earlier. As a result fiber switch is prohibited when
> 	 * GC hook is active and the platform is forced to stop.
> 	 */
> -	if (unlikely(g->hookmask & (HOOK_ACTIVE|HOOK_GC))) {
> +	if (unlikely(g->hookmask & HOOK_GC)) {

It can happens the GC hook is not active? Should we have a second test for HOOK_ACTIVE?

> 		struct lua_State *L = fiber()->storage.lua.stack;
> 		assert(L != NULL);
> 		lua_pushfstring(L, "fiber %d is switched while running GC"
> diff --git a/test/app-tap/gh-5649-yield-in-debug-hook.test.lua b/test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> new file mode 100755
> index 000000000..9c1cca51f
> --- /dev/null
> +++ b/test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> @@ -0,0 +1,25 @@
> +#!/usr/bin/env tarantool
> +
> +local fiber = require('fiber')
> +local tap = require('tap')
> +local test = tap.test('yield-in-debug-hook')
> +
> +test:plan(2)
> +
> +-- Test that HOOK_ACTIVE is not enough to panic and
> +-- fiber still can use general hooks at switches.
> +fiber.create(function()
> +  local old_hook, mask, count = debug.gethook()
> +  debug.sethook(function()
> +    fiber.yield()
> +  end, 'c')
> +  -- All ok if panic doesn't occure.
> +  -- Yield before hook is set back.
> +  debug.sethook(old_hook, mask, count)
> +  test:ok(true)
> +end)
> +-- Return to second fiber.
> +fiber.yield()
> +test:ok(true)
> +
> +os.exit(test:check() and 0 or 1)
> -- 
> 2.28.0
> 

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Sergey Kaplun]

Hi, Sergos!

Thanks for the review!

On 22.12.20, Sergey Ostanevich wrote:
> Hi!
> 
> Thanks for the patch!
> 
> Sergos
> 
> > On 22 Dec 2020, at 16:08, Sergey Kaplun <skaplun@tarantool.org> wrote:
> > 
> > Panic at fiber.yield() occures inside any active hook.
> > 
> > This is the regression from 96dbc49d097a96af5273cce2b5663db5917f4ea9
>   ^^^^^^^^^^ It is a     ^^^ caused by

Fixed. Thanks!

> 
> > ('lua: prohibit fiber yield when GC hook is active').

<snipped>

> > --- a/src/lua/utils.c
> > +++ b/src/lua/utils.c
> > @@ -1375,7 +1375,7 @@ void cord_on_yield(void)
> > 	 * earlier. As a result fiber switch is prohibited when
> > 	 * GC hook is active and the platform is forced to stop.
> > 	 */
> > -	if (unlikely(g->hookmask & (HOOK_ACTIVE|HOOK_GC))) {
> > +	if (unlikely(g->hookmask & HOOK_GC)) {
> 
> It can happens the GC hook is not active? Should we have a second test for HOOK_ACTIVE?

No, it can't. HOOK_GC always is set with HOOK_ACTIVE inside
`gc_call_finalizer()` (see <lj_gc.c>). Old hook is restored after
finalizer's call.

> 
> > 		struct lua_State *L = fiber()->storage.lua.stack;

<snipped>

> 

I've updated commit message and comments in code considering your and
Igor comments. Branch is force-pushed.

New commit message is the follows:
| lua: avoid panic if HOOK_GC is not an active hook
|
| Platform panic occurs when fiber.yield() is used within any active
| (i.e. being executed now) hook.
|
| It is a regression caused by 96dbc49d097a96af5273cce2b5663db5917f4ea9
| ('lua: prohibit fiber yield when GC hook is active').
|
| This patch fixes false positive panic in cases when VM is not running
| a GC hook.
|
| Relates to #4518
| Closes #5649
|
| Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Sergey Kaplun]

Hi, Igor!

Thanks for the review!

On 22.12.20, Igor Munkin wrote:
> Sergey,
> 
> Thanks for the patch! LGTM except a couple of nits below.
> 
> Minor: Strictly saying there is no panic if GC hook is not active
> (literally any Lua code except the finalizers), but panic occurs when
> the active hook is not HOOK_GC. So I propose to tune the commit subject
> the following way:
> | lua: avoid panic if HOOK_GC is not an active hook

Yes, it's better. Thanks!

> 
> On 22.12.20, Sergey Kaplun wrote:
> > Panic at fiber.yield() occures inside any active hook.
> 
> Minor: I rearrange this line a bit, feel free to ignore.
> | Platform panic occurs when fiber.yield() is used within any active
> | (i.e. being executed now) hook.

Look nice for me. Applied :)

> 
> > 
> > This is the regression from 96dbc49d097a96af5273cce2b5663db5917f4ea9
> > ('lua: prohibit fiber yield when GC hook is active').
> > 
> > This patch fixes false positive panic in cases when VM is not running
> > a GC hook.
> > 
> > Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>
> > Follows up #4727
> 
> Minor: I guess this patch doesn't follow #4727 but rather #4518
> (considering the commit message for the patch you mentioned above). So I
> propose to use either "Relates to #4518" or "Follows up #4518" instead.
> I personally prefer the first one, since it occured #4518 relates to
> totally another issue.
> 

Fixed.

> > Closes #5649
> > ---
> > 
> > Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5649-fix-for-debug-hook
> > Issue: https://github.com/tarantool/tarantool/issues/5649
> > CI: https://gitlab.com/tarantool/tarantool/-/pipelines/233249505
> > 
> >  src/lua/utils.c                               |  2 +-
> >  .../gh-5649-yield-in-debug-hook.test.lua      | 25 +++++++++++++++++++

<snipped>

> > +-- Return to second fiber.
> 
> Minor: s/to second/to the second/.

Fixed! Thanks!

> 

<snipped>

> 
> -- 
> Best regards,
> IM

I've updated commit message and comments in code considering your and Igor comments.
Branch is force-pushed.

New commit message is the follows:
| lua: avoid panic if HOOK_GC is not an active hook
|
| Platform panic occurs when fiber.yield() is used within any active
| (i.e. being executed now) hook.
|
| It is a regression caused by 96dbc49d097a96af5273cce2b5663db5917f4ea9
| ('lua: prohibit fiber yield when GC hook is active').
|
| This patch fixes false positive panic in cases when VM is not running
| a GC hook.
|
| Relates to #4518
| Closes #5649
|
| Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Sergey Kaplun]

> I've updated commit message and comments in code considering your and Igor comments.

Sorry, I meant "considering your and Sergos comments".
Apparently I should get some sleep :)

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Sergey Kaplun]

ChangeLog:
 - Fix false positive panic when yield in debug hook (gh-5649).

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Sergey Ostanevich]

LGTM.

Sergos

> On 22 Dec 2020, at 21:01, Sergey Kaplun <skaplun@tarantool.org> wrote:
> 
> ChangeLog:
> - Fix false positive panic when yield in debug hook (gh-5649).
> 
> -- 
> Best regards,
> Sergey Kaplun

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Alexander V. Tikhonov]

Hi Sergey, thanks for the patch, as I see no new degradation found in
gitlab-ci testing commit criteria pipeline [1], patch LGTM.

[1] - https://gitlab.com/tarantool/tarantool/-/pipelines/233455482

On Tue, Dec 22, 2020 at 04:08:58PM +0300, Sergey Kaplun via Tarantool-patches wrote:
> Panic at fiber.yield() occures inside any active hook.
> 
> This is the regression from 96dbc49d097a96af5273cce2b5663db5917f4ea9
> ('lua: prohibit fiber yield when GC hook is active').
> 
> This patch fixes false positive panic in cases when VM is not running
> a GC hook.
> 
> Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>
> Follows up #4727
> Closes #5649
> ---
> 
> Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5649-fix-for-debug-hook
> Issue: https://github.com/tarantool/tarantool/issues/5649
> CI: https://gitlab.com/tarantool/tarantool/-/pipelines/233249505
> 
>  src/lua/utils.c                               |  2 +-
>  .../gh-5649-yield-in-debug-hook.test.lua      | 25 +++++++++++++++++++
>  2 files changed, 26 insertions(+), 1 deletion(-)
>  create mode 100755 test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> 
> diff --git a/src/lua/utils.c b/src/lua/utils.c
> index 23fbdd4ad..b5a6ca5b7 100644
> --- a/src/lua/utils.c
> +++ b/src/lua/utils.c
> @@ -1375,7 +1375,7 @@ void cord_on_yield(void)
>  	 * earlier. As a result fiber switch is prohibited when
>  	 * GC hook is active and the platform is forced to stop.
>  	 */
> -	if (unlikely(g->hookmask & (HOOK_ACTIVE|HOOK_GC))) {
> +	if (unlikely(g->hookmask & HOOK_GC)) {
>  		struct lua_State *L = fiber()->storage.lua.stack;
>  		assert(L != NULL);
>  		lua_pushfstring(L, "fiber %d is switched while running GC"
> diff --git a/test/app-tap/gh-5649-yield-in-debug-hook.test.lua b/test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> new file mode 100755
> index 000000000..9c1cca51f
> --- /dev/null
> +++ b/test/app-tap/gh-5649-yield-in-debug-hook.test.lua
> @@ -0,0 +1,25 @@
> +#!/usr/bin/env tarantool
> +
> +local fiber = require('fiber')
> +local tap = require('tap')
> +local test = tap.test('yield-in-debug-hook')
> +
> +test:plan(2)
> +
> +-- Test that HOOK_ACTIVE is not enough to panic and
> +-- fiber still can use general hooks at switches.
> +fiber.create(function()
> +  local old_hook, mask, count = debug.gethook()
> +  debug.sethook(function()
> +    fiber.yield()
> +  end, 'c')
> +  -- All ok if panic doesn't occure.
> +  -- Yield before hook is set back.
> +  debug.sethook(old_hook, mask, count)
> +  test:ok(true)
> +end)
> +-- Return to second fiber.
> +fiber.yield()
> +test:ok(true)
> +
> +os.exit(test:check() and 0 or 1)
> -- 
> 2.28.0
> 

Re: [Tarantool-patches] [PATCH] lua: avoid panic if HOOK_GC is not active

[Kirill Yukhin]

Hello,

On 22 Dec 16:08, Sergey Kaplun via Tarantool-patches wrote:
> Panic at fiber.yield() occures inside any active hook.
> 
> This is the regression from 96dbc49d097a96af5273cce2b5663db5917f4ea9
> ('lua: prohibit fiber yield when GC hook is active').
> 
> This patch fixes false positive panic in cases when VM is not running
> a GC hook.
> 
> Reported-by: Michael Filonenko <filonenko.mikhail@gmail.com>
> Follows up #4727
> Closes #5649

I've checked your patch into 1.10, 2.5, 2.6 and master.

--
Regards, Kirill Yukhin