From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 7C0836FC85; Tue, 14 Sep 2021 20:56:55 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 7C0836FC85 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1631642215; bh=8oOq0UJBEU4xy0NIRAAGgD9HxbE0HVlNqILafuivCJw=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=gWTQrscmYzuwl0zUGB1Q1qHkd59tyOkCNgn+7jpYnv7zUVzqP3za/7GgaprxljsQx RBiT1AHAfD9y7DEgipPjUJb0qKuKiql0IWK6mdyOuVDnNxHe7WB3rz08gicEkxDCrS KUgm/TTe/hZJV8p7Y14LhWawSPoF6MSIb0DqHvZE= Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 378D96FC85 for ; Tue, 14 Sep 2021 20:56:53 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 378D96FC85 Received: by mail-lf1-f51.google.com with SMTP id i25so164767lfg.6 for ; Tue, 14 Sep 2021 10:56:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=GAaYaGty2MuarcukqHkMjptxMEgfWz+iiKHCYttzPCA=; b=w/YzM56yMcOqD294iPNQzIDt1QvwQRnVbijX6HXDZxdxXR+D12NXpN98TcvPWS8im0 VWnFdn+AHRFsVECjf+RPst9ER2VZ+IyerLfAfXCXfqUranknDDRVJbv+i1XSpLQsiDvO xGw/r3WYltXioSsMUgP0NUKE51VzHOpVmBs9Jm+1Ou4Z4rZAMpQ6lgad1gzDDzotb8Kq wDin1HwtAh18Joe+8RGa3lb14kZTCvEgZl/P1NfGzcY/vVBzVCo6x/Rr499Wq7/N5WJC lK3sIenm2/ccDcWiireYj151CvpX4BieRQfZGWAUwo8ejarZPN2CbqQdnrdOJ8CR0vJt ANxg== X-Gm-Message-State: AOAM530ZrI7xgwzUx2dEao07CAz/HIyHwLTojqlHheBSRkbqtJW2Diyj xHGseNcVaRl+QGzyYH7B5BwZo62HbAZ91A== X-Google-Smtp-Source: ABdhPJyyfviubcx4AupZ8uh6B1v8ObtQ7ncp+0vVwpt7Zmi5WX1GEe8O/Om/5vPzZMGQ+WCBa1u22A== X-Received: by 2002:a05:6512:10d6:: with SMTP id k22mr14147150lfg.575.1631642212170; Tue, 14 Sep 2021 10:56:52 -0700 (PDT) Received: from localhost.localdomain ([93.175.11.199]) by smtp.gmail.com with ESMTPSA id q11sm1401571ljp.91.2021.09.14.10.56.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Sep 2021 10:56:51 -0700 (PDT) X-Google-Original-From: Maxim Kokryashkin To: tarantool-patches@dev.tarantool.org, imun@tarantool.org, skaplun@tarantool.org Date: Tue, 14 Sep 2021 20:56:48 +0300 Message-Id: <20210914175648.2999-1-m.kokryashkin@tarantool.org> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH v2] luajit: proxy -j and -b flags X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Maxim Kokryashkin via Tarantool-patches Reply-To: Maxim Kokryashkin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" There are two flags in the LuaJIT useful for debugging purposes: `-j` and `-b`. However, if you want to check the same Lua code from the Tarantool, you will need to make some adjustments in the script itself, as those flags are not present in the Tarantool's CLI. This patch introduces those flags to the Tarantool, so debugging is much more convenient now. Flags are working the same as they do in LuaJIT. Closes #5541 --- GitHub branch: https://github.com/tarantool/tarantool/tree/fckxorg/gh-5541-proxy-luajit-flags Issues: https://github.com/tarantool/tarantool/issues/5541 Q: Why do we need fiber_sleep here? Please, drop a comment. > + aux_loop_is_run = true; > + for (argv++; *argv != NULL; narg++, argv++) > + lua_pushstring(L, *argv); > + int res = lua_pcall(L, narg, 0, 0); > + if(res) > + goto error; > +end: > + if (!aux_loop_is_run) > + fiber_sleep(0.0); A:That chunk of code is taken almost verbatim from the `run_script_f()` as I need to achieve almost the same thing. So the first `fiber_sleep()` is needed, because `tarantool_lua_dump_bytecode()` needs to start auxiliary event loop and re-schedule this fiber. The second `fiber_sleep()` is needed because event loop should be started before the fiber calls the `ev_break()`. Q: IINM, bcsave module don't run a user's code, just precompile it to the bytecode. Do we need a separate fiber for it? A: Yes, we need it since tarantool has it's own definition of os.exit(). If you call os.exit() when running lua code outside of any fiber, you will get a panic. Since there is no easy way to redefine it temporarily and not break anything, I decided to stick with execution inside a fiber. src/lua/init.c | 185 +++++++++++++++++++++++++++++++++++++++++++++++++ src/lua/init.h | 2 + src/main.cc | 113 ++++++++++++++++++------------ 3 files changed, 255 insertions(+), 45 deletions(-) diff --git a/src/lua/init.c b/src/lua/init.c index f9738025d..54d07926a 100644 --- a/src/lua/init.c +++ b/src/lua/init.c @@ -313,6 +313,154 @@ skip: base = (base == -1 ? 10 : base); /* }}} */ +static void l_message(const char *msg) +{ + printf("%s\n", msg); + fflush(stdout); +} + +/** + * This function checks if there is an error, + * and outputs error message corresponding to the + * error object on the top of the Lua stack, + * if there is one, + * + * This function pops the error message from the Lua stack + */ +static int report(lua_State *L, int status) +{ + if (status && !lua_isnil(L, -1)) { + const char *msg = lua_tostring(L, -1); + if (msg == NULL) + msg = "(error object is not a string)"; + l_message(msg); + lua_pop(L, 1); + } + return status; +} + +/** + * Load add-on module. This function has no + * effects on the Lua stack. + */ +static int loadjitmodule(lua_State* L) +{ + lua_getglobal(L, "require"); + lua_pushliteral(L, "jit."); + lua_pushvalue(L, -3); + lua_concat(L, 2); + if (lua_pcall(L, 1, 1, 0)) { + const char *msg = lua_tostring(L, -1); + if (msg && !strncmp(msg, "module ", 7)) + goto nomodule; + return report(L, 1); + } + lua_getfield(L, -1, "start"); + if (lua_isnil(L, -1)) { + nomodule: + l_message("unknown luaJIT command or jit.* modules not installed"); + return 1; + } + /* Drop module table. */ + lua_remove(L, -2); + return 0; +} +/** + * This function dump bytecode for Lua script. + * Has no effect on the Lua stack + */ +int dobytecode(va_list ap) +{ + char **argv = va_arg(ap, char **); + struct diag *diag = va_arg(ap, struct diag *); + + int narg = 0; + bool aux_loop_is_run = false; + + lua_pushliteral(tarantool_L, "bcsave"); + if (loadjitmodule(tarantool_L)) + goto error; + if (argv[0][2]) { + narg++; + argv[0][1] = '-'; + lua_pushstring(tarantool_L, argv[0] + 1); + } + + fiber_sleep(0.0); + aux_loop_is_run = true; + + for (argv++; *argv != NULL; narg++, argv++) + lua_pushstring(tarantool_L, *argv); + int res = lua_pcall(tarantool_L, narg, 0, 0); + if (res != LUA_OK) + goto error; + end: + if (!aux_loop_is_run) + fiber_sleep(0.0); + ev_break(loop(), EVBREAK_ALL); + return 0; + + error: + diag_move(diag_get(), diag); + goto end; +} + +/** + * Run command with options. + * Has no effect on the Lua stack. + */ +static int runcmdopt(lua_State *L, const char *opt) +{ + int narg = 0; + if (opt && *opt) { + for (;;) { + const char *p = strchr(opt, ','); + narg++; + if (!p) break; + if (p == opt) + lua_pushnil(L); + else + lua_pushlstring(L, opt, (size_t)(p - opt)); + opt = p + 1; + } + if (*opt) + lua_pushstring(L, opt); + else + lua_pushnil(L); + } + return report(L, lua_pcall(L, narg, 0, 0)); +} + +/** + * JIT engine control command: try jit + * library first or load add-on module. + * Has no effect on the Lua stack. + */ +int dojitcmd(const char *cmd) +{ + const char *opt = strchr(cmd, '='); + lua_pushlstring(tarantool_L, cmd, opt ? (size_t)(opt - cmd) : strlen(cmd)); + lua_getfield(tarantool_L, LUA_REGISTRYINDEX, "_LOADED"); + /* Get jit.* module table. */ + lua_getfield(tarantool_L, -1, "jit"); + lua_remove(tarantool_L, -2); + lua_pushvalue(tarantool_L, -2); + /* Lookup library function. */ + lua_gettable(tarantool_L, -2); + if (!lua_isfunction(tarantool_L, -1)) { + /* Drop non-function and jit.* table, keep module name. */ + lua_pop(tarantool_L, 2); + if (loadjitmodule(tarantool_L)) + return 1; + } else { + /* Drop jit.* table. */ + lua_remove(tarantool_L, -2); + } + /* Drop module name. */ + lua_remove(tarantool_L, -2); + return runcmdopt(tarantool_L, opt ? opt + 1 : opt); +} + /** * Original LuaJIT/Lua logic: * @@ -608,6 +756,10 @@ run_script_f(va_list ap) lua_setglobal(L, optv[i + 1]); lua_settop(L, 0); break; + case 'j': + if (dojitcmd(optv[i + 1]) != 0) + goto error; + break; case 'e': /* * Execute chunk @@ -747,6 +899,39 @@ tarantool_lua_run_script(char *path, bool interactive, return diag_is_empty(diag_get()) ? 0 : -1; } +int +tarantool_lua_dump_bytecode(char **argv) { + script_fiber = fiber_new("bcsave", dobytecode); + if (script_fiber == NULL) + panic("%s", diag_last_error(diag_get())->errmsg); + script_fiber->storage.lua.stack = tarantool_L; + /* + * Create a new diag on the stack. Don't pass fiber's diag, because it + * might be overwritten by libev callbacks invoked in the scheduler + * fiber (which is this), and therefore can't be used as a sign of fail + * in the script itself. + */ + struct diag bc_diag; + diag_create(&bc_diag); + fiber_start(script_fiber, argv, &bc_diag); + /* + * Run an auxiliary event loop to re-schedule run_script fiber. + * When this fiber finishes, it will call ev_break to stop the loop. + */ + if (start_loop) + ev_run(loop(), 0); + /* The fiber running the startup script has ended. */ + script_fiber = NULL; + diag_move(&bc_diag, diag_get()); + diag_destroy(&bc_diag); + /* + * Result can't be obtained via fiber_join - script fiber + * never dies if os.exit() was called. This is why diag + * is checked explicitly. + */ + return diag_is_empty(diag_get()) ? 0 : -1; +} + void tarantool_lua_free() { diff --git a/src/lua/init.h b/src/lua/init.h index 7fc0b1a31..3c9489468 100644 --- a/src/lua/init.h +++ b/src/lua/init.h @@ -74,6 +74,8 @@ int tarantool_lua_run_script(char *path, bool force_interactive, int optc, const char **optv, int argc, char **argv); +int +tarantool_lua_dump_bytecode(char **argv); extern char *history; diff --git a/src/main.cc b/src/main.cc index 91ed79fab..65248e655 100644 --- a/src/main.cc +++ b/src/main.cc @@ -576,6 +576,8 @@ print_help(const char *program) puts(" -v, --version\t\t\tprint program version and exit"); puts(" -e EXPR\t\t\texecute string 'EXPR'"); puts(" -l NAME\t\t\trequire library 'NAME'"); + puts(" -j CMD \t\t\tperform LuaJIT control command"); + puts(" -b[options] input output\tsave LuaJIT bytecode"); puts(" -i\t\t\t\tenter interactive mode after executing 'SCRIPT'"); puts(" --\t\t\t\tstop handling options"); puts(" -\t\t\t\texecute stdin and stop handling options"); @@ -584,6 +586,12 @@ print_help(const char *program) puts("to see online documentation, submit bugs or contribute a patch."); } +#define O_INTERACTIVE 0x1 +#define O_BYTECODE 0x2 + +extern "C" void ** +export_syms(void); + int main(int argc, char **argv) { @@ -595,7 +603,7 @@ main(int argc, char **argv) fpconv_check(); /* Enter interactive mode after executing 'script' */ - bool interactive = false; + uint32_t opt_mask = 0; /* Lua interpeter options, e.g. -e and -l */ int optc = 0; const char **optv = NULL; @@ -606,38 +614,47 @@ main(int argc, char **argv) {"version", no_argument, 0, 'v'}, {NULL, 0, 0, 0}, }; - static const char *opts = "+hVvie:l:"; + static const char *opts = "+hVvbij:e:l:"; int ch; + bool lj_arg = false; while ((ch = getopt_long(argc, argv, opts, longopts, NULL)) != -1) { - switch (ch) { - case 'V': - case 'v': - print_version(); - return 0; - case 'h': - print_help(basename(argv[0])); - return 0; - case 'i': - /* Force interactive mode */ - interactive = true; - break; - case 'l': - case 'e': - /* Save Lua interepter options to optv as is */ - if (optc == 0) { - optv = (const char **) calloc(argc, - sizeof(optv[0])); - if (optv == NULL) - panic_syserror("No enough memory for arguments"); - } - optv[optc++] = ch == 'l' ? "-l" : "-e"; - optv[optc++] = optarg; - break; - default: - /* "invalid option" is printed by getopt */ - return EX_USAGE; - } + switch (ch) { + case 'V': + case 'v': + print_version(); + return 0; + case 'h': + print_help(basename(argv[0])); + return 0; + case 'i': + /* Force interactive mode */ + opt_mask |= O_INTERACTIVE; + break; + case 'b': + opt_mask |= O_BYTECODE; + lj_arg = true; + break; + case 'j': + case 'l': + case 'e': + /* Save Lua interepter options to optv as is */ + if (optc == 0) { + optv = (const char **) calloc(argc, sizeof(optv[0])); + if (optv == NULL) + panic_syserror("No enough memory for arguments"); + } + if(ch == 'l') optv[optc++] = "-l"; + else if(ch == 'j') optv[optc++] = "-j"; + else optv[optc++] = "-e"; + optv[optc++] = optarg; + break; + default: + /* "invalid option" is printed by getopt */ + return EX_USAGE; + } + + if(lj_arg) break; } /* Shift arguments */ @@ -645,19 +662,21 @@ main(int argc, char **argv) for (int i = 1; i < argc; i++) argv[i] = argv[optind + i - 1]; - if (argc > 1 && strcmp(argv[1], "-") && access(argv[1], R_OK) != 0) { - /* - * Somebody made a mistake in the file - * name. Be nice: open the file to set - * errno. - */ - int fd = open(argv[1], O_RDONLY); - int save_errno = errno; - if (fd >= 0) - close(fd); - printf("Can't open script %s: %s\n", argv[1], strerror(save_errno)); - return save_errno; - } + if(!(opt_mask & O_BYTECODE)) { + if (argc > 1 && strcmp(argv[1], "-") && access(argv[1], R_OK) != 0) { + /* + * Somebody made a mistake in the file + * name. Be nice: open the file to set + * errno. + */ + int fd = open(argv[1], O_RDONLY); + int save_errno = errno; + if (fd >= 0) + close(fd); + printf("Can't open script %s: %s\n", argv[1], strerror(save_errno)); + return save_errno; + } + } argv = title_init(argc, argv); /* @@ -747,13 +766,17 @@ main(int argc, char **argv) panic("%s", "can't init event loop"); int events = ev_activecnt(loop()); - /* + + if(opt_mask & O_BYTECODE) { + return tarantool_lua_dump_bytecode(argv); + } + /* * Load user init script. The script should have access * to Tarantool Lua API (box.cfg, box.fiber, etc...) that * is why script must run only after the server was fully * initialized. */ - if (tarantool_lua_run_script(script, interactive, optc, optv, + if (tarantool_lua_run_script(script, opt_mask & O_INTERACTIVE, optc, optv, main_argc, main_argv) != 0) diag_raise(); /* -- 2.33.0