From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 39A7A6E454; Wed, 9 Feb 2022 20:55:00 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 39A7A6E454 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1644429300; bh=kesYRob3o4XfiXdY9Pr/8deAksTOhweHfX48TmQNPVs=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=RiQLdKWHt9wuDXkUokvPAcWC903fVTLN9TqVzX0S7NGq+4Zvr6M7j5cORcitcwD8m /Ldmu0u9IYmNiRyXVNJliDeSQ9bZnGUgkTe4vDUTM7gDlkdnuVB5bH9SzvZmYTCpXZ C2M3nBHmAnulwvmZHOp0r6lLne3Z4+efSAvf6wqs= Received: from smtp50.i.mail.ru (smtp50.i.mail.ru [94.100.177.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 482ED6E224 for ; Wed, 9 Feb 2022 20:53:48 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 482ED6E224 Received: by smtp50.i.mail.ru with esmtpa (envelope-from ) id 1nHrA3-0005PT-LA; Wed, 09 Feb 2022 20:53:48 +0300 Message-ID: <99e605f9-a1bb-7b13-01f6-13f5bb40ce81@tarantool.org> Date: Wed, 9 Feb 2022 20:53:47 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.5.1 To: Vladislav Shpilevoy , tarantool-patches@dev.tarantool.org References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9B74B50284A7C1A0BDB3817FC5E3D06AFA2654822E574318B182A05F538085040D169FDB2CE76AEADE7C0FA502F182F1F9B055873A37FAAB28C0E2A33A489D46B X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE779AAD18609327F83EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637C2EE9128AC0EB2C78638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D891D777C19FA5414C47E3A7085F3E66D8117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BF1175FABE1C0F9B6A471835C12D1D977C4224003CC836476EB9C4185024447017B076A6E789B0E975F5C1EE8F4F765FC58E42838E3597E443AA81AA40904B5D9CF19DD082D7633A078D18283394535A93AA81AA40904B5D98AA50765F7900637E643E67EAB77E948D81D268191BDAD3D698AB9A7B718F8C4D1B931868CE1C5781A620F70A64A45A98AA50765F79006372E808ACE2090B5E1725E5C173C3A84C3C5EA940A35A165FF2DBA43225CD8A89F83C798A30B85E16B262FEC7FBD7D1F5BB5C8C57E37DE458BEDA766A37F9254B7 X-8FC586DF: 6EFBBC1D9D64D975 X-C1DE0DAB: 0D63561A33F958A58A8A44DCFEFEA87379BB2E0D5ED122092E860BCD0A9CD021D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75BFC02AB3DF06BA5A410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34F6110710A6CC527F8E684D2A9AB810BAD61B23FE5299F1DC54990DA39C8AD76C82E16447B7E74D0C1D7E09C32AA3244CCEF97E31AC812F9E27F9ACEA9AC16C76A90944CA99CF22E3729B2BEF169E0186 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojptAT+HCMDqOas0ARC1m/Zw== X-Mailru-Sender: 583F1D7ACE8F49BD508BD8DBBD09B14D31DDF73A9EF098EFB5DF9E14446471FFCB345D7DBB60B20B80EE221D05932256AD9BA6614E257C8ED9E51C16F2486AFBE342CF4F05FB7E8CB0DAF586E7D11B3E67EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH vshard 4/4] router: support netbox return_raw X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Oleg Babin via Tarantool-patches Reply-To: Oleg Babin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks for your patch. Two comments below. On 09.02.2022 03:32, Vladislav Shpilevoy wrote: > Netbox return_raw option allows to return remote call result > without decoding, as an msgpack object. It will be introduced > after 2.10.0-beta2. > > It can be used with router working as a proxy. Pass arguments to > it in msgpack object and get back another msgpack object by usage > of return_raw option. This way the router won't decode nor encode > user's function arguments at all. > > Part of #312 > --- > test/instances/storage.lua | 5 ++ > test/reload_evolution/storage.result | 4 -- > test/router-luatest/router_test.lua | 86 ++++++++++++++++++++++++++++ > test/storage/storage.result | 2 - > vshard/router/init.lua | 28 ++++++++- > vshard/storage/init.lua | 15 +++++ > vshard/util.lua | 4 +- > 7 files changed, 136 insertions(+), 8 deletions(-) > > diff --git a/test/instances/storage.lua b/test/instances/storage.lua > index 2d679ba..4a05d82 100755 > --- a/test/instances/storage.lua > +++ b/test/instances/storage.lua > @@ -12,10 +12,15 @@ box.ctl.set_on_shutdown_timeout(0.001) > box.cfg(helpers.box_cfg()) > box.schema.user.grant('guest', 'super', nil, nil, {if_not_exists = true}) > > +local function box_error() > + box.error(box.error.PROC_LUA, 'box_error') > +end > + > local function echo(...) > return ... > end > > +_G.box_error = box_error > _G.echo = echo > > _G.ready = true > diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result > index 77010a2..a282bba 100644 > --- a/test/reload_evolution/storage.result > +++ b/test/reload_evolution/storage.result > @@ -122,8 +122,6 @@ vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42}) > --- > - true > - - [42, 3000] > -- null > -- null > ... > vshard.storage.bucket_send(bucket_id_to_move, util.replicasets[1]) > --- > @@ -155,8 +153,6 @@ vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42}) > --- > - true > - - [42, 3000] > -- null > -- null > ... > -- Check info() does not fail. > vshard.storage.info() ~= nil > diff --git a/test/router-luatest/router_test.lua b/test/router-luatest/router_test.lua > index 13ab74d..23929dc 100644 > --- a/test/router-luatest/router_test.lua > +++ b/test/router-luatest/router_test.lua > @@ -99,3 +99,89 @@ g.test_msgpack_args = function(g) > end > > end -- vutil.feature.msgpack_object > + > +if vutil.feature.netbox_return_raw then > + > +local function test_return_raw_template(g, mode) > + local router = g.router > + -- > + -- Normal call. > + -- > + local res = router:exec(function(timeout, mode) > + return add_details(vshard.router[mode](1, 'echo', {1, 2, 3}, > + {timeout = timeout, return_raw = true})) > + end, {wait_timeout, mode}) > + t.assert_equals(res.val, {1, 2, 3}, 'value value') > + t.assert_equals(res.val_type, 'userdata', 'value type') > + t.assert(not res.err, 'no error') > + > + -- > + -- Route call. > + -- > + res = router:exec(function(timeout, mode) > + local route = vshard.router.route(1) > + return add_details(route[mode](route, 'echo', {1, 2, 3}, > + {timeout = timeout, return_raw = true})) > + end, {wait_timeout, mode}) > + t.assert_equals(res.val, {1, 2, 3}, 'value value') > + t.assert_equals(res.val_type, 'userdata', 'value type') > + t.assert(not res.err, 'no error') > + > + -- > + -- Empty result set. > + -- > + res = router:exec(function(timeout, mode) > + return add_details(vshard.router[mode](1, 'echo', {}, > + {timeout = timeout, return_raw = true})) > + end, {wait_timeout, mode}) > + t.assert(not res.val, 'no value') > + t.assert(not res.err, 'no error') > + > + -- > + -- Error. > + -- > + res = router:exec(function(timeout, mode) > + return add_details(vshard.router[mode](1, 'box_error', {1, 2, 3}, > + {timeout = timeout})) > + end, {wait_timeout, mode}) > + t.assert(not res.val, 'no value') > + t.assert_equals(res.err_type, 'table', 'error type') > + t.assert_covers(res.err, {type = 'ClientError', code = box.error.PROC_LUA}, > + 'error value') > + > + -- > + -- Route error. > + -- > + res = router:exec(function(timeout, mode) > + local route = vshard.router.route(1) > + return add_details(route[mode](route, 'box_error', {1, 2, 3}, > + {timeout = timeout})) > + end, {wait_timeout, mode}) > + t.assert(not res.val, 'no value') > + t.assert_equals(res.err_type, 'table', 'error type') > + t.assert_covers(res.err, {type = 'ClientError', code = box.error.PROC_LUA}, > + 'error value') > +end > + > +g.test_return_raw = function(g) g.skip_if could be used Also probably you could use parametrization (however I won't insist): ``` local pg = t.group('pgroup', {{mode = 'callrw'}, {engine = 'callro'}}) pg.test = function(cg)    ... end ``` > + g.router:exec(function() > + rawset(_G, 'add_details', function(val, err) > + -- Direct return would turn nils into box.NULLs. The tests want to > + -- ensure it doesn't happen. Table wrap makes the actual nils > + -- eliminate themselves. > + return { > + val = val, > + val_type = type(val), > + err = err, > + err_type = type(err), > + } > + end) > + end) > + test_return_raw_template(g, 'callrw') > + test_return_raw_template(g, 'callro') > + g.router:exec(function() > + _G.add_details = nil > + end) > +end > + > +end -- vutil.feature.netbox_return_raw > diff --git a/test/storage/storage.result b/test/storage/storage.result > index dcd1c1f..73c171a 100644 > --- a/test/storage/storage.result > +++ b/test/storage/storage.result > @@ -480,8 +480,6 @@ vshard.storage.call(1, 'read', 'space_get', {'test', {1}}) > --- > - true > - [1, 1] > -- null > -- null > ... > vshard.storage.call(100500, 'read', 'space_get', {'test', {1}}) > --- > diff --git a/vshard/router/init.lua b/vshard/router/init.lua > index 44ed801..d39f489 100644 > --- a/vshard/router/init.lua > +++ b/vshard/router/init.lua > @@ -1,7 +1,10 @@ > local log = require('log') > local lfiber = require('fiber') > +local lmsgpack = require('msgpack') > local table_new = require('table.new') > local fiber_clock = lfiber.clock > +local msgpack_is_object = lmsgpack.is_object > +local msgpack_object = lmsgpack.object I suggest to add something like ``` if msgpack_is_object == nil then     msgpack_is_object = function() error("Msgpack object feature is not supported by current Tarantool version") end end ``` > > local MODULE_INTERNALS = '__module_vshard_router' > -- Reload requirements, in case this module is reloaded manually. > @@ -530,14 +533,17 @@ end > -- > local function router_call_impl(router, bucket_id, mode, prefer_replica, > balance, func, args, opts) > + local do_return_raw > if opts then > if type(opts) ~= 'table' or > (opts.timeout and type(opts.timeout) ~= 'number') then > error('Usage: call(bucket_id, mode, func, args, opts)') > end > opts = table.copy(opts) > - elseif not opts then > + do_return_raw = opts.return_raw > + else > opts = {} > + do_return_raw = false > end > local timeout = opts.timeout or consts.CALL_TIMEOUT_MIN > local replicaset, err > @@ -569,6 +575,26 @@ local function router_call_impl(router, bucket_id, mode, prefer_replica, > local storage_call_status, call_status, call_error = > replicaset[call](replicaset, 'vshard.storage.call', > {bucket_id, mode, func, args}, opts) > + if do_return_raw and msgpack_is_object(storage_call_status) then > + -- Storage.call returns in the first value a flag whether user's > + -- function threw an exception or not. Need to extract it. > + -- Unfortunately, it forces to repack the rest of values into a > + -- new array. But the values themselves are not decoded. > + local it = storage_call_status:iterator() > + local count = it:decode_array_header() > + storage_call_status = it:decode() > + -- When no values, nil is not packed into msgpack object. Same > + -- as in raw netbox. > + if count > 1 then > + count = count - 1 > + local res = table_new(count, 0) > + for i = 1, count do > + res[i] = it:take() > + end > + call_status = msgpack_object(res) > + end > + call_error = nil > + end > if storage_call_status then > if call_status == nil and call_error ~= nil then > return call_status, call_error > diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua > index 6820ad0..5083911 100644 > --- a/vshard/storage/init.lua > +++ b/vshard/storage/init.lua > @@ -2512,6 +2512,21 @@ local function storage_call(bucket_id, mode, name, args) > if not ok then > ret1 = lerror.make(ret1) > end > + -- Truncate nil values. Can't return them all because empty values turn into > + -- box.NULL. Even if user's function actually returned just 1 value, this > + -- would lead to '1, box.NULL, box.NULL' on the client. Not visible on the > + -- router when called normally, but won't help if the router did > + -- 'return_raw' and just forwards everything as is without truncation. > + -- Although this solution truncates really returned nils. > + if ret3 == nil then > + if ret2 == nil then > + if ret1 == nil then > + return ok > + end > + return ok, ret1 > + end > + return ok, ret1, ret2 > + end > return ok, ret1, ret2, ret3 > end > > diff --git a/vshard/util.lua b/vshard/util.lua > index 72176f7..ad469f2 100644 > --- a/vshard/util.lua > +++ b/vshard/util.lua > @@ -234,8 +234,10 @@ end > -- Dictionary of supported core features on the given instance. Try to use it > -- in all the other code rather than direct version check. > -- > +local is_ge_2_10_0_beta2_86 = version_is_at_least(2, 10, 0, 'beta', 2, 86) > local feature = { > - msgpack_object = version_is_at_least(2, 10, 0, 'beta', 2, 86), > + msgpack_object = is_ge_2_10_0_beta2_86, > + netbox_return_raw = is_ge_2_10_0_beta2_86, > } > > return {