From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id B5A816FC8F; Thu, 25 Mar 2021 00:28:09 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B5A816FC8F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1616621289; bh=LEEmwxTDtylQxrdQnUAAc61vPEohXA+KQsH3/A9l6Q0=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=h4y4QdzbOhbpm9zhYlY8Zp4KFAqNS+Ukv6PXtRtV1kH/YIHg4hAmzUwm17yPa87mf KigbVuJBbhtmEn63loRexuOaOfL3ASc7ZubKo3DAUganFWYVGnqhs1l+b4m5B9qJgo W0xM/otVZPCu5ff1wF6NpHGYjMYIcndh53Ev3wnQ= Received: from smtp48.i.mail.ru (smtp48.i.mail.ru [94.100.177.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 8C7D06BD34 for ; Thu, 25 Mar 2021 00:24:44 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 8C7D06BD34 Received: by smtp48.i.mail.ru with esmtpa (envelope-from ) id 1lPAzb-0004ib-Oj; Thu, 25 Mar 2021 00:24:44 +0300 To: tarantool-patches@dev.tarantool.org, kyukhin@tarantool.org Date: Wed, 24 Mar 2021 22:24:27 +0100 Message-Id: <1688b272833639937075ca56cadbe2d228d44512.1616620860.git.v.shpilevoy@tarantool.org> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: EEAE043A70213CC8 X-77F55803: 4F1203BC0FB41BD9064ADF4728AA0EE954173BE606B1EB548800CB16E8E806CE182A05F538085040DC912BFB2CB0D93A2BF4765ADFDA67BD3825D5E47E069EFBFA142EB7FFBD7431 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE746D93DAA4671895CEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637A900FA3C0C04B8698638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95C5DD32608FC869F5DFE25200485B3BB13B149AB61D7DF30B7A471835C12D1D9774AD6D5ED66289B5278DA827A17800CE709B92020B71E24959FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C34C82C86BFC697D19117882F4460429728AD0CFFFB425014E868A13BD56FB6657D81D268191BDAD3DC09775C1D3CA48CF103E9EB61D8D0106BA3038C0950A5D36C8A9BA7A39EFB766EC990983EF5C0329BA3038C0950A5D36D5E8D9A59859A8B65D224A4ECC5A0F1076E601842F6C81A1F004C906525384307823802FF610243DF43C7A68FF6260569E8FC8737B5C2249EC8D19AE6D49635B3BBE47FD9DD3FB59A8DF7F3B2552694A2BEBFE083D3B9BA73A03B725D353964BB11811A4A51E3B096D1867E19FE14079BA9C0B312567BB23089D37D7C0E48F6CA18204E546F3947C83C798A30B85E16BC6EABA9B74D0DA47C8A9BA7A39EFB7666BA297DBC24807EA089D37D7C0E48F6C8AA50765F7900637B8F435DEDE9E76EBEFF80C71ABB335746BA297DBC24807EA27F269C8F02392CD6143B1329D1CCA5627F269C8F02392CD5571747095F342E88FB05168BE4CE3AF X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2368A440D3B0F6089093C9A16E5BC824AC8B6CDF511875BC4E8F7B195E1C97831148145A0506FFD482D680659EFB54793 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975C5DD32608FC869F5DFE25200485B3BB13B149AB61D7DF30B79C2B6934AE262D3EE7EAB7254005DCED7532B743992DF240BDC6A1CF3F042BAD6DF99611D93F60EF0417BEADF48D1460699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34197948450CB5442A61C3DF386F683B11C26EA67468DF443442EA96877429D28590FD433E3BF7F5BE1D7E09C32AA3244CB36A96AD072DD7734B56D01C0D9DDCE91E098CBE561D6343729B2BEF169E0186 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojjqzNotmU+geqrLf7Qx69aQ== X-Mailru-Sender: 504CC1E875BF3E7D9BC0E5172ADA3110A906815429F9D960E154D550F672C520EF5024447D2A43FD07784C02288277CA03E0582D3806FB6A5317862B1921BA260ED6CFD6382C13A6112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH 15/15] buffer: remove Lua registers X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Lua buffer module used to have a couple of preallocated objects of type 'union c_register'. It was a bunch of C scalar and array types intended for use instead of ffi.new() where it was needed to allocate a temporary object like 'int[1]' just to be able to pass 'int *' into a C function via FFI. It was a bit faster than ffi.new() even for small sizes. For instance (when JIT works), getting a register to use it as 'int[1]' cost around 0.2-0.3 ns while ffi.new('int[1]') costs around 0.4 ns. Also the code looked cleaner. But Lua registers were global and therefore had the same issue as IBUF_SHARED and static_alloc() in Lua - no ownership, and sudden reuse when GC starts right the register is still in use in some Lua code. __gc handlers could wipe the register values making the original code behave unpredictably. IBUF_SHARED was fixed by proper ownership implementation, but it is not necessary with Lua registers. It could be done with the buffer.ffi_stash_new() feature, but its performance is about 0.8 ns which is worse than plain ffi.new() for simple scalar types. This patch eliminates Lua registers, and uses ffi.new() instead everywhere. Closes #5632 (cherry picked from commit 911ca60e202986ea283341bb31bfd7a7a5688559) --- changelogs/unreleased/fix-ibuf-static.md | 7 ++++ src/lua/msgpackffi.lua | 39 +++++++++--------- test/app-tap/gh-5632-gc-buf-reuse.test.lua | 46 +++++++++++++++++++++- 3 files changed, 72 insertions(+), 20 deletions(-) create mode 100644 changelogs/unreleased/fix-ibuf-static.md diff --git a/changelogs/unreleased/fix-ibuf-static.md b/changelogs/unreleased/fix-ibuf-static.md new file mode 100644 index 000000000..34450b85d --- /dev/null +++ b/changelogs/unreleased/fix-ibuf-static.md @@ -0,0 +1,7 @@ +## bugfix/core + +* Extensive usage of `uri` and `uuid` modules with debug log level could lead to + a crash or corrupted result of the functions from these modules. Also their + usage from the callbacks passed to `ffi.gc()` could lead to the same but much + easier. The same could happen with some functions from the modules `fio`, + `box.tuple`, `iconv` (gh-5632). diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua index ad7998ed1..b07f0e7f0 100644 --- a/src/lua/msgpackffi.lua +++ b/src/lua/msgpackffi.lua @@ -31,11 +31,6 @@ union tmpint { local strict_alignment = (jit.arch == 'arm') -local tmpint -if strict_alignment then - tmpint = ffi.new('union tmpint[1]') -end - local function bswap_u16(num) return bit.rshift(bit.bswap(tonumber(num)), 16) end @@ -71,7 +66,7 @@ end local encode_u16 if strict_alignment then encode_u16 = function(buf, code, num) - tmpint[0].u16 = bswap_u16(num) + local tmpint = ffi.new('uint16_t[1]', bswap_u16(num)) local p = buf:alloc(3) p[0] = code ffi.copy(p + 1, tmpint, 2) @@ -87,8 +82,9 @@ end local encode_u32 if strict_alignment then encode_u32 = function(buf, code, num) - tmpint[0].u32 = - ffi.cast('uint32_t', bit.bswap(tonumber(num))) + local tmpint = + ffi.new('uint32_t[1]', ffi.cast('uint32_t', + bit.bswap(tonumber(num)))) local p = buf:alloc(5) p[0] = code ffi.copy(p + 1, tmpint, 4) @@ -105,7 +101,8 @@ end local encode_u64 if strict_alignment then encode_u64 = function(buf, code, num) - tmpint[0].u64 = bit.bswap(ffi.cast('uint64_t', num)) + local tmpint = + ffi.new('uint64_t[1]', bit.bswap(ffi.cast('uint64_t', num))) local p = buf:alloc(9) p[0] = code ffi.copy(p + 1, tmpint, 8) @@ -328,9 +325,10 @@ end local decode_u16 if strict_alignment then decode_u16 = function(data) + local tmpint = ffi.new('uint16_t[1]') ffi.copy(tmpint, data[0], 2) data[0] = data[0] + 2 - return tonumber(bswap_u16(tmpint[0].u16)) + return tonumber(bswap_u16(tmpint[0])) end else decode_u16 = function(data) @@ -343,10 +341,11 @@ end local decode_u32 if strict_alignment then decode_u32 = function(data) + local tmpint = ffi.new('uint32_t[1]') ffi.copy(tmpint, data[0], 4) data[0] = data[0] + 4 return tonumber( - ffi.cast('uint32_t', bit.bswap(tonumber(tmpint[0].u32)))) + ffi.cast('uint32_t', bit.bswap(tonumber(tmpint[0])))) end else decode_u32 = function(data) @@ -360,9 +359,10 @@ end local decode_u64 if strict_alignment then decode_u64 = function(data) + local tmpint = ffi.new('uint64_t[1]') ffi.copy(tmpint, data[0], 8); data[0] = data[0] + 8 - local num = bit.bswap(tmpint[0].u64) + local num = bit.bswap(tmpint[0]) if num <= DBL_INT_MAX then return tonumber(num) -- return as 'number' end @@ -389,8 +389,9 @@ end local decode_i16 if strict_alignment then decode_i16 = function(data) + local tmpint = ffi.new('uint16_t[1]') ffi.copy(tmpint, data[0], 2) - local num = bswap_u16(tmpint[0].u16) + local num = bswap_u16(tmpint[0]) data[0] = data[0] + 2 -- note: this double cast is actually necessary return tonumber(ffi.cast('int16_t', ffi.cast('uint16_t', num))) @@ -407,8 +408,9 @@ end local decode_i32 if strict_alignment then decode_i32 = function(data) + local tmpint = ffi.new('uint32_t[1]') ffi.copy(tmpint, data[0], 4) - local num = bit.bswap(tonumber(tmpint[0].u32)) + local num = bit.bswap(tonumber(tmpint[0])) data[0] = data[0] + 4 return num end @@ -423,9 +425,10 @@ end local decode_i64 if strict_alignment then decode_i64 = function(data) + local tmpint = ffi.new('int64_t[1]') ffi.copy(tmpint, data[0], 8) data[0] = data[0] + 8 - local num = bit.bswap(ffi.cast('int64_t', tmpint[0].u64)) + local num = bit.bswap(tmpint[0]) if num >= -DBL_INT_MAX and num <= DBL_INT_MAX then return tonumber(num) -- return as 'number' end @@ -550,13 +553,11 @@ decode_r = function(data) end --- --- A temporary const char ** buffer. -- All decode_XXX functions accept const char **data as its first argument, -- like libmsgpuck does. After decoding data[0] position is changed to the next -- element. It is significally faster on LuaJIT to use double pointer than -- return result, newpos. -- -local bufp = ffi.new('const unsigned char *[1]'); local function check_offset(offset, len) if offset == nil then @@ -576,13 +577,13 @@ local function decode_unchecked(str, offset) if type(str) == "string" then offset = check_offset(offset, #str) local buf = ffi.cast(char_ptr_t, str) - bufp[0] = buf + offset - 1 + local bufp = ffi.new('const unsigned char *[1]', buf + offset - 1) local r = decode_r(bufp) return r, bufp[0] - buf + 1 elseif ffi.istype(char_ptr_t, str) then -- Note: ffi.istype() ignores the const qualifier, so both -- (char *) and (const char *) buffers are valid. - bufp[0] = str + local bufp = ffi.new('const unsigned char *[1]', str) local r = decode_r(bufp) return r, ffi.cast(ffi.typeof(str), bufp[0]) else diff --git a/test/app-tap/gh-5632-gc-buf-reuse.test.lua b/test/app-tap/gh-5632-gc-buf-reuse.test.lua index 81dafd36e..6efddb714 100755 --- a/test/app-tap/gh-5632-gc-buf-reuse.test.lua +++ b/test/app-tap/gh-5632-gc-buf-reuse.test.lua @@ -11,6 +11,7 @@ local tap = require('tap') local ffi = require('ffi') local uuid = require('uuid') local uri = require('uri') +local msgpackffi = require('msgpackffi') local function test_uuid(test) test:plan(1) @@ -99,9 +100,52 @@ local function test_uri(test) test:ok(is_success, 'uri in gc') end +local function test_msgpackffi(test) + test:plan(1) + + local mp_encode = msgpackffi.encode + local mp_decode = msgpackffi.decode + local gc_count = 100 + local iter_count = 1000 + local is_success = true + local data = {0, 1, 1000, 100000000, 'str', true, 1.1} + + local function do_encode() + if not is_success then + return + end + local t = mp_encode(data) + t = mp_decode(t) + if #t ~= #data then + is_success = false + return + end + for i = 1, #t do + if t[i] ~= data[i] then + is_success = false + return + end + end + end + + local function create_gc() + for _ = 1, gc_count do + ffi.gc(ffi.new('char[1]'), do_encode) + end + end + + for _ = 1, iter_count do + create_gc() + do_encode() + end + + test:ok(is_success, 'msgpackffi in gc') +end + local test = tap.test('gh-5632-gc-buf-reuse') -test:plan(2) +test:plan(3) test:test('uuid in __gc', test_uuid) test:test('uri in __gc', test_uri) +test:test('msgpackffi in __gc', test_msgpackffi) os.exit(test:check() and 0 or 1) -- 2.24.3 (Apple Git-128)