From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp48.i.mail.ru (smtp48.i.mail.ru [94.100.177.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 3E13D469719 for ; Wed, 23 Sep 2020 04:14:16 +0300 (MSK) From: Alexander Turenko Date: Wed, 23 Sep 2020 04:14:05 +0300 Message-Id: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH 00/14] RFC: module api: extend for external key_def Lua module List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Vladislav Shpilevoy Cc: tarantool-patches@dev.tarantool.org, Alexander Turenko A lot of code to review... but nobody will do it better than you, Vlad! Please, look when your time will permit. This patchset extends the module API with a set of functions (mainly key_def and tuple related) in order to implement an external key_def Lua module on top of them. Timur will do the same for functions he need for the external merger module (outside of this patchset, but on top of it). I don't wrote necessary tests ATM, so the patchset is not ready to push and marked as RFC (request for comments). However I did my best for all other aspects and decided to send the patchset for review to collect a feedback earlier. I'll work on tests while waiting for review. (I promise I'll not ask to push it until all tests will be written.) There is backport of this patchset to 1.10 (see the links below). It adds two extra commits and reworks some of those fourteen ones. Please, consider it as part of this work and take glance too (I can send it too if there will be demand). Regarding those two extra commits: - d7c8a7888 collation: allow to find a collation by a name The alternative would be a function like [1], but direct call to the database from key_def_api.c looks strange for me. I think it is better to backport the hashmap by collation names: the patch is simple and proven by the time. A long time ago we agreed with Vladimir Davydov to backport this patch to 1.10 if we'll backport merger to 1.10. It didn't occur, but the situation is similar. - 078239e37 refactoring: adjust contract of luaT_tuple_new() Here I'm in doubts. The alternative would be duplicate the code: rename old luaT_tuple_new() to somewhat like luaT_tuple_new_110() and add the new luaT_tuple_new() to expose it into the module API. I don't know, to be honest. The code is proven by time and the risk is minimal. I would prefer to avoid code duplication, but I'm not strict on this position here: I'll implement the alternative if there will be reasoned complains against the current variant. What also worries me: I found that actual behaviour of luaT_tuple_new() is different from what I expected in the past: it may raise an error when serialization fails. I didn't expect this, so I'll look how much it may affect key_def and merger modules (built-in and external ones). For now I just described the actual contract in the API comments for luaT_tuple_new() and luaT_tuple_encode(). Issues: https://github.com/tarantool/tarantool/issues/5313 ('module api: module API requires C11') https://github.com/tarantool/tarantool/issues/5273 ('module api: expose everything that is needed for external key_def module') Branches: https://github.com/tarantool/tarantool/tree/Totktonada/gh-5273-expand-module-api (top 14 commits) https://github.com/tarantool/tarantool/tree/Totktonada/gh-5273-expand-module-api-1.10 (top 16 commits) The module based on this API: https://github.com/Totktonada/key_def The module repository will be moved to the tarantool organization when it'll be ready (surely not before this patchset). Please, consider the module as PoC (everything work, though; even on Mac OS). References: [1]: https://github.com/tarantool/tarantool/blob/b7c787152d8984523d0b8b4d5b08f79173c57115/src/lua/merger.c#L1686-L1708 Alexander Turenko (14): module api: get rid of typedef redefinitions WIP: module api: expose box region WIP: module api/lua: add luaL_iscdata() function WIP: module api/lua: expose luaT_tuple_new() WIP: module api/lua: add luaT_tuple_encode() WIP: refactoring: add API_EXPORT to lua/tuple functions WIP: refactoring: add API_EXPORT to key_def functions WIP: refactoring: extract key_def module API functions WIP: module api: add box_key_def_new_ex() WIP: module api: add box_key_def_dump_parts() WIP: module api: expose box_tuple_validate_key_parts() WIP: module api: expose box_key_def_merge() WIP: module api: expose box_tuple_extract_key_ex() WIP: module api: add box_key_def_validate_key() src/CMakeLists.txt | 7 +- src/box/CMakeLists.txt | 1 + src/box/field_map.h | 4 + src/box/index.h | 5 +- src/box/key_def.c | 62 ++---- src/box/key_def.h | 85 +++------ src/box/key_def_api.c | 323 ++++++++++++++++++++++++++++++++ src/box/key_def_api.h | 295 +++++++++++++++++++++++++++++ src/box/lua/tuple.c | 21 ++- src/box/lua/tuple.h | 56 +++++- src/exports.h | 14 ++ src/lib/core/fiber.c | 24 +++ src/lib/core/fiber.h | 101 ++++++++++ src/lua/utils.c | 6 + src/lua/utils.h | 20 ++ test/app-tap/CMakeLists.txt | 6 + test/app-tap/module_api.c | 2 + test/unit/vy_iterators_helper.c | 1 + test/unit/vy_mem.c | 1 + test/unit/vy_point_lookup.c | 1 + test/unit/vy_write_iterator.c | 1 + 21 files changed, 915 insertions(+), 121 deletions(-) create mode 100644 src/box/key_def_api.c create mode 100644 src/box/key_def_api.h -- 2.25.0