[Tarantool-patches] [PATCH vshard 09/11] ref: introduce vshard.storage.ref module

[Oleg Babin]

Hi! Thanks for fixes. LGTM.

On 06.03.2021 01:06, Vladislav Shpilevoy wrote:
> Hi! Thanks for the review!
>
>>> +local function ref_session_new(sid)
>>> +    -- Session object does store its internal hot attributes in a table. Because
>>> +    -- it would mean access to any session attribute would cost at least one
>>> +    -- table indexing operation. Instead, all internal fields are stored as
>>> +    -- upvalues referenced by the methods defined as closures.
>>> +    --
>>> +    -- This means session creation may not very suitable for jitting, but it is
>>> +    -- very rare and attempts to optimize the most common case.
>>> +    --
>>> +    -- Still the public functions take 'self' object to make it look normally.
>>> +    -- They even use it a bit.
>>> +
>>> +    -- Ref map to get ref object by its ID.
>>> +    local ref_map = {}
>>> +    -- Ref heap sorted by their deadlines.
>>> +    local ref_heap = lheap.new(heap_min_deadline_cmp)
>>> +    -- Total number of refs of the session. Is used to drop the session without
>>> +    -- fullscan of the ref map. Heap size can't be used because not all refs are
>>> +    -- stored here. See more on that below.
>>> +    local count = 0
>> Maybe it's better to rename it to "global_count". Sometimes it's quite confusing to see `M.count +=` near `count += `.
>>
>> Also you have "global_map" and "global_heap" so no reasons to call it just "count".
> I have global_map and global_heap variables because I also have normal map and
> heap, local to the session. To distinguish between them I added 'global_'
> prefix to the global ones.
>
> The count here is not global. It is local to the session. But I see the point.
> I renamed it to `ref_count` to be consistent with `ref_map` and `ref_heap`.
>
> ====================
> diff --git a/vshard/storage/ref.lua b/vshard/storage/ref.lua
> index 7589cb9..27f7804 100644
> --- a/vshard/storage/ref.lua
> +++ b/vshard/storage/ref.lua
> @@ -84,7 +84,7 @@ local function ref_session_new(sid)
>       -- Total number of refs of the session. Is used to drop the session without
>       -- fullscan of the ref map. Heap size can't be used because not all refs are
>       -- stored here. See more on that below.
> -    local count = 0
> +    local ref_count = 0
>       -- Cache global session storages as upvalues to save on M indexing.
>       local global_heap = M.session_heap
>       local global_map = M.session_map
> @@ -94,9 +94,9 @@ local function ref_session_new(sid)
>           assert(new_count >= 0)
>           M.count = new_count
>   
> -        new_count = count - del_count
> +        new_count = ref_count - del_count
>           assert(new_count >= 0)
> -        count = new_count
> +        ref_count = new_count
>       end
>   
>       local function ref_session_update_deadline(self)
> @@ -224,7 +224,7 @@ local function ref_session_new(sid)
>               self.deadline = deadline
>               global_heap:update(self)
>           end
> -        count = count + 1
> +        ref_count = ref_count + 1
>           M.count = M.count + 1
>           return true
>       end
> @@ -260,7 +260,7 @@ local function ref_session_new(sid)
>       local function ref_session_kill(self)
>           global_map[sid] = nil
>           global_heap:remove(self)
> -        ref_session_discount(self, count)
> +        ref_session_discount(self, ref_count)
>       end
>   
>       -- Don't use __index. It is useless since all sessions use closures as
> ====================
>
>>> +
>>> +    --
>>> +    -- GC expired refs until they end or the limit on the number of iterations
>>> +    -- is exhausted. The limit is supposed to prevent too long GC which would
>>> +    -- occupy TX thread unfairly.
>>> +    --
>>> +    -- Returns false if nothing to GC, or number of iterations left from the
>>> +    -- limit. The caller is supposed to yield when 0 is returned, and retry GC
>>> +    -- until it returns false.
>>> +    -- The function itself does not yield, because it is used from a more
>>> +    -- generic function GCing all sessions. It would not ever yield if all
>>> +    -- sessions would have less than limit refs, even if total ref count would
>>> +    -- be much bigger.
>>> +    --
>>> +    -- Besides, the session might be killed during general GC. There must not be
>>> +    -- any yields in session methods so as not to introduce a support of dead
>>> +    -- sessions.
>>> +    --
>>> +    local function ref_session_gc(self, limit, now)
>>> +        if self.deadline >= now then
>>> +            return false
>>> +        end
>> Here you mix "booleans" and "numbers" as return values. Maybe it's better to return "nil" here?
> No problem:
>
> ====================
> diff --git a/vshard/storage/ref.lua b/vshard/storage/ref.lua
> index 27f7804..d31e3ed 100644
> --- a/vshard/storage/ref.lua
> +++ b/vshard/storage/ref.lua
> @@ -164,9 +164,9 @@ local function ref_session_new(sid)
>       -- is exhausted. The limit is supposed to prevent too long GC which would
>       -- occupy TX thread unfairly.
>       --
> -    -- Returns false if nothing to GC, or number of iterations left from the
> +    -- Returns nil if nothing to GC, or number of iterations left from the
>       -- limit. The caller is supposed to yield when 0 is returned, and retry GC
> -    -- until it returns false.
> +    -- until it returns nil.
>       -- The function itself does not yield, because it is used from a more
>       -- generic function GCing all sessions. It would not ever yield if all
>       -- sessions would have less than limit refs, even if total ref count would
> @@ -178,7 +178,7 @@ local function ref_session_new(sid)
>       --
>       local function ref_session_gc(self, limit, now)
>           if self.deadline >= now then
> -            return false
> +            return nil
>           end
>           local top = ref_heap:top()
>           local del = 1
> ====================
>
>>> +
>>> +    -- Don't use __index. It is useless since all sessions use closures as
>>> +    -- methods. Also it is probably slower because on each method call would
>>> +    -- need to get the metatable, get __index, find the method here. While now
>>> +    -- it is only an index operation on the session object.
>> Side note: for heap you still use "__index" even heap uses closures as methods.
> Indeed, I should have thought of this. I updated the part1 branch, and rebased the
> part2 branch. See the part1 email thread for the diff.