Re: [PATCH 01/18] vinyl: rework internal quota API

[Vladimir Davydov <vdavydov.dev@gmail.com>]

We decided to leave all method names as is and only introduce
vy_quota_adjust helper in this patch. Here's the final version
that was pushed to 1.10:

From 89a14237337e220b3373bda978ee5453ba70abee Mon Sep 17 00:00:00 2001
From: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon, 27 Aug 2018 20:41:49 +0300
Subject: [PATCH] vinyl: add vy_quota_adjust helper

Let's introduce this helper to avoid code duplication and keep comments
regarding quota consumption protocol in one place.

diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 6d5ea380..27a93c81 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -2384,21 +2384,7 @@ vinyl_engine_prepare(struct engine *engine, struct txn *txn)
 	size_t mem_used_after = lsregion_used(&env->mem_env.allocator);
 	assert(mem_used_after >= mem_used_before);
 	size_t write_size = mem_used_after - mem_used_before;
-	/*
-	 * Insertion of a statement into an in-memory tree can trigger
-	 * an allocation of a new tree block. This should not normally
-	 * result in a noticeable excess of the memory limit, because
-	 * most memory is occupied by statements anyway, but we need to
-	 * adjust the quota accordingly in this case.
-	 *
-	 * The actual allocation size can also be less than reservation
-	 * if a statement is allocated from an lsregion slab allocated
-	 * by a previous transaction. Take this into account, too.
-	 */
-	if (write_size >= tx->write_size)
-		vy_quota_force_use(&env->quota, write_size - tx->write_size);
-	else
-		vy_quota_release(&env->quota, tx->write_size - write_size);
+	vy_quota_adjust(&env->quota, tx->write_size, write_size);
 
 	if (rc != 0)
 		return -1;
@@ -3292,11 +3278,7 @@ vinyl_space_apply_initial_join_row(struct space *space, struct request *request)
 	size_t mem_used_after = lsregion_used(&env->mem_env.allocator);
 	assert(mem_used_after >= mem_used_before);
 	size_t used = mem_used_after - mem_used_before;
-	if (used >= reserved)
-		vy_quota_force_use(&env->quota, used - reserved);
-	else
-		vy_quota_release(&env->quota, reserved - used);
-
+	vy_quota_adjust(&env->quota, reserved, used);
 	return rc;
 }
 
diff --git a/src/box/vy_quota.h b/src/box/vy_quota.h
index d741c34a..726d3d44 100644
--- a/src/box/vy_quota.h
+++ b/src/box/vy_quota.h
@@ -153,6 +153,35 @@ vy_quota_release(struct vy_quota *q, size_t size)
  * Try to consume @size bytes of memory, throttle the caller
  * if the limit is exceeded. @timeout specifies the maximal
  * time to wait. Return 0 on success, -1 on timeout.
+ *
+ * Usage pattern:
+ *
+ *   size_t reserved = <estimate>;
+ *   if (vy_quota_use(q, reserved, timeout) != 0)
+ *           return -1;
+ *   <allocate memory>
+ *   size_t used = <actually allocated>;
+ *   vy_quota_adjust(q, reserved, used);
+ *
+ * We use two-step quota allocation strategy (reserve-consume),
+ * because we may not yield after we start inserting statements
+ * into a space so we estimate the allocation size and wait for
+ * quota before committing statements. At the same time, we
+ * cannot precisely estimate the size of memory we are going to
+ * consume so we adjust the quota after the allocation.
+ *
+ * The size of memory allocated while committing a transaction
+ * may be greater than an estimate, because insertion of a
+ * statement into an in-memory index can trigger allocation
+ * of a new index extent. This should not normally result in a
+ * noticeable breach in the memory limit, because most memory
+ * is occupied by statements, but we need to adjust the quota
+ * accordingly after the allocation in this case.
+ *
+ * The actual memory allocation size may also be less than an
+ * estimate if the space has multiple indexes, because statements
+ * are stored in the common memory level, which isn't taken into
+ * account while estimating the size of a memory allocation.
  */
 static inline int
 vy_quota_use(struct vy_quota *q, size_t size, double timeout)
@@ -178,6 +207,27 @@ vy_quota_use(struct vy_quota *q, size_t size, double timeout)
 }
 
 /**
+ * Adjust quota after allocating memory.
+ *
+ * @reserved: size of quota reserved by vy_quota_use().
+ * @used: size of memory actually allocated.
+ *
+ * See also vy_quota_use().
+ */
+static inline void
+vy_quota_adjust(struct vy_quota *q, size_t reserved, size_t used)
+{
+	if (reserved > used) {
+		size_t excess = reserved - used;
+		assert(q->used >= excess);
+		q->used -= excess;
+		fiber_cond_broadcast(&q->cond);
+	}
+	if (reserved < used)
+		vy_quota_force_use(q, used - reserved);
+}
+
+/**
  * Block the caller until the quota is not exceeded.
  */
 static inline void