[Tarantool-patches] [PATCH v4] memtx: fix out of memory handling for rtree

[Tarantool-patches] [PATCH v4] memtx: fix out of memory handling for rtree

[Olga Arkhangelskaia]

When tarantool tries to recover rtree from a snapshot and memtx_memory value
is lower than it has been when the snapshot was created, server suffers from
segmentation fault. This happens because there is no out of memory error
handling in rtree lib. In another words, we do not check the result of
malloc operation.
The execution flow in case of recovery uses different way and the secondary
keys are build in batches. There is no separate implementations for rtree of
build_next function. And the way with no checks and reservation is used
(insex_replace). The patch adds memtx_rtree_index_reserve implementation to
make sure that any memory allocation in rtree will fail. Although this gives
us no additional optimization as in case of memtx_tree, the memory
reservation prevents tarantool from segmentation fault.
If there is not enough memory to be reserved - server will fail
gently with the "Failed to allocate" error message.

Closes #4619
---
Branch: https://github.com/tarantool/tarantool/tree/OKriw/gh-RTREE-doesnt-handle-OOM-properly
Issue:https://github.com/tarantool/tarantool/issues/4619

v1:https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012391.html
v2:https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/012958.html
v3:https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013182.html

Changes in v2:
- changed way of error handling, now we reserve pages before go to rtree
  lib
- changed test
- changed commit msg

Changes in v3:
- added memtx_rtree_build_next function
- memory reservation is moved to memtx_rtree_build_next

Changes in v4:
- added index reservation in build_next
- added memtx_rtree_reserve implementation

 src/box/index.cc       |   3 +-
 src/box/memtx_engine.h |  12 +++++
 src/box/memtx_rtree.c  |  15 +++++-
 src/box/memtx_space.c  |  12 -----
 test/box/cfg.result    | 101 +++++++++++++++++++++++++++++++++++++++++
 test/box/cfg.test.lua  |  28 ++++++++++++
 6 files changed, 157 insertions(+), 14 deletions(-)

diff --git a/src/box/index.cc b/src/box/index.cc
index 4e4867118..d0aa62d67 100644
--- a/src/box/index.cc
+++ b/src/box/index.cc
@@ -733,7 +733,8 @@ int
 generic_index_build_next(struct index *index, struct tuple *tuple)
 {
 	struct tuple *unused;
-	return index_replace(index, NULL, tuple, DUP_INSERT, &unused);
+	int res = index_reserve(index, 0);
+	return res == 0 ? index_replace(index, NULL, tuple, DUP_INSERT, &unused) : res;
 }
 
 void
diff --git a/src/box/memtx_engine.h b/src/box/memtx_engine.h
index f562c66df..f6335cc66 100644
--- a/src/box/memtx_engine.h
+++ b/src/box/memtx_engine.h
@@ -87,6 +87,18 @@ enum memtx_recovery_state {
 /** Memtx extents pool, available to statistics. */
 extern struct mempool memtx_index_extent_pool;
 
+enum memtx_reserve_extents_num{
+	/**
+	 * This number is calculated based on the
+	 * max (realistic) number of insertions
+	 * a deletion from a B-tree or an R-tree
+	 * can lead to, and, as a result, the max
+	 * number of new block allocations.
+	 */
+	RESERVE_EXTENTS_BEFORE_DELETE = 8,
+	RESERVE_EXTENTS_BEFORE_REPLACE = 16
+};
+
 /**
  * The size of the biggest memtx iterator. Used with
  * mempool_create. This is the size of the block that will be
diff --git a/src/box/memtx_rtree.c b/src/box/memtx_rtree.c
index 8badad797..805fdf54c 100644
--- a/src/box/memtx_rtree.c
+++ b/src/box/memtx_rtree.c
@@ -242,6 +242,19 @@ memtx_rtree_index_replace(struct index *base, struct tuple *old_tuple,
 	return 0;
 }
 
+static int memtx_rtree_index_reserve(struct index *base, uint32_t size_hint)
+{
+	/**
+	 * In case of rtree we use reserve to make sure that memory allocation
+	 * will not fail durig any operationin rtree, because there is no
+	 * error handling in rtree lib. It is just a check not index reservation.
+	 */
+	(void)size_hint;
+	struct memtx_engine *memtx = (struct memtx_engine *)base->engine;
+	return memtx_index_extent_reserve(memtx,
+                                    RESERVE_EXTENTS_BEFORE_REPLACE);
+}
+
 static struct iterator *
 memtx_rtree_index_create_iterator(struct index *base,  enum iterator_type type,
 				  const char *key, uint32_t part_count)
@@ -333,7 +346,7 @@ static const struct index_vtab memtx_rtree_index_vtab = {
 	/* .compact = */ generic_index_compact,
 	/* .reset_stat = */ generic_index_reset_stat,
 	/* .begin_build = */ generic_index_begin_build,
-	/* .reserve = */ generic_index_reserve,
+	/* .reserve = */ memtx_rtree_index_reserve,
 	/* .build_next = */ generic_index_build_next,
 	/* .end_build = */ generic_index_end_build,
 };
diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
index 6ef84e045..da20b9196 100644
--- a/src/box/memtx_space.c
+++ b/src/box/memtx_space.c
@@ -103,18 +103,6 @@ memtx_space_replace_no_keys(struct space *space, struct tuple *old_tuple,
 	return -1;
 }
 
-enum {
-	/**
-	 * This number is calculated based on the
-	 * max (realistic) number of insertions
-	 * a deletion from a B-tree or an R-tree
-	 * can lead to, and, as a result, the max
-	 * number of new block allocations.
-	 */
-	RESERVE_EXTENTS_BEFORE_DELETE = 8,
-	RESERVE_EXTENTS_BEFORE_REPLACE = 16
-};
-
 /**
  * A short-cut version of replace() used during bulk load
  * from snapshot.
diff --git a/test/box/cfg.result b/test/box/cfg.result
index 5370bb870..9a42c04b6 100644
--- a/test/box/cfg.result
+++ b/test/box/cfg.result
@@ -580,3 +580,104 @@ test_run:cmd("cleanup server cfg_tester6")
  | ---
  | - true
  | ...
+
+--
+-- gh-4619-RTREE-doesn't-handle-OOM-properly
+--
+test_run:cmd('create server rtree with script = "box/lua/cfg_test1.lua"')
+ | ---
+ | - true
+ | ...
+test_run:cmd("start server rtree")
+ | ---
+ | - true
+ | ...
+test_run:cmd('switch rtree')
+ | ---
+ | - true
+ | ...
+box.cfg{memtx_memory = 3221225472}
+ | ---
+ | ...
+math = require("math")
+ | ---
+ | ...
+rtreespace = box.schema.create_space('rtree', {if_not_exists = true})
+ | ---
+ | ...
+rtreespace:create_index('pk', {if_not_exists = true})
+ | ---
+ | - unique: true
+ |   parts:
+ |   - type: unsigned
+ |     is_nullable: false
+ |     fieldno: 1
+ |   id: 0
+ |   space_id: 512
+ |   type: TREE
+ |   name: pk
+ | ...
+rtreespace:create_index('target', {type='rtree', dimension = 3, parts={2, 'array'},unique = false, if_not_exists = true,})
+ | ---
+ | - parts:
+ |   - type: array
+ |     is_nullable: false
+ |     fieldno: 2
+ |   dimension: 3
+ |   id: 1
+ |   type: RTREE
+ |   space_id: 512
+ |   name: target
+ | ...
+count = 2e6
+ | ---
+ | ...
+for i = 1, count do box.space.rtree:insert{i, {(i + 1) -\
+    math.floor((i + 1)/7000) * 7000, (i + 2) - math.floor((i + 2)/7000) * 7000,\
+    (i + 3) - math.floor((i + 3)/7000) * 7000}} end
+ | ---
+ | ...
+rtreespace:count()
+ | ---
+ | - 2000000
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+test_run:cmd('switch default')
+ | ---
+ | - true
+ | ...
+test_run:cmd("stop server rtree")
+ | ---
+ | - true
+ | ...
+test_run:cmd("start server rtree with crash_expected=True")
+ | ---
+ | - false
+ | ...
+fio = require('fio')
+ | ---
+ | ...
+fh = fio.open(fio.pathjoin(fio.cwd(), 'cfg_test1.log'), {'O_RDONLY'})
+ | ---
+ | ...
+size = fh:seek(0, 'SEEK_END')
+ | ---
+ | ...
+fh:seek(-256, 'SEEK_END') ~= nil
+ | ---
+ | - true
+ | ...
+line = fh:read(256)
+ | ---
+ | ...
+fh:close()
+ | ---
+ | - true
+ | ...
+string.match(line, 'Failed to allocate') ~= nil
+ | ---
+ | - true
+ | ...
diff --git a/test/box/cfg.test.lua b/test/box/cfg.test.lua
index 56ccb6767..6f8450496 100644
--- a/test/box/cfg.test.lua
+++ b/test/box/cfg.test.lua
@@ -141,3 +141,31 @@ test_run:cmd("start server cfg_tester6")
 test_run:grep_log('cfg_tester6', 'set \'vinyl_memory\' configuration option to 1073741824', 1000)
 test_run:cmd("stop server cfg_tester6")
 test_run:cmd("cleanup server cfg_tester6")
+
+--
+-- gh-4619-RTREE-doesn't-handle-OOM-properly
+--
+test_run:cmd('create server rtree with script = "box/lua/cfg_test1.lua"')
+test_run:cmd("start server rtree")
+test_run:cmd('switch rtree')
+box.cfg{memtx_memory = 3221225472}
+math = require("math")
+rtreespace = box.schema.create_space('rtree', {if_not_exists = true})
+rtreespace:create_index('pk', {if_not_exists = true})
+rtreespace:create_index('target', {type='rtree', dimension = 3, parts={2, 'array'},unique = false, if_not_exists = true,})
+count = 2e6
+for i = 1, count do box.space.rtree:insert{i, {(i + 1) -\
+    math.floor((i + 1)/7000) * 7000, (i + 2) - math.floor((i + 2)/7000) * 7000,\
+    (i + 3) - math.floor((i + 3)/7000) * 7000}} end
+rtreespace:count()
+box.snapshot()
+test_run:cmd('switch default')
+test_run:cmd("stop server rtree")
+test_run:cmd("start server rtree with crash_expected=True")
+fio = require('fio')
+fh = fio.open(fio.pathjoin(fio.cwd(), 'cfg_test1.log'), {'O_RDONLY'})
+size = fh:seek(0, 'SEEK_END')
+fh:seek(-256, 'SEEK_END') ~= nil
+line = fh:read(256)
+fh:close()
+string.match(line, 'Failed to allocate') ~= nil
-- 
2.17.2 (Apple Git-113)

Re: [Tarantool-patches] [PATCH v4] memtx: fix out of memory handling for rtree

[Nikita Pettik]

On 20 Dec 16:44, Olga Arkhangelskaia wrote:

Hello. Below are a few nits concerning code itself.

> diff --git a/src/box/index.cc b/src/box/index.cc
> index 4e4867118..d0aa62d67 100644
> --- a/src/box/index.cc
> +++ b/src/box/index.cc
> @@ -733,7 +733,8 @@ int
>  generic_index_build_next(struct index *index, struct tuple *tuple)
>  {
>  	struct tuple *unused;
> -	return index_replace(index, NULL, tuple, DUP_INSERT, &unused);
> +	int res = index_reserve(index, 0);
> +	return res == 0 ? index_replace(index, NULL, tuple, DUP_INSERT, &unused) : res;
>  }

Why not:

if (index_reserve(index, 0) != 0)
	return -1;
return index_replace(...);

>  
>  void
> diff --git a/src/box/memtx_engine.h b/src/box/memtx_engine.h
> index f562c66df..f6335cc66 100644
> --- a/src/box/memtx_engine.h
> +++ b/src/box/memtx_engine.h
> @@ -87,6 +87,18 @@ enum memtx_recovery_state {
>  /** Memtx extents pool, available to statistics. */
>  extern struct mempool memtx_index_extent_pool;
>  
> +enum memtx_reserve_extents_num{

                                 ^ missed space

> +	/**
> +	 * This number is calculated based on the
> +	 * max (realistic) number of insertions
> +	 * a deletion from a B-tree or an R-tree
> +	 * can lead to, and, as a result, the max
> +	 * number of new block allocations.
> +	 */
> +	RESERVE_EXTENTS_BEFORE_DELETE = 8,
> +	RESERVE_EXTENTS_BEFORE_REPLACE = 16
> +};
> +
>  /**
>   * The size of the biggest memtx iterator. Used with
>   * mempool_create. This is the size of the block that will be
> diff --git a/src/box/memtx_rtree.c b/src/box/memtx_rtree.c
> index 8badad797..805fdf54c 100644
> --- a/src/box/memtx_rtree.c
> +++ b/src/box/memtx_rtree.c
> @@ -242,6 +242,19 @@ memtx_rtree_index_replace(struct index *base, struct tuple *old_tuple,
>  	return 0;
>  }
>  
> +static int memtx_rtree_index_reserve(struct index *base, uint32_t size_hint)

static int
memtx_rtree_index_reserve(...)

> +{
> +	/**

Comments inside functions start from /* and should be restricted
with 66 characters.

> +	 * In case of rtree we use reserve to make sure that memory allocation
> +	 * will not fail durig any operationin rtree, because there is no
> +	 * error handling in rtree lib. It is just a check not index reservation.
> +	 */
> +	(void)size_hint;
> +	struct memtx_engine *memtx = (struct memtx_engine *)base->engine;
> +	return memtx_index_extent_reserve(memtx,
> +                                    RESERVE_EXTENTS_BEFORE_REPLACE);

Looks like broken indentation.

> diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
> index 6ef84e045..da20b9196 100644
> --- a/src/box/memtx_space.c
> +++ b/src/box/memtx_space.c
> @@ -103,18 +103,6 @@ memtx_space_replace_no_keys(struct space *space, struct tuple *old_tuple,
>  	return -1;
>  }
>  
> -enum {
> -	/**
> -	 * This number is calculated based on the
> -	 * max (realistic) number of insertions
> -	 * a deletion from a B-tree or an R-tree
> -	 * can lead to, and, as a result, the max
> -	 * number of new block allocations.
> -	 */
> -	RESERVE_EXTENTS_BEFORE_DELETE = 8,
> -	RESERVE_EXTENTS_BEFORE_REPLACE = 16
> -};
> -
>  /**
>   * A short-cut version of replace() used during bulk load
>   * from snapshot.

I don't dive into reasons why you avoid using error injection
in this case, but still we have separate 'long_run' tests
(you can specify long_run option in suite.ini config).
 

Re: [Tarantool-patches] [PATCH v4] memtx: fix out of memory handling for rtree

[Olga Arkhangelskaia]

Hi Nikita, thanks for review.
On 20/12/2019 18:35, Nikita Pettik wrote:
> On 20 Dec 16:44, Olga Arkhangelskaia wrote:
>
> Hello. Below are a few nits concerning code itself.
>
>> diff --git a/src/box/index.cc b/src/box/index.cc
>> index 4e4867118..d0aa62d67 100644
>> --- a/src/box/index.cc
>> +++ b/src/box/index.cc
>> @@ -733,7 +733,8 @@ int
>>   generic_index_build_next(struct index *index, struct tuple *tuple)
>>   {
>>   	struct tuple *unused;
>> -	return index_replace(index, NULL, tuple, DUP_INSERT, &unused);
>> +	int res = index_reserve(index, 0);
>> +	return res == 0 ? index_replace(index, NULL, tuple, DUP_INSERT, &unused) : res;
>>   }
> Why not:
>
> if (index_reserve(index, 0) != 0)
> 	return -1;
> return index_replace(...);
Ok, I will change. I guess it did't come on my mind.
>
>>   
>>   void
>> diff --git a/src/box/memtx_engine.h b/src/box/memtx_engine.h
>> index f562c66df..f6335cc66 100644
>> --- a/src/box/memtx_engine.h
>> +++ b/src/box/memtx_engine.h
>> @@ -87,6 +87,18 @@ enum memtx_recovery_state {
>>   /** Memtx extents pool, available to statistics. */
>>   extern struct mempool memtx_index_extent_pool;
>>   
>> +enum memtx_reserve_extents_num{
>                                   ^ missed space
>
>> +	/**
>> +	 * This number is calculated based on the
>> +	 * max (realistic) number of insertions
>> +	 * a deletion from a B-tree or an R-tree
>> +	 * can lead to, and, as a result, the max
>> +	 * number of new block allocations.
>> +	 */
>> +	RESERVE_EXTENTS_BEFORE_DELETE = 8,
>> +	RESERVE_EXTENTS_BEFORE_REPLACE = 16
>> +};
>> +
>>   /**
>>    * The size of the biggest memtx iterator. Used with
>>    * mempool_create. This is the size of the block that will be
>> diff --git a/src/box/memtx_rtree.c b/src/box/memtx_rtree.c
>> index 8badad797..805fdf54c 100644
>> --- a/src/box/memtx_rtree.c
>> +++ b/src/box/memtx_rtree.c
>> @@ -242,6 +242,19 @@ memtx_rtree_index_replace(struct index *base, struct tuple *old_tuple,
>>   	return 0;
>>   }
>>   
>> +static int memtx_rtree_index_reserve(struct index *base, uint32_t size_hint)
> static int
> memtx_rtree_index_reserve(...)
>
>> +{
>> +	/**
> Comments inside functions start from /* and should be restricted
> with 66 characters.
Will fix, and all above too.
>
>> +	 * In case of rtree we use reserve to make sure that memory allocation
>> +	 * will not fail durig any operationin rtree, because there is no
>> +	 * error handling in rtree lib. It is just a check not index reservation.
>> +	 */
>> +	(void)size_hint;
>> +	struct memtx_engine *memtx = (struct memtx_engine *)base->engine;
>> +	return memtx_index_extent_reserve(memtx,
>> +                                    RESERVE_EXTENTS_BEFORE_REPLACE);
> Looks like broken indentation.

It is likely, I have change my work machine and some options may have 
been missed, will pay more attention.


>> diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
>> index 6ef84e045..da20b9196 100644
>> --- a/src/box/memtx_space.c
>> +++ b/src/box/memtx_space.c
>> @@ -103,18 +103,6 @@ memtx_space_replace_no_keys(struct space *space, struct tuple *old_tuple,
>>   	return -1;
>>   }
>>   
>> -enum {
>> -	/**
>> -	 * This number is calculated based on the
>> -	 * max (realistic) number of insertions
>> -	 * a deletion from a B-tree or an R-tree
>> -	 * can lead to, and, as a result, the max
>> -	 * number of new block allocations.
>> -	 */
>> -	RESERVE_EXTENTS_BEFORE_DELETE = 8,
>> -	RESERVE_EXTENTS_BEFORE_REPLACE = 16
>> -};
>> -
>>   /**
>>    * A short-cut version of replace() used during bulk load
>>    * from snapshot.
> I don't dive into reasons why you avoid using error injection
> in this case, but still we have separate 'long_run' tests
> (you can specify long_run option in suite.ini config).

Actually it is long on my mac =)) However, I am not sure what to use

84     _(ERRINJ_INDEX_ALLOC, ERRINJ_BOOL
85     _(ERRINJ_TUPLE_ALLOC, ERRINJ_BOOL

because it is extent allocation that fail. Will be glad to hear what 
should be chosen.

>   

Re: [Tarantool-patches] [PATCH v4] memtx: fix out of memory handling for rtree

[Olga Arkhangelskaia]

On 24/12/2019 11:06, Olga Arkhangelskaia wrote:
> Hi Nikita, thanks for review.
> On 20/12/2019 18:35, Nikita Pettik wrote:
>> On 20 Dec 16:44, Olga Arkhangelskaia wrote:
>>
>> Hello. Below are a few nits concerning code itself.
>>
>>> diff --git a/src/box/index.cc b/src/box/index.cc
>>> index 4e4867118..d0aa62d67 100644
>>> --- a/src/box/index.cc
>>> +++ b/src/box/index.cc
>>> @@ -733,7 +733,8 @@ int
>>>   generic_index_build_next(struct index *index, struct tuple *tuple)
>>>   {
>>>       struct tuple *unused;
>>> -    return index_replace(index, NULL, tuple, DUP_INSERT, &unused);
>>> +    int res = index_reserve(index, 0);
>>> +    return res == 0 ? index_replace(index, NULL, tuple, DUP_INSERT, 
>>> &unused) : res;
>>>   }
>> Why not:
>>
>> if (index_reserve(index, 0) != 0)
>>     return -1;
>> return index_replace(...);
> Ok, I will change. I guess it did't come on my mind.
>>
>>>     void
>>> diff --git a/src/box/memtx_engine.h b/src/box/memtx_engine.h
>>> index f562c66df..f6335cc66 100644
>>> --- a/src/box/memtx_engine.h
>>> +++ b/src/box/memtx_engine.h
>>> @@ -87,6 +87,18 @@ enum memtx_recovery_state {
>>>   /** Memtx extents pool, available to statistics. */
>>>   extern struct mempool memtx_index_extent_pool;
>>>   +enum memtx_reserve_extents_num{
>>                                   ^ missed space
>>
>>> +    /**
>>> +     * This number is calculated based on the
>>> +     * max (realistic) number of insertions
>>> +     * a deletion from a B-tree or an R-tree
>>> +     * can lead to, and, as a result, the max
>>> +     * number of new block allocations.
>>> +     */
>>> +    RESERVE_EXTENTS_BEFORE_DELETE = 8,
>>> +    RESERVE_EXTENTS_BEFORE_REPLACE = 16
>>> +};
>>> +
>>>   /**
>>>    * The size of the biggest memtx iterator. Used with
>>>    * mempool_create. This is the size of the block that will be
>>> diff --git a/src/box/memtx_rtree.c b/src/box/memtx_rtree.c
>>> index 8badad797..805fdf54c 100644
>>> --- a/src/box/memtx_rtree.c
>>> +++ b/src/box/memtx_rtree.c
>>> @@ -242,6 +242,19 @@ memtx_rtree_index_replace(struct index *base, 
>>> struct tuple *old_tuple,
>>>       return 0;
>>>   }
>>>   +static int memtx_rtree_index_reserve(struct index *base, uint32_t 
>>> size_hint)
>> static int
>> memtx_rtree_index_reserve(...)
>>
>>> +{
>>> +    /**
>> Comments inside functions start from /* and should be restricted
>> with 66 characters.
> Will fix, and all above too.
>>
>>> +     * In case of rtree we use reserve to make sure that memory 
>>> allocation
>>> +     * will not fail durig any operationin rtree, because there is no
>>> +     * error handling in rtree lib. It is just a check not index 
>>> reservation.
>>> +     */
>>> +    (void)size_hint;
>>> +    struct memtx_engine *memtx = (struct memtx_engine *)base->engine;
>>> +    return memtx_index_extent_reserve(memtx,
>>> + RESERVE_EXTENTS_BEFORE_REPLACE);
>> Looks like broken indentation.
>
> It is likely, I have change my work machine and some options may have 
> been missed, will pay more attention.
>
>
>>> diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
>>> index 6ef84e045..da20b9196 100644
>>> --- a/src/box/memtx_space.c
>>> +++ b/src/box/memtx_space.c
>>> @@ -103,18 +103,6 @@ memtx_space_replace_no_keys(struct space 
>>> *space, struct tuple *old_tuple,
>>>       return -1;
>>>   }
>>>   -enum {
>>> -    /**
>>> -     * This number is calculated based on the
>>> -     * max (realistic) number of insertions
>>> -     * a deletion from a B-tree or an R-tree
>>> -     * can lead to, and, as a result, the max
>>> -     * number of new block allocations.
>>> -     */
>>> -    RESERVE_EXTENTS_BEFORE_DELETE = 8,
>>> -    RESERVE_EXTENTS_BEFORE_REPLACE = 16
>>> -};
>>> -
>>>   /**
>>>    * A short-cut version of replace() used during bulk load
>>>    * from snapshot.
>> I don't dive into reasons why you avoid using error injection
>> in this case, but still we have separate 'long_run' tests
>> (you can specify long_run option in suite.ini config).
>
> Actually it is long on my mac =)) However, I am not sure what to use
>
> 84     _(ERRINJ_INDEX_ALLOC, ERRINJ_BOOL
> 85     _(ERRINJ_TUPLE_ALLOC, ERRINJ_BOOL
>
> because it is extent allocation that fail. Will be glad to hear what 
> should be chosen.
>
I have looked closer - it is _(ERRINJ_INDEX_ALLOC, ERRINJ_BOOL actually.
>
>