[Tarantool-discussions] Addition of NUMBER field type

[Tarantool-discussions] Addition of NUMBER field type

[Mergen Imeev]

Hi all,
I would like to discuss once again the problem of adding a DOUBLE
field type.

Suppose we have added a DOUBLE field type to which we can insert
data of type MP_DOUBLE in the format MsgPack. The problem is that
inserting an integer from Lua, we get an error. This is not so bad
for integer cdata, but it is bad for integers od Lua type NUMBER.
Currently, we have 5 ways to solve the problem:

1) Re-encode tuple when box receives it.

In this case, we must run the check when box receives the tuple.
After that, we re-encode the tuple, if necessary. The main problem
is that we have to run the check for any front-end, although the
problem exists only for Lua. This case suggests that we have an
implicit cast from MP_INT/MP_UINT to MP_DOUBLE. The idea is simple
to understand and not too difficult to implement. I already have
an implementation.

2) Use space format when data from Lua is serialized.

In this case, we get the space format before calling the
serializer and use the format in the serializer to convert INTEGER
to DOUBLE. In case we get a tuple, we must re-encode it. This case
also suggests that we have an implicit cast from MP_INT/MP_UINT to
MP_DOUBLE. Also, the implementation looks a bit complicated,
though the idea is not hard to understand. The implementation
already exists.

3) Re-encode tuple when tuple is being saved.

In this case, we must replace the INTEGER field with DOUBLE while
checking the tuple in box before inserting into a space. 
Implementation should be done for both engines that support
insertion. This case also suggests that we have an implicit cast
from MP_INT/MP_UINT to MP_DOUBLE. I did not create an
implementation.

4) Give to user a function to convert a number to DOUBLE in Lua
before inserting into space.

In this case, the user must convert the number to DOUBLE before
inserting. We do not add any implicit cast. Very easy to
understand and implement. Although the user may be a little
confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE
field, but this can be solved by writing about this in the
documentation.

In addition, if we convert to LuaJIT DOUBLE cdata, then we can use
these cdata as the operand of an arithmetic operation. However,
arithmetic operations for this type do not work correctly. And it
will not be fixed, most likely. It should also be written in the
documentation, I think.

5) Add an option to insert() method in Lua, which says if user
allows to cast number to DOUBLE if needed.

I have not considered this case before. But in this case, we can
use the same implementation as in the first or second cases. Or we
can create a function from the fourth case and do everything that
we need to do in Lua before calling the C functions.

This case doesn't lookas a good solution, since we are currently
directly calling C functions on INSERT/REPLACE.



It is also worth noting that in any case, when decoding a tuple,
we get a number of Lua type NUMBER.

I think, since we want to add this type of field mainly because of
SQL, the simplest solution is the right one. Since we do all the
operations in Lua, it doesn't matter how we insert the number into
the system space, since we get the same thing with get()/select().

I suggest using the fourth way to solve the problem. It allows us
to avoid adding implicit cast.

Re: [Tarantool-discussions] Addition of NUMBER field type

[Konstantin Osipov]

* Mergen Imeev <imeevma@tarantool.org> [19/12/19 16:44]:
> Hi all,
> I would like to discuss once again the problem of adding a DOUBLE
> field type.
> 
> Suppose we have added a DOUBLE field type to which we can insert
> data of type MP_DOUBLE in the format MsgPack. The problem is that
> inserting an integer from Lua, we get an error. This is not so bad
> for integer cdata, but it is bad for integers od Lua type NUMBER.
> Currently, we have 5 ways to solve the problem:
> 
> 1) Re-encode tuple when box receives it.
> 
> In this case, we must run the check when box receives the tuple.
> After that, we re-encode the tuple, if necessary. The main problem
> is that we have to run the check for any front-end, although the
> problem exists only for Lua. This case suggests that we have an
> implicit cast from MP_INT/MP_UINT to MP_DOUBLE. The idea is simple
> to understand and not too difficult to implement. I already have
> an implementation.
> 
> 2) Use space format when data from Lua is serialized.
> 
> In this case, we get the space format before calling the
> serializer and use the format in the serializer to convert INTEGER
> to DOUBLE. In case we get a tuple, we must re-encode it. This case
> also suggests that we have an implicit cast from MP_INT/MP_UINT to
> MP_DOUBLE. Also, the implementation looks a bit complicated,
> though the idea is not hard to understand. The implementation
> already exists.
> 
> 3) Re-encode tuple when tuple is being saved.
> 
> In this case, we must replace the INTEGER field with DOUBLE while
> checking the tuple in box before inserting into a space. 
> Implementation should be done for both engines that support
> insertion. This case also suggests that we have an implicit cast
> from MP_INT/MP_UINT to MP_DOUBLE. I did not create an
> implementation.
> 
> 4) Give to user a function to convert a number to DOUBLE in Lua
> before inserting into space.
> 
> In this case, the user must convert the number to DOUBLE before
> inserting. We do not add any implicit cast. Very easy to
> understand and implement. Although the user may be a little
> confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE
> field, but this can be solved by writing about this in the
> documentation.
> 
> In addition, if we convert to LuaJIT DOUBLE cdata, then we can use
> these cdata as the operand of an arithmetic operation. However,
> arithmetic operations for this type do not work correctly. And it
> will not be fixed, most likely. It should also be written in the
> documentation, I think.
> 
> 5) Add an option to insert() method in Lua, which says if user
> allows to cast number to DOUBLE if needed.
> 
> I have not considered this case before. But in this case, we can
> use the same implementation as in the first or second cases. Or we
> can create a function from the fourth case and do everything that
> we need to do in Lua before calling the C functions.
> 
> This case doesn't lookas a good solution, since we are currently
> directly calling C functions on INSERT/REPLACE.
> 
> 
> 
> It is also worth noting that in any case, when decoding a tuple,
> we get a number of Lua type NUMBER.
> 
> I think, since we want to add this type of field mainly because of
> SQL, the simplest solution is the right one. Since we do all the
> operations in Lua, it doesn't matter how we insert the number into
> the system space, since we get the same thing with get()/select().
> 
> I suggest using the fourth way to solve the problem. It allows us
> to avoid adding implicit cast.

I like it #4 too - it's cheap & efficient.

-- 
Konstantin Osipov, Moscow, Russia

Re: [Tarantool-discussions] Addition of NUMBER field type

[Igor Munkin]

On 19.12.19, Konstantin Osipov wrote:
> * Mergen Imeev <imeevma@tarantool.org> [19/12/19 16:44]:
> > Hi all,
> > I would like to discuss once again the problem of adding a DOUBLE
> > field type.
> > 
> > Suppose we have added a DOUBLE field type to which we can insert
> > data of type MP_DOUBLE in the format MsgPack. The problem is that
> > inserting an integer from Lua, we get an error. This is not so bad
> > for integer cdata, but it is bad for integers od Lua type NUMBER.
> > Currently, we have 5 ways to solve the problem:
> > 

<snipped>

> > 
> > 4) Give to user a function to convert a number to DOUBLE in Lua
> > before inserting into space.
> > 
> > In this case, the user must convert the number to DOUBLE before
> > inserting. We do not add any implicit cast. Very easy to
> > understand and implement. Although the user may be a little
> > confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE
> > field, but this can be solved by writing about this in the
> > documentation.
> > 
> > In addition, if we convert to LuaJIT DOUBLE cdata, then we can use
> > these cdata as the operand of an arithmetic operation. However,
> > arithmetic operations for this type do not work correctly. And it
> > will not be fixed, most likely. It should also be written in the
> > documentation, I think.
> > 

<snipped>

> 
> I like it #4 too - it's cheap & efficient.

Despite the broken arithmetics and assuming no one need this type in
Lua -- yes, this is a good one.

> 
> -- 
> Konstantin Osipov, Moscow, Russia

-- 
Best regards,
IM

Re: [Tarantool-discussions] Addition of NUMBER field type

[Mergen Imeev]

On Fri, Dec 20, 2019 at 05:35:20PM +0300, Igor Munkin wrote:
> On 19.12.19, Konstantin Osipov wrote:
> > * Mergen Imeev <imeevma@tarantool.org> [19/12/19 16:44]:
> > > Hi all,
> > > I would like to discuss once again the problem of adding a DOUBLE
> > > field type.
> > > 
> > > Suppose we have added a DOUBLE field type to which we can insert
> > > data of type MP_DOUBLE in the format MsgPack. The problem is that
> > > inserting an integer from Lua, we get an error. This is not so bad
> > > for integer cdata, but it is bad for integers od Lua type NUMBER.
> > > Currently, we have 5 ways to solve the problem:
> > > 
> 
> <snipped>
> 
> > > 
> > > 4) Give to user a function to convert a number to DOUBLE in Lua
> > > before inserting into space.
> > > 
> > > In this case, the user must convert the number to DOUBLE before
> > > inserting. We do not add any implicit cast. Very easy to
> > > understand and implement. Although the user may be a little
> > > confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE
> > > field, but this can be solved by writing about this in the
> > > documentation.
> > > 
> > > In addition, if we convert to LuaJIT DOUBLE cdata, then we can use
> > > these cdata as the operand of an arithmetic operation. However,
> > > arithmetic operations for this type do not work correctly. And it
> > > will not be fixed, most likely. It should also be written in the
> > > documentation, I think.
> > > 
> 
> <snipped>
> 
> > 
> > I like it #4 too - it's cheap & efficient.
> 
> Despite the broken arithmetics and assuming no one need this type in
> Lua -- yes, this is a good one.
> 
After discussion, it was decided that the fourth solution
is the most suitable, since the main reason for adding
this field is the need for a DOUBLE type in SQL.

However, most likely, questions will arise about why it
works in this way, so we should be able to answer them. In
this regard, in order to correctly add this field, we must
describe how to work with it in Lua, in the documentation.
We must say that any number can be converted to a number
that can definitely be inserted using the ffi.cast()
function. And so that not any objects of other numeric
types can be inserted.
> > 
> > -- 
> > Konstantin Osipov, Moscow, Russia
> 
> -- 
> Best regards,
> IM

Re: [Tarantool-discussions] Addition of NUMBER field type

[Alexander Turenko]

> 4) Give to user a function to convert a number to DOUBLE in Lua
> before inserting into space.
> 
> In this case, the user must convert the number to DOUBLE before
> inserting. We do not add any implicit cast. Very easy to
> understand and implement. Although the user may be a little
> confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE
> field, but this can be solved by writing about this in the
> documentation.
> 
> In addition, if we convert to LuaJIT DOUBLE cdata, then we can use
> these cdata as the operand of an arithmetic operation. However,
> arithmetic operations for this type do not work correctly. And it
> will not be fixed, most likely. It should also be written in the
> documentation, I think.

> It is also worth noting that in any case, when decoding a tuple,
> we get a number of Lua type NUMBER.

The question is about inserting the tuple back, because it is usual case
for migrations: get a tuple from one space, transform it, put it to
another space, repeat on all tuples, drop old space, rename the new one.

If a double value is changed, then a user should do ffi.cast('double',
value) (or other wrapper we'll provide). No problem.

If tuple will not be decoded (say, a migration is performed using just
tuple:transform()), then I guess everything will work too.

However if we'll decode the tuple in order to perform some
transformation, we'll unable to insert it w/o manual casting of all
double field.

This may be confusing, especially if a user does not touch this double
value explicitly.

However I agree that 4th variant is the best one, because others are
harder to explain. We should pay attention to cases like one above and
cleanly describe them in the documentation.

> I think, since we want to add this type of field mainly because of
> SQL, the simplest solution is the right one. Since we do all the
> operations in Lua, it doesn't matter how we insert the number into
> the system space, since we get the same thing with get()/select().
> 
> I suggest using the fourth way to solve the problem. It allows us
> to avoid adding implicit cast.