Hi all, I would like to discuss once again the problem of adding a DOUBLE field type. Suppose we have added a DOUBLE field type to which we can insert data of type MP_DOUBLE in the format MsgPack. The problem is that inserting an integer from Lua, we get an error. This is not so bad for integer cdata, but it is bad for integers od Lua type NUMBER. Currently, we have 5 ways to solve the problem: 1) Re-encode tuple when box receives it. In this case, we must run the check when box receives the tuple. After that, we re-encode the tuple, if necessary. The main problem is that we have to run the check for any front-end, although the problem exists only for Lua. This case suggests that we have an implicit cast from MP_INT/MP_UINT to MP_DOUBLE. The idea is simple to understand and not too difficult to implement. I already have an implementation. 2) Use space format when data from Lua is serialized. In this case, we get the space format before calling the serializer and use the format in the serializer to convert INTEGER to DOUBLE. In case we get a tuple, we must re-encode it. This case also suggests that we have an implicit cast from MP_INT/MP_UINT to MP_DOUBLE. Also, the implementation looks a bit complicated, though the idea is not hard to understand. The implementation already exists. 3) Re-encode tuple when tuple is being saved. In this case, we must replace the INTEGER field with DOUBLE while checking the tuple in box before inserting into a space. Implementation should be done for both engines that support insertion. This case also suggests that we have an implicit cast from MP_INT/MP_UINT to MP_DOUBLE. I did not create an implementation. 4) Give to user a function to convert a number to DOUBLE in Lua before inserting into space. In this case, the user must convert the number to DOUBLE before inserting. We do not add any implicit cast. Very easy to understand and implement. Although the user may be a little confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE field, but this can be solved by writing about this in the documentation. In addition, if we convert to LuaJIT DOUBLE cdata, then we can use these cdata as the operand of an arithmetic operation. However, arithmetic operations for this type do not work correctly. And it will not be fixed, most likely. It should also be written in the documentation, I think. 5) Add an option to insert() method in Lua, which says if user allows to cast number to DOUBLE if needed. I have not considered this case before. But in this case, we can use the same implementation as in the first or second cases. Or we can create a function from the fourth case and do everything that we need to do in Lua before calling the C functions. This case doesn't lookas a good solution, since we are currently directly calling C functions on INSERT/REPLACE. It is also worth noting that in any case, when decoding a tuple, we get a number of Lua type NUMBER. I think, since we want to add this type of field mainly because of SQL, the simplest solution is the right one. Since we do all the operations in Lua, it doesn't matter how we insert the number into the system space, since we get the same thing with get()/select(). I suggest using the fourth way to solve the problem. It allows us to avoid adding implicit cast.

```
* Mergen Imeev <imeevma@tarantool.org> [19/12/19 16:44]:
> Hi all,
> I would like to discuss once again the problem of adding a DOUBLE
> field type.
>
> Suppose we have added a DOUBLE field type to which we can insert
> data of type MP_DOUBLE in the format MsgPack. The problem is that
> inserting an integer from Lua, we get an error. This is not so bad
> for integer cdata, but it is bad for integers od Lua type NUMBER.
> Currently, we have 5 ways to solve the problem:
>
> 1) Re-encode tuple when box receives it.
>
> In this case, we must run the check when box receives the tuple.
> After that, we re-encode the tuple, if necessary. The main problem
> is that we have to run the check for any front-end, although the
> problem exists only for Lua. This case suggests that we have an
> implicit cast from MP_INT/MP_UINT to MP_DOUBLE. The idea is simple
> to understand and not too difficult to implement. I already have
> an implementation.
>
> 2) Use space format when data from Lua is serialized.
>
> In this case, we get the space format before calling the
> serializer and use the format in the serializer to convert INTEGER
> to DOUBLE. In case we get a tuple, we must re-encode it. This case
> also suggests that we have an implicit cast from MP_INT/MP_UINT to
> MP_DOUBLE. Also, the implementation looks a bit complicated,
> though the idea is not hard to understand. The implementation
> already exists.
>
> 3) Re-encode tuple when tuple is being saved.
>
> In this case, we must replace the INTEGER field with DOUBLE while
> checking the tuple in box before inserting into a space.
> Implementation should be done for both engines that support
> insertion. This case also suggests that we have an implicit cast
> from MP_INT/MP_UINT to MP_DOUBLE. I did not create an
> implementation.
>
> 4) Give to user a function to convert a number to DOUBLE in Lua
> before inserting into space.
>
> In this case, the user must convert the number to DOUBLE before
> inserting. We do not add any implicit cast. Very easy to
> understand and implement. Although the user may be a little
> confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE
> field, but this can be solved by writing about this in the
> documentation.
>
> In addition, if we convert to LuaJIT DOUBLE cdata, then we can use
> these cdata as the operand of an arithmetic operation. However,
> arithmetic operations for this type do not work correctly. And it
> will not be fixed, most likely. It should also be written in the
> documentation, I think.
>
> 5) Add an option to insert() method in Lua, which says if user
> allows to cast number to DOUBLE if needed.
>
> I have not considered this case before. But in this case, we can
> use the same implementation as in the first or second cases. Or we
> can create a function from the fourth case and do everything that
> we need to do in Lua before calling the C functions.
>
> This case doesn't lookas a good solution, since we are currently
> directly calling C functions on INSERT/REPLACE.
>
>
>
> It is also worth noting that in any case, when decoding a tuple,
> we get a number of Lua type NUMBER.
>
> I think, since we want to add this type of field mainly because of
> SQL, the simplest solution is the right one. Since we do all the
> operations in Lua, it doesn't matter how we insert the number into
> the system space, since we get the same thing with get()/select().
>
> I suggest using the fourth way to solve the problem. It allows us
> to avoid adding implicit cast.
I like it #4 too - it's cheap & efficient.
--
Konstantin Osipov, Moscow, Russia
```

On 19.12.19, Konstantin Osipov wrote: > * Mergen Imeev <imeevma@tarantool.org> [19/12/19 16:44]: > > Hi all, > > I would like to discuss once again the problem of adding a DOUBLE > > field type. > > > > Suppose we have added a DOUBLE field type to which we can insert > > data of type MP_DOUBLE in the format MsgPack. The problem is that > > inserting an integer from Lua, we get an error. This is not so bad > > for integer cdata, but it is bad for integers od Lua type NUMBER. > > Currently, we have 5 ways to solve the problem: > > <snipped> > > > > 4) Give to user a function to convert a number to DOUBLE in Lua > > before inserting into space. > > > > In this case, the user must convert the number to DOUBLE before > > inserting. We do not add any implicit cast. Very easy to > > understand and implement. Although the user may be a little > > confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE > > field, but this can be solved by writing about this in the > > documentation. > > > > In addition, if we convert to LuaJIT DOUBLE cdata, then we can use > > these cdata as the operand of an arithmetic operation. However, > > arithmetic operations for this type do not work correctly. And it > > will not be fixed, most likely. It should also be written in the > > documentation, I think. > > <snipped> > > I like it #4 too - it's cheap & efficient. Despite the broken arithmetics and assuming no one need this type in Lua -- yes, this is a good one. > > -- > Konstantin Osipov, Moscow, Russia -- Best regards, IM

On Fri, Dec 20, 2019 at 05:35:20PM +0300, Igor Munkin wrote: > On 19.12.19, Konstantin Osipov wrote: > > * Mergen Imeev <imeevma@tarantool.org> [19/12/19 16:44]: > > > Hi all, > > > I would like to discuss once again the problem of adding a DOUBLE > > > field type. > > > > > > Suppose we have added a DOUBLE field type to which we can insert > > > data of type MP_DOUBLE in the format MsgPack. The problem is that > > > inserting an integer from Lua, we get an error. This is not so bad > > > for integer cdata, but it is bad for integers od Lua type NUMBER. > > > Currently, we have 5 ways to solve the problem: > > > > > <snipped> > > > > > > > 4) Give to user a function to convert a number to DOUBLE in Lua > > > before inserting into space. > > > > > > In this case, the user must convert the number to DOUBLE before > > > inserting. We do not add any implicit cast. Very easy to > > > understand and implement. Although the user may be a little > > > confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE > > > field, but this can be solved by writing about this in the > > > documentation. > > > > > > In addition, if we convert to LuaJIT DOUBLE cdata, then we can use > > > these cdata as the operand of an arithmetic operation. However, > > > arithmetic operations for this type do not work correctly. And it > > > will not be fixed, most likely. It should also be written in the > > > documentation, I think. > > > > > <snipped> > > > > > I like it #4 too - it's cheap & efficient. > > Despite the broken arithmetics and assuming no one need this type in > Lua -- yes, this is a good one. > After discussion, it was decided that the fourth solution is the most suitable, since the main reason for adding this field is the need for a DOUBLE type in SQL. However, most likely, questions will arise about why it works in this way, so we should be able to answer them. In this regard, in order to correctly add this field, we must describe how to work with it in Lua, in the documentation. We must say that any number can be converted to a number that can definitely be inserted using the ffi.cast() function. And so that not any objects of other numeric types can be inserted. > > > > -- > > Konstantin Osipov, Moscow, Russia > > -- > Best regards, > IM

> 4) Give to user a function to convert a number to DOUBLE in Lua > before inserting into space. > > In this case, the user must convert the number to DOUBLE before > inserting. We do not add any implicit cast. Very easy to > understand and implement. Although the user may be a little > confused that he cannot insert INTEGER/UNSIGNED into a DOUBLE > field, but this can be solved by writing about this in the > documentation. > > In addition, if we convert to LuaJIT DOUBLE cdata, then we can use > these cdata as the operand of an arithmetic operation. However, > arithmetic operations for this type do not work correctly. And it > will not be fixed, most likely. It should also be written in the > documentation, I think. > It is also worth noting that in any case, when decoding a tuple, > we get a number of Lua type NUMBER. The question is about inserting the tuple back, because it is usual case for migrations: get a tuple from one space, transform it, put it to another space, repeat on all tuples, drop old space, rename the new one. If a double value is changed, then a user should do ffi.cast('double', value) (or other wrapper we'll provide). No problem. If tuple will not be decoded (say, a migration is performed using just tuple:transform()), then I guess everything will work too. However if we'll decode the tuple in order to perform some transformation, we'll unable to insert it w/o manual casting of all double field. This may be confusing, especially if a user does not touch this double value explicitly. However I agree that 4th variant is the best one, because others are harder to explain. We should pay attention to cases like one above and cleanly describe them in the documentation. > I think, since we want to add this type of field mainly because of > SQL, the simplest solution is the right one. Since we do all the > operations in Lua, it doesn't matter how we insert the number into > the system space, since we get the same thing with get()/select(). > > I suggest using the fourth way to solve the problem. It allows us > to avoid adding implicit cast.