[tarantool-patches] [PATCH 2/2] sql: fix antisymmetric boolean comparison in VDBE

[Nikita Pettik <korablev@tarantool.org>]

There are a few situations when booleans can be compared with values of
other types. To process them, we assume that booleans are always less
than numbers, which in turn are less than strings. On the other hand,
function which implements internal comparison of values -
sqlMemCompare() always returns 'less' result if one of values is boolean
and another one is not, ignoring the order of values. For instance:

... max (false, 'abc') -> 'abc'
... max ('abc', false) -> false

This patch fixes this misbehaviour making boolean values always less
than values of other types.
---
 src/box/sql/vdbeaux.c       |  2 ++
 test/sql-tap/func5.test.lua | 30 +++++++++++++++++++++++++++++-
 test/sql-tap/in1.test.lua   |  2 +-
 3 files changed, 32 insertions(+), 2 deletions(-)

diff --git a/src/box/sql/vdbeaux.c b/src/box/sql/vdbeaux.c
index baeeb4624..d1388f20d 100644
--- a/src/box/sql/vdbeaux.c
+++ b/src/box/sql/vdbeaux.c
@@ -2955,6 +2955,8 @@ sqlMemCompare(const Mem * pMem1, const Mem * pMem2, const struct coll * pColl)
 				return 1;
 			return -1;
 		}
+		if ((f2 & MEM_Bool) != 0)
+			return +1;
 		return -1;
 	}
 
diff --git a/test/sql-tap/func5.test.lua b/test/sql-tap/func5.test.lua
index f3706e631..6da089994 100755
--- a/test/sql-tap/func5.test.lua
+++ b/test/sql-tap/func5.test.lua
@@ -1,6 +1,6 @@
 #!/usr/bin/env tarantool
 test = require("sqltester")
-test:plan(15)
+test:plan(19)
 
 --!./tcltestrunner.lua
 -- 2010 August 27
@@ -229,4 +229,32 @@ test:do_catchsql_test(
     }
 )
 
+-- Order of arguments of min/max functions doesn't affect
+-- the result: boolean is always less than numbers, which
+-- are less than strings.
+--
+test:do_execsql_test(
+    "func-5-4.1",
+    [[
+        SELECT max (false, 'STR', 1, 0.5);
+    ]], { "STR" } )
+
+test:do_execsql_test(
+    "func-5-4.2",
+    [[
+        SELECT max ('STR', 1, 0.5, false);
+    ]], { "STR" } )
+
+test:do_execsql_test(
+    "func-5-4.3",
+    [[
+        SELECT min ('STR', 1, 0.5, false);
+    ]], { false } )
+
+test:do_execsql_test(
+    "func-5-4.4",
+    [[
+        SELECT min (false, 'STR', 1, 0.5);
+    ]], { false } )
+
 test:finish_test()
diff --git a/test/sql-tap/in1.test.lua b/test/sql-tap/in1.test.lua
index eca7c15ba..50c4e4837 100755
--- a/test/sql-tap/in1.test.lua
+++ b/test/sql-tap/in1.test.lua
@@ -207,7 +207,7 @@ test:do_execsql_test(
 test:do_execsql_test(
     "in-2.10",
     [[
-        SELECT a FROM t1 WHERE min(0,b IN (a,30)) <> 0
+        SELECT a FROM t1 WHERE min(0, CAST(b IN (a,30) AS INT)) <> 0
     ]], {
         -- <in-2.10>
         
-- 
2.15.1