<div dir="ltr"><div><div>commit 5f2c0b8aa78d3638473a2a12b6d1059657144d58</div><div>Author: N.Tatunov <<a href="mailto:hollow653@gmail.com">hollow653@gmail.com</a>></div><div>Date:   Thu Jun 28 15:17:32 2018 +0300</div><div><br></div><div>    sql: LIKE & GLOB pattern comparison issue</div><div>    </div><div>    Currently function that compares pattern and string for GLOB & LIKE</div><div>    operators doesn't work properly. It uses ICU reading function which</div><div>    was assumed having other return codes and the implementation for the</div><div>    comparison ending isn't paying attention to some special cases, hence</div><div>    in those cases it works improperly.</div><div>    </div><div>    With the patch applied an error will be returned in case there's an</div><div>    invalid UTF-8 symbol in pattern & pattern containing only valid UTF-8</div><div>    symbols will not be matched with the string that contains invalid</div><div>    symbol.</div><div>    </div><div>    Сloses #3251</div><div>    Сloses #3334</div><div>    Part of #3572</div><div><br></div><div>diff --git a/src/box/sql/func.c b/src/box/sql/func.c</div><div>index c06e3bd..27c6f42 100644</div><div>--- a/src/box/sql/func.c</div><div>+++ b/src/box/sql/func.c</div><div>@@ -617,13 +617,16 @@ struct compareInfo {</div><div> <span style="white-space:pre">      </span>u8 noCase;<span style="white-space:pre">           </span>/* true to ignore case differences */</div><div> };</div><div> </div><div>-/*</div><div>- * For LIKE and GLOB matching on EBCDIC machines, assume that every</div><div>- * character is exactly one byte in size.  Also, provde the Utf8Read()</div><div>- * macro for fast reading of the next character in the common case where</div><div>- * the next character is ASCII.</div><div>+/**</div><div>+ * Providing there are symbols in string s this</div><div>+ * macro returns UTF-8 code of character and</div><div>+ * promotes pointer to the next symbol in the string.</div><div>+ * Otherwise return code is SQL_END_OF_STRING.</div><div>  */</div><div>-#define Utf8Read(s, e)    ucnv_getNextUChar(pUtf8conv, &s, e, &status)</div><div>+#define Utf8Read(s, e) ucnv_getNextUChar(pUtf8conv, &(s), (e), &(status))</div><div>+</div><div>+#define SQL_END_OF_STRING        0xffff</div><div>+#define SQL_INVALID_UTF8_SYMBOL  0xfffd</div><div> </div><div> static const struct compareInfo globInfo = { '*', '?', '[', 0 };</div><div> </div><div>@@ -638,19 +641,16 @@ static const struct compareInfo likeInfoNorm = { '%', '_', 0, 1 };</div><div> static const struct compareInfo likeInfoAlt = { '%', '_', 0, 0 };</div><div> </div><div> /*</div><div>- * Possible error returns from patternMatch()</div><div>+ * Possible error returns from sql_utf8_pattern_compare()</div><div>  */</div><div> #define SQLITE_MATCH             0</div><div> #define SQLITE_NOMATCH           1</div><div> #define SQLITE_NOWILDCARDMATCH   2</div><div>+#define SQL_INVALID_PATTERN      3</div><div> </div><div>-/*</div><div>- * Compare two UTF-8 strings for equality where the first string is</div><div>- * a GLOB or LIKE expression.  Return values:</div><div>- *</div><div>- *    SQLITE_MATCH:            Match</div><div>- *    SQLITE_NOMATCH:          No match</div><div>- *    SQLITE_NOWILDCARDMATCH:  No match in spite of having * or % wildcards.</div><div>+/**</div><div>+ * Compare two UTF-8 strings for equality where the first string</div><div>+ * is a GLOB or LIKE expression.</div><div>  *</div><div>  * Globbing rules:</div><div>  *</div><div>@@ -663,92 +663,136 @@ static const struct compareInfo likeInfoAlt = { '%', '_', 0, 0 };</div><div>  *</div><div>  *     [^...]     Matches one character not in the enclosed list.</div><div>  *</div><div>- * With the [...] and [^...] matching, a ']' character can be included</div><div>- * in the list by making it the first character after '[' or '^'.  A</div><div>- * range of characters can be specified using '-'.  Example:</div><div>- * "[a-z]" matches any single lower-case letter.  To match a '-', make</div><div>- * it the last character in the list.</div><div>+ * With the [...] and [^...] matching, a ']' character can be</div><div>+ * included in the list by making it the first character after</div><div>+ * '[' or '^'. A range of characters can be specified using '-'.</div><div>+ * Example: "[a-z]" matches any single lower-case letter.</div><div>+ * To match a '-', make it the last character in the list.</div><div>  *</div><div>  * Like matching rules:</div><div>  *</div><div>- *      '%'       Matches any sequence of zero or more characters</div><div>+ *      '%'       Matches any sequence of zero or more characters.</div><div>  *</div><div>- **     '_'       Matches any one character</div><div>+ **     '_'       Matches any one character.</div><div>  *</div><div>  *      Ec        Where E is the "esc" character and c is any other</div><div>- *                character, including '%', '_', and esc, match exactly c.</div><div>+ *                character, including '%', '_', and esc, match</div><div>+ *                exactly c.</div><div>  *</div><div>  * The comments within this routine usually assume glob matching.</div><div>  *</div><div>- * This routine is usually quick, but can be N**2 in the worst case.</div><div>+ * This routine is usually quick, but can be N**2 in the worst</div><div>+ * case.</div><div>+ *</div><div>+ * @param pattern String containing comparison pattern.</div><div>+ * @param string String being compared.</div><div>+ * @param compareInfo Information about how to compare.</div><div>+ * @param matchOther The escape char (LIKE) or '[' (GLOB).</div><div>+ *</div><div>+ * @retval SQLITE_MATCH:            Match.</div><div>+ *<span style="white-space:pre">        </span>   SQLITE_NOMATCH:          No match.</div><div>+ *<span style="white-space:pre">       </span>   SQLITE_NOWILDCARDMATCH:  No match in spite of having *</div><div>+ *<span style="white-space:pre">                               </span>    or % wildcards.</div><div>+ *<span style="white-space:pre">      </span>   SQL_INVALID_PATTERN:     Pattern contains invalid</div><div>+ *<span style="white-space:pre">                          </span>    symbol.</div><div>  */</div><div> static int</div><div>-patternCompare(const char * pattern,<span style="white-space:pre">     </span>/* The glob pattern */</div><div>-<span style="white-space:pre">       </span>       const char * string,<span style="white-space:pre">      </span>/* The string to compare against the glob */</div><div>-<span style="white-space:pre"> </span>       const struct compareInfo *pInfo,<span style="white-space:pre">  </span>/* Information about how to do the compare */</div><div>-<span style="white-space:pre">        </span>       UChar32 matchOther<span style="white-space:pre">        </span>/* The escape char (LIKE) or '[' (GLOB) */</div><div>-    )</div><div>+sql_utf8_pattern_compare(const char *pattern,</div><div>+<span style="white-space:pre">                       </span> const char *string,</div><div>+<span style="white-space:pre">                 </span> const struct compareInfo *pInfo,</div><div>+<span style="white-space:pre">                    </span> UChar32 matchOther)</div><div> {</div><div>-<span style="white-space:pre">       </span>UChar32 c, c2;<span style="white-space:pre">               </span>/* Next pattern and input string chars */</div><div>-<span style="white-space:pre">    </span>UChar32 matchOne = pInfo->matchOne;<span style="white-space:pre">       </span>/* "?" or "_" */</div><div>-<span style="white-space:pre"> </span>UChar32 matchAll = pInfo->matchAll;<span style="white-space:pre">       </span>/* "*" or "%" */</div><div>-<span style="white-space:pre"> </span>UChar32 noCase = pInfo->noCase;<span style="white-space:pre">   </span>/* True if uppercase==lowercase */</div><div>-<span style="white-space:pre">   </span>const char *zEscaped = 0;<span style="white-space:pre">    </span>/* One past the last escaped input char */</div><div>+<span style="white-space:pre">   </span>/* Next pattern and input string chars */</div><div>+<span style="white-space:pre">    </span>UChar32 c, c2;</div><div>+<span style="white-space:pre">       </span>/* "?" or "_" */</div><div>+<span style="white-space:pre"> </span>UChar32 matchOne = pInfo->matchOne;</div><div>+<span style="white-space:pre">       </span>/* "*" or "%" */</div><div>+<span style="white-space:pre"> </span>UChar32 matchAll = pInfo->matchAll;</div><div>+<span style="white-space:pre">       </span>/* True if uppercase==lowercase */</div><div>+<span style="white-space:pre">   </span>UChar32 noCase = pInfo->noCase;</div><div>+<span style="white-space:pre">   </span>/* One past the last escaped input char */</div><div>+<span style="white-space:pre">   </span>const char *zEscaped = 0;</div><div> <span style="white-space:pre">   </span>const char * pattern_end = pattern + strlen(pattern);</div><div> <span style="white-space:pre">       </span>const char * string_end = string + strlen(string);</div><div> <span style="white-space:pre">  </span>UErrorCode status = U_ZERO_ERROR;</div><div> </div><div>-<span style="white-space:pre">   </span>while (pattern < pattern_end){</div><div>-<span style="white-space:pre">            </span>c = Utf8Read(pattern, pattern_end);</div><div>+<span style="white-space:pre">  </span>while ((c = Utf8Read(pattern, pattern_end)) != SQL_END_OF_STRING) {</div><div>+<span style="white-space:pre">          </span>if (c == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                    </span>return SQL_INVALID_PATTERN;</div><div> <span style="white-space:pre">         </span>if (c == matchAll) {<span style="white-space:pre"> </span>/* Match "*" */</div><div>-<span style="white-space:pre">                    </span>/* Skip over multiple "*" characters in the pattern.  If there</div><div>-<span style="white-space:pre">                    </span> * are also "?" characters, skip those as well, but consume a</div><div>-<span style="white-space:pre">                      </span> * single character of the input string for each "?" skipped</div><div>+<span style="white-space:pre">                       </span>/* Skip over multiple "*" characters in</div><div>+<span style="white-space:pre">                    </span> * the pattern. If there are also "?"</div><div>+<span style="white-space:pre">                      </span> * characters, skip those as well, but</div><div>+<span style="white-space:pre">                       </span> * consume a single character of the</div><div>+<span style="white-space:pre">                 </span> * input string for each "?" skipped.</div><div> <span style="white-space:pre">                     </span> */</div><div>-<span style="white-space:pre">                  </span>while (pattern < pattern_end){</div><div>-<span style="white-space:pre">                            </span>c = Utf8Read(pattern, pattern_end);</div><div>+<span style="white-space:pre">                  </span>while ((c = Utf8Read(pattern, pattern_end)) !=</div><div>+<span style="white-space:pre">                       </span>       SQL_END_OF_STRING) {</div><div>+<span style="white-space:pre">                              </span>if (c == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                    </span>return SQL_INVALID_PATTERN;</div><div> <span style="white-space:pre">                         </span>if (c != matchAll && c != matchOne)</div><div> <span style="white-space:pre">                                 </span>break;</div><div>-<span style="white-space:pre">                               </span>if (c == matchOne</div><div>-<span style="white-space:pre">                            </span>    && Utf8Read(string, string_end) == 0) {</div><div>+<span style="white-space:pre">                                </span>if (c == matchOne &&</div><div>+<span style="white-space:pre">                         </span>    (c2 = Utf8Read(string, string_end)) ==</div><div>+<span style="white-space:pre">                         </span>    SQL_END_OF_STRING)</div><div> <span style="white-space:pre">                                    </span>return SQLITE_NOWILDCARDMATCH;</div><div>-<span style="white-space:pre">                               </span>}</div><div>+<span style="white-space:pre">                            </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                   </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">                      </span>}</div><div>-<span style="white-space:pre">                    </span>/* "*" at the end of the pattern matches */</div><div>-<span style="white-space:pre">                        </span>if (pattern == pattern_end)</div><div>+<span style="white-space:pre">                  </span>/*</div><div>+<span style="white-space:pre">                   </span> * "*" at the end of the pattern matches.</div><div>+<span style="white-space:pre">                  </span> */</div><div>+<span style="white-space:pre">                  </span>if (c == SQL_END_OF_STRING) {</div><div>+<span style="white-space:pre">                                </span>while ((c2 = Utf8Read(string, string_end)) !=</div><div>+<span style="white-space:pre">                                </span>       SQL_END_OF_STRING)</div><div>+<span style="white-space:pre">                                        </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                           </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">                              </span>return SQLITE_MATCH;</div><div>+<span style="white-space:pre">                 </span>}</div><div> <span style="white-space:pre">                   </span>if (c == matchOther) {</div><div> <span style="white-space:pre">                              </span>if (pInfo->matchSet == 0) {</div><div> <span style="white-space:pre">                                      </span>c = Utf8Read(pattern, pattern_end);</div><div>-<span style="white-space:pre">                                  </span>if (c == 0)</div><div>+<span style="white-space:pre">                                  </span>if (c == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                            </span>return SQL_INVALID_PATTERN;</div><div>+<span style="white-space:pre">                                  </span>if (c == SQL_END_OF_STRING)</div><div> <span style="white-space:pre">                                         </span>return SQLITE_NOWILDCARDMATCH;</div><div> <span style="white-space:pre">                              </span>} else {</div><div>-<span style="white-space:pre">                                     </span>/* "[...]" immediately follows the "*".  We have to do a slow</div><div>-<span style="white-space:pre">                                   </span> * recursive search in this case, but it is an unusual case.</div><div>+<span style="white-space:pre">                                 </span>/* "[...]" immediately</div><div>+<span style="white-space:pre">                                     </span> * follows the "*". We</div><div>+<span style="white-space:pre">                                     </span> * have to do a slow</div><div>+<span style="white-space:pre">                                 </span> * recursive search in</div><div>+<span style="white-space:pre">                                       </span> * this case, but it is</div><div>+<span style="white-space:pre">                                      </span> * an unusual case.</div><div> <span style="white-space:pre">                                 </span> */</div><div>-<span style="white-space:pre">                                  </span>assert(matchOther < 0x80);<span style="white-space:pre">        </span>/* '[' is a single-byte character */</div><div>-<span style="white-space:pre">                                 </span>while (string < string_end) {</div><div>+<span style="white-space:pre">                                     </span>assert(matchOther < 0x80);</div><div>+<span style="white-space:pre">                                        </span>while (c != SQL_END_OF_STRING) {</div><div> <span style="white-space:pre">                                            </span>int bMatch =</div><div>-<span style="white-space:pre">                                         </span>    patternCompare(&pattern[-1],</div><div>-<span style="white-space:pre">                                                               </span>   string,</div><div>-<span style="white-space:pre">                                                         </span>   pInfo,</div><div>-<span style="white-space:pre">                                                          </span>   matchOther);</div><div>+<span style="white-space:pre">                                            </span>    sql_utf8_pattern_compare(</div><div>+<span style="white-space:pre">                                                              </span>&pattern[-1],</div><div>+<span style="white-space:pre">                                                            </span>string,</div><div>+<span style="white-space:pre">                                                              </span>pInfo,</div><div>+<span style="white-space:pre">                                                               </span>matchOther);</div><div> <span style="white-space:pre">                                                </span>if (bMatch != SQLITE_NOMATCH)</div><div> <span style="white-space:pre">                                                       </span>return bMatch;</div><div>-<span style="white-space:pre">                                               </span>Utf8Read(string, string_end);</div><div>+<span style="white-space:pre">                                                </span>c = Utf8Read(string, string_end);</div><div>+<span style="white-space:pre">                                            </span>if (c == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                                    </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">                                      </span>}</div><div> <span style="white-space:pre">                                   </span>return SQLITE_NOWILDCARDMATCH;</div><div> <span style="white-space:pre">                              </span>}</div><div> <span style="white-space:pre">                   </span>}</div><div> </div><div>-<span style="white-space:pre">                   </span>/* At this point variable c contains the first character of the</div><div>-<span style="white-space:pre">                      </span> * pattern string past the "*".  Search in the input string for the</div><div>-<span style="white-space:pre">                       </span> * first matching character and recursively continue the match from</div><div>-<span style="white-space:pre">                  </span> * that point.</div><div>+<span style="white-space:pre">                       </span>/* At this point variable c contains the</div><div>+<span style="white-space:pre">                     </span> * first character of the pattern string</div><div>+<span style="white-space:pre">                     </span> * past the "*". Search in the input</div><div>+<span style="white-space:pre">                       </span> * string for the first matching</div><div>+<span style="white-space:pre">                     </span> * character and recursively continue the</div><div>+<span style="white-space:pre">                    </span> * match from that point.</div><div> <span style="white-space:pre">                   </span> *</div><div>-<span style="white-space:pre">                   </span> * For a case-insensitive search, set variable cx to be the same as</div><div>-<span style="white-space:pre">                  </span> * c but in the other case and search the input string for either</div><div>-<span style="white-space:pre">                    </span> * c or cx.</div><div>+<span style="white-space:pre">                  </span> * For a case-insensitive search, set</div><div>+<span style="white-space:pre">                        </span> * variable cx to be the same as c but in</div><div>+<span style="white-space:pre">                    </span> * the other case and search the input</div><div>+<span style="white-space:pre">                       </span> * string for either c or cx.</div><div> <span style="white-space:pre">                       </span> */</div><div> </div><div> <span style="white-space:pre">                        </span>int bMatch;</div><div>@@ -756,14 +800,18 @@ patternCompare(const char * pattern,<span style="white-space:pre"> </span>/* The glob pattern */</div><div> <span style="white-space:pre">                              </span>c = u_tolower(c);</div><div> <span style="white-space:pre">                   </span>while (string < string_end){</div><div> <span style="white-space:pre">                             </span>/**</div><div>-<span style="white-space:pre">                          </span> * This loop could have been implemented</div><div>-<span style="white-space:pre">                             </span> * without if converting c2 to lower case</div><div>-<span style="white-space:pre">                            </span> * (by holding c_upper and c_lower), however</div><div>-<span style="white-space:pre">                         </span> * it is implemented this way because lower</div><div>-<span style="white-space:pre">                          </span> * works better with German and Turkish</div><div>-<span style="white-space:pre">                              </span> * languages.</div><div>+<span style="white-space:pre">                                </span> * This loop could have been</div><div>+<span style="white-space:pre">                         </span> * implemented without if</div><div>+<span style="white-space:pre">                            </span> * converting c2 to lower case</div><div>+<span style="white-space:pre">                               </span> * by holding c_upper and</div><div>+<span style="white-space:pre">                            </span> * c_lower,however it is</div><div>+<span style="white-space:pre">                             </span> * implemented this way because</div><div>+<span style="white-space:pre">                              </span> * lower works better with German</div><div>+<span style="white-space:pre">                            </span> * and Turkish languages.</div><div> <span style="white-space:pre">                           </span> */</div><div> <span style="white-space:pre">                         </span>c2 = Utf8Read(string, string_end);</div><div>+<span style="white-space:pre">                           </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                   </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">                              </span>if (!noCase) {</div><div> <span style="white-space:pre">                                      </span>if (c2 != c)</div><div> <span style="white-space:pre">                                                </span>continue;</div><div>@@ -771,9 +819,10 @@ patternCompare(const char * pattern,<span style="white-space:pre">    </span>/* The glob pattern */</div><div> <span style="white-space:pre">                                      </span>if (c2 != c && u_tolower(c2) != c)</div><div> <span style="white-space:pre">                                          </span>continue;</div><div> <span style="white-space:pre">                           </span>}</div><div>-<span style="white-space:pre">                            </span>bMatch =</div><div>-<span style="white-space:pre">                             </span>    patternCompare(pattern, string,</div><div>-<span style="white-space:pre">                                                </span>   pInfo, matchOther);</div><div>+<span style="white-space:pre">                             </span>bMatch = sql_utf8_pattern_compare(pattern,</div><div>+<span style="white-space:pre">                                                           </span>  string,</div><div>+<span style="white-space:pre">                                                           </span>  pInfo,</div><div>+<span style="white-space:pre">                                                            </span>  matchOther);</div><div> <span style="white-space:pre">                             </span>if (bMatch != SQLITE_NOMATCH)</div><div> <span style="white-space:pre">                                       </span>return bMatch;</div><div> <span style="white-space:pre">                      </span>}</div><div>@@ -782,7 +831,9 @@ patternCompare(const char * pattern,<span style="white-space:pre">     </span>/* The glob pattern */</div><div> <span style="white-space:pre">              </span>if (c == matchOther) {</div><div> <span style="white-space:pre">                      </span>if (pInfo->matchSet == 0) {</div><div> <span style="white-space:pre">                              </span>c = Utf8Read(pattern, pattern_end);</div><div>-<span style="white-space:pre">                          </span>if (c == 0)</div><div>+<span style="white-space:pre">                          </span>if (c == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                    </span>return SQL_INVALID_PATTERN;</div><div>+<span style="white-space:pre">                          </span>if (c == SQL_END_OF_STRING)</div><div> <span style="white-space:pre">                                 </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">                              </span>zEscaped = pattern;</div><div> <span style="white-space:pre">                 </span>} else {</div><div>@@ -790,23 +841,33 @@ patternCompare(const char * pattern,<span style="white-space:pre">    </span>/* The glob pattern */</div><div> <span style="white-space:pre">                              </span>int seen = 0;</div><div> <span style="white-space:pre">                               </span>int invert = 0;</div><div> <span style="white-space:pre">                             </span>c = Utf8Read(string, string_end);</div><div>+<span style="white-space:pre">                            </span>if (c == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                    </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">                              </span>if (string == string_end)</div><div> <span style="white-space:pre">                                   </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">                              </span>c2 = Utf8Read(pattern, pattern_end);</div><div>+<span style="white-space:pre">                         </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                   </span>return SQL_INVALID_PATTERN;</div><div> <span style="white-space:pre">                         </span>if (c2 == '^') {</div><div> <span style="white-space:pre">                                    </span>invert = 1;</div><div> <span style="white-space:pre">                                 </span>c2 = Utf8Read(pattern, pattern_end);</div><div>+<span style="white-space:pre">                                 </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                           </span>return SQL_INVALID_PATTERN;</div><div> <span style="white-space:pre">                         </span>}</div><div> <span style="white-space:pre">                           </span>if (c2 == ']') {</div><div> <span style="white-space:pre">                                    </span>if (c == ']')</div><div> <span style="white-space:pre">                                               </span>seen = 1;</div><div> <span style="white-space:pre">                                   </span>c2 = Utf8Read(pattern, pattern_end);</div><div>+<span style="white-space:pre">                                 </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                           </span>return SQL_INVALID_PATTERN;</div><div> <span style="white-space:pre">                         </span>}</div><div>-<span style="white-space:pre">                            </span>while (c2 && c2 != ']') {</div><div>+<span style="white-space:pre">                            </span>while (c2 != SQL_END_OF_STRING && c2 != ']') {</div><div> <span style="white-space:pre">                                      </span>if (c2 == '-' && pattern[0] != ']'</div><div> <span style="white-space:pre">                                  </span>    && pattern < pattern_end</div><div> <span style="white-space:pre">                                   </span>    && prior_c > 0) {</div><div> <span style="white-space:pre">                                          </span>c2 = Utf8Read(pattern, pattern_end);</div><div>+<span style="white-space:pre">                                         </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                                   </span>return SQL_INVALID_PATTERN;</div><div> <span style="white-space:pre">                                         </span>if (c >= prior_c && c <= c2)</div><div> <span style="white-space:pre">                                                  </span>seen = 1;</div><div> <span style="white-space:pre">                                           </span>prior_c = 0;</div><div>@@ -817,29 +878,36 @@ patternCompare(const char * pattern,<span style="white-space:pre">        </span>/* The glob pattern */</div><div> <span style="white-space:pre">                                              </span>prior_c = c2;</div><div> <span style="white-space:pre">                                       </span>}</div><div> <span style="white-space:pre">                                   </span>c2 = Utf8Read(pattern, pattern_end);</div><div>+<span style="white-space:pre">                                 </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                                           </span>return SQL_INVALID_PATTERN;</div><div> <span style="white-space:pre">                         </span>}</div><div>-<span style="white-space:pre">                            </span>if (pattern == pattern_end || (seen ^ invert) == 0) {</div><div>+<span style="white-space:pre">                                </span>if (pattern == pattern_end ||</div><div>+<span style="white-space:pre">                                </span>    (seen ^ invert) == 0) {</div><div> <span style="white-space:pre">                                       </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">                              </span>}</div><div> <span style="white-space:pre">                           </span>continue;</div><div> <span style="white-space:pre">                   </span>}</div><div> <span style="white-space:pre">           </span>}</div><div> <span style="white-space:pre">           </span>c2 = Utf8Read(string, string_end);</div><div>+<span style="white-space:pre">           </span>if (c2 == SQL_INVALID_UTF8_SYMBOL)</div><div>+<span style="white-space:pre">                   </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">              </span>if (c == c2)</div><div> <span style="white-space:pre">                        </span>continue;</div><div> <span style="white-space:pre">           </span>if (noCase){</div><div> <span style="white-space:pre">                        </span>/**</div><div>-<span style="white-space:pre">                  </span> * Small optimisation. Reduce number of calls</div><div>-<span style="white-space:pre">                        </span> * to u_tolower function.</div><div>-<span style="white-space:pre">                    </span> * SQL standards suggest use to_upper for symbol</div><div>-<span style="white-space:pre">                     </span> * normalisation. However, using to_lower allows to</div><div>-<span style="white-space:pre">                  </span> * respect Turkish 'İ' in default locale.</div><div>+<span style="white-space:pre">                   </span> * Small optimisation. Reduce number of</div><div>+<span style="white-space:pre">                      </span> * calls to u_tolower function. SQL</div><div>+<span style="white-space:pre">                  </span> * standards suggest use to_upper for</div><div>+<span style="white-space:pre">                        </span> * symbol normalisation. However, using</div><div>+<span style="white-space:pre">                      </span> * to_lower allows to respect Turkish 'İ'</div><div>+<span style="white-space:pre">                   </span> * in default locale.</div><div> <span style="white-space:pre">                       </span> */</div><div> <span style="white-space:pre">                 </span>if (u_tolower(c) == c2 ||</div><div> <span style="white-space:pre">                   </span>    c == u_tolower(c2))</div><div> <span style="white-space:pre">                           </span>continue;</div><div> <span style="white-space:pre">           </span>}</div><div>-<span style="white-space:pre">            </span>if (c == matchOne && pattern != zEscaped && c2 != 0)</div><div>+<span style="white-space:pre">         </span>if (c == matchOne && pattern != zEscaped &&</div><div>+<span style="white-space:pre">          </span>    c2 != SQL_END_OF_STRING)</div><div> <span style="white-space:pre">                      </span>continue;</div><div> <span style="white-space:pre">           </span>return SQLITE_NOMATCH;</div><div> <span style="white-space:pre">      </span>}</div><div>@@ -853,8 +921,7 @@ patternCompare(const char * pattern,<span style="white-space:pre">     </span>/* The glob pattern */</div><div> int</div><div> sqlite3_strglob(const char *zGlobPattern, const char *zString)</div><div> {</div><div>-<span style="white-space:pre">  </span>return patternCompare(zGlobPattern, zString, &globInfo,</div><div>-<span style="white-space:pre">                  </span>      '[');</div><div>+<span style="white-space:pre">       </span>return sql_utf8_pattern_compare(zGlobPattern, zString, &globInfo, '[');</div><div> }</div><div> </div><div> /*</div><div>@@ -864,7 +931,7 @@ sqlite3_strglob(const char *zGlobPattern, const char *zString)</div><div> int</div><div> sqlite3_strlike(const char *zPattern, const char *zStr, unsigned int esc)</div><div> {</div><div>-<span style="white-space:pre">   </span>return patternCompare(zPattern, zStr, &likeInfoNorm, esc);</div><div>+<span style="white-space:pre">       </span>return sql_utf8_pattern_compare(zPattern, zStr, &likeInfoNorm, esc);</div><div> }</div><div> </div><div> /*</div><div>@@ -910,8 +977,9 @@ likeFunc(sqlite3_context * context, int argc, sqlite3_value ** argv)</div><div> <span style="white-space:pre">       </span>zB = (const char *) sqlite3_value_text(argv[0]);</div><div> <span style="white-space:pre">    </span>zA = (const char *) sqlite3_value_text(argv[1]);</div><div> </div><div>-<span style="white-space:pre">    </span>/* Limit the length of the LIKE or GLOB pattern to avoid problems</div><div>-<span style="white-space:pre">    </span> * of deep recursion and N*N behavior in patternCompare().</div><div>+<span style="white-space:pre">   </span>/* Limit the length of the LIKE or GLOB pattern to avoid</div><div>+<span style="white-space:pre">     </span> * problems of deep recursion and N*N behavior in</div><div>+<span style="white-space:pre">    </span> * sql_utf8_pattern_compare().</div><div> <span style="white-space:pre">      </span> */</div><div> <span style="white-space:pre"> </span>nPat = sqlite3_value_bytes(argv[0]);</div><div> <span style="white-space:pre">        </span>testcase(nPat == db->aLimit[SQLITE_LIMIT_LIKE_PATTERN_LENGTH]);</div><div>@@ -947,7 +1015,12 @@ likeFunc(sqlite3_context * context, int argc, sqlite3_value ** argv)</div><div> <span style="white-space:pre"> </span>sqlite3_like_count++;</div><div> #endif</div><div> <span style="white-space:pre">        </span>int res;</div><div>-<span style="white-space:pre">     </span>res = patternCompare(zB, zA, pInfo, escape);</div><div>+<span style="white-space:pre"> </span>res = sql_utf8_pattern_compare(zB, zA, pInfo, escape);</div><div>+<span style="white-space:pre">       </span>if (res == SQL_INVALID_PATTERN) {</div><div>+<span style="white-space:pre">            </span>sqlite3_result_error(context, "LIKE or GLOB pattern can only"</div><div>+<span style="white-space:pre">                              </span>     " contain UTF-8 characters", -1);</div><div>+<span style="white-space:pre">          </span>return;</div><div>+<span style="white-space:pre">      </span>}</div><div> <span style="white-space:pre">   </span>sqlite3_result_int(context, res == SQLITE_MATCH);</div><div> }</div><div> </div><div>diff --git a/test/sql-tap/e_expr.test.lua b/test/sql-tap/e_expr.test.lua</div><div>index 13d3a96..9780d2c 100755</div><div>--- a/test/sql-tap/e_expr.test.lua</div><div>+++ b/test/sql-tap/e_expr.test.lua</div><div>@@ -1,6 +1,6 @@</div><div> #!/usr/bin/env tarantool</div><div> test = require("sqltester")</div><div>-test:plan(12431)</div><div>+test:plan(10665)</div><div> </div><div> --!./tcltestrunner.lua</div><div> -- 2010 July 16</div><div>@@ -77,8 +77,10 @@ local operations = {</div><div>     {"<>", "ne1"},</div><div>     {"!=", "ne2"},</div><div>     {"IS", "is"},</div><div>-    {"LIKE", "like"},</div><div>-    {"GLOB", "glob"},</div><div>+-- NOTE: This test needs refactoring after deletion of GLOB &</div><div>+--<span style="white-space:pre">        </span> type restrictions for LIKE. (See #3572)</div><div>+--    {"LIKE", "like"},</div><div>+--    {"GLOB", "glob"},</div><div>     {"AND", "and"},</div><div>     {"OR", "or"},</div><div>     {"MATCH", "match"},</div><div>@@ -96,7 +98,12 @@ operations = {</div><div>     {"+", "-"},</div><div>     {"<<", ">>", "&", "|"},</div><div>     {"<", "<=", ">", ">="},</div><div>-    {"=", "==", "!=", "<>", "LIKE", "GLOB"}, --"MATCH", "REGEXP"},</div><div>+-- NOTE: This test needs refactoring after deletion of GLOB &</div><div>+--<span style="white-space:pre">        </span> type restrictions for LIKE. (See #3572)</div><div>+-- Another NOTE: MATCH & REGEXP aren't supported in Tarantool &</div><div>+-- <span style="white-space:pre">               </span> are waiting for their hour, don't confuse them</div><div>+--<span style="white-space:pre">                </span> being commented with ticket above.</div><div>+    {"=", "==", "!=", "<>"}, --"LIKE", "GLOB"}, --"MATCH", "REGEXP"},</div><div>     {"AND"},</div><div>     {"OR"},</div><div> }</div><div>@@ -475,6 +482,7 @@ for _, op in ipairs(oplist) do</div><div>         end</div><div>     end</div><div> end</div><div>+</div><div> ---------------------------------------------------------------------------</div><div> -- Test the IS and IS NOT operators.</div><div> --</div><div>diff --git a/test/sql-tap/gh-3251-string-pattern-comparison.test.lua b/test/sql-tap/gh-3251-string-pattern-comparison.test.lua</div><div>new file mode 100755</div><div>index 0000000..addf0e3</div><div>--- /dev/null</div><div>+++ b/test/sql-tap/gh-3251-string-pattern-comparison.test.lua</div><div>@@ -0,0 +1,213 @@</div><div>+#!/usr/bin/env tarantool</div><div>+test = require("sqltester")</div><div>+test:plan(128)</div><div>+</div><div>+local prefix = "like-test-"</div><div>+</div><div>+local like_test_cases =</div><div>+{</div><div>+    {"1.1",</div><div>+        "SELECT 'AB' LIKE '_B';",</div><div>+        {0, {1}} },</div><div>+    {"1.2",</div><div>+        "SELECT 'CD' LIKE '_B';",</div><div>+        {0, {0}} },</div><div>+    {"1.3",</div><div>+        "SELECT '' LIKE '_B';",</div><div>+        {0, {0}} },</div><div>+    {"1.4",</div><div>+        "SELECT 'AB' LIKE '%B';",</div><div>+        {0, {1}} },</div><div>+    {"1.5",</div><div>+        "SELECT 'CD' LIKE '%B';",</div><div>+        {0, {0}} },</div><div>+    {"1.6",</div><div>+        "SELECT '' LIKE '%B';",</div><div>+        {0, {0}} },</div><div>+    {"1.7",</div><div>+        "SELECT 'AB' LIKE 'A__';",</div><div>+        {0, {0}} },</div><div>+    {"1.8",</div><div>+        "SELECT 'CD' LIKE 'A__';",</div><div>+        {0, {0}} },</div><div>+    {"1.9",</div><div>+        "SELECT '' LIKE 'A__';",</div><div>+        {0, {0}} },</div><div>+    {"1.10",</div><div>+        "SELECT 'AB' LIKE 'A_';",</div><div>+        {0, {1}} },</div><div>+    {"1.11",</div><div>+        "SELECT 'CD' LIKE 'A_';",</div><div>+        {0, {0}} },</div><div>+    {"1.12",</div><div>+        "SELECT '' LIKE 'A_';",</div><div>+        {0, {0}} },</div><div>+    {"1.13",</div><div>+        "SELECT 'AB' LIKE 'A';",</div><div>+        {0, {0}} },</div><div>+    {"1.14",</div><div>+        "SELECT 'CD' LIKE 'A';",</div><div>+        {0, {0}} },</div><div>+    {"1.15",</div><div>+        "SELECT '' LIKE 'A';",</div><div>+        {0, {0}} },</div><div>+    {"1.16",</div><div>+        "SELECT 'AB' LIKE '_';",</div><div>+        {0, {0}} },</div><div>+    {"1.17",</div><div>+        "SELECT 'CD' LIKE '_';",</div><div>+        {0, {0}} },</div><div>+    {"1.18",</div><div>+        "SELECT '' LIKE '_';",</div><div>+        {0, {0}} },</div><div>+    {"1.19",</div><div>+        "SELECT 'AB' LIKE '__';",</div><div>+        {0, {1}} },</div><div>+    {"1.20",</div><div>+        "SELECT 'CD' LIKE '__';",</div><div>+        {0, {1}} },</div><div>+    {"1.21",</div><div>+        "SELECT '' LIKE '__';",</div><div>+        {0, {0}} },</div><div>+    {"1.22",</div><div>+        "SELECT 'AB' LIKE '%A';",</div><div>+        {0, {0}} },</div><div>+    {"1.23",</div><div>+        "SELECT 'AB' LIKE '%C';",</div><div>+        {0, {0}} },</div><div>+    {"1.24",</div><div>+        "SELECT 'ёф' LIKE '%œش';",</div><div>+        {0, {0}} },</div><div>+    {"1.25",</div><div>+        "SELECT 'ёфÅŒش' LIKE '%œش';",</div><div>+        {0, {1}} },</div><div>+    {"1.26",</div><div>+        "SELECT 'ÅŒش' LIKE '%œش';",</div><div>+        {0, {1}} },</div><div>+    {"1.27",</div><div>+        "SELECT 'ёф' LIKE 'ё_';",</div><div>+        {0, {1}} },</div><div>+    {"1.28",</div><div>+        "SELECT 'ёфÅŒش' LIKE 'ё_';",</div><div>+        {0, {0}} },</div><div>+    {"1.29",</div><div>+        "SELECT 'ÅŒش' LIKE 'ё_';",</div><div>+        {0, {0}} },</div><div>+    {"1.30",</div><div>+        "SELECT 'ёф' LIKE 'ёф%';",</div><div>+        {0, {1}} },</div><div>+    {"1.31",</div><div>+        "SELECT 'ёфÅŒش' LIKE 'ёф%';",</div><div>+        {0, {1}} },</div><div>+    {"1.32",</div><div>+        "SELECT 'ÅŒش' LIKE 'ёф%';",</div><div>+        {0, {0}} },</div><div>+    {"1.33",</div><div>+        "SELECT 'ёф' LIKE 'ёфÅ%';",</div><div>+        {0, {0}} },</div><div>+    {"1.34",</div><div>+        "SELECT 'ёфÅŒش' LIKE 'ёфÅ%';",</div><div>+        {0, {1}} },</div><div>+    {"1.35",</div><div>+        "SELECT 'ÅŒش' LIKE 'ёфش%';",</div><div>+        {0, {0}} },</div><div>+    {"1.36",</div><div>+        "SELECT 'ёф' LIKE 'ё_%';",</div><div>+        {0, {1}} },</div><div>+    {"1.37",</div><div>+        "SELECT 'ёфÅŒش' LIKE 'ё_%';",</div><div>+        {0, {1}} },</div><div>+    {"1.38",</div><div>+        "SELECT 'ÅŒش' LIKE 'ё_%';",</div><div>+        {0, {0}} },</div><div>+}</div><div>+</div><div>+test:do_catchsql_set_test(like_test_cases, prefix)</div><div>+</div><div>+-- Non-Unicode byte sequences.</div><div>+local invalid_testcases = {</div><div>+    '\xE2\x80',</div><div>+    '\xFE\xFF',</div><div>+    '\xC2',</div><div>+    '\xED\xB0\x80',</div><div>+    '\xD0',</div><div>+}</div><div>+</div><div>+-- Invalid testcases.</div><div>+for i, tested_string in ipairs(invalid_testcases) do</div><div>+</div><div>+    -- We should raise an error in case</div><div>+    -- pattern contains invalid characters.</div><div>+</div><div>+    local test_name = prefix .. "2." .. tostring(i)</div><div>+    local test_itself = "SELECT 'abc' LIKE 'ab" .. tested_string .. "';"</div><div>+    test:do_catchsql_test(test_name, test_itself,</div><div>+                          {1, "LIKE or GLOB pattern can only contain UTF-8 characters"})</div><div>+</div><div>+    test_name = prefix .. "3." .. tostring(i)</div><div>+    test_itself = "SELECT 'abc' LIKE 'abc" .. tested_string .. "';"</div><div>+    test:do_catchsql_test(test_name, test_itself,</div><div>+                          {1, "LIKE or GLOB pattern can only contain UTF-8 characters"})</div><div>+</div><div>+    test_name = prefix .. "4." .. tostring(i)</div><div>+    test_itself = "SELECT 'abc' LIKE 'ab" .. tested_string .. "c';"</div><div>+    test:do_catchsql_test(test_name, test_itself,</div><div>+                          {1, "LIKE or GLOB pattern can only contain UTF-8 characters"})</div><div>+</div><div>+    -- Just skipping if row value predicand contains invalid character.</div><div>+</div><div>+    test_name = prefix .. "5." .. tostring(i)</div><div>+    test_itself = "SELECT 'ab" .. tested_string .. "' LIKE 'abc';"</div><div>+    test:do_execsql_test(test_name, test_itself, {0})</div><div>+</div><div>+    test_name = prefix .. "6." .. tostring(i)</div><div>+    test_itself = "SELECT 'abc" .. tested_string .. "' LIKE 'abc';"</div><div>+    test:do_execsql_test(test_name, test_itself, {0})</div><div>+</div><div>+    test_name = prefix .. "7." .. tostring(i)</div><div>+    test_itself = "SELECT 'ab" .. tested_string .. "c' LIKE 'abc';"</div><div>+    test:do_execsql_test(test_name, test_itself, {0})</div><div>+end</div><div>+</div><div>+-- Unicode byte sequences.</div><div>+local valid_testcases = {</div><div>+    '\x01',</div><div>+    '\x09',</div><div>+    '\x1F',</div><div>+    '\x7F',</div><div>+    '\xC2\x80',</div><div>+    '\xC2\x90',</div><div>+    '\xC2\x9F',</div><div>+    '\xE2\x80\xA8',</div><div>+    '\x20\x0B',</div><div>+    '\xE2\x80\xA9',</div><div>+}</div><div>+</div><div>+-- Valid testcases.</div><div>+for i, tested_string in ipairs(valid_testcases) do</div><div>+    test_name = prefix .. "8." .. tostring(i)</div><div>+    local test_itself = "SELECT 'abc' LIKE 'ab" .. tested_string .. "';"</div><div>+    test:do_execsql_test(test_name, test_itself, {0})</div><div>+</div><div>+    test_name = prefix .. "9." .. tostring(i)</div><div>+    test_itself = "SELECT 'abc' LIKE 'abc" .. tested_string .. "';"</div><div>+    test:do_execsql_test(test_name, test_itself, {0})</div><div>+</div><div>+    test_name = prefix .. "10." .. tostring(i)</div><div>+    test_itself = "SELECT 'abc' LIKE 'ab" .. tested_string .. "c';"</div><div>+    test:do_execsql_test(test_name,<span style="white-space:pre">        </span>test_itself, {0})</div><div>+</div><div>+    test_name = prefix .. "11." .. tostring(i)</div><div>+    test_itself = "SELECT 'ab" .. tested_string .. "' LIKE 'abc';"</div><div>+    test:do_execsql_test(test_name,<span style="white-space:pre">     </span>test_itself, {0})</div><div>+</div><div>+    test_name = prefix .. "12." .. tostring(i)</div><div>+    test_itself = "SELECT 'abc" .. tested_string .. "' LIKE 'abc';"</div><div>+    test:do_execsql_test(test_name, test_itself, {0})</div><div>+</div><div>+    test_name = prefix .. "13." .. tostring(i)</div><div>+    test_itself = "SELECT 'ab" .. tested_string .. "c' LIKE 'abc';"</div><div>+    test:do_execsql_test(test_name, test_itself, {0})</div><div>+end</div><div>+</div><div>+test:finish_test()</div></div><div><br></div></div>