<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 11 Sep 2018, at 16:31, Nikita Tatunov <<a href="mailto:n.tatunov@tarantool.org" class="">n.tatunov@tarantool.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="Apple-interchange-newline"><br class=""><blockquote type="cite" class=""><div class="">On 11 Sep 2018, at 13:06, Alex Khatskevich <<a href="mailto:avkhatskevich@tarantool.org" class="">avkhatskevich@tarantool.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div text="#000000" bgcolor="#FFFFFF" class=""><p class=""><br class=""></p><br class=""><div class="moz-cite-prefix">On 11.09.2018 09:06, Nikita Tatunov wrote:<br class=""></div><blockquote type="cite" cite="mid:58B407E2-AF5D-4531-A9FF-9DC57CE0070B@tarantool.org" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 11 Sep 2018, at 01:20, Alex Khatskevich <<a href="mailto:avkhatskevich@tarantool.org" class="" moz-do-not-send="true">avkhatskevich@tarantool.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><blockquote type="cite" cite="mid:87897608-173E-45EB-80A1-8B249706D8A1@tarantool.org" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration: none;"><div class=""><br class="Apple-interchange-newline"><br class=""><blockquote type="cite" class=""><div class="">On 17 Aug 2018, at 14:42, Alex Khatskevich <<a href="mailto:avkhatskevich@tarantool.org" class="" moz-do-not-send="true">avkhatskevich@tarantool.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">On 17.08.2018 14:17, Alexander Turenko wrote:</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">0xffff is the result of 'end of a string' check as well as internal buffer<br class="">overflow error. I have the relevant code pasted in the first review of<br class="">the patch (July, 18).<br class=""><br class="">// source/common/ucnv.c::ucnv_getNextUChar<br class="">1860 s=*source;<br class="">1861 if(sourceLimit<s) {<br class="">1862 *err=U_ILLEGAL_ARGUMENT_ERROR;<br class="">1863 return 0xffff;<br class="">1864 }<br class=""><br class="">We should not handle the buffer overflow case as an invalid symbol. Of<br class="">course we should not handle it as the 'end of the string' situation.<br class="">Ideally we should perform pointer myself and raise an error in case of<br class="">0xffff. I had thought that a buffer overflow error is unlikely to meet,<br class="">but you are right: we should differentiate these situations.<br class=""><br class="">In one of the previous version of a patch we perform this check like so:<br class=""><br class="">#define Utf8Read(s, e) (((s) < (e)) ?\<br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>ucnv_getNextUChar(pUtf8conv, &s, e, &status) : 0)<br class=""><br class="">Don't sure why it was changed. Maybe it is try to correctly handle '\0'<br class="">symbol (it is valid unicode character)?<br class=""></blockquote><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">The define you have pasted can return 0xffff.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">The reasons to change it back are described in the previous patchset.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">In short:</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">1. It is equivalent to</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"> <span class="Apple-converted-space"> </span>a. check s < e in a while loop</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"> <span class="Apple-converted-space"> </span>b. read next character inside of where loop body.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">2. In some usages of the code this check (s<e) was redundant (it was performed a couple lines above)</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">3. There is no reason to rewrite the old version of this function. (So, we decided to use old version of the function)</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">So I see two ways to proceed:<br class=""><br class="">1. Lean on icu's check and ignore possibility of the buffer overflow.<br class="">2. Use our own check and possibly meet '\0' problems.<br class="">3. Check for U_ILLEGAL_ARGUMENT_ERROR to treat as end of a string, raise<br class=""> the error for other 0xffff.<br class=""><br class="">Alex, what do you suggests here?<br class=""></blockquote><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">As I understand, by now the 0xffff is used ONLY to handle the case of unexpectedly ended symbol.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">E.g. some symbol consists of 2 characters, but the length of the input buffer is 1.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">In my opinion this is the same as an invalid symbol.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">I guess that internal buffer overflow cannot occur in the `ucnv_getNextChar` function.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">I suppose that it is Nikitas duty to investigate this problem and explain it to us all. I just have noticed a strange usage.</span></div></blockquote></div><div class=""><br class=""></div><div class="">Hello, please consider my comments.</div><div class=""><br class=""></div><div class="">There are some cases when 0xffff can occur, but:</div><div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>1) <span class="" style="font-family: HelveticaNeue;">Cannot trigger in our context.</span></div><div class=""><span class="" style="font-family: HelveticaNeue;"><span class="Apple-tab-span" style="white-space: pre;"> </span>2) C</span><span class="" style="font-family: HelveticaNeue;">annot trigger in our context.</span></div><div class=""><span class="" style="font-family: HelveticaNeue;"><span class="Apple-tab-span" style="white-space: pre;"> </span>3) O</span><span class="" style="font-family: HelveticaNeue;">nly triggers if end < start. (Cannot happen in sql_utf8_pattern_compare, i guess)</span></div><div class=""><span class="" style="font-family: HelveticaNeue;"><span class="Apple-tab-span" style="white-space: pre;"> </span>4) O</span><span class="" style="font-family: HelveticaNeue;">nly triggers if string length > (size_t) 0x7ffffffff (can it actually happen? I don’t think so).</span></div><div class=""><span class="" style="font-family: HelveticaNeue;"><span class="Apple-tab-span" style="white-space: pre;"> </span>5) O</span><span class="" style="font-family: HelveticaNeue;">ccurs when trying to access to not unindexed data.</span></div><div class=""><span class="" style="font-family: HelveticaNeue;"><span class="Apple-tab-span" style="white-space: pre;"> </span>6) Cannot occur in our context.</span></div><div class=""><span class="" style="font-family: HelveticaNeue;"><span class="Apple-tab-span" style="white-space: pre;"> </span>7) </span><span class="" style="font-family: HelveticaNeue;">Cannot occur in our context.</span></div></blockquote><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration: none; float: none; display: inline !important;">I do not understand what are those numbers related to. Please, describe it.</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration: none;"></div></blockquote><div class=""><br class=""></div><div class="">They are related to possible cases returning 0xffff from icu source code (function ucnv_getNextUChar()).</div></div></blockquote>Can you just copy it here, so that anyone interested in that conversation can<br class="">analyze it without looking for source files?<br class=""></div></div></blockquote><br class=""></div></div></blockquote><div><br class=""></div><div>Hello Alexander! Could you please consider some following explanations?</div><div>I think we can treat 0xffff as a string end in sql_utf8_pattern_compare.</div><div>Though we can use `status` for differentiating different errors from</div><div>`ucnv_getNextUChar(pUtf8conv, &s, e, &status)`. I would like to know your opinion.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Ok then:</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class=""></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class="">U_CAPI UChar32 U_EXPORT2</div><div class="">ucnv_getNextUChar(UConverter *cnv,</div><div class=""> const char **source, const char *sourceLimit,</div><div class=""> UErrorCode *err) {</div><div class=""> UConverterToUnicodeArgs args;</div><div class=""> UChar buffer[U16_MAX_LENGTH];</div><div class=""> const char *s;</div><div class=""> UChar32 c;</div><div class=""> int32_t i, length;</div><div class=""><br class=""></div><div class=""> /* check parameters */</div><div class=""> if(err==NULL || U_FAILURE(*err)) {</div><div class=""> return 0xffff;</div><div class=""> }</div><div class=""><br class=""></div></div></div></blockquote><div><br class=""></div><div><div>1. This one cannot trigger in sql_utf8_pattern_compare():</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>1) err == &status</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>2) *err == U_ZERO_ERROR</div></div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""> if(cnv==NULL || source==NULL) {</div><div class=""> *err=U_ILLEGAL_ARGUMENT_ERROR;</div><div class=""> return 0xffff;</div><div class=""> }</div><div class=""><br class=""></div></div></div></blockquote><div><br class=""></div><div><div>2. This one cannot trigger in sql_utf8_pattern_compare():</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>1) cnv != NULL</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>2) source != NULL</div></div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""> s=*source;</div><div class=""> if(sourceLimit<s) {</div><div class=""> *err=U_ILLEGAL_ARGUMENT_ERROR;</div><div class=""> return 0xffff;</div></div></div></blockquote><div><br class=""></div><div><div>3. This one cannot trigger in sql_utf8_pattern_compare():</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>1) ucnv_getNextUChar is only called when !(sourceLimit<s).</div></div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""> }</div><div class=""><br class=""></div><div class=""> /*</div><div class=""> * Make sure that the buffer sizes do not exceed the number range for</div><div class=""> * int32_t because some functions use the size (in units or bytes)</div><div class=""> * rather than comparing pointers, and because offsets are int32_t values.</div><div class=""> *</div><div class=""> * size_t is guaranteed to be unsigned and large enough for the job.</div><div class=""> *</div><div class=""> * Return with an error instead of adjusting the limits because we would</div><div class=""> * not be able to maintain the semantics that either the source must be</div><div class=""> * consumed or the target filled (unless an error occurs).</div><div class=""> * An adjustment would be sourceLimit=t+0x7fffffff; for example.</div><div class=""> */</div><div class=""> if(((size_t)(sourceLimit-s)>(size_t)0x7fffffff && sourceLimit>s)) {</div><div class=""> *err=U_ILLEGAL_ARGUMENT_ERROR;</div><div class=""> return 0xffff;</div></div></div></blockquote><div><br class=""></div><div>4. I’m not sure if string data can be this long in our context. </div><div> (<span style="font-family: HelveticaNeue;" class="">string length > (size_t) 0x7ffffffff</span>)</div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""> }</div><div class=""><br class=""></div><div class=""> c=U_SENTINEL;</div><div class=""><br class=""></div><div class=""> /* flush the target overflow buffer */</div><div class=""> if(cnv->UCharErrorBufferLength>0) {</div><div class=""> UChar *overflow;</div><div class=""><br class=""></div><div class=""> overflow=cnv->UCharErrorBuffer;</div><div class=""> i=0;</div><div class=""> length=cnv->UCharErrorBufferLength;</div><div class=""> U16_NEXT(overflow, i, length, c);</div><div class=""><br class=""></div><div class=""> /* move the remaining overflow contents up to the beginning */</div><div class=""> if((cnv->UCharErrorBufferLength=(int8_t)(length-i))>0) {</div><div class=""> uprv_memmove(cnv->UCharErrorBuffer, cnv->UCharErrorBuffer+i,</div><div class=""> cnv->UCharErrorBufferLength*U_SIZEOF_UCHAR);</div><div class=""> }</div><div class=""><br class=""></div><div class=""> if(!U16_IS_LEAD(c) || i<length) {</div><div class=""> return c;</div><div class=""> }</div><div class=""> /*</div><div class=""> * Continue if the overflow buffer contained only a lead surrogate,</div><div class=""> * in case the converter outputs single surrogates from complete</div><div class=""> * input sequences.</div><div class=""> */</div><div class=""> }</div><div class=""><br class=""></div><div class=""> /*</div><div class=""> * flush==TRUE is implied for ucnv_getNextUChar()</div><div class=""> *</div><div class=""> * do not simply return even if s==sourceLimit because the converter may</div><div class=""> * not have seen flush==TRUE before</div><div class=""> */</div><div class=""><br class=""></div><div class=""> /* prepare the converter arguments */</div><div class=""> args.converter=cnv;</div><div class=""> args.flush=TRUE;</div><div class=""> args.offsets=NULL;</div><div class=""> args.source=s;</div><div class=""> args.sourceLimit=sourceLimit;</div><div class=""> args.target=buffer;</div><div class=""> args.targetLimit=buffer+1;</div><div class=""> args.size=sizeof(args);</div><div class=""><br class=""></div></div></div></blockquote><div><br class=""></div><div>c == U_SENTINEL == -1</div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""> if(c<0) {</div><div class=""> /*</div><div class=""> * call the native getNextUChar() implementation if we are</div><div class=""> * at a character boundary (toULength==0)</div><div class=""> *</div><div class=""> * unlike with _toUnicode(), getNextUChar() implementations must set</div><div class=""> * U_TRUNCATED_CHAR_FOUND for truncated input,</div><div class=""> * in addition to setting toULength/toUBytes[]</div><div class=""> */</div><div class=""> if(cnv->toULength==0 && cnv->sharedData->impl->getNextUChar!=NULL) {</div><div class=""> c=cnv->sharedData->impl->getNextUChar(&args, err);</div><div class=""> *source=s=args.source;</div><div class=""> if(*err==U_INDEX_OUTOFBOUNDS_ERROR) {</div><div class=""> /* reset the converter without calling the callback function */</div><div class=""> _reset(cnv, UCNV_RESET_TO_UNICODE, FALSE);</div><div class=""> return 0xffff; /* no output */</div></div></div></blockquote><div><br class=""></div><div>5. Occurs when trying to access unindexed data.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""> } else if(U_SUCCESS(*err) && c>=0) {</div><div class=""> return c;</div></div></div></blockquote><div><br class=""></div><div>6. Returns symbol (can also be 0xfffd, as it is not treated as an actual error).</div><div><br class=""></div><div>So if I’m not mistaken we will get results in our function either from</div><div>‘return’ number 5 or number 6 and the following code will not be executed.</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class=""> /*</div><div class=""> * else fall through to use _toUnicode() because</div><div class=""> * UCNV_GET_NEXT_UCHAR_USE_TO_U: the native function did not want to handle it after all</div><div class=""> * U_FAILURE: call _toUnicode() for callback handling (do not output c)</div><div class=""> */</div><div class=""> }</div><div class=""> }</div><div class=""><br class=""></div><div class=""> /* convert to one UChar in buffer[0], or handle getNextUChar() errors */</div><div class=""> _toUnicodeWithCallback(&args, err);</div><div class=""><br class=""></div><div class=""> if(*err==U_BUFFER_OVERFLOW_ERROR) {</div><div class=""> *err=U_ZERO_ERROR;</div><div class=""> }</div><div class=""><br class=""></div><div class=""> i=0;</div><div class=""> length=(int32_t)(args.target-buffer);</div><div class=""> } else {</div><div class=""> /* write the lead surrogate from the overflow buffer */</div><div class=""> buffer[0]=(UChar)c;</div><div class=""> args.target=buffer+1;</div><div class=""> i=0;</div><div class=""> length=1;</div><div class=""> }</div><div class=""><br class=""></div><div class=""> /* buffer contents starts at i and ends before length */</div><div class=""><br class=""></div><div class=""> if(U_FAILURE(*err)) {</div><div class=""> c=0xffff; /* no output */</div><div class=""> } else if(length==0) {</div><div class=""> /* no input or only state changes */</div><div class=""> *err=U_INDEX_OUTOFBOUNDS_ERROR;</div><div class=""> /* no need to reset explicitly because _toUnicodeWithCallback() did it */</div><div class=""> c=0xffff; /* no output */</div><div class=""> } else {</div><div class=""> c=buffer[0];</div><div class=""> i=1;</div><div class=""> if(!U16_IS_LEAD(c)) {</div><div class=""> /* consume c=buffer[0], done */</div><div class=""> } else {</div><div class=""> /* got a lead surrogate, see if a trail surrogate follows */</div><div class=""> UChar c2;</div><div class=""><br class=""></div><div class=""> if(cnv->UCharErrorBufferLength>0) {</div><div class=""> /* got overflow output from the conversion */</div><div class=""> if(U16_IS_TRAIL(c2=cnv->UCharErrorBuffer[0])) {</div><div class=""> /* got a trail surrogate, too */</div><div class=""> c=U16_GET_SUPPLEMENTARY(c, c2);</div><div class=""><br class=""></div><div class=""> /* move the remaining overflow contents up to the beginning */</div><div class=""> if((--cnv->UCharErrorBufferLength)>0) {</div><div class=""> uprv_memmove(cnv->UCharErrorBuffer, cnv->UCharErrorBuffer+1,</div><div class=""> cnv->UCharErrorBufferLength*U_SIZEOF_UCHAR);</div><div class=""> }</div><div class=""> } else {</div><div class=""> /* c is an unpaired lead surrogate, just return it */</div><div class=""> }</div><div class=""> } else if(args.source<sourceLimit) {</div><div class=""> /* convert once more, to buffer[1] */</div><div class=""> args.targetLimit=buffer+2;</div><div class=""> _toUnicodeWithCallback(&args, err);</div><div class=""> if(*err==U_BUFFER_OVERFLOW_ERROR) {</div><div class=""> *err=U_ZERO_ERROR;</div><div class=""> }</div><div class=""><br class=""></div><div class=""> length=(int32_t)(args.target-buffer);</div><div class=""> if(U_SUCCESS(*err) && length==2 && U16_IS_TRAIL(c2=buffer[1])) {</div><div class=""> /* got a trail surrogate, too */</div><div class=""> c=U16_GET_SUPPLEMENTARY(c, c2);</div><div class=""> i=2;</div><div class=""> }</div><div class=""> }</div><div class=""> }</div><div class=""> }</div><div class=""><br class=""></div><div class=""> /*</div><div class=""> * move leftover output from buffer[i..length[</div><div class=""> * into the beginning of the overflow buffer</div><div class=""> */</div><div class=""> if(i<length) {</div><div class=""> /* move further overflow back */</div><div class=""> int32_t delta=length-i;</div><div class=""> if((length=cnv->UCharErrorBufferLength)>0) {</div><div class=""> uprv_memmove(cnv->UCharErrorBuffer+delta, cnv->UCharErrorBuffer,</div><div class=""> length*U_SIZEOF_UCHAR);</div><div class=""> }</div><div class=""> cnv->UCharErrorBufferLength=(int8_t)(length+delta);</div><div class=""><br class=""></div><div class=""> cnv->UCharErrorBuffer[0]=buffer[i++];</div><div class=""> if(delta>1) {</div><div class=""> cnv->UCharErrorBuffer[1]=buffer[i];</div><div class=""> }</div><div class=""> }</div><div class=""><br class=""></div><div class=""> *source=args.source;</div><div class=""> return c;</div><div class="">}</div></div><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div dir="auto" class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">--</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">WBR, Nikita Tatunov.</div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="mailto:n.tatunov@tarantool.org" class="">n.tatunov@tarantool.org</a></div></div></div></div></blockquote></div><br class=""><div class="">
<div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">--</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">WBR, Nikita Tatunov.</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><a href="mailto:n.tatunov@tarantool.org" class="">n.tatunov@tarantool.org</a></div></div>
</div>
<br class=""></div></body></html>