Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts

[Igor Munkin via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

Sasha,

Thanks a lot for your review!

On 15.03.21, Alexander Turenko wrote:
> I looked over the WIP version of the script and nothing confuses me:
> 
> https://lists.tarantool.org/tarantool-patches/20210220104630.wiwxhkjfimsx2swa@tkn_work_nb/
> 
> So I guess I should do some basic testing to get and share my user
> experience.

Nice, then I'll go through your user experience and adjust the script
considering the problems you're facing.

> 
> So I'll step into another environment to create coredump and archive it:

<snipped the manipulations with installing and running Tarantool>

> 
>  | tarantool> require('ffi').new('int[?]', 1)[1024 ^ 3]
>  | ~ <...>
>  | ~ Aborted (core dumped)
>  | # ls -hl core
>  | ~ -rw------- 1 root root 180M Mar 15 12:49 core
>  | # file core
>  | ~ bash: file: command not found
> 
> (I'll ignore that 'file' is not installed, because it is possible
> situation on a production server, I guess.)

I didn't figure out, whether <file> package is required, recommended or
optional. So if your Docker image doesn't have it, then the production
server can also lack this package.

> 
> Okay, now I got the tarabrt.sh script somehow:
> 
>  | # curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/tarabrt.sh >tarabrt.sh
>  | # chmod a+x tarabrt.sh
>  | (Don't read the content, just run.)
>  | # ./tarabrt.sh --help
>  | ~ tarabrt.sh: unrecognized option '--help'

Unfortunately, I have nothing to do here: this is returned by <getopt>.

>  | # ./tarabrt.sh -h  
>  | ~ <..skipped..>
>  | ~   - Manual usage. User can simply pack all necessary artefacts by running the
>  | ~     following command.
>  | ~     $ /path/to/tarabrt.sh -c ./core -d /tmp
>  | ~ <..skipped..>
>  | # ./tarabrt.sh -c ./core -d /tmp
>  | ~ ./tarabrt.sh: line 115: file: command not found
>  | ~ gdb is not installed, but it is obligatory for collecting the
>  | ~ loaded shared libraries from the core dump.
>  | ~ 
>  | ~ You can proceed collecting the artefacts manually later by running
>  | ~ the following command:
>  | ~ $ tarabrt.sh -e /usr/bin/tarantool -c ./core
>  | (Okay, I'll install file and gdb now.)

I'll added the check that file is installed. Diff is below:

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 372ff80c1..005add707 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -112,6 +112,17 @@ NOCOREDUMP
 	exit 1;
 fi
 
+if ! command -v file >/dev/null; then
+	[ -t 1 ] && cat <<NOFILE
+file is not installed, but it is obligatory for checking the
+artefacts to be collected.
+
+You can proceed collecting the artefacts manually later by running
+the following command:
+$ ${TOOL} -e ${BINARY} -c ${COREFILE}
+NOFILE
+fi
+
 if file "${COREFILE}" | grep -qv 'core file'; then
 	[ -t 1 ] && cat <<NOTACOREDUMP
 Not a core dump: ${COREFILE}

================================================================================

>  | # yum install -y file gdb
>  | # file core
>  | ~ core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'tarantool', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/tarantool', platform: 'x86_64'
>  | (Repeat the attempt to create the archive.)
>  | # ./tarabrt.sh -c ./core -d /tmp

Side note: Heh, I see you didn't read the error message, since your
command differs from the suggested one.

>  | ~ The resulting archive is located here: /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ The archive contents are listed below:
>  | ~ /tmp/tmp.kqC76QSrpF/tarlist
>  | ~ /tmp/tmp.kqC76QSrpF/version
>  | ~ /usr/bin/tarantool
>  | ~ ./core
>  | ~ /lib64/libicui18n.so.50
>  | ~ /lib64/libicuuc.so.50
>  | ~ /lib64/libicudata.so.50
>  | ~ /lib64/libdl.so.2
>  | ~ /lib64/libreadline.so.6
>  | ~ /lib64/libncurses.so.5
>  | ~ /lib64/libtinfo.so.5
>  | ~ /lib64/libform.so.5
>  | ~ /lib64/libz.so.1
>  | ~ /lib64/librt.so.1
>  | ~ /lib64/libssl.so.10
>  | ~ /lib64/libcrypto.so.10
>  | ~ /lib64/libgomp.so.1
>  | ~ /lib64/libpthread.so.0
>  | ~ /lib64/libgcc_s.so.1
>  | ~ /lib64/libunwind-x86_64.so.8
>  | ~ /lib64/libunwind.so.8
>  | ~ /lib64/libstdc++.so.6
>  | ~ /lib64/libm.so.6
>  | ~ /lib64/libc.so.6
>  | ~ /lib64/ld-linux-x86-64.so.2
>  | ~ /lib64/libgssapi_krb5.so.2
>  | ~ /lib64/libkrb5.so.3
>  | ~ /lib64/libcom_err.so.2
>  | ~ /lib64/libk5crypto.so.3
>  | ~ /lib64/libkrb5support.so.0
>  | ~ /lib64/libkeyutils.so.1
>  | ~ /lib64/libresolv.so.2
>  | ~ /lib64/libselinux.so.1
>  | ~ /lib64/libpcre.so.1
>  | ~ /.rocks/lib/tarantool/tuple/keydef.so
>  | ~ /usr/lib64/tarantool/avro_schema_rt_c.so
>  | ~ 
>  | ~ If you want to upload it, choose the available resource
>  | ~ (e.g. http://transfer.sh) and run the following command:
>  | ~ $ curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz <resource-uri>
> 
> Okay, let's spin around this output for a while:
> 
>  | (The /tmp/ entries looks are something I can glance.)
>  | # ll /tmp/tmp.kqC76QSrpF/tarlist
>  | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/tarlist: No such file or directory
>  | ll /tmp/tmp.kqC76QSrpF/version
>  | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/version: No such file or directory
>  | (But they're not. Okay, pass it over.)

These files are temporary (considering their names) and are removed when
the script finishes. I can list only their basename, but not listing them
at all looks like stealing them... Hence, consider the changes below.

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 005add707..e66621895 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -200,6 +200,7 @@ ${BINARY} --version >"${VERSION}"
 
 # Collect the most important artefacts.
 {
+	echo "${TARLIST}"
 	echo "${VERSION}"
 	echo "${BINARY}"
 	echo "${COREFILE}"
@@ -239,13 +240,15 @@ tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
 	--transform="s|${COREFILE}|coredump|" \
 	--transform="s|${TARLIST}|checklist|" \
 	--transform="s|${VERSION}|version|"   \
-	--add-file="${TARLIST}"
+
+# Strip TMPDIR prefix from the files in TARLIST, since this
+# directory will be removed at the end.
+sed "s|${TMPDIR}||g" -i "${TARLIST}"
 
 [ -t 1 ] && cat <<FINALIZE
 The resulting archive is located here: ${ARCHIVENAME}
 The archive contents are listed below:
-${TARLIST}
-$(cat ${TARLIST})
+$(cat "${TARLIST}")
 
 If you want to upload it, choose the available resource
 (e.g. http://transfer.sh) and run the following command:

================================================================================

>  | (keydef.so and avro_schema_rt_c.so are here, nice!)
>  | (Let's look at the archive file.)
>  | # ls -hl /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ -rw-r--r-- 1 root root 27M Mar 15 13:05 /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | # tar -vtf /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ -rw-r--r-- root/root      1085 2021-03-15 13:05 version
>  | ~ -rwxr-xr-x root/root  34431952 2020-12-30 14:08 tarantool
>  | ~ -rw------- root/root 188039168 2021-03-15 12:49 coredump
>  | ~ tar: Removing leading `/' from member names
>  | ~ -rwxr-xr-x root/root   2096104 2020-03-21 00:49 /lib64/libicui18n.so.50
>  | ~ -rwxr-xr-x root/root   1539520 2020-03-21 00:49 /lib64/libicuuc.so.50
>  | ~ -rwxr-xr-x root/root  20785824 2020-03-21 00:49 /lib64/libicudata.so.50
>  | ~ -rwxr-xr-x root/root     19248 2021-02-08 15:46 /lib64/libdl.so.2
>  | ~ -rwxr-xr-x root/root    285136 2019-08-08 12:07 /lib64/libreadline.so.6
>  | ~ -rwxr-xr-x root/root    163696 2017-09-06 22:08 /lib64/libncurses.so.5
>  | ~ -rwxr-xr-x root/root    174576 2017-09-06 22:08 /lib64/libtinfo.so.5
>  | ~ -rwxr-xr-x root/root     67864 2017-09-06 22:08 /lib64/libform.so.5
>  | ~ -rwxr-xr-x root/root     90248 2018-10-30 20:24 /lib64/libz.so.1
>  | ~ -rwxr-xr-x root/root     43712 2021-02-08 15:46 /lib64/librt.so.1
>  | ~ -rwxr-xr-x root/root    470376 2020-12-16 21:24 /lib64/libssl.so.10
>  | ~ -rwxr-xr-x root/root   2520768 2020-12-16 21:24 /lib64/libcrypto.so.10
>  | ~ -rwxr-xr-x root/root    154840 2020-09-30 02:17 /lib64/libgomp.so.1
>  | ~ -rwxr-xr-x root/root    142144 2021-02-08 15:46 /lib64/libpthread.so.0
>  | ~ -rwxr-xr-x root/root     88720 2020-09-30 02:18 /lib64/libgcc_s.so.1
>  | ~ -rwxr-xr-x root/root     73416 2017-08-02 15:45 /lib64/libunwind-x86_64.so.8
>  | ~ -rwxr-xr-x root/root     48904 2017-08-02 15:45 /lib64/libunwind.so.8
>  | ~ -rwxr-xr-x root/root    995840 2020-09-30 02:17 /lib64/libstdc++.so.6
>  | ~ -rwxr-xr-x root/root   1136944 2021-02-08 15:46 /lib64/libm.so.6
>  | ~ -rwxr-xr-x root/root   2156344 2021-02-08 15:46 /lib64/libc.so.6
>  | ~ -rwxr-xr-x root/root    163312 2021-02-08 15:46 /lib64/ld-linux-x86-64.so.2
>  | ~ -rwxr-xr-x root/root    320720 2020-09-30 16:51 /lib64/libgssapi_krb5.so.2
>  | ~ -rwxr-xr-x root/root    967840 2020-09-30 16:51 /lib64/libkrb5.so.3
>  | ~ -rwxr-xr-x root/root     15856 2020-09-30 15:58 /lib64/libcom_err.so.2
>  | ~ -rwxr-xr-x root/root    210784 2020-09-30 16:51 /lib64/libk5crypto.so.3
>  | ~ -rwxr-xr-x root/root     67104 2020-09-30 16:51 /lib64/libkrb5support.so.0
>  | ~ -rwxr-xr-x root/root     15688 2014-06-10 02:17 /lib64/libkeyutils.so.1
>  | ~ -rwxr-xr-x root/root    109976 2021-02-08 15:46 /lib64/libresolv.so.2
>  | ~ -rwxr-xr-x root/root    155744 2020-04-01 03:16 /lib64/libselinux.so.1
>  | ~ -rwxr-xr-x root/root    402384 2017-08-02 03:08 /lib64/libpcre.so.1
>  | ~ -rwxr-xr-x root/root     52168 2021-03-15 12:45 /.rocks/lib/tarantool/tuple/keydef.so
>  | ~ -rwxr-xr-x root/root     23736 2020-08-05 23:10 /usr/lib64/tarantool/avro_schema_rt_c.so
>  | ~ -rw-r--r-- root/root       796 2021-03-15 13:05 checklist
>  | (Can we avoid the the warning re `/`? Just curious.)

It turns out yes! The change is quite simple though the diff looks huge:

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 46e3016f0..63211f9a8 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -193,7 +193,8 @@ HOSTNAME=$(hostname 2>/dev/null || echo hostname)
 TMPDIR=$(mktemp -d -p "${COREDIR}")
 TARLIST=${TMPDIR}/tarlist
 VERSION=${TMPDIR}/version
-ARCHIVENAME=${COREDIR}/tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz
+ARCHIVENAME=tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}
+ARCHIVE=${COREDIR}/${ARCHIVENAME}.tar.gz
 
 # Dump the version to checkout the right commit later.
 ${BINARY} --version >"${VERSION}"
@@ -235,24 +236,25 @@ fi
 # Pack everything listed in TARLIST file into a tarball. To unify
 # the archive format BINARY, COREFILE, VERSION and TARLIST are
 # renamed while packing.
-tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
-	--transform="s|${BINARY}|tarantool|"  \
-	--transform="s|${COREFILE}|coredump|" \
-	--transform="s|${TARLIST}|checklist|" \
-	--transform="s|${VERSION}|version|"   \
+tar -czhf "${ARCHIVE}" -P -T "${TARLIST}" \
+	--transform="s|${BINARY}|/tarantool|"  \
+	--transform="s|${COREFILE}|/coredump|" \
+	--transform="s|${TARLIST}|/checklist|" \
+	--transform="s|${VERSION}|/version|"   \
+	--transform="s|^|${ARCHIVENAME}|"
 
 # Strip TMPDIR prefix from the files in TARLIST, since this
 # directory will be removed at the end.
 sed "s|${TMPDIR}||g" -i "${TARLIST}"
 
 [ -t 1 ] && cat <<FINALIZE
-The resulting archive is located here: ${ARCHIVENAME}
+The resulting archive is located here: ${ARCHIVE}
 The archive contents are listed below:
 $(cat "${TARLIST}")
 
 If you want to upload it, choose the available resource
 (e.g. http://transfer.sh) and run the following command:
-$ curl -T ${ARCHIVENAME} <resource-uri>
+$ curl -T ${ARCHIVE} <resource-uri>
 FINALIZE
 
 # Cleanup temporary files.

================================================================================

>  | (Okay, I see that transfer.sh is suggested. I never use it, so I'm
>  |  curious.)
>  | # curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz http://transfer.sh
>  | ~ http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | (Hey, something occurs! But I'll save it to my system anyway.)
>  | # cp /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz /cores/
>  | (And, while I'm here...)
>  | # md5sum /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ bec8f81c69ed992a258a6d9f0b8a04dc  /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> 
> Okay, let's try gdb.sh on my Gentoo box:
> 
>  | $ mkdir ~/tmp/try-gdb-sh
>  | $ cd ~/tmp/try-gdb-sh
>  | $ curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/gdb.sh > gdb.sh
>  | $ chmod a+x gdb.sh
>  | $ curl -LfsS http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz > tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | $ md5sum tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ bec8f81c69ed992a258a6d9f0b8a04dc  tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | (Matches!)
>  | (Hm, now I'm worry, whether the archive will be uncompressed into the
>  |  current directory or will create its own directory?)

Now (as a result of the patch above) it creates its own directory, that
is archive basename without the suffix.

>  | $ tar -xf tarantool-core-N-202103151305-cc18d0726f6a.tar.gz 
>  | ~ tar: Removing leading `/' from member names
>  | $ ll
>  | ~ total 239M
>  | ~ drwxr-xr-x   5 alex users 4.0K Mar 15 16:56 .
>  | ~ drwxr-xr-x 164 alex users  20K Mar 15 16:50 ..
>  | ~ -rw-r--r--   1 alex users  796 Mar 15 16:05 checklist
>  | ~ -rw-------   1 alex users 180M Mar 15 15:49 coredump
>  | ~ -rwxr-xr-x   1 alex users 2.0K Mar 15 16:51 gdb.sh
>  | ~ drwxr-xr-x   2 alex users 4.0K Mar 15 16:56 lib64
>  | ~ drwxr-xr-x   3 alex users 4.0K Mar 15 16:56 .rocks
>  | ~ -rwxr-xr-x   1 alex users  33M Dec 30 17:08 tarantool
>  | ~ -rw-r--r--   1 alex users  27M Mar 15 16:54 tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ drwxr-xr-x   3 alex users 4.0K Mar 15 16:56 usr
>  | ~ -rw-r--r--   1 alex users 1.1K Mar 15 16:05 version
>  | (Meh. Let's move it around.)
>  | $ mkdir tarantool-core-N-202103151305-cc18d0726f6a
>  | $ mv checklist coredump lib64 .rocks tarantool usr version tarantool-core-N-202103151305-cc18d0726f6a
>  | $ tree
>  | ~ .
>  | ~ ├── gdb.sh
>  | ~ ├── tarantool-core-N-202103151305-cc18d0726f6a
>  | ~ │   ├── checklist
>  | ~ │   ├── coredump
>  | ~ │   ├── lib64
>  | ~ │   │   ├── <..skipped..>
>  | ~ │   ├── tarantool
>  | ~ │   ├── usr
>  | ~ │   │   └── lib64
>  | ~ │   │       └── tarantool
>  | ~ │   │           └── avro_schema_rt_c.so
>  | ~ │   └── version
>  | ~ └── tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~
>  | ~ 5 directories, 37 files
>  | (Now looks okay. Let's try.)
>  | $ cd tarantool-core-N-202103151305-cc18d0726f6a
>  | $ ../gdb.sh --help
>  | ~ ================================================================================
>  | ~
>  | ~ Do not forget to properly setup the environment:
>  | ~ * git clone https://github.com/tarantool/tarantool.git sources
>  | ~ * cd !$
>  | ~ * git checkout 34d504d
>  | ~ * git submodule update --recursive --init
>  | ~
>  | ~ ================================================================================
>  | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1
>  | ~ <..skipped..>
>  | ~ Reading symbols from ./tarantool...
>  | ~ lj-arch command initialized
>  | ~ lj-tv command initialized
>  | ~ lj-str command initialized
>  | ~ lj-tab command initialized
>  | ~ lj-stack command initialized
>  | ~ lj-state command initialized
>  | ~ lj-gc command initialized
>  | ~ luajit-gdb.py is successfully loaded
>  | ~ warning: Can't open file /usr/bin/tarantool during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib/locale/locale-archive (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libpcre.so.1.2.0 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libselinux.so.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libresolv-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libkeyutils.so.1.5 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libkrb5support.so.0.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libk5crypto.so.3.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libcom_err.so.2.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libkrb5.so.3.3 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libgssapi_krb5.so.2.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libc-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libm-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libstdc++.so.6.0.19 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libunwind.so.8.0.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libunwind-x86_64.so.8.0.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libgcc_s-4.8.5-20150702.so.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libpthread-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libgomp.so.1.0.0 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libcrypto.so.1.0.2k during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libssl.so.1.0.2k during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/librt-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libz.so.1.2.7 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libform.so.5.9 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libtinfo.so.5.9 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libncurses.so.5.9 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libreadline.so.6.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libdl-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libicudata.so.50.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libicuuc.so.50.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libicui18n.so.50.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/ld-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/gconv/gconv-modules.cache (deleted) during file-backed mapping note processing
>  | ~ [New LWP 246]
>  | ~ [New LWP 247]
>  | ~ [New LWP 248]
>  | ~ [New LWP 250]
>  | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
>  | ~ warning: the debug information found in "/usr/lib64/debug/usr/lib64/tarantool/avro_schema_rt_c.so.debug" does not match "/home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so" (CRC mismatch).
>  | ~ 
>  | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
>  | ~ Core was generated by `tarantool'.
>  | ~ Program terminated with signal SIGABRT, Aborted.
>  | ~ #0  0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
>  | ~ [Current thread is 1 (LWP 246)]
>  | (gdb) Ctrl+D
>  | (Hey, --help runs it.)

Lol, I didn't even check this. Thanks for the example. I've added
arguments handling and the usage for the script. Diff is below:

================================================================================

diff --git a/tools/gdb.sh b/tools/gdb.sh
index a58c47cab..55fa73d7a 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -1,6 +1,40 @@
 #!/bin/sh
 set -eu
 
+TOOL=$(basename "$0")
+HELP=$(cat <<HELP
+${TOOL} - gdb wrapper loading artefacts collected via tarabrt.sh
+
+	Soon her eye fell upon a little glass box lying underneath
+	the table. She opened it and found in it a very small
+	cake, on which the words "EAT ME" were beautifully marked
+	in currants.
+
+	- Alice's Adventures in Wonderland.
+HELP
+)
+
+# Parse CLI options.
+OPTIONS=$(getopt -o h -n "${TOOL}" -- "$@")
+eval set -- "${OPTIONS}"
+while true; do
+	case "$1" in
+		--) shift; break;;
+		-h) printf "%s\n" "${HELP}";
+			exit 0;;
+		*)  printf "Invalid option: $1\n%s\n" "${HELP}";
+			exit 1;;
+	esac
+done
+
+# Do not proceed if there are some CLI arguments left. Everything
+# should be parsed before this line. Refer to the help message in
+# a funny way.
+if [ $# -ne 0 ]; then
+	echo "RUN ME"
+	exit 1;
+fi
+
 # Check that gdb is installed.
 if ! command -v gdb >/dev/null; then
 	cat <<NOGDB

================================================================================

>  | (Anyway, I should checkout sources. Nice reminder.)
>  | $ git clone https://github.com/tarantool/tarantool.git sources
>  | $ cd sources
>  | $ git checkout 34d504d
>  | $ git submodule update --recursive --init
>  | $ cd ..
>  | (Let's try again?)
>  | $ ../gdb.sh
>  | <..I'll skip the output, it is quite similar to one above. But will
>  |  comment below.>
>  | (The reminder regarding tarantool sources still shown. But I already
>  |  placed the sources here.)

Added the check whether the sources are already setup. Diff is below:

================================================================================

diff --git a/tools/gdb.sh b/tools/gdb.sh
index fd310b99b..63e80c8f3 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -67,18 +67,30 @@ NOARTEFACTS
 	exit 1;
 fi
 
-REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
-cat <<SOURCES
+SOURCES=${PWD}/sources
+
+# Check whether Tarantool sources are setup. Otherwise, leave a
+# recipe for user, how to do it.
+# FIXME: This can be done automatically (simply run the commands
+# below), but this obliges git to be installed. For now, it makes
+# the wrapper more complex by additional checks, so this activity
+# is left for the user.
+if [ ! -d "${SOURCES}" ]; then
+	REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
+	cat <<SOURCES
 ================================================================================
 
 Do not forget to properly setup the environment:
-* git clone https://github.com/tarantool/tarantool.git sources
+* git clone https://github.com/tarantool/tarantool.git ${SOURCES}
 * cd !$
 * git checkout ${REVISION}
 * git submodule update --recursive --init
+* cd -
 
 ================================================================================
 SOURCES
+	exit 1;
+fi
 
 # Define the build path to be substituted with the source path.
 # XXX: Check the absolute path on the function <main> definition

================================================================================

>  | (All library warnings above are still here. However it seems they're
>     just warnings.)
>  | $ (gdb) bt
>  | ~ #0  0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
>  | ~ #1  0x00007f340a5f5a78 in abort () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
>  | ~ #2  0x0000000000453c43 in sig_fatal_cb (signo=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/main.cc:302
>  | ~ #3  <signal handler called>
>  | ~ #4  0x0000000000621ce7 in lj_cconv_ct_ct (cts=<optimized out>, d=<optimized out>, s=0x41508e70, dp=0x41d7a0b8 "", sp=0x140535248 <error: Cannot access memory at address 0x140535248>, 
>  | ~     flags=<optimized out>) at lj_cconv.c:234
>  | ~ #5  0x0000000000621e61 in lj_cconv_tv_ct (cts=0x40fb9858, s=<optimized out>, sid=<optimized out>, o=0x41d7a0b8, sp=<optimized out>) at lj_cconv.c:389
>  | ~ #6  0x0000000000621298 in lj_cdata_get (cts=<optimized out>, s=<optimized out>, o=<optimized out>, sp=<optimized out>) at lj_cdata.c:254
>  | ~ #7  0x0000000000631021 in lj_cf_ffi_meta___index (L=0x40fb1378) at lib_ffi.c:158
>  | ~ #8  0x00000000006338a7 in lj_BC_FUNCC () at buildvm_x86.dasc:809
>  | ~ #9  0x0000000000615c82 in lua_pcall (L=L@entry=0x40fb1378, nargs=<optimized out>, nresults=<optimized out>, errfunc=errfunc@entry=0) at lj_api.c:1158
>  | ~ #10 0x00000000005c4663 in luaT_call (L=0x40fb1378, nargs=<optimized out>, nreturns=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/utils.c:1022
>  | ~ #11 0x00000000005bd259 in lua_main (L=L@entry=0x40fb1378, argc=argc@entry=1, argv=argv@entry=0x20d5fc0) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:546
>  | ~ #12 0x00000000005bd6b3 in run_script_f (ap=<error reading variable: value has been optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:647
>  | ~ #13 0x00000000004530dc in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=<optimized out>, ap=<optimized out>)
>  | ~     at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.h:882
>  | ~ #14 0x00000000005def90 in fiber_loop (data=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.c:879
>  | ~ #15 0x00000000007c7ddf in coro_init () at /build/usr/src/debug/tarantool-2.6.2.0/third_party/coro/coro.c:110
>  | (Looks good!)
>  | (gdb) info sharedlibrary 
>  | ~ From                To                  Syms Read   Shared Object Library
>  | ~ 0x00007f340e9aa5f0  0x00007f340eab0b62  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicui18n.so.50
>  | ~ 0x00007f340e5eb2f0  0x00007f340e69dad8  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicuuc.so.50
>  | ~ 0x00007f340cfc6550  0x00007f340cfc6635  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicudata.so.50
>  | ~ 0x00007f340cdc2e50  0x00007f340cdc394e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libdl.so.2
>  | ~ 0x00007f340cb90de0  0x00007f340cbaf775  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libreadline.so.6
>  | ~ 0x00007f340c95bd50  0x00007f340c975462  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libncurses.so.5
>  | ~ 0x00007f340c737e40  0x00007f340c743bb8  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libtinfo.so.5
>  | ~ 0x00007f340c51e6e0  0x00007f340c525ed0  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libform.so.5
>  | ~ 0x00007f340c307110  0x00007f340c313698  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libz.so.1
>  | ~ 0x00007f340c0ff230  0x00007f340c1021bc  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/librt.so.1
>  | ~ 0x00007f340bea5b90  0x00007f340bee164d  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libssl.so.10
>  | ~ 0x00007f340ba95200  0x00007f340bbce7b7  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcrypto.so.10
>  | ~ 0x00007f340b809060  0x00007f340b81fc2a  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgomp.so.1
>  | ~ 0x00007f340b5eb8f0  0x00007f340b5f6db1  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpthread.so.0
>  | ~ 0x00007f340b3d2ad0  0x00007f340b3e2285  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgcc_s.so.1
>  | ~ 0x00007f340b1b1570  0x00007f340b1be20e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind-x86_64.so.8
>  | ~ 0x00007f340af975b0  0x00007f340af9e95e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind.so.8
>  | ~ 0x00007f340ace94e0  0x00007f340ad5059a  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libstdc++.so.6
>  | ~ 0x00007f340a991350  0x00007f340a9fc336  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libm.so.6
>  | ~ 0x00007f340a5dd9f0  0x00007f340a72da5f  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
>  | ~ 0x00007f340ed11af0  0x00007f340ed2d060  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/ld-linux-x86-64.so.2
>  | ~ 0x00007f340a37d720  0x00007f340a3afbca  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgssapi_krb5.so.2
>  | ~ 0x00007f340a0adc20  0x00007f340a1159fa  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5.so.3
>  | ~ 0x00007f3409e85540  0x00007f3409e86113  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcom_err.so.2
>  | ~ 0x00007f3409c558d0  0x00007f3409c73bbf  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libk5crypto.so.3
>  | ~ 0x00007f3409a44bf0  0x00007f3409a4c36b  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5support.so.0
>  | ~ 0x00007f340983e5b0  0x00007f340983f1cc  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkeyutils.so.1
>  | ~ 0x00007f3409626980  0x00007f34096351e2  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libresolv.so.2
>  | ~ 0x00007f3409402a90  0x00007f3409418896  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libselinux.so.1
>  | ~ 0x00007f340919b5f0  0x00007f34091e15b0  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpcre.so.1
>  | ~ 0x00007f3408c429f0  0x00007f3408c43896  Yes         /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/.rocks/lib/tarantool/tuple/keydef.so
>  | ~ 0x00007f33eedf8d80  0x00007f33eedfc174  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so
>  | ~ (*): Shared library is missing debugging information.
>  | (Looks good too!)
> 
> > Can't open file <...> (deleted) during file-backed mapping note
> > processing.
> 
> I don't know what it is. But I'll share my gdb version just in case
> (maybe you'll be able to reproduce it). I didn't meet this warning
> before.

I tried the script on some machines I have access to: my Gentoo
workstation, our CentOS 7 dev stand, CentOS 8 docker and my personal
Gentoo laptop. I faced the issue only on the latter machine. I guess it
doesn't relate to the script but rather to the environment, since the
problem also occurs when I simply run the following command:
| $ gdb ./tarantool coredump

> 
>  | $ gdb --version | head -n 1
>  | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1

I have the same version on my laptop. My workstation wasn't updated for
some time, so there is only 8.3.1. Let's proceed with this problem in
private, since AFAICS it relates to Gentoo with the quite new gdb.

> 
> However I would note that those warnings misleads me the first time: I
> thought that something is wrong with library paths.
> 
> > Unable to find libthread_db matching inferior's thread library, thread
> > debugging will not be available.
> 
> Sure, it is not listed in the lld output (neither in 'info
> sharedlibrary'), but maybe we should include it into the tarabrt.sh
> generated archive.

Ouch, it was a tough one. However, I've chosen the simplest solution for
this issue. Diff is below:

================================================================================

diff --git a/tools/gdb.sh b/tools/gdb.sh
index 63e80c8f3..669be3c4f 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -102,4 +102,6 @@ SUBPATH=$(gdb -batch -n ./tarantool -ex 'info line main' | \
 gdb ./tarantool \
     -ex "set sysroot $(realpath .)" \
     -ex "set substitute-path ${SUBPATH} sources" \
+    -ex "add-auto-load-safe-path $(realpath .)" \
+    -ex "set auto-load libthread-db on" \
     -ex 'core coredump'
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 8a07b88a6..aadfde129 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -242,6 +242,32 @@ COREMISMATCH
 	exit 1;
 fi
 
+# XXX: This is poor man's libthread_db location resolution.
+# I'm sure there are many ways to implement this, but most of them
+# requires additional tools, libs, etc. The one implemented below
+# is quite simple, robust and requires nothing more than being
+# already used above.
+GREETING=$(gdb "${BINARY}" -c "${COREFILE}" -ex 'quit' 2>/dev/null)
+LIBTHREAD_DB=
+PATTERN1='[Thread debugging using libthread_db enabled]'
+PATTERN2='Using host libthread_db library ".*"\.'
+
+if echo "${GREETING}" | grep -F "${PATTERN1}" >/dev/null; then
+	LIBTHREAD_DB=$(echo "${GREETING}" | grep "${PATTERN2}" | \
+		awk -F'"' '{ print $2 }')
+	echo "${LIBTHREAD_DB}" >>"${TARLIST}"
+fi
+
+if [ -z "${LIBTHREAD_DB}" ]; then
+	[ -t 1 ] && cat <<NOLIBTHREAD_DB
+Failed to find libthread_db.so. Brace yourselves, thread debugging
+will be a challenging exercise.
+
+This is just a notice, not an error.
+NOLIBTHREAD_DB
+	# XXX: Not an error, proceed packing.
+fi
+
 # Pack everything listed in TARLIST file into a tarball. To unify
 # the archive format BINARY, COREFILE, VERSION and TARLIST are
 # renamed while packing.

================================================================================

It's a really useful notice: now one can even work with TLS variables
with no prisedanie at all (I've checked this by transferring coredump
artefacts from C7 host to my Gentoo workstation and playing with them a
little). Thanks!

> 
> > the debug information found in <...> does not match <...> (CRC mismatch).
> 
> It is just because I have local avro schema debuginfo installed. I think
> this warning can be ignored.

I believe we can do nothing with it in general case, so user (or
developer) should fix this warning by himself.

> 
> To sum up: my experience was successful. You may polish some tiny
> details, but in fact the scripts seem to be ready for use.

Glad to hear this, hope they'll be useful.

> 
> ----
> 
> We also briefly discussed the topic about stripped executables with
> Igor.
> 
> Now we ship stripped executables in RPM packages for CentOS 8 (only for
> CentOS 8), but the separate debug package is stored in our repos.
> 
> First point: we can include /etc/os-release into the generated archive.

I see no criminal in this, hence here are the changes:

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 6fc76e8f3..e09d051a4 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -7,11 +7,12 @@ ${TOOL} - Tarantool Automatic Bug Reporting Tool
 
 This tool collects all required artefacts (listed below) and packs them into
 a single archive with unified format:
-  - /checklist - the plain text file with the list of tarball contents
-  - /version   - the plain text file containing \`tarantool --version' output
-  - /tarantool - the executable binary file produced the core dump
-  - /coredump  - the core dump file produced by the executable
-  - all shared libraries loaded (even via dlopen(3)) at the crash moment.
+  - /checklist      - the plain text file with the list of tarball contents
+  - /version        - the plain text file with \`tarantool --version' output
+  - /tarantool      - the executable binary file produced the core dump
+  - /coredump       - the core dump file produced by the executable
+  - /etc/os-release - the plain text file with OS identification data
+  - all shared libraries loaded (even via dlopen(3)) at the crash moment
 
 SYNOPSIS
 
@@ -200,6 +201,7 @@ HOSTNAME=$(hostname 2>/dev/null || echo hostname)
 TMPDIR=$(mktemp -d -p "${COREDIR}")
 TARLIST=${TMPDIR}/tarlist
 VERSION=${TMPDIR}/version
+OSRELEASE=/etc/os-release
 ARCHIVENAME=tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}
 ARCHIVE=${COREDIR}/${ARCHIVENAME}.tar.gz
 
@@ -210,6 +212,7 @@ ${BINARY} --version >"${VERSION}"
 {
 	echo "${TARLIST}"
 	echo "${VERSION}"
+	echo "${OSRELEASE}"
 	echo "${BINARY}"
 	echo "${COREFILE}"
 } >>"${TARLIST}"

================================================================================

> 
> Second point: we can give URL from where debuginfo package may be
> downloaded and give advice how to unpack it (in gdb.sh output, when we
> see the stripped executable).

I'm totally for such helpers, but I am tuning this *draft* for such a
long time... Since this is not such popular case for now, I ignore
everything below and will create an issue for this.

> 
> Examples:
> 
>  | $ curl -LfsS https://download.tarantool.org/tarantool/release/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm
>  | $ mkdir tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
>  | $ cd tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
>  | $ rpm2cpio ../tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm | cpio -idmv
>  | ~ ./usr/lib/debug
>  | ~ ./usr/lib/debug/.build-id
>  | ~ ./usr/lib/debug/.build-id/a3
>  | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60
>  | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60.debug
>  | ~ ./usr/lib/debug/usr
>  | ~ ./usr/lib/debug/usr/bin
>  | ~ ./usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
>  | ~ 64743 blocks
>  | (All are symlinks except tarantool-2.8.0.0-1.el8.x86_64.debug.)
>  | $ file usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
>  | ~ usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter \004, for GNU/Linux 3.2.0, BuildID[sha1]=a3ad84db98a9b14e70d2915271904ddfe26b5f60, with debug_info, not stripped, too many notes (256)
> 
>  | $ curl -LfsS https://download.tarantool.org/tarantool/live/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm
> 
> Third: we can give advice how to enable the ELF with debug symbols in
> the interactive gdb session.
> 
> Or we should do it within gdb.sh to call 'info line main'? Expect it in
> some predefined location or accept an option?
> 
> At least it looks worthful to warn about stripped executable explicitly.
> 
> ----
> 
> It seems, I reached maximum value that I can give as reviewer, so I'll
> just say that the patch looks good to me (and I don't mind small changes
> around usability if you want to do them).

Again, thanks a lot! Added your tag:
| Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

> 
> WBR, Alexander Turenko.

-- 
Best regards,
IM