[Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
Igor Munkin
imun at tarantool.org
Mon Apr 19 23:42:09 MSK 2021
Sasha,
Thanks a lot for your review!
On 15.03.21, Alexander Turenko wrote:
> I looked over the WIP version of the script and nothing confuses me:
>
> https://lists.tarantool.org/tarantool-patches/20210220104630.wiwxhkjfimsx2swa@tkn_work_nb/
>
> So I guess I should do some basic testing to get and share my user
> experience.
Nice, then I'll go through your user experience and adjust the script
considering the problems you're facing.
>
> So I'll step into another environment to create coredump and archive it:
<snipped the manipulations with installing and running Tarantool>
>
> | tarantool> require('ffi').new('int[?]', 1)[1024 ^ 3]
> | ~ <...>
> | ~ Aborted (core dumped)
> | # ls -hl core
> | ~ -rw------- 1 root root 180M Mar 15 12:49 core
> | # file core
> | ~ bash: file: command not found
>
> (I'll ignore that 'file' is not installed, because it is possible
> situation on a production server, I guess.)
I didn't figure out, whether <file> package is required, recommended or
optional. So if your Docker image doesn't have it, then the production
server can also lack this package.
>
> Okay, now I got the tarabrt.sh script somehow:
>
> | # curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/tarabrt.sh >tarabrt.sh
> | # chmod a+x tarabrt.sh
> | (Don't read the content, just run.)
> | # ./tarabrt.sh --help
> | ~ tarabrt.sh: unrecognized option '--help'
Unfortunately, I have nothing to do here: this is returned by <getopt>.
> | # ./tarabrt.sh -h
> | ~ <..skipped..>
> | ~ - Manual usage. User can simply pack all necessary artefacts by running the
> | ~ following command.
> | ~ $ /path/to/tarabrt.sh -c ./core -d /tmp
> | ~ <..skipped..>
> | # ./tarabrt.sh -c ./core -d /tmp
> | ~ ./tarabrt.sh: line 115: file: command not found
> | ~ gdb is not installed, but it is obligatory for collecting the
> | ~ loaded shared libraries from the core dump.
> | ~
> | ~ You can proceed collecting the artefacts manually later by running
> | ~ the following command:
> | ~ $ tarabrt.sh -e /usr/bin/tarantool -c ./core
> | (Okay, I'll install file and gdb now.)
I'll added the check that file is installed. Diff is below:
================================================================================
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 372ff80c1..005add707 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -112,6 +112,17 @@ NOCOREDUMP
exit 1;
fi
+if ! command -v file >/dev/null; then
+ [ -t 1 ] && cat <<NOFILE
+file is not installed, but it is obligatory for checking the
+artefacts to be collected.
+
+You can proceed collecting the artefacts manually later by running
+the following command:
+$ ${TOOL} -e ${BINARY} -c ${COREFILE}
+NOFILE
+fi
+
if file "${COREFILE}" | grep -qv 'core file'; then
[ -t 1 ] && cat <<NOTACOREDUMP
Not a core dump: ${COREFILE}
================================================================================
> | # yum install -y file gdb
> | # file core
> | ~ core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'tarantool', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/tarantool', platform: 'x86_64'
> | (Repeat the attempt to create the archive.)
> | # ./tarabrt.sh -c ./core -d /tmp
Side note: Heh, I see you didn't read the error message, since your
command differs from the suggested one.
> | ~ The resulting archive is located here: /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | ~ The archive contents are listed below:
> | ~ /tmp/tmp.kqC76QSrpF/tarlist
> | ~ /tmp/tmp.kqC76QSrpF/version
> | ~ /usr/bin/tarantool
> | ~ ./core
> | ~ /lib64/libicui18n.so.50
> | ~ /lib64/libicuuc.so.50
> | ~ /lib64/libicudata.so.50
> | ~ /lib64/libdl.so.2
> | ~ /lib64/libreadline.so.6
> | ~ /lib64/libncurses.so.5
> | ~ /lib64/libtinfo.so.5
> | ~ /lib64/libform.so.5
> | ~ /lib64/libz.so.1
> | ~ /lib64/librt.so.1
> | ~ /lib64/libssl.so.10
> | ~ /lib64/libcrypto.so.10
> | ~ /lib64/libgomp.so.1
> | ~ /lib64/libpthread.so.0
> | ~ /lib64/libgcc_s.so.1
> | ~ /lib64/libunwind-x86_64.so.8
> | ~ /lib64/libunwind.so.8
> | ~ /lib64/libstdc++.so.6
> | ~ /lib64/libm.so.6
> | ~ /lib64/libc.so.6
> | ~ /lib64/ld-linux-x86-64.so.2
> | ~ /lib64/libgssapi_krb5.so.2
> | ~ /lib64/libkrb5.so.3
> | ~ /lib64/libcom_err.so.2
> | ~ /lib64/libk5crypto.so.3
> | ~ /lib64/libkrb5support.so.0
> | ~ /lib64/libkeyutils.so.1
> | ~ /lib64/libresolv.so.2
> | ~ /lib64/libselinux.so.1
> | ~ /lib64/libpcre.so.1
> | ~ /.rocks/lib/tarantool/tuple/keydef.so
> | ~ /usr/lib64/tarantool/avro_schema_rt_c.so
> | ~
> | ~ If you want to upload it, choose the available resource
> | ~ (e.g. http://transfer.sh) and run the following command:
> | ~ $ curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz <resource-uri>
>
> Okay, let's spin around this output for a while:
>
> | (The /tmp/ entries looks are something I can glance.)
> | # ll /tmp/tmp.kqC76QSrpF/tarlist
> | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/tarlist: No such file or directory
> | ll /tmp/tmp.kqC76QSrpF/version
> | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/version: No such file or directory
> | (But they're not. Okay, pass it over.)
These files are temporary (considering their names) and are removed when
the script finishes. I can list only their basename, but not listing them
at all looks like stealing them... Hence, consider the changes below.
================================================================================
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 005add707..e66621895 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -200,6 +200,7 @@ ${BINARY} --version >"${VERSION}"
# Collect the most important artefacts.
{
+ echo "${TARLIST}"
echo "${VERSION}"
echo "${BINARY}"
echo "${COREFILE}"
@@ -239,13 +240,15 @@ tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
--transform="s|${COREFILE}|coredump|" \
--transform="s|${TARLIST}|checklist|" \
--transform="s|${VERSION}|version|" \
- --add-file="${TARLIST}"
+
+# Strip TMPDIR prefix from the files in TARLIST, since this
+# directory will be removed at the end.
+sed "s|${TMPDIR}||g" -i "${TARLIST}"
[ -t 1 ] && cat <<FINALIZE
The resulting archive is located here: ${ARCHIVENAME}
The archive contents are listed below:
-${TARLIST}
-$(cat ${TARLIST})
+$(cat "${TARLIST}")
If you want to upload it, choose the available resource
(e.g. http://transfer.sh) and run the following command:
================================================================================
> | (keydef.so and avro_schema_rt_c.so are here, nice!)
> | (Let's look at the archive file.)
> | # ls -hl /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | ~ -rw-r--r-- 1 root root 27M Mar 15 13:05 /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | # tar -vtf /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | ~ -rw-r--r-- root/root 1085 2021-03-15 13:05 version
> | ~ -rwxr-xr-x root/root 34431952 2020-12-30 14:08 tarantool
> | ~ -rw------- root/root 188039168 2021-03-15 12:49 coredump
> | ~ tar: Removing leading `/' from member names
> | ~ -rwxr-xr-x root/root 2096104 2020-03-21 00:49 /lib64/libicui18n.so.50
> | ~ -rwxr-xr-x root/root 1539520 2020-03-21 00:49 /lib64/libicuuc.so.50
> | ~ -rwxr-xr-x root/root 20785824 2020-03-21 00:49 /lib64/libicudata.so.50
> | ~ -rwxr-xr-x root/root 19248 2021-02-08 15:46 /lib64/libdl.so.2
> | ~ -rwxr-xr-x root/root 285136 2019-08-08 12:07 /lib64/libreadline.so.6
> | ~ -rwxr-xr-x root/root 163696 2017-09-06 22:08 /lib64/libncurses.so.5
> | ~ -rwxr-xr-x root/root 174576 2017-09-06 22:08 /lib64/libtinfo.so.5
> | ~ -rwxr-xr-x root/root 67864 2017-09-06 22:08 /lib64/libform.so.5
> | ~ -rwxr-xr-x root/root 90248 2018-10-30 20:24 /lib64/libz.so.1
> | ~ -rwxr-xr-x root/root 43712 2021-02-08 15:46 /lib64/librt.so.1
> | ~ -rwxr-xr-x root/root 470376 2020-12-16 21:24 /lib64/libssl.so.10
> | ~ -rwxr-xr-x root/root 2520768 2020-12-16 21:24 /lib64/libcrypto.so.10
> | ~ -rwxr-xr-x root/root 154840 2020-09-30 02:17 /lib64/libgomp.so.1
> | ~ -rwxr-xr-x root/root 142144 2021-02-08 15:46 /lib64/libpthread.so.0
> | ~ -rwxr-xr-x root/root 88720 2020-09-30 02:18 /lib64/libgcc_s.so.1
> | ~ -rwxr-xr-x root/root 73416 2017-08-02 15:45 /lib64/libunwind-x86_64.so.8
> | ~ -rwxr-xr-x root/root 48904 2017-08-02 15:45 /lib64/libunwind.so.8
> | ~ -rwxr-xr-x root/root 995840 2020-09-30 02:17 /lib64/libstdc++.so.6
> | ~ -rwxr-xr-x root/root 1136944 2021-02-08 15:46 /lib64/libm.so.6
> | ~ -rwxr-xr-x root/root 2156344 2021-02-08 15:46 /lib64/libc.so.6
> | ~ -rwxr-xr-x root/root 163312 2021-02-08 15:46 /lib64/ld-linux-x86-64.so.2
> | ~ -rwxr-xr-x root/root 320720 2020-09-30 16:51 /lib64/libgssapi_krb5.so.2
> | ~ -rwxr-xr-x root/root 967840 2020-09-30 16:51 /lib64/libkrb5.so.3
> | ~ -rwxr-xr-x root/root 15856 2020-09-30 15:58 /lib64/libcom_err.so.2
> | ~ -rwxr-xr-x root/root 210784 2020-09-30 16:51 /lib64/libk5crypto.so.3
> | ~ -rwxr-xr-x root/root 67104 2020-09-30 16:51 /lib64/libkrb5support.so.0
> | ~ -rwxr-xr-x root/root 15688 2014-06-10 02:17 /lib64/libkeyutils.so.1
> | ~ -rwxr-xr-x root/root 109976 2021-02-08 15:46 /lib64/libresolv.so.2
> | ~ -rwxr-xr-x root/root 155744 2020-04-01 03:16 /lib64/libselinux.so.1
> | ~ -rwxr-xr-x root/root 402384 2017-08-02 03:08 /lib64/libpcre.so.1
> | ~ -rwxr-xr-x root/root 52168 2021-03-15 12:45 /.rocks/lib/tarantool/tuple/keydef.so
> | ~ -rwxr-xr-x root/root 23736 2020-08-05 23:10 /usr/lib64/tarantool/avro_schema_rt_c.so
> | ~ -rw-r--r-- root/root 796 2021-03-15 13:05 checklist
> | (Can we avoid the the warning re `/`? Just curious.)
It turns out yes! The change is quite simple though the diff looks huge:
================================================================================
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 46e3016f0..63211f9a8 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -193,7 +193,8 @@ HOSTNAME=$(hostname 2>/dev/null || echo hostname)
TMPDIR=$(mktemp -d -p "${COREDIR}")
TARLIST=${TMPDIR}/tarlist
VERSION=${TMPDIR}/version
-ARCHIVENAME=${COREDIR}/tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz
+ARCHIVENAME=tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}
+ARCHIVE=${COREDIR}/${ARCHIVENAME}.tar.gz
# Dump the version to checkout the right commit later.
${BINARY} --version >"${VERSION}"
@@ -235,24 +236,25 @@ fi
# Pack everything listed in TARLIST file into a tarball. To unify
# the archive format BINARY, COREFILE, VERSION and TARLIST are
# renamed while packing.
-tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
- --transform="s|${BINARY}|tarantool|" \
- --transform="s|${COREFILE}|coredump|" \
- --transform="s|${TARLIST}|checklist|" \
- --transform="s|${VERSION}|version|" \
+tar -czhf "${ARCHIVE}" -P -T "${TARLIST}" \
+ --transform="s|${BINARY}|/tarantool|" \
+ --transform="s|${COREFILE}|/coredump|" \
+ --transform="s|${TARLIST}|/checklist|" \
+ --transform="s|${VERSION}|/version|" \
+ --transform="s|^|${ARCHIVENAME}|"
# Strip TMPDIR prefix from the files in TARLIST, since this
# directory will be removed at the end.
sed "s|${TMPDIR}||g" -i "${TARLIST}"
[ -t 1 ] && cat <<FINALIZE
-The resulting archive is located here: ${ARCHIVENAME}
+The resulting archive is located here: ${ARCHIVE}
The archive contents are listed below:
$(cat "${TARLIST}")
If you want to upload it, choose the available resource
(e.g. http://transfer.sh) and run the following command:
-$ curl -T ${ARCHIVENAME} <resource-uri>
+$ curl -T ${ARCHIVE} <resource-uri>
FINALIZE
# Cleanup temporary files.
================================================================================
> | (Okay, I see that transfer.sh is suggested. I never use it, so I'm
> | curious.)
> | # curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz http://transfer.sh
> | ~ http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | (Hey, something occurs! But I'll save it to my system anyway.)
> | # cp /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz /cores/
> | (And, while I'm here...)
> | # md5sum /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | ~ bec8f81c69ed992a258a6d9f0b8a04dc /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>
> Okay, let's try gdb.sh on my Gentoo box:
>
> | $ mkdir ~/tmp/try-gdb-sh
> | $ cd ~/tmp/try-gdb-sh
> | $ curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/gdb.sh > gdb.sh
> | $ chmod a+x gdb.sh
> | $ curl -LfsS http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz > tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | $ md5sum tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | ~ bec8f81c69ed992a258a6d9f0b8a04dc tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | (Matches!)
> | (Hm, now I'm worry, whether the archive will be uncompressed into the
> | current directory or will create its own directory?)
Now (as a result of the patch above) it creates its own directory, that
is archive basename without the suffix.
> | $ tar -xf tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | ~ tar: Removing leading `/' from member names
> | $ ll
> | ~ total 239M
> | ~ drwxr-xr-x 5 alex users 4.0K Mar 15 16:56 .
> | ~ drwxr-xr-x 164 alex users 20K Mar 15 16:50 ..
> | ~ -rw-r--r-- 1 alex users 796 Mar 15 16:05 checklist
> | ~ -rw------- 1 alex users 180M Mar 15 15:49 coredump
> | ~ -rwxr-xr-x 1 alex users 2.0K Mar 15 16:51 gdb.sh
> | ~ drwxr-xr-x 2 alex users 4.0K Mar 15 16:56 lib64
> | ~ drwxr-xr-x 3 alex users 4.0K Mar 15 16:56 .rocks
> | ~ -rwxr-xr-x 1 alex users 33M Dec 30 17:08 tarantool
> | ~ -rw-r--r-- 1 alex users 27M Mar 15 16:54 tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | ~ drwxr-xr-x 3 alex users 4.0K Mar 15 16:56 usr
> | ~ -rw-r--r-- 1 alex users 1.1K Mar 15 16:05 version
> | (Meh. Let's move it around.)
> | $ mkdir tarantool-core-N-202103151305-cc18d0726f6a
> | $ mv checklist coredump lib64 .rocks tarantool usr version tarantool-core-N-202103151305-cc18d0726f6a
> | $ tree
> | ~ .
> | ~ ├── gdb.sh
> | ~ ├── tarantool-core-N-202103151305-cc18d0726f6a
> | ~ │ ├── checklist
> | ~ │ ├── coredump
> | ~ │ ├── lib64
> | ~ │ │ ├── <..skipped..>
> | ~ │ ├── tarantool
> | ~ │ ├── usr
> | ~ │ │ └── lib64
> | ~ │ │ └── tarantool
> | ~ │ │ └── avro_schema_rt_c.so
> | ~ │ └── version
> | ~ └── tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> | ~
> | ~ 5 directories, 37 files
> | (Now looks okay. Let's try.)
> | $ cd tarantool-core-N-202103151305-cc18d0726f6a
> | $ ../gdb.sh --help
> | ~ ================================================================================
> | ~
> | ~ Do not forget to properly setup the environment:
> | ~ * git clone https://github.com/tarantool/tarantool.git sources
> | ~ * cd !$
> | ~ * git checkout 34d504d
> | ~ * git submodule update --recursive --init
> | ~
> | ~ ================================================================================
> | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1
> | ~ <..skipped..>
> | ~ Reading symbols from ./tarantool...
> | ~ lj-arch command initialized
> | ~ lj-tv command initialized
> | ~ lj-str command initialized
> | ~ lj-tab command initialized
> | ~ lj-stack command initialized
> | ~ lj-state command initialized
> | ~ lj-gc command initialized
> | ~ luajit-gdb.py is successfully loaded
> | ~ warning: Can't open file /usr/bin/tarantool during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib/locale/locale-archive (deleted) during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libpcre.so.1.2.0 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libselinux.so.1 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libresolv-2.17.so (deleted) during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libkeyutils.so.1.5 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libkrb5support.so.0.1 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libk5crypto.so.3.1 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libcom_err.so.2.1 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libkrb5.so.3.3 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libgssapi_krb5.so.2.2 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libc-2.17.so (deleted) during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libm-2.17.so (deleted) during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libstdc++.so.6.0.19 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libunwind.so.8.0.1 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libunwind-x86_64.so.8.0.1 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libgcc_s-4.8.5-20150702.so.1 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libpthread-2.17.so (deleted) during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libgomp.so.1.0.0 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libcrypto.so.1.0.2k during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libssl.so.1.0.2k during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/librt-2.17.so (deleted) during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libz.so.1.2.7 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libform.so.5.9 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libtinfo.so.5.9 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libncurses.so.5.9 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libreadline.so.6.2 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libdl-2.17.so (deleted) during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libicudata.so.50.2 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libicuuc.so.50.2 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/libicui18n.so.50.2 during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/ld-2.17.so (deleted) during file-backed mapping note processing
> | ~ warning: Can't open file /usr/lib64/gconv/gconv-modules.cache (deleted) during file-backed mapping note processing
> | ~ [New LWP 246]
> | ~ [New LWP 247]
> | ~ [New LWP 248]
> | ~ [New LWP 250]
> | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> | ~ warning: the debug information found in "/usr/lib64/debug/usr/lib64/tarantool/avro_schema_rt_c.so.debug" does not match "/home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so" (CRC mismatch).
> | ~
> | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> | ~ Core was generated by `tarantool'.
> | ~ Program terminated with signal SIGABRT, Aborted.
> | ~ #0 0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
> | ~ [Current thread is 1 (LWP 246)]
> | (gdb) Ctrl+D
> | (Hey, --help runs it.)
Lol, I didn't even check this. Thanks for the example. I've added
arguments handling and the usage for the script. Diff is below:
================================================================================
diff --git a/tools/gdb.sh b/tools/gdb.sh
index a58c47cab..55fa73d7a 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -1,6 +1,40 @@
#!/bin/sh
set -eu
+TOOL=$(basename "$0")
+HELP=$(cat <<HELP
+${TOOL} - gdb wrapper loading artefacts collected via tarabrt.sh
+
+ Soon her eye fell upon a little glass box lying underneath
+ the table. She opened it and found in it a very small
+ cake, on which the words "EAT ME" were beautifully marked
+ in currants.
+
+ - Alice's Adventures in Wonderland.
+HELP
+)
+
+# Parse CLI options.
+OPTIONS=$(getopt -o h -n "${TOOL}" -- "$@")
+eval set -- "${OPTIONS}"
+while true; do
+ case "$1" in
+ --) shift; break;;
+ -h) printf "%s\n" "${HELP}";
+ exit 0;;
+ *) printf "Invalid option: $1\n%s\n" "${HELP}";
+ exit 1;;
+ esac
+done
+
+# Do not proceed if there are some CLI arguments left. Everything
+# should be parsed before this line. Refer to the help message in
+# a funny way.
+if [ $# -ne 0 ]; then
+ echo "RUN ME"
+ exit 1;
+fi
+
# Check that gdb is installed.
if ! command -v gdb >/dev/null; then
cat <<NOGDB
================================================================================
> | (Anyway, I should checkout sources. Nice reminder.)
> | $ git clone https://github.com/tarantool/tarantool.git sources
> | $ cd sources
> | $ git checkout 34d504d
> | $ git submodule update --recursive --init
> | $ cd ..
> | (Let's try again?)
> | $ ../gdb.sh
> | <..I'll skip the output, it is quite similar to one above. But will
> | comment below.>
> | (The reminder regarding tarantool sources still shown. But I already
> | placed the sources here.)
Added the check whether the sources are already setup. Diff is below:
================================================================================
diff --git a/tools/gdb.sh b/tools/gdb.sh
index fd310b99b..63e80c8f3 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -67,18 +67,30 @@ NOARTEFACTS
exit 1;
fi
-REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
-cat <<SOURCES
+SOURCES=${PWD}/sources
+
+# Check whether Tarantool sources are setup. Otherwise, leave a
+# recipe for user, how to do it.
+# FIXME: This can be done automatically (simply run the commands
+# below), but this obliges git to be installed. For now, it makes
+# the wrapper more complex by additional checks, so this activity
+# is left for the user.
+if [ ! -d "${SOURCES}" ]; then
+ REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
+ cat <<SOURCES
================================================================================
Do not forget to properly setup the environment:
-* git clone https://github.com/tarantool/tarantool.git sources
+* git clone https://github.com/tarantool/tarantool.git ${SOURCES}
* cd !$
* git checkout ${REVISION}
* git submodule update --recursive --init
+* cd -
================================================================================
SOURCES
+ exit 1;
+fi
# Define the build path to be substituted with the source path.
# XXX: Check the absolute path on the function <main> definition
================================================================================
> | (All library warnings above are still here. However it seems they're
> just warnings.)
> | $ (gdb) bt
> | ~ #0 0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
> | ~ #1 0x00007f340a5f5a78 in abort () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
> | ~ #2 0x0000000000453c43 in sig_fatal_cb (signo=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/main.cc:302
> | ~ #3 <signal handler called>
> | ~ #4 0x0000000000621ce7 in lj_cconv_ct_ct (cts=<optimized out>, d=<optimized out>, s=0x41508e70, dp=0x41d7a0b8 "", sp=0x140535248 <error: Cannot access memory at address 0x140535248>,
> | ~ flags=<optimized out>) at lj_cconv.c:234
> | ~ #5 0x0000000000621e61 in lj_cconv_tv_ct (cts=0x40fb9858, s=<optimized out>, sid=<optimized out>, o=0x41d7a0b8, sp=<optimized out>) at lj_cconv.c:389
> | ~ #6 0x0000000000621298 in lj_cdata_get (cts=<optimized out>, s=<optimized out>, o=<optimized out>, sp=<optimized out>) at lj_cdata.c:254
> | ~ #7 0x0000000000631021 in lj_cf_ffi_meta___index (L=0x40fb1378) at lib_ffi.c:158
> | ~ #8 0x00000000006338a7 in lj_BC_FUNCC () at buildvm_x86.dasc:809
> | ~ #9 0x0000000000615c82 in lua_pcall (L=L at entry=0x40fb1378, nargs=<optimized out>, nresults=<optimized out>, errfunc=errfunc at entry=0) at lj_api.c:1158
> | ~ #10 0x00000000005c4663 in luaT_call (L=0x40fb1378, nargs=<optimized out>, nreturns=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/utils.c:1022
> | ~ #11 0x00000000005bd259 in lua_main (L=L at entry=0x40fb1378, argc=argc at entry=1, argv=argv at entry=0x20d5fc0) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:546
> | ~ #12 0x00000000005bd6b3 in run_script_f (ap=<error reading variable: value has been optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:647
> | ~ #13 0x00000000004530dc in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=<optimized out>, ap=<optimized out>)
> | ~ at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.h:882
> | ~ #14 0x00000000005def90 in fiber_loop (data=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.c:879
> | ~ #15 0x00000000007c7ddf in coro_init () at /build/usr/src/debug/tarantool-2.6.2.0/third_party/coro/coro.c:110
> | (Looks good!)
> | (gdb) info sharedlibrary
> | ~ From To Syms Read Shared Object Library
> | ~ 0x00007f340e9aa5f0 0x00007f340eab0b62 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicui18n.so.50
> | ~ 0x00007f340e5eb2f0 0x00007f340e69dad8 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicuuc.so.50
> | ~ 0x00007f340cfc6550 0x00007f340cfc6635 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicudata.so.50
> | ~ 0x00007f340cdc2e50 0x00007f340cdc394e Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libdl.so.2
> | ~ 0x00007f340cb90de0 0x00007f340cbaf775 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libreadline.so.6
> | ~ 0x00007f340c95bd50 0x00007f340c975462 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libncurses.so.5
> | ~ 0x00007f340c737e40 0x00007f340c743bb8 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libtinfo.so.5
> | ~ 0x00007f340c51e6e0 0x00007f340c525ed0 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libform.so.5
> | ~ 0x00007f340c307110 0x00007f340c313698 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libz.so.1
> | ~ 0x00007f340c0ff230 0x00007f340c1021bc Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/librt.so.1
> | ~ 0x00007f340bea5b90 0x00007f340bee164d Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libssl.so.10
> | ~ 0x00007f340ba95200 0x00007f340bbce7b7 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcrypto.so.10
> | ~ 0x00007f340b809060 0x00007f340b81fc2a Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgomp.so.1
> | ~ 0x00007f340b5eb8f0 0x00007f340b5f6db1 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpthread.so.0
> | ~ 0x00007f340b3d2ad0 0x00007f340b3e2285 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgcc_s.so.1
> | ~ 0x00007f340b1b1570 0x00007f340b1be20e Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind-x86_64.so.8
> | ~ 0x00007f340af975b0 0x00007f340af9e95e Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind.so.8
> | ~ 0x00007f340ace94e0 0x00007f340ad5059a Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libstdc++.so.6
> | ~ 0x00007f340a991350 0x00007f340a9fc336 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libm.so.6
> | ~ 0x00007f340a5dd9f0 0x00007f340a72da5f Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
> | ~ 0x00007f340ed11af0 0x00007f340ed2d060 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/ld-linux-x86-64.so.2
> | ~ 0x00007f340a37d720 0x00007f340a3afbca Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgssapi_krb5.so.2
> | ~ 0x00007f340a0adc20 0x00007f340a1159fa Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5.so.3
> | ~ 0x00007f3409e85540 0x00007f3409e86113 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcom_err.so.2
> | ~ 0x00007f3409c558d0 0x00007f3409c73bbf Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libk5crypto.so.3
> | ~ 0x00007f3409a44bf0 0x00007f3409a4c36b Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5support.so.0
> | ~ 0x00007f340983e5b0 0x00007f340983f1cc Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkeyutils.so.1
> | ~ 0x00007f3409626980 0x00007f34096351e2 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libresolv.so.2
> | ~ 0x00007f3409402a90 0x00007f3409418896 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libselinux.so.1
> | ~ 0x00007f340919b5f0 0x00007f34091e15b0 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpcre.so.1
> | ~ 0x00007f3408c429f0 0x00007f3408c43896 Yes /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/.rocks/lib/tarantool/tuple/keydef.so
> | ~ 0x00007f33eedf8d80 0x00007f33eedfc174 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so
> | ~ (*): Shared library is missing debugging information.
> | (Looks good too!)
>
> > Can't open file <...> (deleted) during file-backed mapping note
> > processing.
>
> I don't know what it is. But I'll share my gdb version just in case
> (maybe you'll be able to reproduce it). I didn't meet this warning
> before.
I tried the script on some machines I have access to: my Gentoo
workstation, our CentOS 7 dev stand, CentOS 8 docker and my personal
Gentoo laptop. I faced the issue only on the latter machine. I guess it
doesn't relate to the script but rather to the environment, since the
problem also occurs when I simply run the following command:
| $ gdb ./tarantool coredump
>
> | $ gdb --version | head -n 1
> | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1
I have the same version on my laptop. My workstation wasn't updated for
some time, so there is only 8.3.1. Let's proceed with this problem in
private, since AFAICS it relates to Gentoo with the quite new gdb.
>
> However I would note that those warnings misleads me the first time: I
> thought that something is wrong with library paths.
>
> > Unable to find libthread_db matching inferior's thread library, thread
> > debugging will not be available.
>
> Sure, it is not listed in the lld output (neither in 'info
> sharedlibrary'), but maybe we should include it into the tarabrt.sh
> generated archive.
Ouch, it was a tough one. However, I've chosen the simplest solution for
this issue. Diff is below:
================================================================================
diff --git a/tools/gdb.sh b/tools/gdb.sh
index 63e80c8f3..669be3c4f 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -102,4 +102,6 @@ SUBPATH=$(gdb -batch -n ./tarantool -ex 'info line main' | \
gdb ./tarantool \
-ex "set sysroot $(realpath .)" \
-ex "set substitute-path ${SUBPATH} sources" \
+ -ex "add-auto-load-safe-path $(realpath .)" \
+ -ex "set auto-load libthread-db on" \
-ex 'core coredump'
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 8a07b88a6..aadfde129 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -242,6 +242,32 @@ COREMISMATCH
exit 1;
fi
+# XXX: This is poor man's libthread_db location resolution.
+# I'm sure there are many ways to implement this, but most of them
+# requires additional tools, libs, etc. The one implemented below
+# is quite simple, robust and requires nothing more than being
+# already used above.
+GREETING=$(gdb "${BINARY}" -c "${COREFILE}" -ex 'quit' 2>/dev/null)
+LIBTHREAD_DB=
+PATTERN1='[Thread debugging using libthread_db enabled]'
+PATTERN2='Using host libthread_db library ".*"\.'
+
+if echo "${GREETING}" | grep -F "${PATTERN1}" >/dev/null; then
+ LIBTHREAD_DB=$(echo "${GREETING}" | grep "${PATTERN2}" | \
+ awk -F'"' '{ print $2 }')
+ echo "${LIBTHREAD_DB}" >>"${TARLIST}"
+fi
+
+if [ -z "${LIBTHREAD_DB}" ]; then
+ [ -t 1 ] && cat <<NOLIBTHREAD_DB
+Failed to find libthread_db.so. Brace yourselves, thread debugging
+will be a challenging exercise.
+
+This is just a notice, not an error.
+NOLIBTHREAD_DB
+ # XXX: Not an error, proceed packing.
+fi
+
# Pack everything listed in TARLIST file into a tarball. To unify
# the archive format BINARY, COREFILE, VERSION and TARLIST are
# renamed while packing.
================================================================================
It's a really useful notice: now one can even work with TLS variables
with no prisedanie at all (I've checked this by transferring coredump
artefacts from C7 host to my Gentoo workstation and playing with them a
little). Thanks!
>
> > the debug information found in <...> does not match <...> (CRC mismatch).
>
> It is just because I have local avro schema debuginfo installed. I think
> this warning can be ignored.
I believe we can do nothing with it in general case, so user (or
developer) should fix this warning by himself.
>
> To sum up: my experience was successful. You may polish some tiny
> details, but in fact the scripts seem to be ready for use.
Glad to hear this, hope they'll be useful.
>
> ----
>
> We also briefly discussed the topic about stripped executables with
> Igor.
>
> Now we ship stripped executables in RPM packages for CentOS 8 (only for
> CentOS 8), but the separate debug package is stored in our repos.
>
> First point: we can include /etc/os-release into the generated archive.
I see no criminal in this, hence here are the changes:
================================================================================
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 6fc76e8f3..e09d051a4 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -7,11 +7,12 @@ ${TOOL} - Tarantool Automatic Bug Reporting Tool
This tool collects all required artefacts (listed below) and packs them into
a single archive with unified format:
- - /checklist - the plain text file with the list of tarball contents
- - /version - the plain text file containing \`tarantool --version' output
- - /tarantool - the executable binary file produced the core dump
- - /coredump - the core dump file produced by the executable
- - all shared libraries loaded (even via dlopen(3)) at the crash moment.
+ - /checklist - the plain text file with the list of tarball contents
+ - /version - the plain text file with \`tarantool --version' output
+ - /tarantool - the executable binary file produced the core dump
+ - /coredump - the core dump file produced by the executable
+ - /etc/os-release - the plain text file with OS identification data
+ - all shared libraries loaded (even via dlopen(3)) at the crash moment
SYNOPSIS
@@ -200,6 +201,7 @@ HOSTNAME=$(hostname 2>/dev/null || echo hostname)
TMPDIR=$(mktemp -d -p "${COREDIR}")
TARLIST=${TMPDIR}/tarlist
VERSION=${TMPDIR}/version
+OSRELEASE=/etc/os-release
ARCHIVENAME=tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}
ARCHIVE=${COREDIR}/${ARCHIVENAME}.tar.gz
@@ -210,6 +212,7 @@ ${BINARY} --version >"${VERSION}"
{
echo "${TARLIST}"
echo "${VERSION}"
+ echo "${OSRELEASE}"
echo "${BINARY}"
echo "${COREFILE}"
} >>"${TARLIST}"
================================================================================
>
> Second point: we can give URL from where debuginfo package may be
> downloaded and give advice how to unpack it (in gdb.sh output, when we
> see the stripped executable).
I'm totally for such helpers, but I am tuning this *draft* for such a
long time... Since this is not such popular case for now, I ignore
everything below and will create an issue for this.
>
> Examples:
>
> | $ curl -LfsS https://download.tarantool.org/tarantool/release/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm
> | $ mkdir tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
> | $ cd tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
> | $ rpm2cpio ../tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm | cpio -idmv
> | ~ ./usr/lib/debug
> | ~ ./usr/lib/debug/.build-id
> | ~ ./usr/lib/debug/.build-id/a3
> | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60
> | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60.debug
> | ~ ./usr/lib/debug/usr
> | ~ ./usr/lib/debug/usr/bin
> | ~ ./usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
> | ~ 64743 blocks
> | (All are symlinks except tarantool-2.8.0.0-1.el8.x86_64.debug.)
> | $ file usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
> | ~ usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter \004, for GNU/Linux 3.2.0, BuildID[sha1]=a3ad84db98a9b14e70d2915271904ddfe26b5f60, with debug_info, not stripped, too many notes (256)
>
> | $ curl -LfsS https://download.tarantool.org/tarantool/live/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm
>
> Third: we can give advice how to enable the ELF with debug symbols in
> the interactive gdb session.
>
> Or we should do it within gdb.sh to call 'info line main'? Expect it in
> some predefined location or accept an option?
>
> At least it looks worthful to warn about stripped executable explicitly.
>
> ----
>
> It seems, I reached maximum value that I can give as reviewer, so I'll
> just say that the patch looks good to me (and I don't mind small changes
> around usability if you want to do them).
Again, thanks a lot! Added your tag:
| Reviewed-by: Alexander Turenko <alexander.turenko at tarantool.org>
>
> WBR, Alexander Turenko.
--
Best regards,
IM
More information about the Tarantool-patches
mailing list