[Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
Alexander Turenko
alexander.turenko at tarantool.org
Mon Mar 15 19:30:46 MSK 2021
I looked over the WIP version of the script and nothing confuses me:
https://lists.tarantool.org/tarantool-patches/20210220104630.wiwxhkjfimsx2swa@tkn_work_nb/
So I guess I should do some basic testing to get and share my user
experience.
So I'll step into another environment to create coredump and archive it:
| $ mkdir ~/tmp/cores && docker run --volume $(realpath ~/tmp/cores):/cores -it centos:7 /bin/bash
| # curl -LfsS https://tarantool.io/release/2.6/installer.sh >installer.sh
| # chmod a+x installer.sh
| # ./installer.sh
| # yum install -y tarantool
| # ulimit -c unlimited
| # tarantool
Now we're in tarantool in a foreign environment. Let's initialize the
database and fill it with some data to imitate a real situation:
| tarantool> box.cfg{}
| tarantool> box.schema.create_space('s')
| tarantool> box.space.s:create_index('pk')
| tarantool> for i = 1, 1000 do box.space.s:insert({i}) end
And load some external C module (one from rocks, one from packages):
| tarantool> Ctrl+Z
| # yum install -y tarantool-devel git cmake gcc
| # tarantoolctl rocks install tuple-keydef
| # yum install -y tarantool-avro-schema
| # fg
| tarantool> require('tuple.keydef')
| tarantool> require('avro_schema')
Let's coredump myself (I'll prepend commands output with '~' symbols to
ease reading):
| tarantool> require('ffi').new('int[?]', 1)[1024 ^ 3]
| ~ <...>
| ~ Aborted (core dumped)
| # ls -hl core
| ~ -rw------- 1 root root 180M Mar 15 12:49 core
| # file core
| ~ bash: file: command not found
(I'll ignore that 'file' is not installed, because it is possible
situation on a production server, I guess.)
Okay, now I got the tarabrt.sh script somehow:
| # curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/tarabrt.sh >tarabrt.sh
| # chmod a+x tarabrt.sh
| (Don't read the content, just run.)
| # ./tarabrt.sh --help
| ~ tarabrt.sh: unrecognized option '--help'
| # ./tarabrt.sh -h
| ~ <..skipped..>
| ~ - Manual usage. User can simply pack all necessary artefacts by running the
| ~ following command.
| ~ $ /path/to/tarabrt.sh -c ./core -d /tmp
| ~ <..skipped..>
| # ./tarabrt.sh -c ./core -d /tmp
| ~ ./tarabrt.sh: line 115: file: command not found
| ~ gdb is not installed, but it is obligatory for collecting the
| ~ loaded shared libraries from the core dump.
| ~
| ~ You can proceed collecting the artefacts manually later by running
| ~ the following command:
| ~ $ tarabrt.sh -e /usr/bin/tarantool -c ./core
| (Okay, I'll install file and gdb now.)
| # yum install -y file gdb
| # file core
| ~ core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'tarantool', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/tarantool', platform: 'x86_64'
| (Repeat the attempt to create the archive.)
| # ./tarabrt.sh -c ./core -d /tmp
| ~ The resulting archive is located here: /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| ~ The archive contents are listed below:
| ~ /tmp/tmp.kqC76QSrpF/tarlist
| ~ /tmp/tmp.kqC76QSrpF/version
| ~ /usr/bin/tarantool
| ~ ./core
| ~ /lib64/libicui18n.so.50
| ~ /lib64/libicuuc.so.50
| ~ /lib64/libicudata.so.50
| ~ /lib64/libdl.so.2
| ~ /lib64/libreadline.so.6
| ~ /lib64/libncurses.so.5
| ~ /lib64/libtinfo.so.5
| ~ /lib64/libform.so.5
| ~ /lib64/libz.so.1
| ~ /lib64/librt.so.1
| ~ /lib64/libssl.so.10
| ~ /lib64/libcrypto.so.10
| ~ /lib64/libgomp.so.1
| ~ /lib64/libpthread.so.0
| ~ /lib64/libgcc_s.so.1
| ~ /lib64/libunwind-x86_64.so.8
| ~ /lib64/libunwind.so.8
| ~ /lib64/libstdc++.so.6
| ~ /lib64/libm.so.6
| ~ /lib64/libc.so.6
| ~ /lib64/ld-linux-x86-64.so.2
| ~ /lib64/libgssapi_krb5.so.2
| ~ /lib64/libkrb5.so.3
| ~ /lib64/libcom_err.so.2
| ~ /lib64/libk5crypto.so.3
| ~ /lib64/libkrb5support.so.0
| ~ /lib64/libkeyutils.so.1
| ~ /lib64/libresolv.so.2
| ~ /lib64/libselinux.so.1
| ~ /lib64/libpcre.so.1
| ~ /.rocks/lib/tarantool/tuple/keydef.so
| ~ /usr/lib64/tarantool/avro_schema_rt_c.so
| ~
| ~ If you want to upload it, choose the available resource
| ~ (e.g. http://transfer.sh) and run the following command:
| ~ $ curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz <resource-uri>
Okay, let's spin around this output for a while:
| (The /tmp/ entries looks are something I can glance.)
| # ll /tmp/tmp.kqC76QSrpF/tarlist
| ~ ls: cannot access /tmp/tmp.kqC76QSrpF/tarlist: No such file or directory
| ll /tmp/tmp.kqC76QSrpF/version
| ~ ls: cannot access /tmp/tmp.kqC76QSrpF/version: No such file or directory
| (But they're not. Okay, pass it over.)
| (keydef.so and avro_schema_rt_c.so are here, nice!)
| (Let's look at the archive file.)
| # ls -hl /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| ~ -rw-r--r-- 1 root root 27M Mar 15 13:05 /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| # tar -vtf /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| ~ -rw-r--r-- root/root 1085 2021-03-15 13:05 version
| ~ -rwxr-xr-x root/root 34431952 2020-12-30 14:08 tarantool
| ~ -rw------- root/root 188039168 2021-03-15 12:49 coredump
| ~ tar: Removing leading `/' from member names
| ~ -rwxr-xr-x root/root 2096104 2020-03-21 00:49 /lib64/libicui18n.so.50
| ~ -rwxr-xr-x root/root 1539520 2020-03-21 00:49 /lib64/libicuuc.so.50
| ~ -rwxr-xr-x root/root 20785824 2020-03-21 00:49 /lib64/libicudata.so.50
| ~ -rwxr-xr-x root/root 19248 2021-02-08 15:46 /lib64/libdl.so.2
| ~ -rwxr-xr-x root/root 285136 2019-08-08 12:07 /lib64/libreadline.so.6
| ~ -rwxr-xr-x root/root 163696 2017-09-06 22:08 /lib64/libncurses.so.5
| ~ -rwxr-xr-x root/root 174576 2017-09-06 22:08 /lib64/libtinfo.so.5
| ~ -rwxr-xr-x root/root 67864 2017-09-06 22:08 /lib64/libform.so.5
| ~ -rwxr-xr-x root/root 90248 2018-10-30 20:24 /lib64/libz.so.1
| ~ -rwxr-xr-x root/root 43712 2021-02-08 15:46 /lib64/librt.so.1
| ~ -rwxr-xr-x root/root 470376 2020-12-16 21:24 /lib64/libssl.so.10
| ~ -rwxr-xr-x root/root 2520768 2020-12-16 21:24 /lib64/libcrypto.so.10
| ~ -rwxr-xr-x root/root 154840 2020-09-30 02:17 /lib64/libgomp.so.1
| ~ -rwxr-xr-x root/root 142144 2021-02-08 15:46 /lib64/libpthread.so.0
| ~ -rwxr-xr-x root/root 88720 2020-09-30 02:18 /lib64/libgcc_s.so.1
| ~ -rwxr-xr-x root/root 73416 2017-08-02 15:45 /lib64/libunwind-x86_64.so.8
| ~ -rwxr-xr-x root/root 48904 2017-08-02 15:45 /lib64/libunwind.so.8
| ~ -rwxr-xr-x root/root 995840 2020-09-30 02:17 /lib64/libstdc++.so.6
| ~ -rwxr-xr-x root/root 1136944 2021-02-08 15:46 /lib64/libm.so.6
| ~ -rwxr-xr-x root/root 2156344 2021-02-08 15:46 /lib64/libc.so.6
| ~ -rwxr-xr-x root/root 163312 2021-02-08 15:46 /lib64/ld-linux-x86-64.so.2
| ~ -rwxr-xr-x root/root 320720 2020-09-30 16:51 /lib64/libgssapi_krb5.so.2
| ~ -rwxr-xr-x root/root 967840 2020-09-30 16:51 /lib64/libkrb5.so.3
| ~ -rwxr-xr-x root/root 15856 2020-09-30 15:58 /lib64/libcom_err.so.2
| ~ -rwxr-xr-x root/root 210784 2020-09-30 16:51 /lib64/libk5crypto.so.3
| ~ -rwxr-xr-x root/root 67104 2020-09-30 16:51 /lib64/libkrb5support.so.0
| ~ -rwxr-xr-x root/root 15688 2014-06-10 02:17 /lib64/libkeyutils.so.1
| ~ -rwxr-xr-x root/root 109976 2021-02-08 15:46 /lib64/libresolv.so.2
| ~ -rwxr-xr-x root/root 155744 2020-04-01 03:16 /lib64/libselinux.so.1
| ~ -rwxr-xr-x root/root 402384 2017-08-02 03:08 /lib64/libpcre.so.1
| ~ -rwxr-xr-x root/root 52168 2021-03-15 12:45 /.rocks/lib/tarantool/tuple/keydef.so
| ~ -rwxr-xr-x root/root 23736 2020-08-05 23:10 /usr/lib64/tarantool/avro_schema_rt_c.so
| ~ -rw-r--r-- root/root 796 2021-03-15 13:05 checklist
| (Can we avoid the the warning re `/`? Just curious.)
| (Okay, I see that transfer.sh is suggested. I never use it, so I'm
| curious.)
| # curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz http://transfer.sh
| ~ http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| (Hey, something occurs! But I'll save it to my system anyway.)
| # cp /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz /cores/
| (And, while I'm here...)
| # md5sum /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| ~ bec8f81c69ed992a258a6d9f0b8a04dc /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
Okay, let's try gdb.sh on my Gentoo box:
| $ mkdir ~/tmp/try-gdb-sh
| $ cd ~/tmp/try-gdb-sh
| $ curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/gdb.sh > gdb.sh
| $ chmod a+x gdb.sh
| $ curl -LfsS http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz > tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| $ md5sum tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| ~ bec8f81c69ed992a258a6d9f0b8a04dc tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| (Matches!)
| (Hm, now I'm worry, whether the archive will be uncompressed into the
| current directory or will create its own directory?)
| $ tar -xf tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| ~ tar: Removing leading `/' from member names
| $ ll
| ~ total 239M
| ~ drwxr-xr-x 5 alex users 4.0K Mar 15 16:56 .
| ~ drwxr-xr-x 164 alex users 20K Mar 15 16:50 ..
| ~ -rw-r--r-- 1 alex users 796 Mar 15 16:05 checklist
| ~ -rw------- 1 alex users 180M Mar 15 15:49 coredump
| ~ -rwxr-xr-x 1 alex users 2.0K Mar 15 16:51 gdb.sh
| ~ drwxr-xr-x 2 alex users 4.0K Mar 15 16:56 lib64
| ~ drwxr-xr-x 3 alex users 4.0K Mar 15 16:56 .rocks
| ~ -rwxr-xr-x 1 alex users 33M Dec 30 17:08 tarantool
| ~ -rw-r--r-- 1 alex users 27M Mar 15 16:54 tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| ~ drwxr-xr-x 3 alex users 4.0K Mar 15 16:56 usr
| ~ -rw-r--r-- 1 alex users 1.1K Mar 15 16:05 version
| (Meh. Let's move it around.)
| $ mkdir tarantool-core-N-202103151305-cc18d0726f6a
| $ mv checklist coredump lib64 .rocks tarantool usr version tarantool-core-N-202103151305-cc18d0726f6a
| $ tree
| ~ .
| ~ ├── gdb.sh
| ~ ├── tarantool-core-N-202103151305-cc18d0726f6a
| ~ │ ├── checklist
| ~ │ ├── coredump
| ~ │ ├── lib64
| ~ │ │ ├── <..skipped..>
| ~ │ ├── tarantool
| ~ │ ├── usr
| ~ │ │ └── lib64
| ~ │ │ └── tarantool
| ~ │ │ └── avro_schema_rt_c.so
| ~ │ └── version
| ~ └── tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
| ~
| ~ 5 directories, 37 files
| (Now looks okay. Let's try.)
| $ cd tarantool-core-N-202103151305-cc18d0726f6a
| $ ../gdb.sh --help
| ~ ================================================================================
| ~
| ~ Do not forget to properly setup the environment:
| ~ * git clone https://github.com/tarantool/tarantool.git sources
| ~ * cd !$
| ~ * git checkout 34d504d
| ~ * git submodule update --recursive --init
| ~
| ~ ================================================================================
| ~ GNU gdb (Gentoo 10.1 vanilla) 10.1
| ~ <..skipped..>
| ~ Reading symbols from ./tarantool...
| ~ lj-arch command initialized
| ~ lj-tv command initialized
| ~ lj-str command initialized
| ~ lj-tab command initialized
| ~ lj-stack command initialized
| ~ lj-state command initialized
| ~ lj-gc command initialized
| ~ luajit-gdb.py is successfully loaded
| ~ warning: Can't open file /usr/bin/tarantool during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib/locale/locale-archive (deleted) during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libpcre.so.1.2.0 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libselinux.so.1 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libresolv-2.17.so (deleted) during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libkeyutils.so.1.5 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libkrb5support.so.0.1 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libk5crypto.so.3.1 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libcom_err.so.2.1 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libkrb5.so.3.3 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libgssapi_krb5.so.2.2 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libc-2.17.so (deleted) during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libm-2.17.so (deleted) during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libstdc++.so.6.0.19 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libunwind.so.8.0.1 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libunwind-x86_64.so.8.0.1 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libgcc_s-4.8.5-20150702.so.1 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libpthread-2.17.so (deleted) during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libgomp.so.1.0.0 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libcrypto.so.1.0.2k during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libssl.so.1.0.2k during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/librt-2.17.so (deleted) during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libz.so.1.2.7 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libform.so.5.9 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libtinfo.so.5.9 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libncurses.so.5.9 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libreadline.so.6.2 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libdl-2.17.so (deleted) during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libicudata.so.50.2 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libicuuc.so.50.2 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/libicui18n.so.50.2 during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/ld-2.17.so (deleted) during file-backed mapping note processing
| ~ warning: Can't open file /usr/lib64/gconv/gconv-modules.cache (deleted) during file-backed mapping note processing
| ~ [New LWP 246]
| ~ [New LWP 247]
| ~ [New LWP 248]
| ~ [New LWP 250]
| ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
| ~ warning: the debug information found in "/usr/lib64/debug/usr/lib64/tarantool/avro_schema_rt_c.so.debug" does not match "/home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so" (CRC mismatch).
| ~
| ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
| ~ Core was generated by `tarantool'.
| ~ Program terminated with signal SIGABRT, Aborted.
| ~ #0 0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
| ~ [Current thread is 1 (LWP 246)]
| (gdb) Ctrl+D
| (Hey, --help runs it.)
| (Anyway, I should checkout sources. Nice reminder.)
| $ git clone https://github.com/tarantool/tarantool.git sources
| $ cd sources
| $ git checkout 34d504d
| $ git submodule update --recursive --init
| $ cd ..
| (Let's try again?)
| $ ../gdb.sh
| <..I'll skip the output, it is quite similar to one above. But will
| comment below.>
| (The reminder regarding tarantool sources still shown. But I already
| placed the sources here.)
| (All library warnings above are still here. However it seems they're
just warnings.)
| $ (gdb) bt
| ~ #0 0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
| ~ #1 0x00007f340a5f5a78 in abort () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
| ~ #2 0x0000000000453c43 in sig_fatal_cb (signo=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/main.cc:302
| ~ #3 <signal handler called>
| ~ #4 0x0000000000621ce7 in lj_cconv_ct_ct (cts=<optimized out>, d=<optimized out>, s=0x41508e70, dp=0x41d7a0b8 "", sp=0x140535248 <error: Cannot access memory at address 0x140535248>,
| ~ flags=<optimized out>) at lj_cconv.c:234
| ~ #5 0x0000000000621e61 in lj_cconv_tv_ct (cts=0x40fb9858, s=<optimized out>, sid=<optimized out>, o=0x41d7a0b8, sp=<optimized out>) at lj_cconv.c:389
| ~ #6 0x0000000000621298 in lj_cdata_get (cts=<optimized out>, s=<optimized out>, o=<optimized out>, sp=<optimized out>) at lj_cdata.c:254
| ~ #7 0x0000000000631021 in lj_cf_ffi_meta___index (L=0x40fb1378) at lib_ffi.c:158
| ~ #8 0x00000000006338a7 in lj_BC_FUNCC () at buildvm_x86.dasc:809
| ~ #9 0x0000000000615c82 in lua_pcall (L=L at entry=0x40fb1378, nargs=<optimized out>, nresults=<optimized out>, errfunc=errfunc at entry=0) at lj_api.c:1158
| ~ #10 0x00000000005c4663 in luaT_call (L=0x40fb1378, nargs=<optimized out>, nreturns=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/utils.c:1022
| ~ #11 0x00000000005bd259 in lua_main (L=L at entry=0x40fb1378, argc=argc at entry=1, argv=argv at entry=0x20d5fc0) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:546
| ~ #12 0x00000000005bd6b3 in run_script_f (ap=<error reading variable: value has been optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:647
| ~ #13 0x00000000004530dc in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=<optimized out>, ap=<optimized out>)
| ~ at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.h:882
| ~ #14 0x00000000005def90 in fiber_loop (data=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.c:879
| ~ #15 0x00000000007c7ddf in coro_init () at /build/usr/src/debug/tarantool-2.6.2.0/third_party/coro/coro.c:110
| (Looks good!)
| (gdb) info sharedlibrary
| ~ From To Syms Read Shared Object Library
| ~ 0x00007f340e9aa5f0 0x00007f340eab0b62 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicui18n.so.50
| ~ 0x00007f340e5eb2f0 0x00007f340e69dad8 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicuuc.so.50
| ~ 0x00007f340cfc6550 0x00007f340cfc6635 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicudata.so.50
| ~ 0x00007f340cdc2e50 0x00007f340cdc394e Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libdl.so.2
| ~ 0x00007f340cb90de0 0x00007f340cbaf775 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libreadline.so.6
| ~ 0x00007f340c95bd50 0x00007f340c975462 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libncurses.so.5
| ~ 0x00007f340c737e40 0x00007f340c743bb8 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libtinfo.so.5
| ~ 0x00007f340c51e6e0 0x00007f340c525ed0 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libform.so.5
| ~ 0x00007f340c307110 0x00007f340c313698 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libz.so.1
| ~ 0x00007f340c0ff230 0x00007f340c1021bc Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/librt.so.1
| ~ 0x00007f340bea5b90 0x00007f340bee164d Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libssl.so.10
| ~ 0x00007f340ba95200 0x00007f340bbce7b7 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcrypto.so.10
| ~ 0x00007f340b809060 0x00007f340b81fc2a Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgomp.so.1
| ~ 0x00007f340b5eb8f0 0x00007f340b5f6db1 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpthread.so.0
| ~ 0x00007f340b3d2ad0 0x00007f340b3e2285 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgcc_s.so.1
| ~ 0x00007f340b1b1570 0x00007f340b1be20e Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind-x86_64.so.8
| ~ 0x00007f340af975b0 0x00007f340af9e95e Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind.so.8
| ~ 0x00007f340ace94e0 0x00007f340ad5059a Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libstdc++.so.6
| ~ 0x00007f340a991350 0x00007f340a9fc336 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libm.so.6
| ~ 0x00007f340a5dd9f0 0x00007f340a72da5f Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
| ~ 0x00007f340ed11af0 0x00007f340ed2d060 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/ld-linux-x86-64.so.2
| ~ 0x00007f340a37d720 0x00007f340a3afbca Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgssapi_krb5.so.2
| ~ 0x00007f340a0adc20 0x00007f340a1159fa Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5.so.3
| ~ 0x00007f3409e85540 0x00007f3409e86113 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcom_err.so.2
| ~ 0x00007f3409c558d0 0x00007f3409c73bbf Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libk5crypto.so.3
| ~ 0x00007f3409a44bf0 0x00007f3409a4c36b Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5support.so.0
| ~ 0x00007f340983e5b0 0x00007f340983f1cc Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkeyutils.so.1
| ~ 0x00007f3409626980 0x00007f34096351e2 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libresolv.so.2
| ~ 0x00007f3409402a90 0x00007f3409418896 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libselinux.so.1
| ~ 0x00007f340919b5f0 0x00007f34091e15b0 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpcre.so.1
| ~ 0x00007f3408c429f0 0x00007f3408c43896 Yes /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/.rocks/lib/tarantool/tuple/keydef.so
| ~ 0x00007f33eedf8d80 0x00007f33eedfc174 Yes (*) /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so
| ~ (*): Shared library is missing debugging information.
| (Looks good too!)
> Can't open file <...> (deleted) during file-backed mapping note
> processing.
I don't know what it is. But I'll share my gdb version just in case
(maybe you'll be able to reproduce it). I didn't meet this warning
before.
| $ gdb --version | head -n 1
| ~ GNU gdb (Gentoo 10.1 vanilla) 10.1
However I would note that those warnings misleads me the first time: I
thought that something is wrong with library paths.
> Unable to find libthread_db matching inferior's thread library, thread
> debugging will not be available.
Sure, it is not listed in the lld output (neither in 'info
sharedlibrary'), but maybe we should include it into the tarabrt.sh
generated archive.
> the debug information found in <...> does not match <...> (CRC mismatch).
It is just because I have local avro schema debuginfo installed. I think
this warning can be ignored.
To sum up: my experience was successful. You may polish some tiny
details, but in fact the scripts seem to be ready for use.
----
We also briefly discussed the topic about stripped executables with
Igor.
Now we ship stripped executables in RPM packages for CentOS 8 (only for
CentOS 8), but the separate debug package is stored in our repos.
First point: we can include /etc/os-release into the generated archive.
Second point: we can give URL from where debuginfo package may be
downloaded and give advice how to unpack it (in gdb.sh output, when we
see the stripped executable).
Examples:
| $ curl -LfsS https://download.tarantool.org/tarantool/release/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm
| $ mkdir tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
| $ cd tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
| $ rpm2cpio ../tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm | cpio -idmv
| ~ ./usr/lib/debug
| ~ ./usr/lib/debug/.build-id
| ~ ./usr/lib/debug/.build-id/a3
| ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60
| ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60.debug
| ~ ./usr/lib/debug/usr
| ~ ./usr/lib/debug/usr/bin
| ~ ./usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
| ~ 64743 blocks
| (All are symlinks except tarantool-2.8.0.0-1.el8.x86_64.debug.)
| $ file usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
| ~ usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter \004, for GNU/Linux 3.2.0, BuildID[sha1]=a3ad84db98a9b14e70d2915271904ddfe26b5f60, with debug_info, not stripped, too many notes (256)
| $ curl -LfsS https://download.tarantool.org/tarantool/live/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm
Third: we can give advice how to enable the ELF with debug symbols in
the interactive gdb session.
Or we should do it within gdb.sh to call 'info line main'? Expect it in
some predefined location or accept an option?
At least it looks worthful to warn about stripped executable explicitly.
----
It seems, I reached maximum value that I can give as reviewer, so I'll
just say that the patch looks good to me (and I don't mind small changes
around usability if you want to do them).
WBR, Alexander Turenko.
More information about the Tarantool-patches
mailing list