Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts

[Alexander Turenko via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

I looked over the WIP version of the script and nothing confuses me:

https://lists.tarantool.org/tarantool-patches/20210220104630.wiwxhkjfimsx2swa@tkn_work_nb/

So I guess I should do some basic testing to get and share my user
experience.

So I'll step into another environment to create coredump and archive it:

 | $ mkdir ~/tmp/cores && docker run --volume $(realpath ~/tmp/cores):/cores -it centos:7 /bin/bash
 | # curl -LfsS https://tarantool.io/release/2.6/installer.sh >installer.sh
 | # chmod a+x installer.sh
 | # ./installer.sh
 | # yum install -y tarantool
 | # ulimit -c unlimited
 | # tarantool

Now we're in tarantool in a foreign environment. Let's initialize the
database and fill it with some data to imitate a real situation:

 | tarantool> box.cfg{}
 | tarantool> box.schema.create_space('s')
 | tarantool> box.space.s:create_index('pk')
 | tarantool> for i = 1, 1000 do box.space.s:insert({i}) end

And load some external C module (one from rocks, one from packages):

 | tarantool> Ctrl+Z
 | # yum install -y tarantool-devel git cmake gcc
 | # tarantoolctl rocks install tuple-keydef
 | # yum install -y tarantool-avro-schema
 | # fg
 | tarantool> require('tuple.keydef')
 | tarantool> require('avro_schema')

Let's coredump myself (I'll prepend commands output with '~' symbols to
ease reading):

 | tarantool> require('ffi').new('int[?]', 1)[1024 ^ 3]
 | ~ <...>
 | ~ Aborted (core dumped)
 | # ls -hl core
 | ~ -rw------- 1 root root 180M Mar 15 12:49 core
 | # file core
 | ~ bash: file: command not found

(I'll ignore that 'file' is not installed, because it is possible
situation on a production server, I guess.)

Okay, now I got the tarabrt.sh script somehow:

 | # curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/tarabrt.sh >tarabrt.sh
 | # chmod a+x tarabrt.sh
 | (Don't read the content, just run.)
 | # ./tarabrt.sh --help
 | ~ tarabrt.sh: unrecognized option '--help'
 | # ./tarabrt.sh -h  
 | ~ <..skipped..>
 | ~   - Manual usage. User can simply pack all necessary artefacts by running the
 | ~     following command.
 | ~     $ /path/to/tarabrt.sh -c ./core -d /tmp
 | ~ <..skipped..>
 | # ./tarabrt.sh -c ./core -d /tmp
 | ~ ./tarabrt.sh: line 115: file: command not found
 | ~ gdb is not installed, but it is obligatory for collecting the
 | ~ loaded shared libraries from the core dump.
 | ~ 
 | ~ You can proceed collecting the artefacts manually later by running
 | ~ the following command:
 | ~ $ tarabrt.sh -e /usr/bin/tarantool -c ./core
 | (Okay, I'll install file and gdb now.)
 | # yum install -y file gdb
 | # file core
 | ~ core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'tarantool', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/tarantool', platform: 'x86_64'
 | (Repeat the attempt to create the archive.)
 | # ./tarabrt.sh -c ./core -d /tmp
 | ~ The resulting archive is located here: /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ The archive contents are listed below:
 | ~ /tmp/tmp.kqC76QSrpF/tarlist
 | ~ /tmp/tmp.kqC76QSrpF/version
 | ~ /usr/bin/tarantool
 | ~ ./core
 | ~ /lib64/libicui18n.so.50
 | ~ /lib64/libicuuc.so.50
 | ~ /lib64/libicudata.so.50
 | ~ /lib64/libdl.so.2
 | ~ /lib64/libreadline.so.6
 | ~ /lib64/libncurses.so.5
 | ~ /lib64/libtinfo.so.5
 | ~ /lib64/libform.so.5
 | ~ /lib64/libz.so.1
 | ~ /lib64/librt.so.1
 | ~ /lib64/libssl.so.10
 | ~ /lib64/libcrypto.so.10
 | ~ /lib64/libgomp.so.1
 | ~ /lib64/libpthread.so.0
 | ~ /lib64/libgcc_s.so.1
 | ~ /lib64/libunwind-x86_64.so.8
 | ~ /lib64/libunwind.so.8
 | ~ /lib64/libstdc++.so.6
 | ~ /lib64/libm.so.6
 | ~ /lib64/libc.so.6
 | ~ /lib64/ld-linux-x86-64.so.2
 | ~ /lib64/libgssapi_krb5.so.2
 | ~ /lib64/libkrb5.so.3
 | ~ /lib64/libcom_err.so.2
 | ~ /lib64/libk5crypto.so.3
 | ~ /lib64/libkrb5support.so.0
 | ~ /lib64/libkeyutils.so.1
 | ~ /lib64/libresolv.so.2
 | ~ /lib64/libselinux.so.1
 | ~ /lib64/libpcre.so.1
 | ~ /.rocks/lib/tarantool/tuple/keydef.so
 | ~ /usr/lib64/tarantool/avro_schema_rt_c.so
 | ~ 
 | ~ If you want to upload it, choose the available resource
 | ~ (e.g. http://transfer.sh) and run the following command:
 | ~ $ curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz <resource-uri>

Okay, let's spin around this output for a while:

 | (The /tmp/ entries looks are something I can glance.)
 | # ll /tmp/tmp.kqC76QSrpF/tarlist
 | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/tarlist: No such file or directory
 | ll /tmp/tmp.kqC76QSrpF/version
 | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/version: No such file or directory
 | (But they're not. Okay, pass it over.)
 | (keydef.so and avro_schema_rt_c.so are here, nice!)
 | (Let's look at the archive file.)
 | # ls -hl /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ -rw-r--r-- 1 root root 27M Mar 15 13:05 /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | # tar -vtf /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ -rw-r--r-- root/root      1085 2021-03-15 13:05 version
 | ~ -rwxr-xr-x root/root  34431952 2020-12-30 14:08 tarantool
 | ~ -rw------- root/root 188039168 2021-03-15 12:49 coredump
 | ~ tar: Removing leading `/' from member names
 | ~ -rwxr-xr-x root/root   2096104 2020-03-21 00:49 /lib64/libicui18n.so.50
 | ~ -rwxr-xr-x root/root   1539520 2020-03-21 00:49 /lib64/libicuuc.so.50
 | ~ -rwxr-xr-x root/root  20785824 2020-03-21 00:49 /lib64/libicudata.so.50
 | ~ -rwxr-xr-x root/root     19248 2021-02-08 15:46 /lib64/libdl.so.2
 | ~ -rwxr-xr-x root/root    285136 2019-08-08 12:07 /lib64/libreadline.so.6
 | ~ -rwxr-xr-x root/root    163696 2017-09-06 22:08 /lib64/libncurses.so.5
 | ~ -rwxr-xr-x root/root    174576 2017-09-06 22:08 /lib64/libtinfo.so.5
 | ~ -rwxr-xr-x root/root     67864 2017-09-06 22:08 /lib64/libform.so.5
 | ~ -rwxr-xr-x root/root     90248 2018-10-30 20:24 /lib64/libz.so.1
 | ~ -rwxr-xr-x root/root     43712 2021-02-08 15:46 /lib64/librt.so.1
 | ~ -rwxr-xr-x root/root    470376 2020-12-16 21:24 /lib64/libssl.so.10
 | ~ -rwxr-xr-x root/root   2520768 2020-12-16 21:24 /lib64/libcrypto.so.10
 | ~ -rwxr-xr-x root/root    154840 2020-09-30 02:17 /lib64/libgomp.so.1
 | ~ -rwxr-xr-x root/root    142144 2021-02-08 15:46 /lib64/libpthread.so.0
 | ~ -rwxr-xr-x root/root     88720 2020-09-30 02:18 /lib64/libgcc_s.so.1
 | ~ -rwxr-xr-x root/root     73416 2017-08-02 15:45 /lib64/libunwind-x86_64.so.8
 | ~ -rwxr-xr-x root/root     48904 2017-08-02 15:45 /lib64/libunwind.so.8
 | ~ -rwxr-xr-x root/root    995840 2020-09-30 02:17 /lib64/libstdc++.so.6
 | ~ -rwxr-xr-x root/root   1136944 2021-02-08 15:46 /lib64/libm.so.6
 | ~ -rwxr-xr-x root/root   2156344 2021-02-08 15:46 /lib64/libc.so.6
 | ~ -rwxr-xr-x root/root    163312 2021-02-08 15:46 /lib64/ld-linux-x86-64.so.2
 | ~ -rwxr-xr-x root/root    320720 2020-09-30 16:51 /lib64/libgssapi_krb5.so.2
 | ~ -rwxr-xr-x root/root    967840 2020-09-30 16:51 /lib64/libkrb5.so.3
 | ~ -rwxr-xr-x root/root     15856 2020-09-30 15:58 /lib64/libcom_err.so.2
 | ~ -rwxr-xr-x root/root    210784 2020-09-30 16:51 /lib64/libk5crypto.so.3
 | ~ -rwxr-xr-x root/root     67104 2020-09-30 16:51 /lib64/libkrb5support.so.0
 | ~ -rwxr-xr-x root/root     15688 2014-06-10 02:17 /lib64/libkeyutils.so.1
 | ~ -rwxr-xr-x root/root    109976 2021-02-08 15:46 /lib64/libresolv.so.2
 | ~ -rwxr-xr-x root/root    155744 2020-04-01 03:16 /lib64/libselinux.so.1
 | ~ -rwxr-xr-x root/root    402384 2017-08-02 03:08 /lib64/libpcre.so.1
 | ~ -rwxr-xr-x root/root     52168 2021-03-15 12:45 /.rocks/lib/tarantool/tuple/keydef.so
 | ~ -rwxr-xr-x root/root     23736 2020-08-05 23:10 /usr/lib64/tarantool/avro_schema_rt_c.so
 | ~ -rw-r--r-- root/root       796 2021-03-15 13:05 checklist
 | (Can we avoid the the warning re `/`? Just curious.)
 | (Okay, I see that transfer.sh is suggested. I never use it, so I'm
 |  curious.)
 | # curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz http://transfer.sh
 | ~ http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | (Hey, something occurs! But I'll save it to my system anyway.)
 | # cp /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz /cores/
 | (And, while I'm here...)
 | # md5sum /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ bec8f81c69ed992a258a6d9f0b8a04dc  /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz

Okay, let's try gdb.sh on my Gentoo box:

 | $ mkdir ~/tmp/try-gdb-sh
 | $ cd ~/tmp/try-gdb-sh
 | $ curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/gdb.sh > gdb.sh
 | $ chmod a+x gdb.sh
 | $ curl -LfsS http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz > tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | $ md5sum tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ bec8f81c69ed992a258a6d9f0b8a04dc  tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | (Matches!)
 | (Hm, now I'm worry, whether the archive will be uncompressed into the
 |  current directory or will create its own directory?)
 | $ tar -xf tarantool-core-N-202103151305-cc18d0726f6a.tar.gz 
 | ~ tar: Removing leading `/' from member names
 | $ ll
 | ~ total 239M
 | ~ drwxr-xr-x   5 alex users 4.0K Mar 15 16:56 .
 | ~ drwxr-xr-x 164 alex users  20K Mar 15 16:50 ..
 | ~ -rw-r--r--   1 alex users  796 Mar 15 16:05 checklist
 | ~ -rw-------   1 alex users 180M Mar 15 15:49 coredump
 | ~ -rwxr-xr-x   1 alex users 2.0K Mar 15 16:51 gdb.sh
 | ~ drwxr-xr-x   2 alex users 4.0K Mar 15 16:56 lib64
 | ~ drwxr-xr-x   3 alex users 4.0K Mar 15 16:56 .rocks
 | ~ -rwxr-xr-x   1 alex users  33M Dec 30 17:08 tarantool
 | ~ -rw-r--r--   1 alex users  27M Mar 15 16:54 tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ drwxr-xr-x   3 alex users 4.0K Mar 15 16:56 usr
 | ~ -rw-r--r--   1 alex users 1.1K Mar 15 16:05 version
 | (Meh. Let's move it around.)
 | $ mkdir tarantool-core-N-202103151305-cc18d0726f6a
 | $ mv checklist coredump lib64 .rocks tarantool usr version tarantool-core-N-202103151305-cc18d0726f6a
 | $ tree
 | ~ .
 | ~ ├── gdb.sh
 | ~ ├── tarantool-core-N-202103151305-cc18d0726f6a
 | ~ │   ├── checklist
 | ~ │   ├── coredump
 | ~ │   ├── lib64
 | ~ │   │   ├── <..skipped..>
 | ~ │   ├── tarantool
 | ~ │   ├── usr
 | ~ │   │   └── lib64
 | ~ │   │       └── tarantool
 | ~ │   │           └── avro_schema_rt_c.so
 | ~ │   └── version
 | ~ └── tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~
 | ~ 5 directories, 37 files
 | (Now looks okay. Let's try.)
 | $ cd tarantool-core-N-202103151305-cc18d0726f6a
 | $ ../gdb.sh --help
 | ~ ================================================================================
 | ~
 | ~ Do not forget to properly setup the environment:
 | ~ * git clone https://github.com/tarantool/tarantool.git sources
 | ~ * cd !$
 | ~ * git checkout 34d504d
 | ~ * git submodule update --recursive --init
 | ~
 | ~ ================================================================================
 | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1
 | ~ <..skipped..>
 | ~ Reading symbols from ./tarantool...
 | ~ lj-arch command initialized
 | ~ lj-tv command initialized
 | ~ lj-str command initialized
 | ~ lj-tab command initialized
 | ~ lj-stack command initialized
 | ~ lj-state command initialized
 | ~ lj-gc command initialized
 | ~ luajit-gdb.py is successfully loaded
 | ~ warning: Can't open file /usr/bin/tarantool during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib/locale/locale-archive (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libpcre.so.1.2.0 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libselinux.so.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libresolv-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libkeyutils.so.1.5 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libkrb5support.so.0.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libk5crypto.so.3.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libcom_err.so.2.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libkrb5.so.3.3 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libgssapi_krb5.so.2.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libc-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libm-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libstdc++.so.6.0.19 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libunwind.so.8.0.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libunwind-x86_64.so.8.0.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libgcc_s-4.8.5-20150702.so.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libpthread-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libgomp.so.1.0.0 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libcrypto.so.1.0.2k during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libssl.so.1.0.2k during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/librt-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libz.so.1.2.7 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libform.so.5.9 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libtinfo.so.5.9 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libncurses.so.5.9 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libreadline.so.6.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libdl-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libicudata.so.50.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libicuuc.so.50.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libicui18n.so.50.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/ld-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/gconv/gconv-modules.cache (deleted) during file-backed mapping note processing
 | ~ [New LWP 246]
 | ~ [New LWP 247]
 | ~ [New LWP 248]
 | ~ [New LWP 250]
 | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 | ~ warning: the debug information found in "/usr/lib64/debug/usr/lib64/tarantool/avro_schema_rt_c.so.debug" does not match "/home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so" (CRC mismatch).
 | ~ 
 | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 | ~ Core was generated by `tarantool'.
 | ~ Program terminated with signal SIGABRT, Aborted.
 | ~ #0  0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
 | ~ [Current thread is 1 (LWP 246)]
 | (gdb) Ctrl+D
 | (Hey, --help runs it.)
 | (Anyway, I should checkout sources. Nice reminder.)
 | $ git clone https://github.com/tarantool/tarantool.git sources
 | $ cd sources
 | $ git checkout 34d504d
 | $ git submodule update --recursive --init
 | $ cd ..
 | (Let's try again?)
 | $ ../gdb.sh
 | <..I'll skip the output, it is quite similar to one above. But will
 |  comment below.>
 | (The reminder regarding tarantool sources still shown. But I already
 |  placed the sources here.)
 | (All library warnings above are still here. However it seems they're
    just warnings.)
 | $ (gdb) bt
 | ~ #0  0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
 | ~ #1  0x00007f340a5f5a78 in abort () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
 | ~ #2  0x0000000000453c43 in sig_fatal_cb (signo=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/main.cc:302
 | ~ #3  <signal handler called>
 | ~ #4  0x0000000000621ce7 in lj_cconv_ct_ct (cts=<optimized out>, d=<optimized out>, s=0x41508e70, dp=0x41d7a0b8 "", sp=0x140535248 <error: Cannot access memory at address 0x140535248>, 
 | ~     flags=<optimized out>) at lj_cconv.c:234
 | ~ #5  0x0000000000621e61 in lj_cconv_tv_ct (cts=0x40fb9858, s=<optimized out>, sid=<optimized out>, o=0x41d7a0b8, sp=<optimized out>) at lj_cconv.c:389
 | ~ #6  0x0000000000621298 in lj_cdata_get (cts=<optimized out>, s=<optimized out>, o=<optimized out>, sp=<optimized out>) at lj_cdata.c:254
 | ~ #7  0x0000000000631021 in lj_cf_ffi_meta___index (L=0x40fb1378) at lib_ffi.c:158
 | ~ #8  0x00000000006338a7 in lj_BC_FUNCC () at buildvm_x86.dasc:809
 | ~ #9  0x0000000000615c82 in lua_pcall (L=L@entry=0x40fb1378, nargs=<optimized out>, nresults=<optimized out>, errfunc=errfunc@entry=0) at lj_api.c:1158
 | ~ #10 0x00000000005c4663 in luaT_call (L=0x40fb1378, nargs=<optimized out>, nreturns=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/utils.c:1022
 | ~ #11 0x00000000005bd259 in lua_main (L=L@entry=0x40fb1378, argc=argc@entry=1, argv=argv@entry=0x20d5fc0) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:546
 | ~ #12 0x00000000005bd6b3 in run_script_f (ap=<error reading variable: value has been optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:647
 | ~ #13 0x00000000004530dc in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=<optimized out>, ap=<optimized out>)
 | ~     at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.h:882
 | ~ #14 0x00000000005def90 in fiber_loop (data=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.c:879
 | ~ #15 0x00000000007c7ddf in coro_init () at /build/usr/src/debug/tarantool-2.6.2.0/third_party/coro/coro.c:110
 | (Looks good!)
 | (gdb) info sharedlibrary 
 | ~ From                To                  Syms Read   Shared Object Library
 | ~ 0x00007f340e9aa5f0  0x00007f340eab0b62  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicui18n.so.50
 | ~ 0x00007f340e5eb2f0  0x00007f340e69dad8  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicuuc.so.50
 | ~ 0x00007f340cfc6550  0x00007f340cfc6635  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicudata.so.50
 | ~ 0x00007f340cdc2e50  0x00007f340cdc394e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libdl.so.2
 | ~ 0x00007f340cb90de0  0x00007f340cbaf775  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libreadline.so.6
 | ~ 0x00007f340c95bd50  0x00007f340c975462  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libncurses.so.5
 | ~ 0x00007f340c737e40  0x00007f340c743bb8  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libtinfo.so.5
 | ~ 0x00007f340c51e6e0  0x00007f340c525ed0  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libform.so.5
 | ~ 0x00007f340c307110  0x00007f340c313698  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libz.so.1
 | ~ 0x00007f340c0ff230  0x00007f340c1021bc  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/librt.so.1
 | ~ 0x00007f340bea5b90  0x00007f340bee164d  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libssl.so.10
 | ~ 0x00007f340ba95200  0x00007f340bbce7b7  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcrypto.so.10
 | ~ 0x00007f340b809060  0x00007f340b81fc2a  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgomp.so.1
 | ~ 0x00007f340b5eb8f0  0x00007f340b5f6db1  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpthread.so.0
 | ~ 0x00007f340b3d2ad0  0x00007f340b3e2285  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgcc_s.so.1
 | ~ 0x00007f340b1b1570  0x00007f340b1be20e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind-x86_64.so.8
 | ~ 0x00007f340af975b0  0x00007f340af9e95e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind.so.8
 | ~ 0x00007f340ace94e0  0x00007f340ad5059a  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libstdc++.so.6
 | ~ 0x00007f340a991350  0x00007f340a9fc336  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libm.so.6
 | ~ 0x00007f340a5dd9f0  0x00007f340a72da5f  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
 | ~ 0x00007f340ed11af0  0x00007f340ed2d060  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/ld-linux-x86-64.so.2
 | ~ 0x00007f340a37d720  0x00007f340a3afbca  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgssapi_krb5.so.2
 | ~ 0x00007f340a0adc20  0x00007f340a1159fa  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5.so.3
 | ~ 0x00007f3409e85540  0x00007f3409e86113  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcom_err.so.2
 | ~ 0x00007f3409c558d0  0x00007f3409c73bbf  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libk5crypto.so.3
 | ~ 0x00007f3409a44bf0  0x00007f3409a4c36b  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5support.so.0
 | ~ 0x00007f340983e5b0  0x00007f340983f1cc  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkeyutils.so.1
 | ~ 0x00007f3409626980  0x00007f34096351e2  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libresolv.so.2
 | ~ 0x00007f3409402a90  0x00007f3409418896  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libselinux.so.1
 | ~ 0x00007f340919b5f0  0x00007f34091e15b0  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpcre.so.1
 | ~ 0x00007f3408c429f0  0x00007f3408c43896  Yes         /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/.rocks/lib/tarantool/tuple/keydef.so
 | ~ 0x00007f33eedf8d80  0x00007f33eedfc174  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so
 | ~ (*): Shared library is missing debugging information.
 | (Looks good too!)

> Can't open file <...> (deleted) during file-backed mapping note
> processing.

I don't know what it is. But I'll share my gdb version just in case
(maybe you'll be able to reproduce it). I didn't meet this warning
before.

 | $ gdb --version | head -n 1
 | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1

However I would note that those warnings misleads me the first time: I
thought that something is wrong with library paths.

> Unable to find libthread_db matching inferior's thread library, thread
> debugging will not be available.

Sure, it is not listed in the lld output (neither in 'info
sharedlibrary'), but maybe we should include it into the tarabrt.sh
generated archive.

> the debug information found in <...> does not match <...> (CRC mismatch).

It is just because I have local avro schema debuginfo installed. I think
this warning can be ignored.

To sum up: my experience was successful. You may polish some tiny
details, but in fact the scripts seem to be ready for use.

----

We also briefly discussed the topic about stripped executables with
Igor.

Now we ship stripped executables in RPM packages for CentOS 8 (only for
CentOS 8), but the separate debug package is stored in our repos.

First point: we can include /etc/os-release into the generated archive.

Second point: we can give URL from where debuginfo package may be
downloaded and give advice how to unpack it (in gdb.sh output, when we
see the stripped executable).

Examples:

 | $ curl -LfsS https://download.tarantool.org/tarantool/release/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm
 | $ mkdir tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
 | $ cd tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
 | $ rpm2cpio ../tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm | cpio -idmv
 | ~ ./usr/lib/debug
 | ~ ./usr/lib/debug/.build-id
 | ~ ./usr/lib/debug/.build-id/a3
 | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60
 | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60.debug
 | ~ ./usr/lib/debug/usr
 | ~ ./usr/lib/debug/usr/bin
 | ~ ./usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
 | ~ 64743 blocks
 | (All are symlinks except tarantool-2.8.0.0-1.el8.x86_64.debug.)
 | $ file usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
 | ~ usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter \004, for GNU/Linux 3.2.0, BuildID[sha1]=a3ad84db98a9b14e70d2915271904ddfe26b5f60, with debug_info, not stripped, too many notes (256)

 | $ curl -LfsS https://download.tarantool.org/tarantool/live/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm

Third: we can give advice how to enable the ELF with debug symbols in
the interactive gdb session.

Or we should do it within gdb.sh to call 'info line main'? Expect it in
some predefined location or accept an option?

At least it looks worthful to warn about stripped executable explicitly.

----

It seems, I reached maximum value that I can give as reviewer, so I'll
just say that the patch looks good to me (and I don't mind small changes
around usability if you want to do them).

WBR, Alexander Turenko.