Tarantool development patches archive
 help / color / mirror / Atom feed
* [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
@ 2021-02-25 13:23 Igor Munkin via Tarantool-patches
  2021-03-03 12:44 ` Sergey Bronnikov via Tarantool-patches
                   ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Igor Munkin via Tarantool-patches @ 2021-02-25 13:23 UTC (permalink / raw)
  To: Alexander Turenko, Sergey Bronnikov; +Cc: tarantool-patches

This patch introduces two scripts to ease crash artefacts collecting and
loading for postmortem analysis:

* tarabrt.sh - the tool collecting a tarball with the crash artefacts
  the right way: the coredump with the binary, all loaded shared libs,
  Tarantool version (this is a separate exercise to get it from the
  binary built with -O2). Besides, the tarball has a unified layout, so
  it can be easily processed with the second script:
  - /coredump - core dump file on the root level
  - /binary - tarantool executable on the root level
  - /version - plain text file on the root level with
    `tarantool --version` output
  - /checklist - plain text file on the root level with
    the list of the collected entities
  - all shared libraries used by the crashed instance - their layout
    respects the one on the host machine, so them can be easily loaded
    with the following gdb command: set sysroot $(realpath .)

  The script can be easily used either manually or via
  kernel.core_pattern variable.

* gdb.sh - the auxiliary script originally written by @Totktonada, but
  needed to be adjusted to the crash artefacts layout every time. Since
  there is a unified layout, the original script is enhanced a bit to
  automatically load the coredump via gdb the right way.

Closes #5569

Signed-off-by: Igor Munkin <imun@tarantool.org>
---

Issue: https://github.com/tarantool/tarantool/issues/5569
Branch: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling

 changelogs/unreleased/tarabrt.md |   3 +
 tools/gdb.sh                     |  59 ++++++++
 tools/tarabrt.sh                 | 234 +++++++++++++++++++++++++++++++
 3 files changed, 296 insertions(+)
 create mode 100644 changelogs/unreleased/tarabrt.md
 create mode 100755 tools/gdb.sh
 create mode 100755 tools/tarabrt.sh

diff --git a/changelogs/unreleased/tarabrt.md b/changelogs/unreleased/tarabrt.md
new file mode 100644
index 000000000..e5e616111
--- /dev/null
+++ b/changelogs/unreleased/tarabrt.md
@@ -0,0 +1,3 @@
+## feature/tools
+
+* Introduced tooling for crash artefacts collecting and postmortem analysis (gh-5569).
diff --git a/tools/gdb.sh b/tools/gdb.sh
new file mode 100755
index 000000000..a58c47cab
--- /dev/null
+++ b/tools/gdb.sh
@@ -0,0 +1,59 @@
+#!/bin/sh
+set -eu
+
+# Check that gdb is installed.
+if ! command -v gdb >/dev/null; then
+	cat <<NOGDB
+gdb is not installed or not found in the PATH.
+
+Install gdb or adjust you PATH if you are using non-system gdb and
+try once more.
+NOGDB
+	exit 1;
+fi
+
+VERSION=${PWD}/version
+
+# Check the location: if the coredump artefacts are collected via
+# `tarabrt.sh' there should be /version file in the root of the
+# unpacked tarball. Otherwise, there is no guarantee the coredump
+# is collected the right way and we can't proceed loading it.
+if [ ! -f "${VERSION}" ]; then
+	cat <<NOARTEFACTS
+${VERSION} file is missing.
+
+If the coredump artefacts are collected via \`tararbrt.sh' tool
+there should be /version file in the root of the unpacked tarball
+(i.e. ${PWD}).
+If version file is missing, there is no guarantee the coredump
+is collected the right way and its loading can't be proceeded
+with this script. Check whether current working directory is the
+tarball root, or try load the core dump file manually.
+NOARTEFACTS
+	exit 1;
+fi
+
+REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
+cat <<SOURCES
+================================================================================
+
+Do not forget to properly setup the environment:
+* git clone https://github.com/tarantool/tarantool.git sources
+* cd !$
+* git checkout $REVISION
+* git submodule update --recursive --init
+
+================================================================================
+SOURCES
+
+# Define the build path to be substituted with the source path.
+# XXX: Check the absolute path on the function <main> definition
+# considering it is located in src/main.cc within Tarantool repo.
+SUBPATH=$(gdb -batch -n ./tarantool -ex 'info line main' | \
+	grep -oP 'Line \d+ of \"\K.+(?=\/src\/main\.cc\")')
+
+# Launch gdb and load coredump with all related artefacts.
+gdb ./tarantool \
+    -ex "set sysroot $(realpath .)" \
+    -ex "set substitute-path $SUBPATH sources" \
+    -ex 'core coredump'
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
new file mode 100755
index 000000000..3d44803be
--- /dev/null
+++ b/tools/tarabrt.sh
@@ -0,0 +1,234 @@
+#!/bin/sh
+set -eu
+
+TOOL=$(basename "$0")
+HELP=$(cat <<HELP
+${TOOL} - Tarantool Automatic Bug Reporting Tool
+
+This tool collects all required artefacts (listed below) and packs them into
+a single archive with unified format:
+  - /checklist - the plain text file with the list of tarball contents
+  - /version   - the plain text file containing \`tarantool --version' output
+  - /tarantool - the executable binary file produced the core dump
+  - /coredump  - the core dump file produced by the executable
+  - all shared libraries loaded (even via dlopen(3)) at the crash moment.
+
+SYNOPSIS
+
+  ${TOOL} [-h] [-c core] [-d dir] [-e executable] [-p procID] [-t datetime]
+
+Supported options are:
+  -c COREDUMP                   Use file COREDUMP as a core dump to examine.
+
+  -d DIRECTORY                  Create the resulting archive with the artefacts
+                                within DIRECTORY.
+
+  -e TARANTOOL                  Use file TARANTOOL as the executable file for
+                                examining with a core dump COREDUMP. If PID is
+                                specified, the one from /proc/PID/exe is chosen
+                                (see proc(5) for more info). If TARANTOOL is
+                                omitted, /usr/bin/tarantool is chosen.
+
+  -p PID                        PID of the dumped process, as seen in the PID
+                                namespace in which the given process resides
+                                (see %p in core(5) for more info). This flag
+                                have to be set when ${TOOL} is used as
+                                kernel.core_pattern pipeline script.
+
+  -t DATETIME                   Time of dump, expressed as seconds since the
+                                epoch, 1970-01-01 00:00:00 +0000 (UTC).
+
+  -h                            Shows this message and exit.
+
+USAGE
+
+  - Manual usage. User can simply pack all necessary artefacts by running the
+    following command.
+    $ /path/to/${TOOL} -c ./core -d /tmp
+
+  - Automatic usage. If user faces the failures often, one can set this script
+    as a pipe reciever in kernel.core_pattern syntax.
+    # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d /var/core -p %p -t %t"
+
+HELP
+)
+
+# Parse CLI options.
+OPTIONS=$(getopt -o c:d:e:hp:t: -n "${TOOL}" -- "$@")
+eval set -- "${OPTIONS}"
+while true; do
+	case "$1" in
+		--) shift; break;;
+		-c) COREFILE=$2; shift 2;;
+		-d) COREDIR=$2;  shift 2;;
+		-e) BINARY=$2;   shift 2;;
+		-p) PID=$2;      shift 2;;
+		-t) TIME=$2;     shift 2;;
+		-h) printf "%s\n", "${HELP}";
+			exit 0;;
+		*)  printf "Invalid option: $1\n%s\n", "${HELP}";
+			exit 1;;
+	esac
+done
+
+# Use the default values for the remaining parameters.
+BINARY=${BINARY:-/usr/bin/tarantool}
+COREDIR=${COREDIR:-${PWD}}
+COREFILE=${COREFILE:-}
+PID=${PID:-}
+TIME=${TIME:-$(date +%s)}
+
+# XXX: This section handles the case when the script is used for
+# kernel.core_pattern. If PID is set and there is a directory in
+# procfs with this PID, the script processes the core dumped by
+# this process. If the process exe (or strictly saying its comm)
+# is not 'tarantool' then the coredump is simply saved to the
+# COREDIR; otherwise the dumped core is packed to the tarball.
+if [ -n "${PID}" ] && [ -d /proc/"${PID}" ]; then
+	BINARY=$(readlink /proc/"${PID}"/exe)
+	CMDNAME=$(sed -z 's/\s$//' /proc/"${PID}"/comm)
+	COREFILE=${COREDIR}/${CMDNAME}-core.${PID}.${TIME}
+	cat >"${COREFILE}"
+	if [ "${CMDNAME}" != 'tarantool' ]; then
+		[ -t 1 ] && cat <<ALIENCOREDUMP
+/proc/${PID}/comm doesn't equal to 'tarantool', so we assume the
+obtained core is dumped by \`${CMDNAME}' and should be packed in
+a different way. As a result it is simply stored to the file, so
+you can process it on your own.
+
+The file with core dump: ${COREFILE}
+ALIENCOREDUMP
+		exit 0;
+	fi
+fi
+
+if [ -z "${COREFILE}" ]; then
+	[ -t 1 ] && cat <<NOCOREDUMP
+There is no core dump file passed to ${TOOL}. The artefacts can't
+be collected. If you see this message, check the usage by running
+\`${TOOL} -h': -c option is the obligatory one.
+NOCOREDUMP
+	exit 1;
+fi
+
+if file "${COREFILE}" | grep -qv 'core file'; then
+	[ -t 1 ] && cat <<NOTACOREDUMP
+Not a core dump: ${COREFILE}
+
+The given COREDUMP file is not a valid core dump (see core(5) for
+more info) or not even an ELF (see elf(5) for more info). If you
+see this message, check the COREDUMP file the following way:
+$ file ${COREFILE}
+NOTACOREDUMP
+	exit 1;
+fi
+
+# Check that gdb is installed.
+if ! command -v gdb >/dev/null; then
+	[ -t 1 ] && cat <<NOGDB
+gdb is not installed, but it is obligatory for collecting the
+loaded shared libraries from the core dump.
+
+You can proceed collecting the artefacts manually later by running
+the following command:
+$ ${TOOL} -e ${BINARY} -c ${COREFILE}
+NOGDB
+	exit 1;
+fi
+
+if file "${BINARY}" | grep -qv 'executable'; then
+	[ -t 1 ] && cat <<NOTELF
+Not an ELF file: ${BINARY}
+
+The given BINARY file is not an ELF (see elf(5) for more info).
+If you see this message, check the BINARY file the following way:
+$ file ${BINARY}
+NOTELF
+	exit 1;
+fi
+
+if gdb -batch -n "${BINARY}" -ex 'info symbol tarantool_version' 2>/dev/null | \
+	grep -q 'tarantool_version in section .text'
+then
+	# XXX: This is a very ugly hack to implement 'unless'
+	# operator in bash for a long pipeline as a conditional.
+	:
+else
+	[ -t 1 ] && cat <<NOTARANTOOL
+Not a Tarantool binary: ${BINARY}
+
+The given BINARY file is not a Tarantool executable: there is no a
+signature symbol in the binary file. If you see this message,
+check the BINARY file the following way:
+$ ${BINARY} --help
+NOTARANTOOL
+	exit 1;
+fi
+
+# Resolve the host name if possible.
+HOSTNAME=$(hostname 2>/dev/null || echo hostname)
+
+# Proceed with collecting and packing artefacts.
+TMPDIR=$(mktemp -d -p "${COREDIR}")
+TARLIST=${TMPDIR}/tarlist
+VERSION=${TMPDIR}/version
+ARCHIVENAME=${COREDIR}/tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz
+
+# Dump the version to checkout the right commit later.
+${BINARY} --version >"${VERSION}"
+
+# Collect the most important artefacts.
+{
+	echo "${BINARY}"
+	echo "${COREFILE}"
+	echo "${VERSION}"
+} >>"${TARLIST}"
+
+SEPARATOR1="Shared Object Library"
+SEPARATOR2="Shared library is missing debugging information"
+# XXX: This is kinda "postmortem ldd": the command below dumps the
+# full list of the shared libraries the binary is linked against
+# or those loaded via dlopen at the platform runtime.
+# This is black voodoo magic. Do not touch. You are warned.
+if gdb -batch -n "${BINARY}" -c "${COREFILE}" -ex 'info shared'    | \
+	sed -n "/${SEPARATOR1}/,/${SEPARATOR2}/p;/${SEPARATOR2}/q" | \
+	awk '{ print $NF }' | grep '^/' >>"${TARLIST}"
+then
+	# XXX: This is a very ugly hack to implement 'unless'
+	# operator in bash for a long pipeline as a conditional.
+	:
+else
+	[ -t 1 ] && cat <<COREMISMATCH
+Core dump file is produced by the different Tarantool executable.
+
+Looks like '${COREFILE}' is not generated by \`${BINARY}'.
+If you see this message, please check that the given COREDUMP
+is produced by the specified BINARY.
+There are some temporary artefacts in ${TMPDIR}.
+Remove it manually if you don't need them anymore.
+COREMISMATCH
+	exit 1;
+fi
+
+# Pack everything listed in TARLIST file into a tarball. To unify
+# the archive format BINARY, COREFILE, VERSION and TARLIST are
+# renamed while packing.
+tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
+	--transform="s|${BINARY}|tarantool|"  \
+	--transform="s|${COREFILE}|coredump|" \
+	--transform="s|${TARLIST}|checklist|" \
+	--transform="s|${VERSION}|version|"   \
+	--add-file="${TARLIST}"
+
+[ -t 1 ] && cat <<FINALIZE
+The resulting is located here: ${ARCHIVENAME}
+
+If you want to upload it, choose the available resourse
+(e.g. http://transfer.sh) and run the following command:
+$ curl -T ${ARCHIVENAME} <resourse-uri>
+FINALIZE
+
+# Cleanup temporary files.
+[ -f "${TARLIST}" ] && rm -f "${TARLIST}"
+[ -f "${VERSION}" ] && rm -f "${VERSION}"
+[ -d "${TMPDIR}" ] && rmdir "${TMPDIR}"
-- 
2.25.0


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-02-25 13:23 [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts Igor Munkin via Tarantool-patches
@ 2021-03-03 12:44 ` Sergey Bronnikov via Tarantool-patches
  2021-03-08  4:56   ` Alexander Turenko via Tarantool-patches
  2021-03-11 12:13   ` Igor Munkin via Tarantool-patches
  2021-03-15 16:30 ` Alexander Turenko via Tarantool-patches
  2021-04-19 22:51 ` Igor Munkin via Tarantool-patches
  2 siblings, 2 replies; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2021-03-03 12:44 UTC (permalink / raw)
  To: Igor Munkin, Alexander Turenko; +Cc: tarantool-patches

Hello!

Igor, thanks for the patch!

I spend a bit on testing script. To create a coredump I run tarantool binary

and generated coredump with GDB: "gdb -batch -ex "generate-core-file" -p 
PID".

To create report I run:

[s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ ./tools/tarabrt.sh -c 
core.2715242 -e build/src/tarantool

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments
The resulting is located here: 
/home/s.bronnikov/work/tarantool/tarantool-core-N-202103031215-tarantool-core-dev-mcs1.tar.gz


If you want to upload it, choose the available resourse
(e.g. http://transfer.sh) and run the following command:
$ curl -T 
/home/s.bronnikov/work/tarantool/tarantool-core-N-202103031222-tarantool-core-dev-mcs1.tar.gz 
<resourse-uri>

1. Can we suppress warnings produced by gdb? I believe it is not useful 
information for users.

2. I propose to show files included to the report and the end of creation.


When I run tarabrt.sh without specifying -e it says that 
/usr/bin/tarantool is not ELF:

(venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ 
./tools/tarabrt.sh -c core.2715242
Not an ELF file: /usr/bin/tarantool

The given BINARY file is not an ELF (see elf(5) for more info).
If you see this message, check the BINARY file the following way:
$ file /usr/bin/tarantool
(venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ file 
/usr/bin/tarantool
/usr/bin/tarantool: ELF 64-bit LSB shared object, x86-64, version 1 
(SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for 
GNU/Linux 3.2.0, BuildID[sha1]=58d86a2b8bf88de3496ea4ead0a8d3c7a73e9c36, 
stripped, too many notes (256)
(venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$

Also see my comments inline.

On 25.02.2021 16:23, Igor Munkin wrote:
> This patch introduces two scripts to ease crash artefacts collecting and
> loading for postmortem analysis:
>
> * tarabrt.sh - the tool collecting a tarball with the crash artefacts
>    the right way: the coredump with the binary, all loaded shared libs,
>    Tarantool version (this is a separate exercise to get it from the
>    binary built with -O2). Besides, the tarball has a unified layout, so
>    it can be easily processed with the second script:
>    - /coredump - core dump file on the root level
>    - /binary - tarantool executable on the root level
>    - /version - plain text file on the root level with
>      `tarantool --version` output
>    - /checklist - plain text file on the root level with
>      the list of the collected entities
>    - all shared libraries used by the crashed instance - their layout
>      respects the one on the host machine, so them can be easily loaded
>      with the following gdb command: set sysroot $(realpath .)
>
>    The script can be easily used either manually or via
>    kernel.core_pattern variable.
>
> * gdb.sh - the auxiliary script originally written by @Totktonada, but
>    needed to be adjusted to the crash artefacts layout every time. Since
>    there is a unified layout, the original script is enhanced a bit to
>    automatically load the coredump via gdb the right way.
2. Looks like gdb.sh is an internal script, so I propose to rename it to 
"_gdb.sh" to reflect it for users.
>
> Closes #5569
>
> Signed-off-by: Igor Munkin <imun@tarantool.org>
> ---
>
> Issue: https://github.com/tarantool/tarantool/issues/5569
> Branch: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling
>
>   changelogs/unreleased/tarabrt.md |   3 +
>   tools/gdb.sh                     |  59 ++++++++
>   tools/tarabrt.sh                 | 234 +++++++++++++++++++++++++++++++
>   3 files changed, 296 insertions(+)
>   create mode 100644 changelogs/unreleased/tarabrt.md
>   create mode 100755 tools/gdb.sh
>   create mode 100755 tools/tarabrt.sh
>
> diff --git a/changelogs/unreleased/tarabrt.md b/changelogs/unreleased/tarabrt.md
> new file mode 100644
> index 000000000..e5e616111
> --- /dev/null
> +++ b/changelogs/unreleased/tarabrt.md
> @@ -0,0 +1,3 @@
> +## feature/tools
> +
> +* Introduced tooling for crash artefacts collecting and postmortem analysis (gh-5569).
> diff --git a/tools/gdb.sh b/tools/gdb.sh
> new file mode 100755
> index 000000000..a58c47cab
> --- /dev/null
> +++ b/tools/gdb.sh
> @@ -0,0 +1,59 @@
> +#!/bin/sh
> +set -eu
3. "set -euo pipefail" is better. Pipe used for SUBPATH var.
> +
> +# Check that gdb is installed.
> +if ! command -v gdb >/dev/null; then
> +	cat <<NOGDB
> +gdb is not installed or not found in the PATH.
> +
> +Install gdb or adjust you PATH if you are using non-system gdb and
> +try once more.
> +NOGDB
> +	exit 1;
> +fi
> +
> +VERSION=${PWD}/version
> +
> +# Check the location: if the coredump artefacts are collected via
> +# `tarabrt.sh' there should be /version file in the root of the
> +# unpacked tarball. Otherwise, there is no guarantee the coredump
> +# is collected the right way and we can't proceed loading it.
> +if [ ! -f "${VERSION}" ]; then
> +	cat <<NOARTEFACTS
> +${VERSION} file is missing.
> +
> +If the coredump artefacts are collected via \`tararbrt.sh' tool
> +there should be /version file in the root of the unpacked tarball
> +(i.e. ${PWD}).
> +If version file is missing, there is no guarantee the coredump
> +is collected the right way and its loading can't be proceeded
> +with this script. Check whether current working directory is the
> +tarball root, or try load the core dump file manually.
> +NOARTEFACTS
> +	exit 1;
> +fi
> +
> +REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
> +cat <<SOURCES
> +================================================================================
> +
> +Do not forget to properly setup the environment:
> +* git clone https://github.com/tarantool/tarantool.git sources
> +* cd !$
> +* git checkout $REVISION
> +* git submodule update --recursive --init
> +
> +================================================================================
> +SOURCES
> +
> +# Define the build path to be substituted with the source path.
> +# XXX: Check the absolute path on the function <main> definition
> +# considering it is located in src/main.cc within Tarantool repo.
> +SUBPATH=$(gdb -batch -n ./tarantool -ex 'info line main' | \
> +	grep -oP 'Line \d+ of \"\K.+(?=\/src\/main\.cc\")')
> +
> +# Launch gdb and load coredump with all related artefacts.
> +gdb ./tarantool \
> +    -ex "set sysroot $(realpath .)" \
> +    -ex "set substitute-path $SUBPATH sources" \
> +    -ex 'core coredump'
> diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
> new file mode 100755
> index 000000000..3d44803be
> --- /dev/null
> +++ b/tools/tarabrt.sh
> @@ -0,0 +1,234 @@
> +#!/bin/sh
> +set -eu
4. "set -euo pipefail" is better. Pipe used for SUBPATH var.
> +
> +TOOL=$(basename "$0")
> +HELP=$(cat <<HELP
> +${TOOL} - Tarantool Automatic Bug Reporting Tool
> +
> +This tool collects all required artefacts (listed below) and packs them into
> +a single archive with unified format:
> +  - /checklist - the plain text file with the list of tarball contents
> +  - /version   - the plain text file containing \`tarantool --version' output
> +  - /tarantool - the executable binary file produced the core dump
> +  - /coredump  - the core dump file produced by the executable
> +  - all shared libraries loaded (even via dlopen(3)) at the crash moment.
> +
> +SYNOPSIS
> +
> +  ${TOOL} [-h] [-c core] [-d dir] [-e executable] [-p procID] [-t datetime]
> +
> +Supported options are:
> +  -c COREDUMP                   Use file COREDUMP as a core dump to examine.

5. I propose to specify somewhere links to documentation how to setup 
coredump paths and where

to find coredumps in a system. For Linux it is core(5) [1].

1. https://man7.org/linux/man-pages/man5/core.5.html


> +
> +  -d DIRECTORY                  Create the resulting archive with the artefacts
> +                                within DIRECTORY.
> +
> +  -e TARANTOOL                  Use file TARANTOOL as the executable file for
> +                                examining with a core dump COREDUMP. If PID is
> +                                specified, the one from /proc/PID/exe is chosen
> +                                (see proc(5) for more info). If TARANTOOL is
> +                                omitted, /usr/bin/tarantool is chosen.
> +
> +  -p PID                        PID of the dumped process, as seen in the PID
> +                                namespace in which the given process resides
> +                                (see %p in core(5) for more info). This flag
> +                                have to be set when ${TOOL} is used as
> +                                kernel.core_pattern pipeline script.
> +
> +  -t DATETIME                   Time of dump, expressed as seconds since the
> +                                epoch, 1970-01-01 00:00:00 +0000 (UTC).
> +

6. Why users may want to specify datetime manually?

We can get datetime in the script with stat:

  $stat -c '%Y' core.2715242

1614773664

> +  -h                            Shows this message and exit.
> +
> +USAGE
> +
> +  - Manual usage. User can simply pack all necessary artefacts by running the
> +    following command.
> +    $ /path/to/${TOOL} -c ./core -d /tmp
> +
> +  - Automatic usage. If user faces the failures often, one can set this script
> +    as a pipe reciever in kernel.core_pattern syntax.
> +    # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d /var/core -p %p -t %t"
> +
> +HELP
> +)
> +
> +# Parse CLI options.
> +OPTIONS=$(getopt -o c:d:e:hp:t: -n "${TOOL}" -- "$@")
> +eval set -- "${OPTIONS}"
> +while true; do
> +	case "$1" in
> +		--) shift; break;;
> +		-c) COREFILE=$2; shift 2;;
> +		-d) COREDIR=$2;  shift 2;;
> +		-e) BINARY=$2;   shift 2;;
> +		-p) PID=$2;      shift 2;;
> +		-t) TIME=$2;     shift 2;;
> +		-h) printf "%s\n", "${HELP}";
> +			exit 0;;
> +		*)  printf "Invalid option: $1\n%s\n", "${HELP}";
> +			exit 1;;
> +	esac
> +done
> +
> +# Use the default values for the remaining parameters.
> +BINARY=${BINARY:-/usr/bin/tarantool}
> +COREDIR=${COREDIR:-${PWD}}
> +COREFILE=${COREFILE:-}
> +PID=${PID:-}
> +TIME=${TIME:-$(date +%s)}
> +
> +# XXX: This section handles the case when the script is used for
> +# kernel.core_pattern. If PID is set and there is a directory in
> +# procfs with this PID, the script processes the core dumped by
> +# this process. If the process exe (or strictly saying its comm)
> +# is not 'tarantool' then the coredump is simply saved to the
> +# COREDIR; otherwise the dumped core is packed to the tarball.
> +if [ -n "${PID}" ] && [ -d /proc/"${PID}" ]; then
> +	BINARY=$(readlink /proc/"${PID}"/exe)
> +	CMDNAME=$(sed -z 's/\s$//' /proc/"${PID}"/comm)
> +	COREFILE=${COREDIR}/${CMDNAME}-core.${PID}.${TIME}
> +	cat >"${COREFILE}"
> +	if [ "${CMDNAME}" != 'tarantool' ]; then
> +		[ -t 1 ] && cat <<ALIENCOREDUMP
> +/proc/${PID}/comm doesn't equal to 'tarantool', so we assume the
> +obtained core is dumped by \`${CMDNAME}' and should be packed in
> +a different way. As a result it is simply stored to the file, so
> +you can process it on your own.
> +
> +The file with core dump: ${COREFILE}
> +ALIENCOREDUMP
> +		exit 0;
> +	fi
> +fi
> +
> +if [ -z "${COREFILE}" ]; then
> +	[ -t 1 ] && cat <<NOCOREDUMP
> +There is no core dump file passed to ${TOOL}. The artefacts can't
> +be collected. If you see this message, check the usage by running
> +\`${TOOL} -h': -c option is the obligatory one.
> +NOCOREDUMP
> +	exit 1;
> +fi
> +
> +if file "${COREFILE}" | grep -qv 'core file'; then
> +	[ -t 1 ] && cat <<NOTACOREDUMP
> +Not a core dump: ${COREFILE}
> +
> +The given COREDUMP file is not a valid core dump (see core(5) for
> +more info) or not even an ELF (see elf(5) for more info). If you
> +see this message, check the COREDUMP file the following way:
> +$ file ${COREFILE}
> +NOTACOREDUMP
> +	exit 1;
> +fi
> +
> +# Check that gdb is installed.
> +if ! command -v gdb >/dev/null; then
> +	[ -t 1 ] && cat <<NOGDB
> +gdb is not installed, but it is obligatory for collecting the
> +loaded shared libraries from the core dump.
> +
> +You can proceed collecting the artefacts manually later by running
> +the following command:
> +$ ${TOOL} -e ${BINARY} -c ${COREFILE}
> +NOGDB
> +	exit 1;
> +fi
> +
> +if file "${BINARY}" | grep -qv 'executable'; then
> +	[ -t 1 ] && cat <<NOTELF
> +Not an ELF file: ${BINARY}
> +
> +The given BINARY file is not an ELF (see elf(5) for more info).
> +If you see this message, check the BINARY file the following way:
> +$ file ${BINARY}
> +NOTELF
> +	exit 1;
> +fi
> +
> +if gdb -batch -n "${BINARY}" -ex 'info symbol tarantool_version' 2>/dev/null | \
> +	grep -q 'tarantool_version in section .text'
> +then
> +	# XXX: This is a very ugly hack to implement 'unless'
> +	# operator in bash for a long pipeline as a conditional.
> +	:
> +else
> +	[ -t 1 ] && cat <<NOTARANTOOL
> +Not a Tarantool binary: ${BINARY}
> +
> +The given BINARY file is not a Tarantool executable: there is no a
> +signature symbol in the binary file. If you see this message,
> +check the BINARY file the following way:
> +$ ${BINARY} --help
> +NOTARANTOOL
> +	exit 1;
> +fi
> +
> +# Resolve the host name if possible.
> +HOSTNAME=$(hostname 2>/dev/null || echo hostname)
> +
> +# Proceed with collecting and packing artefacts.
> +TMPDIR=$(mktemp -d -p "${COREDIR}")
> +TARLIST=${TMPDIR}/tarlist
> +VERSION=${TMPDIR}/version
> +ARCHIVENAME=${COREDIR}/tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz
> +
> +# Dump the version to checkout the right commit later.
> +${BINARY} --version >"${VERSION}"
> +
> +# Collect the most important artefacts.
> +{
> +	echo "${BINARY}"
> +	echo "${COREFILE}"
> +	echo "${VERSION}"
> +} >>"${TARLIST}"
> +
> +SEPARATOR1="Shared Object Library"
> +SEPARATOR2="Shared library is missing debugging information"
> +# XXX: This is kinda "postmortem ldd": the command below dumps the
> +# full list of the shared libraries the binary is linked against
> +# or those loaded via dlopen at the platform runtime.
> +# This is black voodoo magic. Do not touch. You are warned.
> +if gdb -batch -n "${BINARY}" -c "${COREFILE}" -ex 'info shared'    | \
> +	sed -n "/${SEPARATOR1}/,/${SEPARATOR2}/p;/${SEPARATOR2}/q" | \
> +	awk '{ print $NF }' | grep '^/' >>"${TARLIST}"
> +then
> +	# XXX: This is a very ugly hack to implement 'unless'
> +	# operator in bash for a long pipeline as a conditional.
> +	:
> +else
> +	[ -t 1 ] && cat <<COREMISMATCH
> +Core dump file is produced by the different Tarantool executable.
> +
> +Looks like '${COREFILE}' is not generated by \`${BINARY}'.
> +If you see this message, please check that the given COREDUMP
> +is produced by the specified BINARY.
> +There are some temporary artefacts in ${TMPDIR}.
> +Remove it manually if you don't need them anymore.
> +COREMISMATCH
> +	exit 1;
> +fi
> +
> +# Pack everything listed in TARLIST file into a tarball. To unify
> +# the archive format BINARY, COREFILE, VERSION and TARLIST are
> +# renamed while packing.
> +tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
> +	--transform="s|${BINARY}|tarantool|"  \
> +	--transform="s|${COREFILE}|coredump|" \
> +	--transform="s|${TARLIST}|checklist|" \
> +	--transform="s|${VERSION}|version|"   \
> +	--add-file="${TARLIST}"
> +
> +[ -t 1 ] && cat <<FINALIZE
> +The resulting is located here: ${ARCHIVENAME}
> +
> +If you want to upload it, choose the available resourse

7. Typo: resourse -> resource

> +(e.g. http://transfer.sh) and run the following command:
> +$ curl -T ${ARCHIVENAME} <resourse-uri>
8. Typo: resourse-uri -> resource-uri
> +FINALIZE
> +
> +# Cleanup temporary files.
> +[ -f "${TARLIST}" ] && rm -f "${TARLIST}"
> +[ -f "${VERSION}" ] && rm -f "${VERSION}"
> +[ -d "${TMPDIR}" ] && rmdir "${TMPDIR}"

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-03-03 12:44 ` Sergey Bronnikov via Tarantool-patches
@ 2021-03-08  4:56   ` Alexander Turenko via Tarantool-patches
  2021-03-08 11:54     ` Sergey Bronnikov via Tarantool-patches
  2021-03-11 12:13   ` Igor Munkin via Tarantool-patches
  1 sibling, 1 reply; 14+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-03-08  4:56 UTC (permalink / raw)
  To: Sergey Bronnikov; +Cc: tarantool-patches

> > diff --git a/tools/gdb.sh b/tools/gdb.sh
> > new file mode 100755
> > index 000000000..a58c47cab
> > --- /dev/null
> > +++ b/tools/gdb.sh
> > @@ -0,0 +1,59 @@
> > +#!/bin/sh
> > +set -eu
>
> 3. "set -euo pipefail" is better. Pipe used for SUBPATH var.

'-o pipefail' is bash-specific AFAIR.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-03-08  4:56   ` Alexander Turenko via Tarantool-patches
@ 2021-03-08 11:54     ` Sergey Bronnikov via Tarantool-patches
  0 siblings, 0 replies; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2021-03-08 11:54 UTC (permalink / raw)
  To: Alexander Turenko; +Cc: tarantool-patches


On 08.03.2021 07:56, Alexander Turenko wrote:
>>> diff --git a/tools/gdb.sh b/tools/gdb.sh
>>> new file mode 100755
>>> index 000000000..a58c47cab
>>> --- /dev/null
>>> +++ b/tools/gdb.sh
>>> @@ -0,0 +1,59 @@
>>> +#!/bin/sh
>>> +set -eu
>> 3. "set -euo pipefail" is better. Pipe used for SUBPATH var.
> '-o pipefail' is bash-specific AFAIR.

Seem yes. I haven't found anything about pipefail in POSIX standard [1].

1. https://pubs.opengroup.org/onlinepubs/9699919799/


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-03-03 12:44 ` Sergey Bronnikov via Tarantool-patches
  2021-03-08  4:56   ` Alexander Turenko via Tarantool-patches
@ 2021-03-11 12:13   ` Igor Munkin via Tarantool-patches
  2021-03-24 16:02     ` Sergey Bronnikov via Tarantool-patches
  1 sibling, 1 reply; 14+ messages in thread
From: Igor Munkin via Tarantool-patches @ 2021-03-11 12:13 UTC (permalink / raw)
  To: Sergey Bronnikov; +Cc: Alexander Turenko, tarantool-patches

Sergey,

Thanks for your review!

On 03.03.21, Sergey Bronnikov wrote:
> Hello!
> 
> Igor, thanks for the patch!
> 
> I spend a bit on testing script. To create a coredump I run tarantool binary
> 
> and generated coredump with GDB: "gdb -batch -ex "generate-core-file" -p 
> PID".
> 
> To create report I run:
> 
> [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ ./tools/tarabrt.sh -c 
> core.2715242 -e build/src/tarantool
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> 
> warning: Loadable section ".note.gnu.property" outside of ELF segments
> The resulting is located here: 
> /home/s.bronnikov/work/tarantool/tarantool-core-N-202103031215-tarantool-core-dev-mcs1.tar.gz
> 
> 
> If you want to upload it, choose the available resourse
> (e.g. http://transfer.sh) and run the following command:
> $ curl -T 
> /home/s.bronnikov/work/tarantool/tarantool-core-N-202103031222-tarantool-core-dev-mcs1.tar.gz 
> <resourse-uri>
> 
> 1. Can we suppress warnings produced by gdb? I believe it is not useful 
> information for users.

I don't want to do this, since in the normal case there should be no
warnings at all. Otherwise, it's better if customer share them with us.

> 
> 2. I propose to show files included to the report and the end of creation.

Well, I believe I didn't understand you right before (this list is
included to the archive). Anyway, now I added the list also to the
message at the end.

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 212511813..6b88698a2 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -222,7 +222,10 @@ tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
 	--add-file="${TARLIST}"
 
 [ -t 1 ] && cat <<FINALIZE
-The resulting is located here: ${ARCHIVENAME}
+The resulting archive is located here: ${ARCHIVENAME}
+The archive contents are listed below:
+${TARLIST}
+$(cat ${TARLIST})
 
 If you want to upload it, choose the available resourse
 (e.g. http://transfer.sh) and run the following command:

================================================================================

> 
> 
> When I run tarabrt.sh without specifying -e it says that 
> /usr/bin/tarantool is not ELF:
> 
> (venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ 
> ./tools/tarabrt.sh -c core.2715242
> Not an ELF file: /usr/bin/tarantool
> 
> The given BINARY file is not an ELF (see elf(5) for more info).
> If you see this message, check the BINARY file the following way:
> $ file /usr/bin/tarantool
> (venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ file 
> /usr/bin/tarantool
> /usr/bin/tarantool: ELF 64-bit LSB shared object, x86-64, version 1 
> (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for 
> GNU/Linux 3.2.0, BuildID[sha1]=58d86a2b8bf88de3496ea4ead0a8d3c7a73e9c36, 
> stripped, too many notes (256)
> (venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$

Oh, crap... Thank you a lot! I've tried on two hosts: my Gentoo working
station and our CentOS 7 dev1 stand: both are looking good for two
reasons:
* There is a bleeding file package on my Gentoo and Tarantool binary is
  built with PIE enabled.
* There is an ancient file package on our CentOS 7 machine and Tarantool
  binary is built without PIE.

Your case discovered the third case: a buggy file utility used against
Tarantool binary built with PIE. Hence I've adjusted the condition and
added the comment with the link providing the highly detailed answer.
Here are the changes.

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index c3ec9241e..372ff80c1 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -137,7 +137,16 @@ NOGDB
 	exit 1;
 fi
 
-if file "${BINARY}" | grep -qv 'executable'; then
+# XXX: There is annoying bug in file(1), making it claim that PIE
+# is a shared object, but not an executable one. Strictly saying,
+# there was no PIE support until 5.34 at all, but the whole thing
+# didn't work until 5.36. Unfortunately, there are distros (e.g.
+# CentOS 8) providing file utility without valid PIE support, but
+# compiling PIE binaries. As a result we ought to respect both
+# behaviours for the check below. For more info, see the highly
+# detailed answer on Stack Overflow below.
+# https://stackoverflow.com/a/55704865/4609359
+if file "${BINARY}" | grep -qvE 'executable|shared object'; then
 	[ -t 1 ] && cat <<NOTELF
 Not an ELF file: ${BINARY}
 

================================================================================

> 
> Also see my comments inline.
> 
> On 25.02.2021 16:23, Igor Munkin wrote:
> > This patch introduces two scripts to ease crash artefacts collecting and
> > loading for postmortem analysis:
> >
> > * tarabrt.sh - the tool collecting a tarball with the crash artefacts
> >    the right way: the coredump with the binary, all loaded shared libs,
> >    Tarantool version (this is a separate exercise to get it from the
> >    binary built with -O2). Besides, the tarball has a unified layout, so
> >    it can be easily processed with the second script:
> >    - /coredump - core dump file on the root level
> >    - /binary - tarantool executable on the root level
> >    - /version - plain text file on the root level with
> >      `tarantool --version` output
> >    - /checklist - plain text file on the root level with
> >      the list of the collected entities
> >    - all shared libraries used by the crashed instance - their layout
> >      respects the one on the host machine, so them can be easily loaded
> >      with the following gdb command: set sysroot $(realpath .)
> >
> >    The script can be easily used either manually or via
> >    kernel.core_pattern variable.
> >
> > * gdb.sh - the auxiliary script originally written by @Totktonada, but
> >    needed to be adjusted to the crash artefacts layout every time. Since
> >    there is a unified layout, the original script is enhanced a bit to
> >    automatically load the coredump via gdb the right way.
> 2. Looks like gdb.sh is an internal script, so I propose to rename it to 
> "_gdb.sh" to reflect it for users.

No, it's not. I hope this will be useful for the teammates, so its name
should be convenient to be used via shell.

> >
> > Closes #5569
> >
> > Signed-off-by: Igor Munkin <imun@tarantool.org>
> > ---
> >
> > Issue: https://github.com/tarantool/tarantool/issues/5569
> > Branch: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling
> >
> >   changelogs/unreleased/tarabrt.md |   3 +
> >   tools/gdb.sh                     |  59 ++++++++
> >   tools/tarabrt.sh                 | 234 +++++++++++++++++++++++++++++++
> >   3 files changed, 296 insertions(+)
> >   create mode 100644 changelogs/unreleased/tarabrt.md
> >   create mode 100755 tools/gdb.sh
> >   create mode 100755 tools/tarabrt.sh
> >

<snipped>

> > diff --git a/tools/gdb.sh b/tools/gdb.sh
> > new file mode 100755
> > index 000000000..a58c47cab
> > --- /dev/null
> > +++ b/tools/gdb.sh
> > @@ -0,0 +1,59 @@
> > +#!/bin/sh
> > +set -eu
> 3. "set -euo pipefail" is better. Pipe used for SUBPATH var.

There is no "-o pipefail" for /bin/sh, only for /bin/bash.

> > +

<snipped>

> > diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
> > new file mode 100755
> > index 000000000..3d44803be
> > --- /dev/null
> > +++ b/tools/tarabrt.sh
> > @@ -0,0 +1,234 @@
> > +#!/bin/sh
> > +set -eu
> 4. "set -euo pipefail" is better. Pipe used for SUBPATH var.

There is no "-o pipefail" for /bin/sh, only for /bin/bash.

> > +
> > +TOOL=$(basename "$0")
> > +HELP=$(cat <<HELP
> > +${TOOL} - Tarantool Automatic Bug Reporting Tool
> > +
> > +This tool collects all required artefacts (listed below) and packs them into
> > +a single archive with unified format:
> > +  - /checklist - the plain text file with the list of tarball contents
> > +  - /version   - the plain text file containing \`tarantool --version' output
> > +  - /tarantool - the executable binary file produced the core dump
> > +  - /coredump  - the core dump file produced by the executable
> > +  - all shared libraries loaded (even via dlopen(3)) at the crash moment.
> > +
> > +SYNOPSIS
> > +
> > +  ${TOOL} [-h] [-c core] [-d dir] [-e executable] [-p procID] [-t datetime]
> > +
> > +Supported options are:
> > +  -c COREDUMP                   Use file COREDUMP as a core dump to examine.
> 
> 5. I propose to specify somewhere links to documentation how to setup 
> coredump paths and where
> 
> to find coredumps in a system. For Linux it is core(5) [1].
> 
> 1. https://man7.org/linux/man-pages/man5/core.5.html

Added the following line to the help message.

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 3d44803be..212511813 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -19,6 +19,7 @@ SYNOPSIS
 
 Supported options are:
   -c COREDUMP                   Use file COREDUMP as a core dump to examine.
+                                See core(5) for more info.
 
   -d DIRECTORY                  Create the resulting archive with the artefacts
                                 within DIRECTORY.

================================================================================

> 
> 
> > +
> > +  -d DIRECTORY                  Create the resulting archive with the artefacts
> > +                                within DIRECTORY.
> > +
> > +  -e TARANTOOL                  Use file TARANTOOL as the executable file for
> > +                                examining with a core dump COREDUMP. If PID is
> > +                                specified, the one from /proc/PID/exe is chosen
> > +                                (see proc(5) for more info). If TARANTOOL is
> > +                                omitted, /usr/bin/tarantool is chosen.
> > +
> > +  -p PID                        PID of the dumped process, as seen in the PID
> > +                                namespace in which the given process resides
> > +                                (see %p in core(5) for more info). This flag
> > +                                have to be set when ${TOOL} is used as
> > +                                kernel.core_pattern pipeline script.
> > +
> > +  -t DATETIME                   Time of dump, expressed as seconds since the
> > +                                epoch, 1970-01-01 00:00:00 +0000 (UTC).
> > +
> 
> 6. Why users may want to specify datetime manually?

To provide the timing in core pattern via %t. I don't know whether the
value differs from the one you mentioned below. If it's not, I guess
this option can be removed.

> 
> We can get datetime in the script with stat:
> 
>   $stat -c '%Y' core.2715242
> 
> 1614773664
> 
> > +  -h                            Shows this message and exit.
> > +
> > +USAGE
> > +
> > +  - Manual usage. User can simply pack all necessary artefacts by running the
> > +    following command.
> > +    $ /path/to/${TOOL} -c ./core -d /tmp
> > +
> > +  - Automatic usage. If user faces the failures often, one can set this script
> > +    as a pipe reciever in kernel.core_pattern syntax.
> > +    # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d /var/core -p %p -t %t"
> > +
> > +HELP
> > +)
> > +
> > +# Parse CLI options.
> > +OPTIONS=$(getopt -o c:d:e:hp:t: -n "${TOOL}" -- "$@")
> > +eval set -- "${OPTIONS}"
> > +while true; do
> > +	case "$1" in
> > +		--) shift; break;;
> > +		-c) COREFILE=$2; shift 2;;
> > +		-d) COREDIR=$2;  shift 2;;
> > +		-e) BINARY=$2;   shift 2;;
> > +		-p) PID=$2;      shift 2;;
> > +		-t) TIME=$2;     shift 2;;
> > +		-h) printf "%s\n", "${HELP}";
> > +			exit 0;;
> > +		*)  printf "Invalid option: $1\n%s\n", "${HELP}";
> > +			exit 1;;
> > +	esac
> > +done
> > +
> > +# Use the default values for the remaining parameters.
> > +BINARY=${BINARY:-/usr/bin/tarantool}
> > +COREDIR=${COREDIR:-${PWD}}
> > +COREFILE=${COREFILE:-}
> > +PID=${PID:-}
> > +TIME=${TIME:-$(date +%s)}
> > +
> > +# XXX: This section handles the case when the script is used for
> > +# kernel.core_pattern. If PID is set and there is a directory in
> > +# procfs with this PID, the script processes the core dumped by
> > +# this process. If the process exe (or strictly saying its comm)
> > +# is not 'tarantool' then the coredump is simply saved to the
> > +# COREDIR; otherwise the dumped core is packed to the tarball.
> > +if [ -n "${PID}" ] && [ -d /proc/"${PID}" ]; then
> > +	BINARY=$(readlink /proc/"${PID}"/exe)
> > +	CMDNAME=$(sed -z 's/\s$//' /proc/"${PID}"/comm)
> > +	COREFILE=${COREDIR}/${CMDNAME}-core.${PID}.${TIME}
> > +	cat >"${COREFILE}"
> > +	if [ "${CMDNAME}" != 'tarantool' ]; then
> > +		[ -t 1 ] && cat <<ALIENCOREDUMP
> > +/proc/${PID}/comm doesn't equal to 'tarantool', so we assume the
> > +obtained core is dumped by \`${CMDNAME}' and should be packed in
> > +a different way. As a result it is simply stored to the file, so
> > +you can process it on your own.
> > +
> > +The file with core dump: ${COREFILE}
> > +ALIENCOREDUMP
> > +		exit 0;
> > +	fi
> > +fi
> > +
> > +if [ -z "${COREFILE}" ]; then
> > +	[ -t 1 ] && cat <<NOCOREDUMP
> > +There is no core dump file passed to ${TOOL}. The artefacts can't
> > +be collected. If you see this message, check the usage by running
> > +\`${TOOL} -h': -c option is the obligatory one.
> > +NOCOREDUMP
> > +	exit 1;
> > +fi
> > +
> > +if file "${COREFILE}" | grep -qv 'core file'; then
> > +	[ -t 1 ] && cat <<NOTACOREDUMP
> > +Not a core dump: ${COREFILE}
> > +
> > +The given COREDUMP file is not a valid core dump (see core(5) for
> > +more info) or not even an ELF (see elf(5) for more info). If you
> > +see this message, check the COREDUMP file the following way:
> > +$ file ${COREFILE}
> > +NOTACOREDUMP
> > +	exit 1;
> > +fi
> > +
> > +# Check that gdb is installed.
> > +if ! command -v gdb >/dev/null; then
> > +	[ -t 1 ] && cat <<NOGDB
> > +gdb is not installed, but it is obligatory for collecting the
> > +loaded shared libraries from the core dump.
> > +
> > +You can proceed collecting the artefacts manually later by running
> > +the following command:
> > +$ ${TOOL} -e ${BINARY} -c ${COREFILE}
> > +NOGDB
> > +	exit 1;
> > +fi
> > +
> > +if file "${BINARY}" | grep -qv 'executable'; then
> > +	[ -t 1 ] && cat <<NOTELF
> > +Not an ELF file: ${BINARY}
> > +
> > +The given BINARY file is not an ELF (see elf(5) for more info).
> > +If you see this message, check the BINARY file the following way:
> > +$ file ${BINARY}
> > +NOTELF
> > +	exit 1;
> > +fi
> > +
> > +if gdb -batch -n "${BINARY}" -ex 'info symbol tarantool_version' 2>/dev/null | \
> > +	grep -q 'tarantool_version in section .text'
> > +then
> > +	# XXX: This is a very ugly hack to implement 'unless'
> > +	# operator in bash for a long pipeline as a conditional.
> > +	:
> > +else
> > +	[ -t 1 ] && cat <<NOTARANTOOL
> > +Not a Tarantool binary: ${BINARY}
> > +
> > +The given BINARY file is not a Tarantool executable: there is no a
> > +signature symbol in the binary file. If you see this message,
> > +check the BINARY file the following way:
> > +$ ${BINARY} --help
> > +NOTARANTOOL
> > +	exit 1;
> > +fi
> > +
> > +# Resolve the host name if possible.
> > +HOSTNAME=$(hostname 2>/dev/null || echo hostname)
> > +
> > +# Proceed with collecting and packing artefacts.
> > +TMPDIR=$(mktemp -d -p "${COREDIR}")
> > +TARLIST=${TMPDIR}/tarlist
> > +VERSION=${TMPDIR}/version
> > +ARCHIVENAME=${COREDIR}/tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz
> > +
> > +# Dump the version to checkout the right commit later.
> > +${BINARY} --version >"${VERSION}"
> > +
> > +# Collect the most important artefacts.
> > +{
> > +	echo "${BINARY}"
> > +	echo "${COREFILE}"
> > +	echo "${VERSION}"
> > +} >>"${TARLIST}"
> > +
> > +SEPARATOR1="Shared Object Library"
> > +SEPARATOR2="Shared library is missing debugging information"
> > +# XXX: This is kinda "postmortem ldd": the command below dumps the
> > +# full list of the shared libraries the binary is linked against
> > +# or those loaded via dlopen at the platform runtime.
> > +# This is black voodoo magic. Do not touch. You are warned.
> > +if gdb -batch -n "${BINARY}" -c "${COREFILE}" -ex 'info shared'    | \
> > +	sed -n "/${SEPARATOR1}/,/${SEPARATOR2}/p;/${SEPARATOR2}/q" | \
> > +	awk '{ print $NF }' | grep '^/' >>"${TARLIST}"
> > +then
> > +	# XXX: This is a very ugly hack to implement 'unless'
> > +	# operator in bash for a long pipeline as a conditional.
> > +	:
> > +else
> > +	[ -t 1 ] && cat <<COREMISMATCH
> > +Core dump file is produced by the different Tarantool executable.
> > +
> > +Looks like '${COREFILE}' is not generated by \`${BINARY}'.
> > +If you see this message, please check that the given COREDUMP
> > +is produced by the specified BINARY.
> > +There are some temporary artefacts in ${TMPDIR}.
> > +Remove it manually if you don't need them anymore.
> > +COREMISMATCH
> > +	exit 1;
> > +fi
> > +
> > +# Pack everything listed in TARLIST file into a tarball. To unify
> > +# the archive format BINARY, COREFILE, VERSION and TARLIST are
> > +# renamed while packing.
> > +tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
> > +	--transform="s|${BINARY}|tarantool|"  \
> > +	--transform="s|${COREFILE}|coredump|" \
> > +	--transform="s|${TARLIST}|checklist|" \
> > +	--transform="s|${VERSION}|version|"   \
> > +	--add-file="${TARLIST}"
> > +
> > +[ -t 1 ] && cat <<FINALIZE
> > +The resulting is located here: ${ARCHIVENAME}
> > +
> > +If you want to upload it, choose the available resourse
> 
> 7. Typo: resourse -> resource
> 
> > +(e.g. http://transfer.sh) and run the following command:
> > +$ curl -T ${ARCHIVENAME} <resourse-uri>
> 8. Typo: resourse-uri -> resource-uri

Thanks, fixed both typos.

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 6b88698a2..1f9e10867 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -227,9 +227,9 @@ The archive contents are listed below:
 ${TARLIST}
 $(cat ${TARLIST})
 
-If you want to upload it, choose the available resourse
+If you want to upload it, choose the available resource
 (e.g. http://transfer.sh) and run the following command:
-$ curl -T ${ARCHIVENAME} <resourse-uri>
+$ curl -T ${ARCHIVENAME} <resource-uri>
 FINALIZE
 
 # Cleanup temporary files.

================================================================================

> > +FINALIZE
> > +
> > +# Cleanup temporary files.
> > +[ -f "${TARLIST}" ] && rm -f "${TARLIST}"
> > +[ -f "${VERSION}" ] && rm -f "${VERSION}"
> > +[ -d "${TMPDIR}" ] && rmdir "${TMPDIR}"

-- 
Best regards,
IM

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-02-25 13:23 [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts Igor Munkin via Tarantool-patches
  2021-03-03 12:44 ` Sergey Bronnikov via Tarantool-patches
@ 2021-03-15 16:30 ` Alexander Turenko via Tarantool-patches
  2021-04-19 20:42   ` Igor Munkin via Tarantool-patches
  2021-04-19 22:51 ` Igor Munkin via Tarantool-patches
  2 siblings, 1 reply; 14+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-03-15 16:30 UTC (permalink / raw)
  To: Igor Munkin; +Cc: tarantool-patches

I looked over the WIP version of the script and nothing confuses me:

https://lists.tarantool.org/tarantool-patches/20210220104630.wiwxhkjfimsx2swa@tkn_work_nb/

So I guess I should do some basic testing to get and share my user
experience.

So I'll step into another environment to create coredump and archive it:

 | $ mkdir ~/tmp/cores && docker run --volume $(realpath ~/tmp/cores):/cores -it centos:7 /bin/bash
 | # curl -LfsS https://tarantool.io/release/2.6/installer.sh >installer.sh
 | # chmod a+x installer.sh
 | # ./installer.sh
 | # yum install -y tarantool
 | # ulimit -c unlimited
 | # tarantool

Now we're in tarantool in a foreign environment. Let's initialize the
database and fill it with some data to imitate a real situation:

 | tarantool> box.cfg{}
 | tarantool> box.schema.create_space('s')
 | tarantool> box.space.s:create_index('pk')
 | tarantool> for i = 1, 1000 do box.space.s:insert({i}) end

And load some external C module (one from rocks, one from packages):

 | tarantool> Ctrl+Z
 | # yum install -y tarantool-devel git cmake gcc
 | # tarantoolctl rocks install tuple-keydef
 | # yum install -y tarantool-avro-schema
 | # fg
 | tarantool> require('tuple.keydef')
 | tarantool> require('avro_schema')

Let's coredump myself (I'll prepend commands output with '~' symbols to
ease reading):

 | tarantool> require('ffi').new('int[?]', 1)[1024 ^ 3]
 | ~ <...>
 | ~ Aborted (core dumped)
 | # ls -hl core
 | ~ -rw------- 1 root root 180M Mar 15 12:49 core
 | # file core
 | ~ bash: file: command not found

(I'll ignore that 'file' is not installed, because it is possible
situation on a production server, I guess.)

Okay, now I got the tarabrt.sh script somehow:

 | # curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/tarabrt.sh >tarabrt.sh
 | # chmod a+x tarabrt.sh
 | (Don't read the content, just run.)
 | # ./tarabrt.sh --help
 | ~ tarabrt.sh: unrecognized option '--help'
 | # ./tarabrt.sh -h  
 | ~ <..skipped..>
 | ~   - Manual usage. User can simply pack all necessary artefacts by running the
 | ~     following command.
 | ~     $ /path/to/tarabrt.sh -c ./core -d /tmp
 | ~ <..skipped..>
 | # ./tarabrt.sh -c ./core -d /tmp
 | ~ ./tarabrt.sh: line 115: file: command not found
 | ~ gdb is not installed, but it is obligatory for collecting the
 | ~ loaded shared libraries from the core dump.
 | ~ 
 | ~ You can proceed collecting the artefacts manually later by running
 | ~ the following command:
 | ~ $ tarabrt.sh -e /usr/bin/tarantool -c ./core
 | (Okay, I'll install file and gdb now.)
 | # yum install -y file gdb
 | # file core
 | ~ core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'tarantool', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/tarantool', platform: 'x86_64'
 | (Repeat the attempt to create the archive.)
 | # ./tarabrt.sh -c ./core -d /tmp
 | ~ The resulting archive is located here: /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ The archive contents are listed below:
 | ~ /tmp/tmp.kqC76QSrpF/tarlist
 | ~ /tmp/tmp.kqC76QSrpF/version
 | ~ /usr/bin/tarantool
 | ~ ./core
 | ~ /lib64/libicui18n.so.50
 | ~ /lib64/libicuuc.so.50
 | ~ /lib64/libicudata.so.50
 | ~ /lib64/libdl.so.2
 | ~ /lib64/libreadline.so.6
 | ~ /lib64/libncurses.so.5
 | ~ /lib64/libtinfo.so.5
 | ~ /lib64/libform.so.5
 | ~ /lib64/libz.so.1
 | ~ /lib64/librt.so.1
 | ~ /lib64/libssl.so.10
 | ~ /lib64/libcrypto.so.10
 | ~ /lib64/libgomp.so.1
 | ~ /lib64/libpthread.so.0
 | ~ /lib64/libgcc_s.so.1
 | ~ /lib64/libunwind-x86_64.so.8
 | ~ /lib64/libunwind.so.8
 | ~ /lib64/libstdc++.so.6
 | ~ /lib64/libm.so.6
 | ~ /lib64/libc.so.6
 | ~ /lib64/ld-linux-x86-64.so.2
 | ~ /lib64/libgssapi_krb5.so.2
 | ~ /lib64/libkrb5.so.3
 | ~ /lib64/libcom_err.so.2
 | ~ /lib64/libk5crypto.so.3
 | ~ /lib64/libkrb5support.so.0
 | ~ /lib64/libkeyutils.so.1
 | ~ /lib64/libresolv.so.2
 | ~ /lib64/libselinux.so.1
 | ~ /lib64/libpcre.so.1
 | ~ /.rocks/lib/tarantool/tuple/keydef.so
 | ~ /usr/lib64/tarantool/avro_schema_rt_c.so
 | ~ 
 | ~ If you want to upload it, choose the available resource
 | ~ (e.g. http://transfer.sh) and run the following command:
 | ~ $ curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz <resource-uri>

Okay, let's spin around this output for a while:

 | (The /tmp/ entries looks are something I can glance.)
 | # ll /tmp/tmp.kqC76QSrpF/tarlist
 | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/tarlist: No such file or directory
 | ll /tmp/tmp.kqC76QSrpF/version
 | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/version: No such file or directory
 | (But they're not. Okay, pass it over.)
 | (keydef.so and avro_schema_rt_c.so are here, nice!)
 | (Let's look at the archive file.)
 | # ls -hl /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ -rw-r--r-- 1 root root 27M Mar 15 13:05 /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | # tar -vtf /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ -rw-r--r-- root/root      1085 2021-03-15 13:05 version
 | ~ -rwxr-xr-x root/root  34431952 2020-12-30 14:08 tarantool
 | ~ -rw------- root/root 188039168 2021-03-15 12:49 coredump
 | ~ tar: Removing leading `/' from member names
 | ~ -rwxr-xr-x root/root   2096104 2020-03-21 00:49 /lib64/libicui18n.so.50
 | ~ -rwxr-xr-x root/root   1539520 2020-03-21 00:49 /lib64/libicuuc.so.50
 | ~ -rwxr-xr-x root/root  20785824 2020-03-21 00:49 /lib64/libicudata.so.50
 | ~ -rwxr-xr-x root/root     19248 2021-02-08 15:46 /lib64/libdl.so.2
 | ~ -rwxr-xr-x root/root    285136 2019-08-08 12:07 /lib64/libreadline.so.6
 | ~ -rwxr-xr-x root/root    163696 2017-09-06 22:08 /lib64/libncurses.so.5
 | ~ -rwxr-xr-x root/root    174576 2017-09-06 22:08 /lib64/libtinfo.so.5
 | ~ -rwxr-xr-x root/root     67864 2017-09-06 22:08 /lib64/libform.so.5
 | ~ -rwxr-xr-x root/root     90248 2018-10-30 20:24 /lib64/libz.so.1
 | ~ -rwxr-xr-x root/root     43712 2021-02-08 15:46 /lib64/librt.so.1
 | ~ -rwxr-xr-x root/root    470376 2020-12-16 21:24 /lib64/libssl.so.10
 | ~ -rwxr-xr-x root/root   2520768 2020-12-16 21:24 /lib64/libcrypto.so.10
 | ~ -rwxr-xr-x root/root    154840 2020-09-30 02:17 /lib64/libgomp.so.1
 | ~ -rwxr-xr-x root/root    142144 2021-02-08 15:46 /lib64/libpthread.so.0
 | ~ -rwxr-xr-x root/root     88720 2020-09-30 02:18 /lib64/libgcc_s.so.1
 | ~ -rwxr-xr-x root/root     73416 2017-08-02 15:45 /lib64/libunwind-x86_64.so.8
 | ~ -rwxr-xr-x root/root     48904 2017-08-02 15:45 /lib64/libunwind.so.8
 | ~ -rwxr-xr-x root/root    995840 2020-09-30 02:17 /lib64/libstdc++.so.6
 | ~ -rwxr-xr-x root/root   1136944 2021-02-08 15:46 /lib64/libm.so.6
 | ~ -rwxr-xr-x root/root   2156344 2021-02-08 15:46 /lib64/libc.so.6
 | ~ -rwxr-xr-x root/root    163312 2021-02-08 15:46 /lib64/ld-linux-x86-64.so.2
 | ~ -rwxr-xr-x root/root    320720 2020-09-30 16:51 /lib64/libgssapi_krb5.so.2
 | ~ -rwxr-xr-x root/root    967840 2020-09-30 16:51 /lib64/libkrb5.so.3
 | ~ -rwxr-xr-x root/root     15856 2020-09-30 15:58 /lib64/libcom_err.so.2
 | ~ -rwxr-xr-x root/root    210784 2020-09-30 16:51 /lib64/libk5crypto.so.3
 | ~ -rwxr-xr-x root/root     67104 2020-09-30 16:51 /lib64/libkrb5support.so.0
 | ~ -rwxr-xr-x root/root     15688 2014-06-10 02:17 /lib64/libkeyutils.so.1
 | ~ -rwxr-xr-x root/root    109976 2021-02-08 15:46 /lib64/libresolv.so.2
 | ~ -rwxr-xr-x root/root    155744 2020-04-01 03:16 /lib64/libselinux.so.1
 | ~ -rwxr-xr-x root/root    402384 2017-08-02 03:08 /lib64/libpcre.so.1
 | ~ -rwxr-xr-x root/root     52168 2021-03-15 12:45 /.rocks/lib/tarantool/tuple/keydef.so
 | ~ -rwxr-xr-x root/root     23736 2020-08-05 23:10 /usr/lib64/tarantool/avro_schema_rt_c.so
 | ~ -rw-r--r-- root/root       796 2021-03-15 13:05 checklist
 | (Can we avoid the the warning re `/`? Just curious.)
 | (Okay, I see that transfer.sh is suggested. I never use it, so I'm
 |  curious.)
 | # curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz http://transfer.sh
 | ~ http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | (Hey, something occurs! But I'll save it to my system anyway.)
 | # cp /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz /cores/
 | (And, while I'm here...)
 | # md5sum /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ bec8f81c69ed992a258a6d9f0b8a04dc  /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz

Okay, let's try gdb.sh on my Gentoo box:

 | $ mkdir ~/tmp/try-gdb-sh
 | $ cd ~/tmp/try-gdb-sh
 | $ curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/gdb.sh > gdb.sh
 | $ chmod a+x gdb.sh
 | $ curl -LfsS http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz > tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | $ md5sum tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ bec8f81c69ed992a258a6d9f0b8a04dc  tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | (Matches!)
 | (Hm, now I'm worry, whether the archive will be uncompressed into the
 |  current directory or will create its own directory?)
 | $ tar -xf tarantool-core-N-202103151305-cc18d0726f6a.tar.gz 
 | ~ tar: Removing leading `/' from member names
 | $ ll
 | ~ total 239M
 | ~ drwxr-xr-x   5 alex users 4.0K Mar 15 16:56 .
 | ~ drwxr-xr-x 164 alex users  20K Mar 15 16:50 ..
 | ~ -rw-r--r--   1 alex users  796 Mar 15 16:05 checklist
 | ~ -rw-------   1 alex users 180M Mar 15 15:49 coredump
 | ~ -rwxr-xr-x   1 alex users 2.0K Mar 15 16:51 gdb.sh
 | ~ drwxr-xr-x   2 alex users 4.0K Mar 15 16:56 lib64
 | ~ drwxr-xr-x   3 alex users 4.0K Mar 15 16:56 .rocks
 | ~ -rwxr-xr-x   1 alex users  33M Dec 30 17:08 tarantool
 | ~ -rw-r--r--   1 alex users  27M Mar 15 16:54 tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~ drwxr-xr-x   3 alex users 4.0K Mar 15 16:56 usr
 | ~ -rw-r--r--   1 alex users 1.1K Mar 15 16:05 version
 | (Meh. Let's move it around.)
 | $ mkdir tarantool-core-N-202103151305-cc18d0726f6a
 | $ mv checklist coredump lib64 .rocks tarantool usr version tarantool-core-N-202103151305-cc18d0726f6a
 | $ tree
 | ~ .
 | ~ ├── gdb.sh
 | ~ ├── tarantool-core-N-202103151305-cc18d0726f6a
 | ~ │   ├── checklist
 | ~ │   ├── coredump
 | ~ │   ├── lib64
 | ~ │   │   ├── <..skipped..>
 | ~ │   ├── tarantool
 | ~ │   ├── usr
 | ~ │   │   └── lib64
 | ~ │   │       └── tarantool
 | ~ │   │           └── avro_schema_rt_c.so
 | ~ │   └── version
 | ~ └── tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
 | ~
 | ~ 5 directories, 37 files
 | (Now looks okay. Let's try.)
 | $ cd tarantool-core-N-202103151305-cc18d0726f6a
 | $ ../gdb.sh --help
 | ~ ================================================================================
 | ~
 | ~ Do not forget to properly setup the environment:
 | ~ * git clone https://github.com/tarantool/tarantool.git sources
 | ~ * cd !$
 | ~ * git checkout 34d504d
 | ~ * git submodule update --recursive --init
 | ~
 | ~ ================================================================================
 | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1
 | ~ <..skipped..>
 | ~ Reading symbols from ./tarantool...
 | ~ lj-arch command initialized
 | ~ lj-tv command initialized
 | ~ lj-str command initialized
 | ~ lj-tab command initialized
 | ~ lj-stack command initialized
 | ~ lj-state command initialized
 | ~ lj-gc command initialized
 | ~ luajit-gdb.py is successfully loaded
 | ~ warning: Can't open file /usr/bin/tarantool during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib/locale/locale-archive (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libpcre.so.1.2.0 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libselinux.so.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libresolv-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libkeyutils.so.1.5 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libkrb5support.so.0.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libk5crypto.so.3.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libcom_err.so.2.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libkrb5.so.3.3 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libgssapi_krb5.so.2.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libc-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libm-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libstdc++.so.6.0.19 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libunwind.so.8.0.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libunwind-x86_64.so.8.0.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libgcc_s-4.8.5-20150702.so.1 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libpthread-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libgomp.so.1.0.0 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libcrypto.so.1.0.2k during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libssl.so.1.0.2k during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/librt-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libz.so.1.2.7 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libform.so.5.9 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libtinfo.so.5.9 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libncurses.so.5.9 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libreadline.so.6.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libdl-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libicudata.so.50.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libicuuc.so.50.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/libicui18n.so.50.2 during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/ld-2.17.so (deleted) during file-backed mapping note processing
 | ~ warning: Can't open file /usr/lib64/gconv/gconv-modules.cache (deleted) during file-backed mapping note processing
 | ~ [New LWP 246]
 | ~ [New LWP 247]
 | ~ [New LWP 248]
 | ~ [New LWP 250]
 | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 | ~ warning: the debug information found in "/usr/lib64/debug/usr/lib64/tarantool/avro_schema_rt_c.so.debug" does not match "/home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so" (CRC mismatch).
 | ~ 
 | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
 | ~ Core was generated by `tarantool'.
 | ~ Program terminated with signal SIGABRT, Aborted.
 | ~ #0  0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
 | ~ [Current thread is 1 (LWP 246)]
 | (gdb) Ctrl+D
 | (Hey, --help runs it.)
 | (Anyway, I should checkout sources. Nice reminder.)
 | $ git clone https://github.com/tarantool/tarantool.git sources
 | $ cd sources
 | $ git checkout 34d504d
 | $ git submodule update --recursive --init
 | $ cd ..
 | (Let's try again?)
 | $ ../gdb.sh
 | <..I'll skip the output, it is quite similar to one above. But will
 |  comment below.>
 | (The reminder regarding tarantool sources still shown. But I already
 |  placed the sources here.)
 | (All library warnings above are still here. However it seems they're
    just warnings.)
 | $ (gdb) bt
 | ~ #0  0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
 | ~ #1  0x00007f340a5f5a78 in abort () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
 | ~ #2  0x0000000000453c43 in sig_fatal_cb (signo=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/main.cc:302
 | ~ #3  <signal handler called>
 | ~ #4  0x0000000000621ce7 in lj_cconv_ct_ct (cts=<optimized out>, d=<optimized out>, s=0x41508e70, dp=0x41d7a0b8 "", sp=0x140535248 <error: Cannot access memory at address 0x140535248>, 
 | ~     flags=<optimized out>) at lj_cconv.c:234
 | ~ #5  0x0000000000621e61 in lj_cconv_tv_ct (cts=0x40fb9858, s=<optimized out>, sid=<optimized out>, o=0x41d7a0b8, sp=<optimized out>) at lj_cconv.c:389
 | ~ #6  0x0000000000621298 in lj_cdata_get (cts=<optimized out>, s=<optimized out>, o=<optimized out>, sp=<optimized out>) at lj_cdata.c:254
 | ~ #7  0x0000000000631021 in lj_cf_ffi_meta___index (L=0x40fb1378) at lib_ffi.c:158
 | ~ #8  0x00000000006338a7 in lj_BC_FUNCC () at buildvm_x86.dasc:809
 | ~ #9  0x0000000000615c82 in lua_pcall (L=L@entry=0x40fb1378, nargs=<optimized out>, nresults=<optimized out>, errfunc=errfunc@entry=0) at lj_api.c:1158
 | ~ #10 0x00000000005c4663 in luaT_call (L=0x40fb1378, nargs=<optimized out>, nreturns=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/utils.c:1022
 | ~ #11 0x00000000005bd259 in lua_main (L=L@entry=0x40fb1378, argc=argc@entry=1, argv=argv@entry=0x20d5fc0) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:546
 | ~ #12 0x00000000005bd6b3 in run_script_f (ap=<error reading variable: value has been optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:647
 | ~ #13 0x00000000004530dc in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=<optimized out>, ap=<optimized out>)
 | ~     at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.h:882
 | ~ #14 0x00000000005def90 in fiber_loop (data=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.c:879
 | ~ #15 0x00000000007c7ddf in coro_init () at /build/usr/src/debug/tarantool-2.6.2.0/third_party/coro/coro.c:110
 | (Looks good!)
 | (gdb) info sharedlibrary 
 | ~ From                To                  Syms Read   Shared Object Library
 | ~ 0x00007f340e9aa5f0  0x00007f340eab0b62  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicui18n.so.50
 | ~ 0x00007f340e5eb2f0  0x00007f340e69dad8  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicuuc.so.50
 | ~ 0x00007f340cfc6550  0x00007f340cfc6635  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicudata.so.50
 | ~ 0x00007f340cdc2e50  0x00007f340cdc394e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libdl.so.2
 | ~ 0x00007f340cb90de0  0x00007f340cbaf775  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libreadline.so.6
 | ~ 0x00007f340c95bd50  0x00007f340c975462  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libncurses.so.5
 | ~ 0x00007f340c737e40  0x00007f340c743bb8  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libtinfo.so.5
 | ~ 0x00007f340c51e6e0  0x00007f340c525ed0  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libform.so.5
 | ~ 0x00007f340c307110  0x00007f340c313698  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libz.so.1
 | ~ 0x00007f340c0ff230  0x00007f340c1021bc  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/librt.so.1
 | ~ 0x00007f340bea5b90  0x00007f340bee164d  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libssl.so.10
 | ~ 0x00007f340ba95200  0x00007f340bbce7b7  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcrypto.so.10
 | ~ 0x00007f340b809060  0x00007f340b81fc2a  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgomp.so.1
 | ~ 0x00007f340b5eb8f0  0x00007f340b5f6db1  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpthread.so.0
 | ~ 0x00007f340b3d2ad0  0x00007f340b3e2285  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgcc_s.so.1
 | ~ 0x00007f340b1b1570  0x00007f340b1be20e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind-x86_64.so.8
 | ~ 0x00007f340af975b0  0x00007f340af9e95e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind.so.8
 | ~ 0x00007f340ace94e0  0x00007f340ad5059a  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libstdc++.so.6
 | ~ 0x00007f340a991350  0x00007f340a9fc336  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libm.so.6
 | ~ 0x00007f340a5dd9f0  0x00007f340a72da5f  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
 | ~ 0x00007f340ed11af0  0x00007f340ed2d060  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/ld-linux-x86-64.so.2
 | ~ 0x00007f340a37d720  0x00007f340a3afbca  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgssapi_krb5.so.2
 | ~ 0x00007f340a0adc20  0x00007f340a1159fa  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5.so.3
 | ~ 0x00007f3409e85540  0x00007f3409e86113  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcom_err.so.2
 | ~ 0x00007f3409c558d0  0x00007f3409c73bbf  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libk5crypto.so.3
 | ~ 0x00007f3409a44bf0  0x00007f3409a4c36b  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5support.so.0
 | ~ 0x00007f340983e5b0  0x00007f340983f1cc  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkeyutils.so.1
 | ~ 0x00007f3409626980  0x00007f34096351e2  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libresolv.so.2
 | ~ 0x00007f3409402a90  0x00007f3409418896  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libselinux.so.1
 | ~ 0x00007f340919b5f0  0x00007f34091e15b0  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpcre.so.1
 | ~ 0x00007f3408c429f0  0x00007f3408c43896  Yes         /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/.rocks/lib/tarantool/tuple/keydef.so
 | ~ 0x00007f33eedf8d80  0x00007f33eedfc174  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so
 | ~ (*): Shared library is missing debugging information.
 | (Looks good too!)

> Can't open file <...> (deleted) during file-backed mapping note
> processing.

I don't know what it is. But I'll share my gdb version just in case
(maybe you'll be able to reproduce it). I didn't meet this warning
before.

 | $ gdb --version | head -n 1
 | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1

However I would note that those warnings misleads me the first time: I
thought that something is wrong with library paths.

> Unable to find libthread_db matching inferior's thread library, thread
> debugging will not be available.

Sure, it is not listed in the lld output (neither in 'info
sharedlibrary'), but maybe we should include it into the tarabrt.sh
generated archive.

> the debug information found in <...> does not match <...> (CRC mismatch).

It is just because I have local avro schema debuginfo installed. I think
this warning can be ignored.

To sum up: my experience was successful. You may polish some tiny
details, but in fact the scripts seem to be ready for use.

----

We also briefly discussed the topic about stripped executables with
Igor.

Now we ship stripped executables in RPM packages for CentOS 8 (only for
CentOS 8), but the separate debug package is stored in our repos.

First point: we can include /etc/os-release into the generated archive.

Second point: we can give URL from where debuginfo package may be
downloaded and give advice how to unpack it (in gdb.sh output, when we
see the stripped executable).

Examples:

 | $ curl -LfsS https://download.tarantool.org/tarantool/release/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm
 | $ mkdir tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
 | $ cd tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
 | $ rpm2cpio ../tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm | cpio -idmv
 | ~ ./usr/lib/debug
 | ~ ./usr/lib/debug/.build-id
 | ~ ./usr/lib/debug/.build-id/a3
 | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60
 | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60.debug
 | ~ ./usr/lib/debug/usr
 | ~ ./usr/lib/debug/usr/bin
 | ~ ./usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
 | ~ 64743 blocks
 | (All are symlinks except tarantool-2.8.0.0-1.el8.x86_64.debug.)
 | $ file usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
 | ~ usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter \004, for GNU/Linux 3.2.0, BuildID[sha1]=a3ad84db98a9b14e70d2915271904ddfe26b5f60, with debug_info, not stripped, too many notes (256)

 | $ curl -LfsS https://download.tarantool.org/tarantool/live/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm

Third: we can give advice how to enable the ELF with debug symbols in
the interactive gdb session.

Or we should do it within gdb.sh to call 'info line main'? Expect it in
some predefined location or accept an option?

At least it looks worthful to warn about stripped executable explicitly.

----

It seems, I reached maximum value that I can give as reviewer, so I'll
just say that the patch looks good to me (and I don't mind small changes
around usability if you want to do them).

WBR, Alexander Turenko.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-03-11 12:13   ` Igor Munkin via Tarantool-patches
@ 2021-03-24 16:02     ` Sergey Bronnikov via Tarantool-patches
  2021-03-24 20:25       ` Alexander Turenko via Tarantool-patches
  2021-04-19 21:06       ` Igor Munkin via Tarantool-patches
  0 siblings, 2 replies; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2021-03-24 16:02 UTC (permalink / raw)
  To: Igor Munkin; +Cc: Alexander Turenko, tarantool-patches

Igor,

thanks for changes! Generally LGTM with two notes inline.

On 11.03.2021 15:13, Igor Munkin wrote:
> Sergey,
>
> Thanks for your review!
>
> On 03.03.21, Sergey Bronnikov wrote:
<skipped>
>> 2. I propose to show files included to the report and the end of creation.
> Well, I believe I didn't understand you right before (this list is
> included to the archive). Anyway, now I added the list also to the
> message at the end.

You got it right.

<skipped>


1. in a private conversation we decided to remove mention of public 
resource in a message "http://transfer.sh".


 >  - Manual usage. User can simply pack all necessary artefacts by 
running the
 >    following command.
 >    $ /path/to/${TOOL} -c ./core -d /tmp
 >
 >  - Automatic usage. If user faces the failures often, one can set 
this script
 >    as a pipe reciever in kernel.core_pattern syntax.
 >    # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d 
/var/core -p %p -t %t"

2. it would be more convenient to set real path to tarabrt.sh in a 
usage, see patch below

--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -2,6 +2,8 @@
  set -eu

  TOOL=$(basename "$0")
+#TOOLPATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
+TOOLPATH=$(realpath -s $0)
  HELP=$(cat <<HELP
  ${TOOL} - Tarantool Automatic Bug Reporting Tool

@@ -45,11 +47,11 @@ USAGE

    - Manual usage. User can simply pack all necessary artefacts by 
running the
      following command.
-    $ /path/to/${TOOL} -c ./core -d /tmp
+    $ ${TOOLPATH} -c ./core -d /tmp

    - Automatic usage. If user faces the failures often, one can set 
this script
      as a pipe reciever in kernel.core_pattern syntax.
-    # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d 
/var/core -p %p -t %t"
+    # sysctl -w kernel.core_pattern="|${TOOLPATH} -d /var/core -p %p -t %t"

  HELP
  )


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-03-24 16:02     ` Sergey Bronnikov via Tarantool-patches
@ 2021-03-24 20:25       ` Alexander Turenko via Tarantool-patches
  2021-03-25  9:45         ` Sergey Bronnikov via Tarantool-patches
  2021-04-19 21:06       ` Igor Munkin via Tarantool-patches
  1 sibling, 1 reply; 14+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-03-24 20:25 UTC (permalink / raw)
  To: Sergey Bronnikov; +Cc: tarantool-patches

> 1. in a private conversation we decided to remove mention of public resource
> in a message "http://transfer.sh".

Can you (or Igor) share a reason?

>  TOOL=$(basename "$0")
> +#TOOLPATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
> +TOOLPATH=$(realpath -s $0)

'realpath' is not necessarily available. Actually I saw that it may be
not available on Mac OS, but I didn't saw such situation on Linux.
However realpath from busybox does not support -s option (verified on
Alpine 3.10).

My opinion: it is better to don't extend the list of external
dependencies in this case.

I would also highlight that it is quite usual to use just $0 in a usage
message:

 | $ cat --help | head -n 1
 | Usage: cat [OPTION]... [FILE]...
 |
 | $ /bin/cat --help | head -n 1
 | Usage: /bin/cat [OPTION]... [FILE]...

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-03-24 20:25       ` Alexander Turenko via Tarantool-patches
@ 2021-03-25  9:45         ` Sergey Bronnikov via Tarantool-patches
  0 siblings, 0 replies; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2021-03-25  9:45 UTC (permalink / raw)
  To: Alexander Turenko; +Cc: tarantool-patches

Hello,

On 24.03.2021 23:25, Alexander Turenko wrote:
>> 1. in a private conversation we decided to remove mention of public resource
>> in a message "http://transfer.sh".
> Can you (or Igor) share a reason?

We suppose that coredump may contain a sensitive data

and it would be wrong to recommend to share an archive on a public file 
hosting (at least without encryption).

>>   TOOL=$(basename "$0")
>> +#TOOLPATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
>> +TOOLPATH=$(realpath -s $0)
> 'realpath' is not necessarily available. Actually I saw that it may be
> not available on Mac OS, but I didn't saw such situation on Linux.
> However realpath from busybox does not support -s option (verified on
> Alpine 3.10).
>
> My opinion: it is better to don't extend the list of external
> dependencies in this case.
>
> I would also highlight that it is quite usual to use just $0 in a usage
> message:
>
>   | $ cat --help | head -n 1
>   | Usage: cat [OPTION]... [FILE]...
>   |
>   | $ /bin/cat --help | head -n 1
>   | Usage: /bin/cat [OPTION]... [FILE]...

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-03-15 16:30 ` Alexander Turenko via Tarantool-patches
@ 2021-04-19 20:42   ` Igor Munkin via Tarantool-patches
  2021-04-19 23:57     ` Alexander Turenko via Tarantool-patches
  0 siblings, 1 reply; 14+ messages in thread
From: Igor Munkin via Tarantool-patches @ 2021-04-19 20:42 UTC (permalink / raw)
  To: Alexander Turenko; +Cc: tarantool-patches

Sasha,

Thanks a lot for your review!

On 15.03.21, Alexander Turenko wrote:
> I looked over the WIP version of the script and nothing confuses me:
> 
> https://lists.tarantool.org/tarantool-patches/20210220104630.wiwxhkjfimsx2swa@tkn_work_nb/
> 
> So I guess I should do some basic testing to get and share my user
> experience.

Nice, then I'll go through your user experience and adjust the script
considering the problems you're facing.

> 
> So I'll step into another environment to create coredump and archive it:

<snipped the manipulations with installing and running Tarantool>

> 
>  | tarantool> require('ffi').new('int[?]', 1)[1024 ^ 3]
>  | ~ <...>
>  | ~ Aborted (core dumped)
>  | # ls -hl core
>  | ~ -rw------- 1 root root 180M Mar 15 12:49 core
>  | # file core
>  | ~ bash: file: command not found
> 
> (I'll ignore that 'file' is not installed, because it is possible
> situation on a production server, I guess.)

I didn't figure out, whether <file> package is required, recommended or
optional. So if your Docker image doesn't have it, then the production
server can also lack this package.

> 
> Okay, now I got the tarabrt.sh script somehow:
> 
>  | # curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/tarabrt.sh >tarabrt.sh
>  | # chmod a+x tarabrt.sh
>  | (Don't read the content, just run.)
>  | # ./tarabrt.sh --help
>  | ~ tarabrt.sh: unrecognized option '--help'

Unfortunately, I have nothing to do here: this is returned by <getopt>.

>  | # ./tarabrt.sh -h  
>  | ~ <..skipped..>
>  | ~   - Manual usage. User can simply pack all necessary artefacts by running the
>  | ~     following command.
>  | ~     $ /path/to/tarabrt.sh -c ./core -d /tmp
>  | ~ <..skipped..>
>  | # ./tarabrt.sh -c ./core -d /tmp
>  | ~ ./tarabrt.sh: line 115: file: command not found
>  | ~ gdb is not installed, but it is obligatory for collecting the
>  | ~ loaded shared libraries from the core dump.
>  | ~ 
>  | ~ You can proceed collecting the artefacts manually later by running
>  | ~ the following command:
>  | ~ $ tarabrt.sh -e /usr/bin/tarantool -c ./core
>  | (Okay, I'll install file and gdb now.)

I'll added the check that file is installed. Diff is below:

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 372ff80c1..005add707 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -112,6 +112,17 @@ NOCOREDUMP
 	exit 1;
 fi
 
+if ! command -v file >/dev/null; then
+	[ -t 1 ] && cat <<NOFILE
+file is not installed, but it is obligatory for checking the
+artefacts to be collected.
+
+You can proceed collecting the artefacts manually later by running
+the following command:
+$ ${TOOL} -e ${BINARY} -c ${COREFILE}
+NOFILE
+fi
+
 if file "${COREFILE}" | grep -qv 'core file'; then
 	[ -t 1 ] && cat <<NOTACOREDUMP
 Not a core dump: ${COREFILE}

================================================================================

>  | # yum install -y file gdb
>  | # file core
>  | ~ core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'tarantool', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/tarantool', platform: 'x86_64'
>  | (Repeat the attempt to create the archive.)
>  | # ./tarabrt.sh -c ./core -d /tmp

Side note: Heh, I see you didn't read the error message, since your
command differs from the suggested one.

>  | ~ The resulting archive is located here: /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ The archive contents are listed below:
>  | ~ /tmp/tmp.kqC76QSrpF/tarlist
>  | ~ /tmp/tmp.kqC76QSrpF/version
>  | ~ /usr/bin/tarantool
>  | ~ ./core
>  | ~ /lib64/libicui18n.so.50
>  | ~ /lib64/libicuuc.so.50
>  | ~ /lib64/libicudata.so.50
>  | ~ /lib64/libdl.so.2
>  | ~ /lib64/libreadline.so.6
>  | ~ /lib64/libncurses.so.5
>  | ~ /lib64/libtinfo.so.5
>  | ~ /lib64/libform.so.5
>  | ~ /lib64/libz.so.1
>  | ~ /lib64/librt.so.1
>  | ~ /lib64/libssl.so.10
>  | ~ /lib64/libcrypto.so.10
>  | ~ /lib64/libgomp.so.1
>  | ~ /lib64/libpthread.so.0
>  | ~ /lib64/libgcc_s.so.1
>  | ~ /lib64/libunwind-x86_64.so.8
>  | ~ /lib64/libunwind.so.8
>  | ~ /lib64/libstdc++.so.6
>  | ~ /lib64/libm.so.6
>  | ~ /lib64/libc.so.6
>  | ~ /lib64/ld-linux-x86-64.so.2
>  | ~ /lib64/libgssapi_krb5.so.2
>  | ~ /lib64/libkrb5.so.3
>  | ~ /lib64/libcom_err.so.2
>  | ~ /lib64/libk5crypto.so.3
>  | ~ /lib64/libkrb5support.so.0
>  | ~ /lib64/libkeyutils.so.1
>  | ~ /lib64/libresolv.so.2
>  | ~ /lib64/libselinux.so.1
>  | ~ /lib64/libpcre.so.1
>  | ~ /.rocks/lib/tarantool/tuple/keydef.so
>  | ~ /usr/lib64/tarantool/avro_schema_rt_c.so
>  | ~ 
>  | ~ If you want to upload it, choose the available resource
>  | ~ (e.g. http://transfer.sh) and run the following command:
>  | ~ $ curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz <resource-uri>
> 
> Okay, let's spin around this output for a while:
> 
>  | (The /tmp/ entries looks are something I can glance.)
>  | # ll /tmp/tmp.kqC76QSrpF/tarlist
>  | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/tarlist: No such file or directory
>  | ll /tmp/tmp.kqC76QSrpF/version
>  | ~ ls: cannot access /tmp/tmp.kqC76QSrpF/version: No such file or directory
>  | (But they're not. Okay, pass it over.)

These files are temporary (considering their names) and are removed when
the script finishes. I can list only their basename, but not listing them
at all looks like stealing them... Hence, consider the changes below.

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 005add707..e66621895 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -200,6 +200,7 @@ ${BINARY} --version >"${VERSION}"
 
 # Collect the most important artefacts.
 {
+	echo "${TARLIST}"
 	echo "${VERSION}"
 	echo "${BINARY}"
 	echo "${COREFILE}"
@@ -239,13 +240,15 @@ tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
 	--transform="s|${COREFILE}|coredump|" \
 	--transform="s|${TARLIST}|checklist|" \
 	--transform="s|${VERSION}|version|"   \
-	--add-file="${TARLIST}"
+
+# Strip TMPDIR prefix from the files in TARLIST, since this
+# directory will be removed at the end.
+sed "s|${TMPDIR}||g" -i "${TARLIST}"
 
 [ -t 1 ] && cat <<FINALIZE
 The resulting archive is located here: ${ARCHIVENAME}
 The archive contents are listed below:
-${TARLIST}
-$(cat ${TARLIST})
+$(cat "${TARLIST}")
 
 If you want to upload it, choose the available resource
 (e.g. http://transfer.sh) and run the following command:

================================================================================

>  | (keydef.so and avro_schema_rt_c.so are here, nice!)
>  | (Let's look at the archive file.)
>  | # ls -hl /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ -rw-r--r-- 1 root root 27M Mar 15 13:05 /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | # tar -vtf /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ -rw-r--r-- root/root      1085 2021-03-15 13:05 version
>  | ~ -rwxr-xr-x root/root  34431952 2020-12-30 14:08 tarantool
>  | ~ -rw------- root/root 188039168 2021-03-15 12:49 coredump
>  | ~ tar: Removing leading `/' from member names
>  | ~ -rwxr-xr-x root/root   2096104 2020-03-21 00:49 /lib64/libicui18n.so.50
>  | ~ -rwxr-xr-x root/root   1539520 2020-03-21 00:49 /lib64/libicuuc.so.50
>  | ~ -rwxr-xr-x root/root  20785824 2020-03-21 00:49 /lib64/libicudata.so.50
>  | ~ -rwxr-xr-x root/root     19248 2021-02-08 15:46 /lib64/libdl.so.2
>  | ~ -rwxr-xr-x root/root    285136 2019-08-08 12:07 /lib64/libreadline.so.6
>  | ~ -rwxr-xr-x root/root    163696 2017-09-06 22:08 /lib64/libncurses.so.5
>  | ~ -rwxr-xr-x root/root    174576 2017-09-06 22:08 /lib64/libtinfo.so.5
>  | ~ -rwxr-xr-x root/root     67864 2017-09-06 22:08 /lib64/libform.so.5
>  | ~ -rwxr-xr-x root/root     90248 2018-10-30 20:24 /lib64/libz.so.1
>  | ~ -rwxr-xr-x root/root     43712 2021-02-08 15:46 /lib64/librt.so.1
>  | ~ -rwxr-xr-x root/root    470376 2020-12-16 21:24 /lib64/libssl.so.10
>  | ~ -rwxr-xr-x root/root   2520768 2020-12-16 21:24 /lib64/libcrypto.so.10
>  | ~ -rwxr-xr-x root/root    154840 2020-09-30 02:17 /lib64/libgomp.so.1
>  | ~ -rwxr-xr-x root/root    142144 2021-02-08 15:46 /lib64/libpthread.so.0
>  | ~ -rwxr-xr-x root/root     88720 2020-09-30 02:18 /lib64/libgcc_s.so.1
>  | ~ -rwxr-xr-x root/root     73416 2017-08-02 15:45 /lib64/libunwind-x86_64.so.8
>  | ~ -rwxr-xr-x root/root     48904 2017-08-02 15:45 /lib64/libunwind.so.8
>  | ~ -rwxr-xr-x root/root    995840 2020-09-30 02:17 /lib64/libstdc++.so.6
>  | ~ -rwxr-xr-x root/root   1136944 2021-02-08 15:46 /lib64/libm.so.6
>  | ~ -rwxr-xr-x root/root   2156344 2021-02-08 15:46 /lib64/libc.so.6
>  | ~ -rwxr-xr-x root/root    163312 2021-02-08 15:46 /lib64/ld-linux-x86-64.so.2
>  | ~ -rwxr-xr-x root/root    320720 2020-09-30 16:51 /lib64/libgssapi_krb5.so.2
>  | ~ -rwxr-xr-x root/root    967840 2020-09-30 16:51 /lib64/libkrb5.so.3
>  | ~ -rwxr-xr-x root/root     15856 2020-09-30 15:58 /lib64/libcom_err.so.2
>  | ~ -rwxr-xr-x root/root    210784 2020-09-30 16:51 /lib64/libk5crypto.so.3
>  | ~ -rwxr-xr-x root/root     67104 2020-09-30 16:51 /lib64/libkrb5support.so.0
>  | ~ -rwxr-xr-x root/root     15688 2014-06-10 02:17 /lib64/libkeyutils.so.1
>  | ~ -rwxr-xr-x root/root    109976 2021-02-08 15:46 /lib64/libresolv.so.2
>  | ~ -rwxr-xr-x root/root    155744 2020-04-01 03:16 /lib64/libselinux.so.1
>  | ~ -rwxr-xr-x root/root    402384 2017-08-02 03:08 /lib64/libpcre.so.1
>  | ~ -rwxr-xr-x root/root     52168 2021-03-15 12:45 /.rocks/lib/tarantool/tuple/keydef.so
>  | ~ -rwxr-xr-x root/root     23736 2020-08-05 23:10 /usr/lib64/tarantool/avro_schema_rt_c.so
>  | ~ -rw-r--r-- root/root       796 2021-03-15 13:05 checklist
>  | (Can we avoid the the warning re `/`? Just curious.)

It turns out yes! The change is quite simple though the diff looks huge:

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 46e3016f0..63211f9a8 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -193,7 +193,8 @@ HOSTNAME=$(hostname 2>/dev/null || echo hostname)
 TMPDIR=$(mktemp -d -p "${COREDIR}")
 TARLIST=${TMPDIR}/tarlist
 VERSION=${TMPDIR}/version
-ARCHIVENAME=${COREDIR}/tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz
+ARCHIVENAME=tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}
+ARCHIVE=${COREDIR}/${ARCHIVENAME}.tar.gz
 
 # Dump the version to checkout the right commit later.
 ${BINARY} --version >"${VERSION}"
@@ -235,24 +236,25 @@ fi
 # Pack everything listed in TARLIST file into a tarball. To unify
 # the archive format BINARY, COREFILE, VERSION and TARLIST are
 # renamed while packing.
-tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
-	--transform="s|${BINARY}|tarantool|"  \
-	--transform="s|${COREFILE}|coredump|" \
-	--transform="s|${TARLIST}|checklist|" \
-	--transform="s|${VERSION}|version|"   \
+tar -czhf "${ARCHIVE}" -P -T "${TARLIST}" \
+	--transform="s|${BINARY}|/tarantool|"  \
+	--transform="s|${COREFILE}|/coredump|" \
+	--transform="s|${TARLIST}|/checklist|" \
+	--transform="s|${VERSION}|/version|"   \
+	--transform="s|^|${ARCHIVENAME}|"
 
 # Strip TMPDIR prefix from the files in TARLIST, since this
 # directory will be removed at the end.
 sed "s|${TMPDIR}||g" -i "${TARLIST}"
 
 [ -t 1 ] && cat <<FINALIZE
-The resulting archive is located here: ${ARCHIVENAME}
+The resulting archive is located here: ${ARCHIVE}
 The archive contents are listed below:
 $(cat "${TARLIST}")
 
 If you want to upload it, choose the available resource
 (e.g. http://transfer.sh) and run the following command:
-$ curl -T ${ARCHIVENAME} <resource-uri>
+$ curl -T ${ARCHIVE} <resource-uri>
 FINALIZE
 
 # Cleanup temporary files.

================================================================================

>  | (Okay, I see that transfer.sh is suggested. I never use it, so I'm
>  |  curious.)
>  | # curl -T /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz http://transfer.sh
>  | ~ http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | (Hey, something occurs! But I'll save it to my system anyway.)
>  | # cp /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz /cores/
>  | (And, while I'm here...)
>  | # md5sum /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ bec8f81c69ed992a258a6d9f0b8a04dc  /tmp/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
> 
> Okay, let's try gdb.sh on my Gentoo box:
> 
>  | $ mkdir ~/tmp/try-gdb-sh
>  | $ cd ~/tmp/try-gdb-sh
>  | $ curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/gdb.sh > gdb.sh
>  | $ chmod a+x gdb.sh
>  | $ curl -LfsS http://transfer.sh/HeUux/tarantool-core-N-202103151305-cc18d0726f6a.tar.gz > tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | $ md5sum tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ bec8f81c69ed992a258a6d9f0b8a04dc  tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | (Matches!)
>  | (Hm, now I'm worry, whether the archive will be uncompressed into the
>  |  current directory or will create its own directory?)

Now (as a result of the patch above) it creates its own directory, that
is archive basename without the suffix.

>  | $ tar -xf tarantool-core-N-202103151305-cc18d0726f6a.tar.gz 
>  | ~ tar: Removing leading `/' from member names
>  | $ ll
>  | ~ total 239M
>  | ~ drwxr-xr-x   5 alex users 4.0K Mar 15 16:56 .
>  | ~ drwxr-xr-x 164 alex users  20K Mar 15 16:50 ..
>  | ~ -rw-r--r--   1 alex users  796 Mar 15 16:05 checklist
>  | ~ -rw-------   1 alex users 180M Mar 15 15:49 coredump
>  | ~ -rwxr-xr-x   1 alex users 2.0K Mar 15 16:51 gdb.sh
>  | ~ drwxr-xr-x   2 alex users 4.0K Mar 15 16:56 lib64
>  | ~ drwxr-xr-x   3 alex users 4.0K Mar 15 16:56 .rocks
>  | ~ -rwxr-xr-x   1 alex users  33M Dec 30 17:08 tarantool
>  | ~ -rw-r--r--   1 alex users  27M Mar 15 16:54 tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~ drwxr-xr-x   3 alex users 4.0K Mar 15 16:56 usr
>  | ~ -rw-r--r--   1 alex users 1.1K Mar 15 16:05 version
>  | (Meh. Let's move it around.)
>  | $ mkdir tarantool-core-N-202103151305-cc18d0726f6a
>  | $ mv checklist coredump lib64 .rocks tarantool usr version tarantool-core-N-202103151305-cc18d0726f6a
>  | $ tree
>  | ~ .
>  | ~ ├── gdb.sh
>  | ~ ├── tarantool-core-N-202103151305-cc18d0726f6a
>  | ~ │   ├── checklist
>  | ~ │   ├── coredump
>  | ~ │   ├── lib64
>  | ~ │   │   ├── <..skipped..>
>  | ~ │   ├── tarantool
>  | ~ │   ├── usr
>  | ~ │   │   └── lib64
>  | ~ │   │       └── tarantool
>  | ~ │   │           └── avro_schema_rt_c.so
>  | ~ │   └── version
>  | ~ └── tarantool-core-N-202103151305-cc18d0726f6a.tar.gz
>  | ~
>  | ~ 5 directories, 37 files
>  | (Now looks okay. Let's try.)
>  | $ cd tarantool-core-N-202103151305-cc18d0726f6a
>  | $ ../gdb.sh --help
>  | ~ ================================================================================
>  | ~
>  | ~ Do not forget to properly setup the environment:
>  | ~ * git clone https://github.com/tarantool/tarantool.git sources
>  | ~ * cd !$
>  | ~ * git checkout 34d504d
>  | ~ * git submodule update --recursive --init
>  | ~
>  | ~ ================================================================================
>  | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1
>  | ~ <..skipped..>
>  | ~ Reading symbols from ./tarantool...
>  | ~ lj-arch command initialized
>  | ~ lj-tv command initialized
>  | ~ lj-str command initialized
>  | ~ lj-tab command initialized
>  | ~ lj-stack command initialized
>  | ~ lj-state command initialized
>  | ~ lj-gc command initialized
>  | ~ luajit-gdb.py is successfully loaded
>  | ~ warning: Can't open file /usr/bin/tarantool during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib/locale/locale-archive (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libpcre.so.1.2.0 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libselinux.so.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libresolv-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libkeyutils.so.1.5 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libkrb5support.so.0.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libk5crypto.so.3.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libcom_err.so.2.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libkrb5.so.3.3 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libgssapi_krb5.so.2.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libc-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libm-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libstdc++.so.6.0.19 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libunwind.so.8.0.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libunwind-x86_64.so.8.0.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libgcc_s-4.8.5-20150702.so.1 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libpthread-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libgomp.so.1.0.0 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libcrypto.so.1.0.2k during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libssl.so.1.0.2k during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/librt-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libz.so.1.2.7 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libform.so.5.9 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libtinfo.so.5.9 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libncurses.so.5.9 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libreadline.so.6.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libdl-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libicudata.so.50.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libicuuc.so.50.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/libicui18n.so.50.2 during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/ld-2.17.so (deleted) during file-backed mapping note processing
>  | ~ warning: Can't open file /usr/lib64/gconv/gconv-modules.cache (deleted) during file-backed mapping note processing
>  | ~ [New LWP 246]
>  | ~ [New LWP 247]
>  | ~ [New LWP 248]
>  | ~ [New LWP 250]
>  | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
>  | ~ warning: the debug information found in "/usr/lib64/debug/usr/lib64/tarantool/avro_schema_rt_c.so.debug" does not match "/home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so" (CRC mismatch).
>  | ~ 
>  | ~ warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
>  | ~ Core was generated by `tarantool'.
>  | ~ Program terminated with signal SIGABRT, Aborted.
>  | ~ #0  0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
>  | ~ [Current thread is 1 (LWP 246)]
>  | (gdb) Ctrl+D
>  | (Hey, --help runs it.)

Lol, I didn't even check this. Thanks for the example. I've added
arguments handling and the usage for the script. Diff is below:

================================================================================

diff --git a/tools/gdb.sh b/tools/gdb.sh
index a58c47cab..55fa73d7a 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -1,6 +1,40 @@
 #!/bin/sh
 set -eu
 
+TOOL=$(basename "$0")
+HELP=$(cat <<HELP
+${TOOL} - gdb wrapper loading artefacts collected via tarabrt.sh
+
+	Soon her eye fell upon a little glass box lying underneath
+	the table. She opened it and found in it a very small
+	cake, on which the words "EAT ME" were beautifully marked
+	in currants.
+
+	- Alice's Adventures in Wonderland.
+HELP
+)
+
+# Parse CLI options.
+OPTIONS=$(getopt -o h -n "${TOOL}" -- "$@")
+eval set -- "${OPTIONS}"
+while true; do
+	case "$1" in
+		--) shift; break;;
+		-h) printf "%s\n" "${HELP}";
+			exit 0;;
+		*)  printf "Invalid option: $1\n%s\n" "${HELP}";
+			exit 1;;
+	esac
+done
+
+# Do not proceed if there are some CLI arguments left. Everything
+# should be parsed before this line. Refer to the help message in
+# a funny way.
+if [ $# -ne 0 ]; then
+	echo "RUN ME"
+	exit 1;
+fi
+
 # Check that gdb is installed.
 if ! command -v gdb >/dev/null; then
 	cat <<NOGDB

================================================================================

>  | (Anyway, I should checkout sources. Nice reminder.)
>  | $ git clone https://github.com/tarantool/tarantool.git sources
>  | $ cd sources
>  | $ git checkout 34d504d
>  | $ git submodule update --recursive --init
>  | $ cd ..
>  | (Let's try again?)
>  | $ ../gdb.sh
>  | <..I'll skip the output, it is quite similar to one above. But will
>  |  comment below.>
>  | (The reminder regarding tarantool sources still shown. But I already
>  |  placed the sources here.)

Added the check whether the sources are already setup. Diff is below:

================================================================================

diff --git a/tools/gdb.sh b/tools/gdb.sh
index fd310b99b..63e80c8f3 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -67,18 +67,30 @@ NOARTEFACTS
 	exit 1;
 fi
 
-REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
-cat <<SOURCES
+SOURCES=${PWD}/sources
+
+# Check whether Tarantool sources are setup. Otherwise, leave a
+# recipe for user, how to do it.
+# FIXME: This can be done automatically (simply run the commands
+# below), but this obliges git to be installed. For now, it makes
+# the wrapper more complex by additional checks, so this activity
+# is left for the user.
+if [ ! -d "${SOURCES}" ]; then
+	REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
+	cat <<SOURCES
 ================================================================================
 
 Do not forget to properly setup the environment:
-* git clone https://github.com/tarantool/tarantool.git sources
+* git clone https://github.com/tarantool/tarantool.git ${SOURCES}
 * cd !$
 * git checkout ${REVISION}
 * git submodule update --recursive --init
+* cd -
 
 ================================================================================
 SOURCES
+	exit 1;
+fi
 
 # Define the build path to be substituted with the source path.
 # XXX: Check the absolute path on the function <main> definition

================================================================================

>  | (All library warnings above are still here. However it seems they're
>     just warnings.)
>  | $ (gdb) bt
>  | ~ #0  0x00007f340a5f4387 in ssignal () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
>  | ~ #1  0x00007f340a5f5a78 in abort () from /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
>  | ~ #2  0x0000000000453c43 in sig_fatal_cb (signo=<optimized out>, siginfo=<optimized out>, context=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/main.cc:302
>  | ~ #3  <signal handler called>
>  | ~ #4  0x0000000000621ce7 in lj_cconv_ct_ct (cts=<optimized out>, d=<optimized out>, s=0x41508e70, dp=0x41d7a0b8 "", sp=0x140535248 <error: Cannot access memory at address 0x140535248>, 
>  | ~     flags=<optimized out>) at lj_cconv.c:234
>  | ~ #5  0x0000000000621e61 in lj_cconv_tv_ct (cts=0x40fb9858, s=<optimized out>, sid=<optimized out>, o=0x41d7a0b8, sp=<optimized out>) at lj_cconv.c:389
>  | ~ #6  0x0000000000621298 in lj_cdata_get (cts=<optimized out>, s=<optimized out>, o=<optimized out>, sp=<optimized out>) at lj_cdata.c:254
>  | ~ #7  0x0000000000631021 in lj_cf_ffi_meta___index (L=0x40fb1378) at lib_ffi.c:158
>  | ~ #8  0x00000000006338a7 in lj_BC_FUNCC () at buildvm_x86.dasc:809
>  | ~ #9  0x0000000000615c82 in lua_pcall (L=L@entry=0x40fb1378, nargs=<optimized out>, nresults=<optimized out>, errfunc=errfunc@entry=0) at lj_api.c:1158
>  | ~ #10 0x00000000005c4663 in luaT_call (L=0x40fb1378, nargs=<optimized out>, nreturns=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/utils.c:1022
>  | ~ #11 0x00000000005bd259 in lua_main (L=L@entry=0x40fb1378, argc=argc@entry=1, argv=argv@entry=0x20d5fc0) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:546
>  | ~ #12 0x00000000005bd6b3 in run_script_f (ap=<error reading variable: value has been optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lua/init.c:647
>  | ~ #13 0x00000000004530dc in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=<optimized out>, ap=<optimized out>)
>  | ~     at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.h:882
>  | ~ #14 0x00000000005def90 in fiber_loop (data=<optimized out>) at /build/usr/src/debug/tarantool-2.6.2.0/src/lib/core/fiber.c:879
>  | ~ #15 0x00000000007c7ddf in coro_init () at /build/usr/src/debug/tarantool-2.6.2.0/third_party/coro/coro.c:110
>  | (Looks good!)
>  | (gdb) info sharedlibrary 
>  | ~ From                To                  Syms Read   Shared Object Library
>  | ~ 0x00007f340e9aa5f0  0x00007f340eab0b62  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicui18n.so.50
>  | ~ 0x00007f340e5eb2f0  0x00007f340e69dad8  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicuuc.so.50
>  | ~ 0x00007f340cfc6550  0x00007f340cfc6635  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libicudata.so.50
>  | ~ 0x00007f340cdc2e50  0x00007f340cdc394e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libdl.so.2
>  | ~ 0x00007f340cb90de0  0x00007f340cbaf775  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libreadline.so.6
>  | ~ 0x00007f340c95bd50  0x00007f340c975462  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libncurses.so.5
>  | ~ 0x00007f340c737e40  0x00007f340c743bb8  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libtinfo.so.5
>  | ~ 0x00007f340c51e6e0  0x00007f340c525ed0  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libform.so.5
>  | ~ 0x00007f340c307110  0x00007f340c313698  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libz.so.1
>  | ~ 0x00007f340c0ff230  0x00007f340c1021bc  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/librt.so.1
>  | ~ 0x00007f340bea5b90  0x00007f340bee164d  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libssl.so.10
>  | ~ 0x00007f340ba95200  0x00007f340bbce7b7  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcrypto.so.10
>  | ~ 0x00007f340b809060  0x00007f340b81fc2a  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgomp.so.1
>  | ~ 0x00007f340b5eb8f0  0x00007f340b5f6db1  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpthread.so.0
>  | ~ 0x00007f340b3d2ad0  0x00007f340b3e2285  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgcc_s.so.1
>  | ~ 0x00007f340b1b1570  0x00007f340b1be20e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind-x86_64.so.8
>  | ~ 0x00007f340af975b0  0x00007f340af9e95e  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libunwind.so.8
>  | ~ 0x00007f340ace94e0  0x00007f340ad5059a  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libstdc++.so.6
>  | ~ 0x00007f340a991350  0x00007f340a9fc336  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libm.so.6
>  | ~ 0x00007f340a5dd9f0  0x00007f340a72da5f  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libc.so.6
>  | ~ 0x00007f340ed11af0  0x00007f340ed2d060  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/ld-linux-x86-64.so.2
>  | ~ 0x00007f340a37d720  0x00007f340a3afbca  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libgssapi_krb5.so.2
>  | ~ 0x00007f340a0adc20  0x00007f340a1159fa  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5.so.3
>  | ~ 0x00007f3409e85540  0x00007f3409e86113  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libcom_err.so.2
>  | ~ 0x00007f3409c558d0  0x00007f3409c73bbf  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libk5crypto.so.3
>  | ~ 0x00007f3409a44bf0  0x00007f3409a4c36b  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkrb5support.so.0
>  | ~ 0x00007f340983e5b0  0x00007f340983f1cc  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libkeyutils.so.1
>  | ~ 0x00007f3409626980  0x00007f34096351e2  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libresolv.so.2
>  | ~ 0x00007f3409402a90  0x00007f3409418896  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libselinux.so.1
>  | ~ 0x00007f340919b5f0  0x00007f34091e15b0  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/lib64/libpcre.so.1
>  | ~ 0x00007f3408c429f0  0x00007f3408c43896  Yes         /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/.rocks/lib/tarantool/tuple/keydef.so
>  | ~ 0x00007f33eedf8d80  0x00007f33eedfc174  Yes (*)     /home/alex/tmp/try-gdb-sh/tarantool-core-N-202103151305-cc18d0726f6a/usr/lib64/tarantool/avro_schema_rt_c.so
>  | ~ (*): Shared library is missing debugging information.
>  | (Looks good too!)
> 
> > Can't open file <...> (deleted) during file-backed mapping note
> > processing.
> 
> I don't know what it is. But I'll share my gdb version just in case
> (maybe you'll be able to reproduce it). I didn't meet this warning
> before.

I tried the script on some machines I have access to: my Gentoo
workstation, our CentOS 7 dev stand, CentOS 8 docker and my personal
Gentoo laptop. I faced the issue only on the latter machine. I guess it
doesn't relate to the script but rather to the environment, since the
problem also occurs when I simply run the following command:
| $ gdb ./tarantool coredump

> 
>  | $ gdb --version | head -n 1
>  | ~ GNU gdb (Gentoo 10.1 vanilla) 10.1

I have the same version on my laptop. My workstation wasn't updated for
some time, so there is only 8.3.1. Let's proceed with this problem in
private, since AFAICS it relates to Gentoo with the quite new gdb.

> 
> However I would note that those warnings misleads me the first time: I
> thought that something is wrong with library paths.
> 
> > Unable to find libthread_db matching inferior's thread library, thread
> > debugging will not be available.
> 
> Sure, it is not listed in the lld output (neither in 'info
> sharedlibrary'), but maybe we should include it into the tarabrt.sh
> generated archive.

Ouch, it was a tough one. However, I've chosen the simplest solution for
this issue. Diff is below:

================================================================================

diff --git a/tools/gdb.sh b/tools/gdb.sh
index 63e80c8f3..669be3c4f 100755
--- a/tools/gdb.sh
+++ b/tools/gdb.sh
@@ -102,4 +102,6 @@ SUBPATH=$(gdb -batch -n ./tarantool -ex 'info line main' | \
 gdb ./tarantool \
     -ex "set sysroot $(realpath .)" \
     -ex "set substitute-path ${SUBPATH} sources" \
+    -ex "add-auto-load-safe-path $(realpath .)" \
+    -ex "set auto-load libthread-db on" \
     -ex 'core coredump'
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 8a07b88a6..aadfde129 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -242,6 +242,32 @@ COREMISMATCH
 	exit 1;
 fi
 
+# XXX: This is poor man's libthread_db location resolution.
+# I'm sure there are many ways to implement this, but most of them
+# requires additional tools, libs, etc. The one implemented below
+# is quite simple, robust and requires nothing more than being
+# already used above.
+GREETING=$(gdb "${BINARY}" -c "${COREFILE}" -ex 'quit' 2>/dev/null)
+LIBTHREAD_DB=
+PATTERN1='[Thread debugging using libthread_db enabled]'
+PATTERN2='Using host libthread_db library ".*"\.'
+
+if echo "${GREETING}" | grep -F "${PATTERN1}" >/dev/null; then
+	LIBTHREAD_DB=$(echo "${GREETING}" | grep "${PATTERN2}" | \
+		awk -F'"' '{ print $2 }')
+	echo "${LIBTHREAD_DB}" >>"${TARLIST}"
+fi
+
+if [ -z "${LIBTHREAD_DB}" ]; then
+	[ -t 1 ] && cat <<NOLIBTHREAD_DB
+Failed to find libthread_db.so. Brace yourselves, thread debugging
+will be a challenging exercise.
+
+This is just a notice, not an error.
+NOLIBTHREAD_DB
+	# XXX: Not an error, proceed packing.
+fi
+
 # Pack everything listed in TARLIST file into a tarball. To unify
 # the archive format BINARY, COREFILE, VERSION and TARLIST are
 # renamed while packing.

================================================================================

It's a really useful notice: now one can even work with TLS variables
with no prisedanie at all (I've checked this by transferring coredump
artefacts from C7 host to my Gentoo workstation and playing with them a
little). Thanks!

> 
> > the debug information found in <...> does not match <...> (CRC mismatch).
> 
> It is just because I have local avro schema debuginfo installed. I think
> this warning can be ignored.

I believe we can do nothing with it in general case, so user (or
developer) should fix this warning by himself.

> 
> To sum up: my experience was successful. You may polish some tiny
> details, but in fact the scripts seem to be ready for use.

Glad to hear this, hope they'll be useful.

> 
> ----
> 
> We also briefly discussed the topic about stripped executables with
> Igor.
> 
> Now we ship stripped executables in RPM packages for CentOS 8 (only for
> CentOS 8), but the separate debug package is stored in our repos.
> 
> First point: we can include /etc/os-release into the generated archive.

I see no criminal in this, hence here are the changes:

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index 6fc76e8f3..e09d051a4 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -7,11 +7,12 @@ ${TOOL} - Tarantool Automatic Bug Reporting Tool
 
 This tool collects all required artefacts (listed below) and packs them into
 a single archive with unified format:
-  - /checklist - the plain text file with the list of tarball contents
-  - /version   - the plain text file containing \`tarantool --version' output
-  - /tarantool - the executable binary file produced the core dump
-  - /coredump  - the core dump file produced by the executable
-  - all shared libraries loaded (even via dlopen(3)) at the crash moment.
+  - /checklist      - the plain text file with the list of tarball contents
+  - /version        - the plain text file with \`tarantool --version' output
+  - /tarantool      - the executable binary file produced the core dump
+  - /coredump       - the core dump file produced by the executable
+  - /etc/os-release - the plain text file with OS identification data
+  - all shared libraries loaded (even via dlopen(3)) at the crash moment
 
 SYNOPSIS
 
@@ -200,6 +201,7 @@ HOSTNAME=$(hostname 2>/dev/null || echo hostname)
 TMPDIR=$(mktemp -d -p "${COREDIR}")
 TARLIST=${TMPDIR}/tarlist
 VERSION=${TMPDIR}/version
+OSRELEASE=/etc/os-release
 ARCHIVENAME=tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}
 ARCHIVE=${COREDIR}/${ARCHIVENAME}.tar.gz
 
@@ -210,6 +212,7 @@ ${BINARY} --version >"${VERSION}"
 {
 	echo "${TARLIST}"
 	echo "${VERSION}"
+	echo "${OSRELEASE}"
 	echo "${BINARY}"
 	echo "${COREFILE}"
 } >>"${TARLIST}"

================================================================================

> 
> Second point: we can give URL from where debuginfo package may be
> downloaded and give advice how to unpack it (in gdb.sh output, when we
> see the stripped executable).

I'm totally for such helpers, but I am tuning this *draft* for such a
long time... Since this is not such popular case for now, I ignore
everything below and will create an issue for this.

> 
> Examples:
> 
>  | $ curl -LfsS https://download.tarantool.org/tarantool/release/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm
>  | $ mkdir tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
>  | $ cd tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm.unpack
>  | $ rpm2cpio ../tarantool-debuginfo-2.8.0.0-1.el8.x86_64.rpm | cpio -idmv
>  | ~ ./usr/lib/debug
>  | ~ ./usr/lib/debug/.build-id
>  | ~ ./usr/lib/debug/.build-id/a3
>  | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60
>  | ~ ./usr/lib/debug/.build-id/a3/ad84db98a9b14e70d2915271904ddfe26b5f60.debug
>  | ~ ./usr/lib/debug/usr
>  | ~ ./usr/lib/debug/usr/bin
>  | ~ ./usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
>  | ~ 64743 blocks
>  | (All are symlinks except tarantool-2.8.0.0-1.el8.x86_64.debug.)
>  | $ file usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug
>  | ~ usr/lib/debug/usr/bin/tarantool-2.8.0.0-1.el8.x86_64.debug: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter \004, for GNU/Linux 3.2.0, BuildID[sha1]=a3ad84db98a9b14e70d2915271904ddfe26b5f60, with debug_info, not stripped, too many notes (256)
> 
>  | $ curl -LfsS https://download.tarantool.org/tarantool/live/2.8/el/8/x86_64/Packages/tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm > tarantool-debuginfo-2.8.0.116-1.el8.x86_64.rpm
> 
> Third: we can give advice how to enable the ELF with debug symbols in
> the interactive gdb session.
> 
> Or we should do it within gdb.sh to call 'info line main'? Expect it in
> some predefined location or accept an option?
> 
> At least it looks worthful to warn about stripped executable explicitly.
> 
> ----
> 
> It seems, I reached maximum value that I can give as reviewer, so I'll
> just say that the patch looks good to me (and I don't mind small changes
> around usability if you want to do them).

Again, thanks a lot! Added your tag:
| Reviewed-by: Alexander Turenko <alexander.turenko@tarantool.org>

> 
> WBR, Alexander Turenko.

-- 
Best regards,
IM

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-03-24 16:02     ` Sergey Bronnikov via Tarantool-patches
  2021-03-24 20:25       ` Alexander Turenko via Tarantool-patches
@ 2021-04-19 21:06       ` Igor Munkin via Tarantool-patches
  1 sibling, 0 replies; 14+ messages in thread
From: Igor Munkin via Tarantool-patches @ 2021-04-19 21:06 UTC (permalink / raw)
  To: Sergey Bronnikov; +Cc: Alexander Turenko, tarantool-patches

Sergey,

Thanks for your review!

On 24.03.21, Sergey Bronnikov wrote:
> Igor,
> 
> thanks for changes! Generally LGTM with two notes inline.

Added your tag:
| Reviewed-by: Sergey Bronnikov <sergeyb@tarantool.org>

> 
> On 11.03.2021 15:13, Igor Munkin wrote:
> > Sergey,
> >
> > Thanks for your review!
> >
> > On 03.03.21, Sergey Bronnikov wrote:
> <skipped>
> >> 2. I propose to show files included to the report and the end of creation.
> > Well, I believe I didn't understand you right before (this list is
> > included to the archive). Anyway, now I added the list also to the
> > message at the end.
> 
> You got it right.
> 
> <skipped>
> 
> 
> 1. in a private conversation we decided to remove mention of public 
> resource in a message "http://transfer.sh".

Removed. Diff is below:

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index e09d051a4..b484fc9fb 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -289,7 +289,7 @@ The archive contents are listed below:
 $(cat "${TARLIST}")
 
 If you want to upload it, choose the available resource
-(e.g. http://transfer.sh) and run the following command:
+and run the following command:
 $ curl -T ${ARCHIVE} <resource-uri>
 FINALIZE
 

================================================================================

> 
> 
>  >  - Manual usage. User can simply pack all necessary artefacts by 
> running the
>  >    following command.
>  >    $ /path/to/${TOOL} -c ./core -d /tmp
>  >
>  >  - Automatic usage. If user faces the failures often, one can set 
> this script
>  >    as a pipe reciever in kernel.core_pattern syntax.
>  >    # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d 
> /var/core -p %p -t %t"
> 
> 2. it would be more convenient to set real path to tarabrt.sh in a 
> usage, see patch below

I'll follow Sasha's suggestion and simply use the given script name.
Diff is below:

================================================================================

diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
index b484fc9fb..66d6a39d6 100755
--- a/tools/tarabrt.sh
+++ b/tools/tarabrt.sh
@@ -3,7 +3,7 @@ set -eu
 
 TOOL=$(basename "$0")
 HELP=$(cat <<HELP
-${TOOL} - Tarantool Automatic Bug Reporting Tool
+$0 - Tarantool Automatic Bug Reporting Tool
 
 This tool collects all required artefacts (listed below) and packs them into
 a single archive with unified format:
@@ -16,7 +16,7 @@ a single archive with unified format:
 
 SYNOPSIS
 
-  ${TOOL} [-h] [-c core] [-d dir] [-e executable] [-p procID] [-t datetime]
+  $0 [-h] [-c core] [-d dir] [-e executable] [-p procID] [-t datetime]
 
 Supported options are:
   -c COREDUMP                   Use file COREDUMP as a core dump to examine.
@@ -46,11 +46,11 @@ USAGE
 
   - Manual usage. User can simply pack all necessary artefacts by running the
     following command.
-    $ /path/to/${TOOL} -c ./core -d /tmp
+    $ $0 -c ./core -d /tmp
 
   - Automatic usage. If user faces the failures often, one can set this script
     as a pipe reciever in kernel.core_pattern syntax.
-    # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d /var/core -p %p -t %t"
+    # sysctl -w kernel.core_pattern="|$0 -d /var/core -p %p -t %t"
 
 HELP
 )

================================================================================

> 
> --- a/tools/tarabrt.sh
> +++ b/tools/tarabrt.sh
> @@ -2,6 +2,8 @@
>   set -eu
> 
>   TOOL=$(basename "$0")
> +#TOOLPATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
> +TOOLPATH=$(realpath -s $0)
>   HELP=$(cat <<HELP
>   ${TOOL} - Tarantool Automatic Bug Reporting Tool
> 
> @@ -45,11 +47,11 @@ USAGE
> 
>     - Manual usage. User can simply pack all necessary artefacts by 
> running the
>       following command.
> -    $ /path/to/${TOOL} -c ./core -d /tmp
> +    $ ${TOOLPATH} -c ./core -d /tmp
> 
>     - Automatic usage. If user faces the failures often, one can set 
> this script
>       as a pipe reciever in kernel.core_pattern syntax.
> -    # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d 
> /var/core -p %p -t %t"
> +    # sysctl -w kernel.core_pattern="|${TOOLPATH} -d /var/core -p %p -t %t"
> 
>   HELP
>   )
> 

-- 
Best regards,
IM

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-02-25 13:23 [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts Igor Munkin via Tarantool-patches
  2021-03-03 12:44 ` Sergey Bronnikov via Tarantool-patches
  2021-03-15 16:30 ` Alexander Turenko via Tarantool-patches
@ 2021-04-19 22:51 ` Igor Munkin via Tarantool-patches
  2 siblings, 0 replies; 14+ messages in thread
From: Igor Munkin via Tarantool-patches @ 2021-04-19 22:51 UTC (permalink / raw)
  To: Alexander Turenko, Sergey Bronnikov; +Cc: tarantool-patches

I've checked the patch into master.

On 25.02.21, Igor Munkin wrote:
> This patch introduces two scripts to ease crash artefacts collecting and
> loading for postmortem analysis:
> 
> * tarabrt.sh - the tool collecting a tarball with the crash artefacts
>   the right way: the coredump with the binary, all loaded shared libs,
>   Tarantool version (this is a separate exercise to get it from the
>   binary built with -O2). Besides, the tarball has a unified layout, so
>   it can be easily processed with the second script:
>   - /coredump - core dump file on the root level
>   - /binary - tarantool executable on the root level
>   - /version - plain text file on the root level with
>     `tarantool --version` output
>   - /checklist - plain text file on the root level with
>     the list of the collected entities
>   - all shared libraries used by the crashed instance - their layout
>     respects the one on the host machine, so them can be easily loaded
>     with the following gdb command: set sysroot $(realpath .)
> 
>   The script can be easily used either manually or via
>   kernel.core_pattern variable.
> 
> * gdb.sh - the auxiliary script originally written by @Totktonada, but
>   needed to be adjusted to the crash artefacts layout every time. Since
>   there is a unified layout, the original script is enhanced a bit to
>   automatically load the coredump via gdb the right way.
> 
> Closes #5569
> 
> Signed-off-by: Igor Munkin <imun@tarantool.org>
> ---
> 
> Issue: https://github.com/tarantool/tarantool/issues/5569
> Branch: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling
> 
>  changelogs/unreleased/tarabrt.md |   3 +
>  tools/gdb.sh                     |  59 ++++++++
>  tools/tarabrt.sh                 | 234 +++++++++++++++++++++++++++++++
>  3 files changed, 296 insertions(+)
>  create mode 100644 changelogs/unreleased/tarabrt.md
>  create mode 100755 tools/gdb.sh
>  create mode 100755 tools/tarabrt.sh
> 

<snipped>

> -- 
> 2.25.0
> 

-- 
Best regards,
IM

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-04-19 20:42   ` Igor Munkin via Tarantool-patches
@ 2021-04-19 23:57     ` Alexander Turenko via Tarantool-patches
  2021-04-20 12:14       ` Igor Munkin via Tarantool-patches
  0 siblings, 1 reply; 14+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-04-19 23:57 UTC (permalink / raw)
  To: Igor Munkin; +Cc: tarantool-patches

> > Okay, now I got the tarabrt.sh script somehow:
> > 
> >  | # curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/tarabrt.sh >tarabrt.sh
> >  | # chmod a+x tarabrt.sh
> >  | (Don't read the content, just run.)
> >  | # ./tarabrt.sh --help
> >  | ~ tarabrt.sh: unrecognized option '--help'
> 
> Unfortunately, I have nothing to do here: this is returned by <getopt>.

Can we add --help option to <getopt>'s optstring?

> >  | # ./tarabrt.sh -h  
> >  | ~ <..skipped..>
> >  | ~   - Manual usage. User can simply pack all necessary artefacts by running the
> >  | ~     following command.
> >  | ~     $ /path/to/tarabrt.sh -c ./core -d /tmp
> >  | ~ <..skipped..>
> >  | # ./tarabrt.sh -c ./core -d /tmp
> >  | ~ ./tarabrt.sh: line 115: file: command not found
> >  | ~ gdb is not installed, but it is obligatory for collecting the
> >  | ~ loaded shared libraries from the core dump.
> >  | ~ 
> >  | ~ You can proceed collecting the artefacts manually later by running
> >  | ~ the following command:
> >  | ~ $ tarabrt.sh -e /usr/bin/tarantool -c ./core
> >  | (Okay, I'll install file and gdb now.)
> 
> I'll added the check that file is installed. Diff is below:
> 
> <..skipped the diff..>
> 
> >  | # yum install -y file gdb
> >  | # file core
> >  | ~ core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'tarantool', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/tarantool', platform: 'x86_64'
> >  | (Repeat the attempt to create the archive.)
> >  | # ./tarabrt.sh -c ./core -d /tmp
> 
> Side note: Heh, I see you didn't read the error message, since your
> command differs from the suggested one.

Not exactly. I found that the command fails due to lack of <file> and
<gdb> commands, so decided to repeat it as is after installing them. If
this would fail too I would proceed with the suggested command with -e.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts
  2021-04-19 23:57     ` Alexander Turenko via Tarantool-patches
@ 2021-04-20 12:14       ` Igor Munkin via Tarantool-patches
  0 siblings, 0 replies; 14+ messages in thread
From: Igor Munkin via Tarantool-patches @ 2021-04-20 12:14 UTC (permalink / raw)
  To: Alexander Turenko; +Cc: tarantool-patches

Sasha,

I've created the ticket[1] with the follow up activities for these
scripts. Feel free to add your inputs there, so they won't be lost.

On 20.04.21, Alexander Turenko wrote:
> > > Okay, now I got the tarabrt.sh script somehow:
> > > 
> > >  | # curl -LfsS https://raw.githubusercontent.com/tarantool/tarantool/929982bf49e4d177ea33876bee91845c9528d038/tools/tarabrt.sh >tarabrt.sh
> > >  | # chmod a+x tarabrt.sh
> > >  | (Don't read the content, just run.)
> > >  | # ./tarabrt.sh --help
> > >  | ~ tarabrt.sh: unrecognized option '--help'
> > 
> > Unfortunately, I have nothing to do here: this is returned by <getopt>.
> 
> Can we add --help option to <getopt>'s optstring?

Oops, I've re-checked <getopt> manual page and it occurs we can... There
is -l/--longoptions flag, to provide long options to be handled.
Unfortunately, I've already pushed the scripts to the trunk, but I've
added this to the ticket with enhancements.

> 
> > >  | # ./tarabrt.sh -h  
> > >  | ~ <..skipped..>
> > >  | ~   - Manual usage. User can simply pack all necessary artefacts by running the
> > >  | ~     following command.
> > >  | ~     $ /path/to/tarabrt.sh -c ./core -d /tmp
> > >  | ~ <..skipped..>
> > >  | # ./tarabrt.sh -c ./core -d /tmp
> > >  | ~ ./tarabrt.sh: line 115: file: command not found
> > >  | ~ gdb is not installed, but it is obligatory for collecting the
> > >  | ~ loaded shared libraries from the core dump.
> > >  | ~ 
> > >  | ~ You can proceed collecting the artefacts manually later by running
> > >  | ~ the following command:
> > >  | ~ $ tarabrt.sh -e /usr/bin/tarantool -c ./core
> > >  | (Okay, I'll install file and gdb now.)
> > 
> > I'll added the check that file is installed. Diff is below:
> > 
> > <..skipped the diff..>
> > 
> > >  | # yum install -y file gdb
> > >  | # file core
> > >  | ~ core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'tarantool', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/tarantool', platform: 'x86_64'
> > >  | (Repeat the attempt to create the archive.)
> > >  | # ./tarabrt.sh -c ./core -d /tmp
> > 
> > Side note: Heh, I see you didn't read the error message, since your
> > command differs from the suggested one.
> 
> Not exactly. I found that the command fails due to lack of <file> and
> <gdb> commands, so decided to repeat it as is after installing them. If
> this would fail too I would proceed with the suggested command with -e.

OK then :)

[1]: https://github.com/tarantool/tarantool/issues/6026

-- 
Best regards,
IM

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2021-04-20 12:15 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-25 13:23 [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts Igor Munkin via Tarantool-patches
2021-03-03 12:44 ` Sergey Bronnikov via Tarantool-patches
2021-03-08  4:56   ` Alexander Turenko via Tarantool-patches
2021-03-08 11:54     ` Sergey Bronnikov via Tarantool-patches
2021-03-11 12:13   ` Igor Munkin via Tarantool-patches
2021-03-24 16:02     ` Sergey Bronnikov via Tarantool-patches
2021-03-24 20:25       ` Alexander Turenko via Tarantool-patches
2021-03-25  9:45         ` Sergey Bronnikov via Tarantool-patches
2021-04-19 21:06       ` Igor Munkin via Tarantool-patches
2021-03-15 16:30 ` Alexander Turenko via Tarantool-patches
2021-04-19 20:42   ` Igor Munkin via Tarantool-patches
2021-04-19 23:57     ` Alexander Turenko via Tarantool-patches
2021-04-20 12:14       ` Igor Munkin via Tarantool-patches
2021-04-19 22:51 ` Igor Munkin via Tarantool-patches

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox