[Tarantool-patches] [PATCH] WIP: introduce tooling for coredumps

[Tarantool-patches] [PATCH] WIP: introduce tooling for coredumps

[Igor Munkin via Tarantool-patches]

Signed-off-by: Igor Munkin <imun@tarantool.org>
---

WARNING: This patch doesn't require full and precise review now. I
chose those folks from core team, who either works with customer reports
a lot or liked the ticket[1]. I hope I missed nobody, but if I did the
patch is available in our ml.

I've polished a bit two scripts I often use:
* tarabrt.sh (so called tarcore.sh) -- this is the script collecting a
  tarball with the coredump the right way: with the binary, all loaded
  shared libs, Tarantool version (this is a separate exercise to get it
  from the binary built with -O2). Besides, I made the tarball struct
  unified so it can be easily processed with the second script.
* gdb.sh -- the auxiliary script originally written by Sasha Tu, that
  needed to be adjusted to the customer artefacts every time. Since we
  will have a unified layout I enhanced it a bit to automatically load
  the coredump via gdb the right way.

Check this out[2], try both scripts and reach me with your feedback!

[1]: https://github.com/tarantool/tarantool/issues/5569
[2]: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling

 tools/gdb.sh     |  33 +++++++++++++++
 tools/tarabrt.sh | 102 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 135 insertions(+)
 create mode 100755 tools/gdb.sh
 create mode 100755 tools/tarabrt.sh

diff --git a/tools/gdb.sh b/tools/gdb.sh
new file mode 100755
index 000000000..4daac7106
--- /dev/null
+++ b/tools/gdb.sh
@@ -0,0 +1,33 @@
+#!/bin/sh
+set -eu
+
+# Check that gdb is installed.
+type gdb &> /dev/null
+
+VERSION=${PWD}/version
+
+[ -f "$VERSIONFILE" ] || (echo 'Invalid coredump location'; exit 1)
+
+REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
+cat <<EOF
+================================================================================
+
+Do not forget to properly setup the environment:
+* git clone https://github.com/tarantool/tarantool.git sources
+* cd !$
+* git checkout $REVISION
+* git submodule update --recursive --init
+
+================================================================================
+EOF
+
+# Define the build path to be substituted with the source path.
+# XXX: Check the absolute path on the function <main> definition
+# considering it is located in src/main.cc within Tarantool repo.
+SUBPATH=$(gdb -batch ./tarantool -ex 'info line main' | \
+	grep -oP 'Line \d+ of \"\K.+(?=\/src\/main\.cc\")')
+
+gdb ./tarantool \
+    -ex "set sysroot $(realpath .)" \
+    -ex "set substitute-path $SUBPATH sources" \
+    -ex 'core coredump'
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
new file mode 100755
index 000000000..92252899b
--- /dev/null
+++ b/tools/tarabrt.sh
@@ -0,0 +1,102 @@
+#!/bin/sh
+set -eu
+
+USAGE=$(cat <<EOF
+Usage:
+
+$ ./tools/tarabrt.sh -e ./src/tarantool -c core
+# sysctl -w kernel.core_pattern="|/usr/share/tarantool/tools/tarabrt.sh -d /var/core -p %p -t %t"
+
+EOF
+)
+
+# Parse CLI options.
+OPTIONS=$(getopt -o c:d:e:hp:t: -n 'tarabrt.sh' -- "$@")
+eval set -- "$OPTIONS"
+
+while true; do
+	case "$1" in
+		-c) COREFILE=$2; shift 2;;
+		-d) COREDIR=$2;  shift 2;;
+		-e) BINARY=$2;   shift 2;;
+		-p) PID=$2;      shift 2;;
+		-t) TIME=$2;     shift 2;;
+		--) shift; break;;
+		-h) printf "%s\n", "$USAGE";
+			exit 0;;
+		*)  printf "Invalid option: $1\n%s\n", "$USAGE";
+			exit 1;;
+	esac
+done
+
+# Use default values for the remaining parameters.
+COREFILE=${COREFILE:-${COREDIR}/tarantool-core.${PID}.${TIME}}
+COREDIR=${COREDIR:-${PWD}}
+BINARY=${BINARY:-$(/usr/bin/readlink /proc/"${PID}"/exe)}
+TIME=${TIME:-$(date +%s)}
+PID=${PID:-N}
+
+if [ ! -e "${COREFILE}" ]; then
+	cat > "${COREFILE}"
+fi
+
+GDBERROR=$(cat <<EOF
+gdb is not installed, but it is obligatory for collecting the
+loaded shared libraries from the coredump.
+
+You can proceed collecting the artefacts manually later by running
+the following command:
+$ tarabrt.sh -e $BINARY -c $COREFILE
+EOF
+)
+
+# Check that gdb is installed.
+type gdb &> /dev/null || ([ -t 1 ] && echo "$GDBERROR")
+
+# Resolve hostname if possible.
+HOSTNAME=$(timeout 2>/dev/null -s 9 1 hostname -f \
+	|| hostname 2>/dev/null                   \
+	|| echo hostname)
+
+# Proceed with collecting and packing artefacts.
+TMPDIR=$(mktemp -d -p "${COREDIR}")
+TARLIST=${TMPDIR}/tarlist
+VERSION=${TMPDIR}/version
+ARCHIVENAME=${COREDIR}/tarantool-core-${PID}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz
+
+# Dump the version to checkout the right commit later.
+$BINARY --version > "$VERSION"
+
+# Collect the most important artefacts.
+{
+	echo "$BINARY"
+	echo "$COREFILE"
+	echo "$VERSION"
+} >> "${TARLIST}"
+
+SEPARATOR1="Shared Object Library"
+SEPARATOR2="Shared library is missing debugging information"
+# XXX: This is kinda "postmortem ldd": the command below dumps the
+# full list of the shared libraries the binary is linked against
+# or those loaded via dlopen at the platform runtime.
+# This is black woodoo magic. Do not touch. You are warned.
+gdb -batch "${BINARY}" -c "${COREFILE}" -ex "info shared" -ex "quit" | \
+	sed -n "/${SEPARATOR1}/,/${SEPARATOR2}/p;/${SEPARATOR2}/q"   | \
+	awk '{ print $NF }' | grep "^/" >> "${TARLIST}"
+
+# Pack everything listed in TARLIST file into a tarball. To unify
+# the archive format BINARY, COREFILE, VERSION and TARLIST are
+# renamed while packing.
+tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
+	--transform="s|$BINARY|tarantool|"    \
+	--transform="s|$COREFILE|coredump|"   \
+	--transform="s|$TARLIST|checklist|"   \
+	--transform="s|$VERSION|version|"     \
+	--add-file="${TARLIST}"
+
+[ -t 1 ] && echo "Archive: ${ARCHIVENAME}"
+
+# Cleanup temporary files.
+[ -f "${TARLIST}" ] && rm -f "${TARLIST}"
+[ -f "${VERSION}" ] && rm -f "${VERSION}"
+[ -d "${TMPDIR}" ] && rmdir "${TMPDIR}"
-- 
2.25.0

Re: [Tarantool-patches] [PATCH] WIP: introduce tooling for coredumps

[Alexander Turenko via Tarantool-patches]

On Thu, Feb 11, 2021 at 09:48:28PM +0300, Igor Munkin wrote:
> Signed-off-by: Igor Munkin <imun@tarantool.org>
> ---
> 
> WARNING: This patch doesn't require full and precise review now. I
> chose those folks from core team, who either works with customer reports
> a lot or liked the ticket[1]. I hope I missed nobody, but if I did the
> patch is available in our ml.
> 
> I've polished a bit two scripts I often use:
> * tarabrt.sh (so called tarcore.sh) -- this is the script collecting a
>   tarball with the coredump the right way: with the binary, all loaded
>   shared libs, Tarantool version (this is a separate exercise to get it
>   from the binary built with -O2). Besides, I made the tarball struct
>   unified so it can be easily processed with the second script.
> * gdb.sh -- the auxiliary script originally written by Sasha Tu, that
>   needed to be adjusted to the customer artefacts every time. Since we
>   will have a unified layout I enhanced it a bit to automatically load
>   the coredump via gdb the right way.
> 
> Check this out[2], try both scripts and reach me with your feedback!
> 
> [1]: https://github.com/tarantool/tarantool/issues/5569
> [2]: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling
> 
>  tools/gdb.sh     |  33 +++++++++++++++
>  tools/tarabrt.sh | 102 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 135 insertions(+)
>  create mode 100755 tools/gdb.sh
>  create mode 100755 tools/tarabrt.sh

Thanks! I looked over the scripts and nothing confuses me. I didn't
test, though.