Tarantool development patches archive
 help / color / mirror / Atom feed
From: Igor Munkin via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: Sergey Ostanevich <sergos@tarantool.org>,
	Sergey Kaplun <skaplun@tarantool.org>,
	Alexander Turenko <alexander.turenko@tarantool.org>,
	Nikita Pettik <korablev@tarantool.org>,
	Aleksandr Lyapunov <alyapunov@tarantool.org>,
	Serge Petrenko <sergepetrenko@tarantool.org>,
	Sergey Bronnikov <sergeyb@tarantool.org>
Cc: tarantool-patches@dev.tarantool.org
Subject: [Tarantool-patches] [PATCH] WIP: introduce tooling for coredumps
Date: Thu, 11 Feb 2021 21:48:28 +0300
Message-ID: <ec2505c1db7c56aa5fd17990b8a1d24d6de17c9b.1613065868.git.imun@tarantool.org> (raw)

Signed-off-by: Igor Munkin <imun@tarantool.org>
---

WARNING: This patch doesn't require full and precise review now. I
chose those folks from core team, who either works with customer reports
a lot or liked the ticket[1]. I hope I missed nobody, but if I did the
patch is available in our ml.

I've polished a bit two scripts I often use:
* tarabrt.sh (so called tarcore.sh) -- this is the script collecting a
  tarball with the coredump the right way: with the binary, all loaded
  shared libs, Tarantool version (this is a separate exercise to get it
  from the binary built with -O2). Besides, I made the tarball struct
  unified so it can be easily processed with the second script.
* gdb.sh -- the auxiliary script originally written by Sasha Tu, that
  needed to be adjusted to the customer artefacts every time. Since we
  will have a unified layout I enhanced it a bit to automatically load
  the coredump via gdb the right way.

Check this out[2], try both scripts and reach me with your feedback!

[1]: https://github.com/tarantool/tarantool/issues/5569
[2]: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling

 tools/gdb.sh     |  33 +++++++++++++++
 tools/tarabrt.sh | 102 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 135 insertions(+)
 create mode 100755 tools/gdb.sh
 create mode 100755 tools/tarabrt.sh

diff --git a/tools/gdb.sh b/tools/gdb.sh
new file mode 100755
index 000000000..4daac7106
--- /dev/null
+++ b/tools/gdb.sh
@@ -0,0 +1,33 @@
+#!/bin/sh
+set -eu
+
+# Check that gdb is installed.
+type gdb &> /dev/null
+
+VERSION=${PWD}/version
+
+[ -f "$VERSIONFILE" ] || (echo 'Invalid coredump location'; exit 1)
+
+REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION")
+cat <<EOF
+================================================================================
+
+Do not forget to properly setup the environment:
+* git clone https://github.com/tarantool/tarantool.git sources
+* cd !$
+* git checkout $REVISION
+* git submodule update --recursive --init
+
+================================================================================
+EOF
+
+# Define the build path to be substituted with the source path.
+# XXX: Check the absolute path on the function <main> definition
+# considering it is located in src/main.cc within Tarantool repo.
+SUBPATH=$(gdb -batch ./tarantool -ex 'info line main' | \
+	grep -oP 'Line \d+ of \"\K.+(?=\/src\/main\.cc\")')
+
+gdb ./tarantool \
+    -ex "set sysroot $(realpath .)" \
+    -ex "set substitute-path $SUBPATH sources" \
+    -ex 'core coredump'
diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh
new file mode 100755
index 000000000..92252899b
--- /dev/null
+++ b/tools/tarabrt.sh
@@ -0,0 +1,102 @@
+#!/bin/sh
+set -eu
+
+USAGE=$(cat <<EOF
+Usage:
+
+$ ./tools/tarabrt.sh -e ./src/tarantool -c core
+# sysctl -w kernel.core_pattern="|/usr/share/tarantool/tools/tarabrt.sh -d /var/core -p %p -t %t"
+
+EOF
+)
+
+# Parse CLI options.
+OPTIONS=$(getopt -o c:d:e:hp:t: -n 'tarabrt.sh' -- "$@")
+eval set -- "$OPTIONS"
+
+while true; do
+	case "$1" in
+		-c) COREFILE=$2; shift 2;;
+		-d) COREDIR=$2;  shift 2;;
+		-e) BINARY=$2;   shift 2;;
+		-p) PID=$2;      shift 2;;
+		-t) TIME=$2;     shift 2;;
+		--) shift; break;;
+		-h) printf "%s\n", "$USAGE";
+			exit 0;;
+		*)  printf "Invalid option: $1\n%s\n", "$USAGE";
+			exit 1;;
+	esac
+done
+
+# Use default values for the remaining parameters.
+COREFILE=${COREFILE:-${COREDIR}/tarantool-core.${PID}.${TIME}}
+COREDIR=${COREDIR:-${PWD}}
+BINARY=${BINARY:-$(/usr/bin/readlink /proc/"${PID}"/exe)}
+TIME=${TIME:-$(date +%s)}
+PID=${PID:-N}
+
+if [ ! -e "${COREFILE}" ]; then
+	cat > "${COREFILE}"
+fi
+
+GDBERROR=$(cat <<EOF
+gdb is not installed, but it is obligatory for collecting the
+loaded shared libraries from the coredump.
+
+You can proceed collecting the artefacts manually later by running
+the following command:
+$ tarabrt.sh -e $BINARY -c $COREFILE
+EOF
+)
+
+# Check that gdb is installed.
+type gdb &> /dev/null || ([ -t 1 ] && echo "$GDBERROR")
+
+# Resolve hostname if possible.
+HOSTNAME=$(timeout 2>/dev/null -s 9 1 hostname -f \
+	|| hostname 2>/dev/null                   \
+	|| echo hostname)
+
+# Proceed with collecting and packing artefacts.
+TMPDIR=$(mktemp -d -p "${COREDIR}")
+TARLIST=${TMPDIR}/tarlist
+VERSION=${TMPDIR}/version
+ARCHIVENAME=${COREDIR}/tarantool-core-${PID}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz
+
+# Dump the version to checkout the right commit later.
+$BINARY --version > "$VERSION"
+
+# Collect the most important artefacts.
+{
+	echo "$BINARY"
+	echo "$COREFILE"
+	echo "$VERSION"
+} >> "${TARLIST}"
+
+SEPARATOR1="Shared Object Library"
+SEPARATOR2="Shared library is missing debugging information"
+# XXX: This is kinda "postmortem ldd": the command below dumps the
+# full list of the shared libraries the binary is linked against
+# or those loaded via dlopen at the platform runtime.
+# This is black woodoo magic. Do not touch. You are warned.
+gdb -batch "${BINARY}" -c "${COREFILE}" -ex "info shared" -ex "quit" | \
+	sed -n "/${SEPARATOR1}/,/${SEPARATOR2}/p;/${SEPARATOR2}/q"   | \
+	awk '{ print $NF }' | grep "^/" >> "${TARLIST}"
+
+# Pack everything listed in TARLIST file into a tarball. To unify
+# the archive format BINARY, COREFILE, VERSION and TARLIST are
+# renamed while packing.
+tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \
+	--transform="s|$BINARY|tarantool|"    \
+	--transform="s|$COREFILE|coredump|"   \
+	--transform="s|$TARLIST|checklist|"   \
+	--transform="s|$VERSION|version|"     \
+	--add-file="${TARLIST}"
+
+[ -t 1 ] && echo "Archive: ${ARCHIVENAME}"
+
+# Cleanup temporary files.
+[ -f "${TARLIST}" ] && rm -f "${TARLIST}"
+[ -f "${VERSION}" ] && rm -f "${VERSION}"
+[ -d "${TMPDIR}" ] && rmdir "${TMPDIR}"
-- 
2.25.0


             reply	other threads:[~2021-02-11 18:48 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-11 18:48 Igor Munkin via Tarantool-patches [this message]
2021-02-20 10:46 ` Alexander Turenko via Tarantool-patches

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=ec2505c1db7c56aa5fd17990b8a1d24d6de17c9b.1613065868.git.imun@tarantool.org \
    --to=tarantool-patches@dev.tarantool.org \
    --cc=alexander.turenko@tarantool.org \
    --cc=alyapunov@tarantool.org \
    --cc=imun@tarantool.org \
    --cc=korablev@tarantool.org \
    --cc=sergepetrenko@tarantool.org \
    --cc=sergeyb@tarantool.org \
    --cc=sergos@tarantool.org \
    --cc=skaplun@tarantool.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Tarantool development patches archive

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://lists.tarantool.org/tarantool-patches/0 tarantool-patches/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 tarantool-patches tarantool-patches/ https://lists.tarantool.org/tarantool-patches \
		tarantool-patches@dev.tarantool.org.
	public-inbox-index tarantool-patches

Example config snippet for mirrors.


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git