From: Igor Munkin via Tarantool-patches <tarantool-patches@dev.tarantool.org> To: Sergey Ostanevich <sergos@tarantool.org>, Sergey Kaplun <skaplun@tarantool.org>, Alexander Turenko <alexander.turenko@tarantool.org>, Nikita Pettik <korablev@tarantool.org>, Aleksandr Lyapunov <alyapunov@tarantool.org>, Serge Petrenko <sergepetrenko@tarantool.org>, Sergey Bronnikov <sergeyb@tarantool.org> Cc: tarantool-patches@dev.tarantool.org Subject: [Tarantool-patches] [PATCH] WIP: introduce tooling for coredumps Date: Thu, 11 Feb 2021 21:48:28 +0300 [thread overview] Message-ID: <ec2505c1db7c56aa5fd17990b8a1d24d6de17c9b.1613065868.git.imun@tarantool.org> (raw) Signed-off-by: Igor Munkin <imun@tarantool.org> --- WARNING: This patch doesn't require full and precise review now. I chose those folks from core team, who either works with customer reports a lot or liked the ticket[1]. I hope I missed nobody, but if I did the patch is available in our ml. I've polished a bit two scripts I often use: * tarabrt.sh (so called tarcore.sh) -- this is the script collecting a tarball with the coredump the right way: with the binary, all loaded shared libs, Tarantool version (this is a separate exercise to get it from the binary built with -O2). Besides, I made the tarball struct unified so it can be easily processed with the second script. * gdb.sh -- the auxiliary script originally written by Sasha Tu, that needed to be adjusted to the customer artefacts every time. Since we will have a unified layout I enhanced it a bit to automatically load the coredump via gdb the right way. Check this out[2], try both scripts and reach me with your feedback! [1]: https://github.com/tarantool/tarantool/issues/5569 [2]: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling tools/gdb.sh | 33 +++++++++++++++ tools/tarabrt.sh | 102 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+) create mode 100755 tools/gdb.sh create mode 100755 tools/tarabrt.sh diff --git a/tools/gdb.sh b/tools/gdb.sh new file mode 100755 index 000000000..4daac7106 --- /dev/null +++ b/tools/gdb.sh @@ -0,0 +1,33 @@ +#!/bin/sh +set -eu + +# Check that gdb is installed. +type gdb &> /dev/null + +VERSION=${PWD}/version + +[ -f "$VERSIONFILE" ] || (echo 'Invalid coredump location'; exit 1) + +REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION") +cat <<EOF +================================================================================ + +Do not forget to properly setup the environment: +* git clone https://github.com/tarantool/tarantool.git sources +* cd !$ +* git checkout $REVISION +* git submodule update --recursive --init + +================================================================================ +EOF + +# Define the build path to be substituted with the source path. +# XXX: Check the absolute path on the function <main> definition +# considering it is located in src/main.cc within Tarantool repo. +SUBPATH=$(gdb -batch ./tarantool -ex 'info line main' | \ + grep -oP 'Line \d+ of \"\K.+(?=\/src\/main\.cc\")') + +gdb ./tarantool \ + -ex "set sysroot $(realpath .)" \ + -ex "set substitute-path $SUBPATH sources" \ + -ex 'core coredump' diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh new file mode 100755 index 000000000..92252899b --- /dev/null +++ b/tools/tarabrt.sh @@ -0,0 +1,102 @@ +#!/bin/sh +set -eu + +USAGE=$(cat <<EOF +Usage: + +$ ./tools/tarabrt.sh -e ./src/tarantool -c core +# sysctl -w kernel.core_pattern="|/usr/share/tarantool/tools/tarabrt.sh -d /var/core -p %p -t %t" + +EOF +) + +# Parse CLI options. +OPTIONS=$(getopt -o c:d:e:hp:t: -n 'tarabrt.sh' -- "$@") +eval set -- "$OPTIONS" + +while true; do + case "$1" in + -c) COREFILE=$2; shift 2;; + -d) COREDIR=$2; shift 2;; + -e) BINARY=$2; shift 2;; + -p) PID=$2; shift 2;; + -t) TIME=$2; shift 2;; + --) shift; break;; + -h) printf "%s\n", "$USAGE"; + exit 0;; + *) printf "Invalid option: $1\n%s\n", "$USAGE"; + exit 1;; + esac +done + +# Use default values for the remaining parameters. +COREFILE=${COREFILE:-${COREDIR}/tarantool-core.${PID}.${TIME}} +COREDIR=${COREDIR:-${PWD}} +BINARY=${BINARY:-$(/usr/bin/readlink /proc/"${PID}"/exe)} +TIME=${TIME:-$(date +%s)} +PID=${PID:-N} + +if [ ! -e "${COREFILE}" ]; then + cat > "${COREFILE}" +fi + +GDBERROR=$(cat <<EOF +gdb is not installed, but it is obligatory for collecting the +loaded shared libraries from the coredump. + +You can proceed collecting the artefacts manually later by running +the following command: +$ tarabrt.sh -e $BINARY -c $COREFILE +EOF +) + +# Check that gdb is installed. +type gdb &> /dev/null || ([ -t 1 ] && echo "$GDBERROR") + +# Resolve hostname if possible. +HOSTNAME=$(timeout 2>/dev/null -s 9 1 hostname -f \ + || hostname 2>/dev/null \ + || echo hostname) + +# Proceed with collecting and packing artefacts. +TMPDIR=$(mktemp -d -p "${COREDIR}") +TARLIST=${TMPDIR}/tarlist +VERSION=${TMPDIR}/version +ARCHIVENAME=${COREDIR}/tarantool-core-${PID}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz + +# Dump the version to checkout the right commit later. +$BINARY --version > "$VERSION" + +# Collect the most important artefacts. +{ + echo "$BINARY" + echo "$COREFILE" + echo "$VERSION" +} >> "${TARLIST}" + +SEPARATOR1="Shared Object Library" +SEPARATOR2="Shared library is missing debugging information" +# XXX: This is kinda "postmortem ldd": the command below dumps the +# full list of the shared libraries the binary is linked against +# or those loaded via dlopen at the platform runtime. +# This is black woodoo magic. Do not touch. You are warned. +gdb -batch "${BINARY}" -c "${COREFILE}" -ex "info shared" -ex "quit" | \ + sed -n "/${SEPARATOR1}/,/${SEPARATOR2}/p;/${SEPARATOR2}/q" | \ + awk '{ print $NF }' | grep "^/" >> "${TARLIST}" + +# Pack everything listed in TARLIST file into a tarball. To unify +# the archive format BINARY, COREFILE, VERSION and TARLIST are +# renamed while packing. +tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \ + --transform="s|$BINARY|tarantool|" \ + --transform="s|$COREFILE|coredump|" \ + --transform="s|$TARLIST|checklist|" \ + --transform="s|$VERSION|version|" \ + --add-file="${TARLIST}" + +[ -t 1 ] && echo "Archive: ${ARCHIVENAME}" + +# Cleanup temporary files. +[ -f "${TARLIST}" ] && rm -f "${TARLIST}" +[ -f "${VERSION}" ] && rm -f "${VERSION}" +[ -d "${TMPDIR}" ] && rmdir "${TMPDIR}" -- 2.25.0
next reply other threads:[~2021-02-11 18:48 UTC|newest] Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-02-11 18:48 Igor Munkin via Tarantool-patches [this message] 2021-02-20 10:46 ` Alexander Turenko via Tarantool-patches
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=ec2505c1db7c56aa5fd17990b8a1d24d6de17c9b.1613065868.git.imun@tarantool.org \ --to=tarantool-patches@dev.tarantool.org \ --cc=alexander.turenko@tarantool.org \ --cc=alyapunov@tarantool.org \ --cc=imun@tarantool.org \ --cc=korablev@tarantool.org \ --cc=sergepetrenko@tarantool.org \ --cc=sergeyb@tarantool.org \ --cc=sergos@tarantool.org \ --cc=skaplun@tarantool.org \ --subject='Re: [Tarantool-patches] [PATCH] WIP: introduce tooling for coredumps' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox