From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 7D5956B461; Wed, 3 Mar 2021 15:44:28 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 7D5956B461 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614775468; bh=bWg+p19R0fnNAvnDpS2f5jVBjcj1uY+wzhgcuHvNq1k=; h=To:Cc:References:Date:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=Lc7dJmkua9Kaf1rketwfAEavFHdZSDO2ijkb5B41qOkH45ZspFCHrSYnMPyDSgt3/ FkUaN8S77d/jb1roODEOCpX9XwiqljFGeW8MCbwZcTQ0won1Ih0Ia0r/6X7INI/GB1 yazBQppwDlgf07N0fk2wjDNg/fpcDQWAc6xjtES4= Received: from smtpng1.m.smailru.net (smtpng1.m.smailru.net [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id A5F556B461 for ; Wed, 3 Mar 2021 15:44:26 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A5F556B461 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1lHQrZ-0005gu-1C; Wed, 03 Mar 2021 15:44:25 +0300 To: Igor Munkin , Alexander Turenko Cc: tarantool-patches@dev.tarantool.org References: <7f444b92bacefea0e3a0b2d2846737d9791626ea.1614256802.git.imun@tarantool.org> Message-ID: <0a7f111d-7119-bd2d-020d-1938edeb973e@tarantool.org> Date: Wed, 3 Mar 2021 15:44:24 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <7f444b92bacefea0e3a0b2d2846737d9791626ea.1614256802.git.imun@tarantool.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92A98208ECBDD29F5B42DD879439ED63E04973C1061E33475182A05F538085040620871D9D33089C81C195325C56DBFAD5D9E261045396315E327771504D01BA5 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE742D9BD90C58D50E0EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637B05885A782BE20C5EA1F7E6F0F101C67CDEEF6D7F21E0D1D174C73DBBBFC76648A43C42257572E5A13AE23CDB387FB71AC09EB59D2B680C9389733CBF5DBD5E913377AFFFEAFD269A417C69337E82CC2CC7F00164DA146DAFE8445B8C89999729449624AB7ADAF37F6B57BC7E64490611E7FA7ABCAF51C92A417C69337E82CC2CC7F00164DA146DA6F5DAA56C3B73B23C77107234E2CFBA567F23339F89546C55F5C1EE8F4F765FC96390DA516BDDC3475ECD9A6C639B01BBD4B6F7A4D31EC0BC0CAF46E325F83A522CA9DD8327EE4931B544F03EFBC4D57444A83B712AC0148C4224003CC836476C0CAF46E325F83A50BF2EBBBDD9D6B0F05F538519369F3743B503F486389A921A5CC5B56E945C8DA X-C1DE0DAB: 0D63561A33F958A5E6E026DB80B15776AE3F8E233A0A65D330FFF325F49BEF5ED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75448CF9D3A7B2C848410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34FD9114F69CA58D1E59B25DEC9C3F744F7F57A9E9652B53D7D734668DA59D02FEF2661046093BA4971D7E09C32AA3244CCE03619EB23CF6BEBE66A7A47FAE20CE95A9E0DC41E9A4CFFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojnllmSZCgRqusWDOQvF9NVw== X-Mailru-Sender: 689FA8AB762F73936BC43F508A06382294715BA23C5AEBD9496F6523AC89CB16DD788429FD8613638ED9BB8B05EE7B3AFB559BB5D741EB96D19CD4E7312BAA970A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hello! Igor, thanks for the patch! I spend a bit on testing script. To create a coredump I run tarantool binary and generated coredump with GDB: "gdb -batch -ex "generate-core-file" -p PID". To create report I run: [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ ./tools/tarabrt.sh -c core.2715242 -e build/src/tarantool warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments warning: Loadable section ".note.gnu.property" outside of ELF segments The resulting is located here: /home/s.bronnikov/work/tarantool/tarantool-core-N-202103031215-tarantool-core-dev-mcs1.tar.gz If you want to upload it, choose the available resourse (e.g. http://transfer.sh) and run the following command: $ curl -T /home/s.bronnikov/work/tarantool/tarantool-core-N-202103031222-tarantool-core-dev-mcs1.tar.gz 1. Can we suppress warnings produced by gdb? I believe it is not useful information for users. 2. I propose to show files included to the report and the end of creation. When I run tarabrt.sh without specifying -e it says that /usr/bin/tarantool is not ELF: (venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ ./tools/tarabrt.sh -c core.2715242 Not an ELF file: /usr/bin/tarantool The given BINARY file is not an ELF (see elf(5) for more info). If you see this message, check the BINARY file the following way: $ file /usr/bin/tarantool (venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ file /usr/bin/tarantool /usr/bin/tarantool: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=58d86a2b8bf88de3496ea4ead0a8d3c7a73e9c36, stripped, too many notes (256) (venv) [s.bronnikov@tarantool-core-dev-mcs1 tarantool]$ Also see my comments inline. On 25.02.2021 16:23, Igor Munkin wrote: > This patch introduces two scripts to ease crash artefacts collecting and > loading for postmortem analysis: > > * tarabrt.sh - the tool collecting a tarball with the crash artefacts > the right way: the coredump with the binary, all loaded shared libs, > Tarantool version (this is a separate exercise to get it from the > binary built with -O2). Besides, the tarball has a unified layout, so > it can be easily processed with the second script: > - /coredump - core dump file on the root level > - /binary - tarantool executable on the root level > - /version - plain text file on the root level with > `tarantool --version` output > - /checklist - plain text file on the root level with > the list of the collected entities > - all shared libraries used by the crashed instance - their layout > respects the one on the host machine, so them can be easily loaded > with the following gdb command: set sysroot $(realpath .) > > The script can be easily used either manually or via > kernel.core_pattern variable. > > * gdb.sh - the auxiliary script originally written by @Totktonada, but > needed to be adjusted to the crash artefacts layout every time. Since > there is a unified layout, the original script is enhanced a bit to > automatically load the coredump via gdb the right way. 2. Looks like gdb.sh is an internal script, so I propose to rename it to "_gdb.sh" to reflect it for users. > > Closes #5569 > > Signed-off-by: Igor Munkin > --- > > Issue: https://github.com/tarantool/tarantool/issues/5569 > Branch: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling > > changelogs/unreleased/tarabrt.md | 3 + > tools/gdb.sh | 59 ++++++++ > tools/tarabrt.sh | 234 +++++++++++++++++++++++++++++++ > 3 files changed, 296 insertions(+) > create mode 100644 changelogs/unreleased/tarabrt.md > create mode 100755 tools/gdb.sh > create mode 100755 tools/tarabrt.sh > > diff --git a/changelogs/unreleased/tarabrt.md b/changelogs/unreleased/tarabrt.md > new file mode 100644 > index 000000000..e5e616111 > --- /dev/null > +++ b/changelogs/unreleased/tarabrt.md > @@ -0,0 +1,3 @@ > +## feature/tools > + > +* Introduced tooling for crash artefacts collecting and postmortem analysis (gh-5569). > diff --git a/tools/gdb.sh b/tools/gdb.sh > new file mode 100755 > index 000000000..a58c47cab > --- /dev/null > +++ b/tools/gdb.sh > @@ -0,0 +1,59 @@ > +#!/bin/sh > +set -eu 3. "set -euo pipefail" is better. Pipe used for SUBPATH var. > + > +# Check that gdb is installed. > +if ! command -v gdb >/dev/null; then > + cat < +gdb is not installed or not found in the PATH. > + > +Install gdb or adjust you PATH if you are using non-system gdb and > +try once more. > +NOGDB > + exit 1; > +fi > + > +VERSION=${PWD}/version > + > +# Check the location: if the coredump artefacts are collected via > +# `tarabrt.sh' there should be /version file in the root of the > +# unpacked tarball. Otherwise, there is no guarantee the coredump > +# is collected the right way and we can't proceed loading it. > +if [ ! -f "${VERSION}" ]; then > + cat < +${VERSION} file is missing. > + > +If the coredump artefacts are collected via \`tararbrt.sh' tool > +there should be /version file in the root of the unpacked tarball > +(i.e. ${PWD}). > +If version file is missing, there is no guarantee the coredump > +is collected the right way and its loading can't be proceeded > +with this script. Check whether current working directory is the > +tarball root, or try load the core dump file manually. > +NOARTEFACTS > + exit 1; > +fi > + > +REVISION=$(grep -oP 'Tarantool \d+\.\d+\.\d+-\d+-g\K[a-f0-9]+' "$VERSION") > +cat < +================================================================================ > + > +Do not forget to properly setup the environment: > +* git clone https://github.com/tarantool/tarantool.git sources > +* cd !$ > +* git checkout $REVISION > +* git submodule update --recursive --init > + > +================================================================================ > +SOURCES > + > +# Define the build path to be substituted with the source path. > +# XXX: Check the absolute path on the function
definition > +# considering it is located in src/main.cc within Tarantool repo. > +SUBPATH=$(gdb -batch -n ./tarantool -ex 'info line main' | \ > + grep -oP 'Line \d+ of \"\K.+(?=\/src\/main\.cc\")') > + > +# Launch gdb and load coredump with all related artefacts. > +gdb ./tarantool \ > + -ex "set sysroot $(realpath .)" \ > + -ex "set substitute-path $SUBPATH sources" \ > + -ex 'core coredump' > diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh > new file mode 100755 > index 000000000..3d44803be > --- /dev/null > +++ b/tools/tarabrt.sh > @@ -0,0 +1,234 @@ > +#!/bin/sh > +set -eu 4. "set -euo pipefail" is better. Pipe used for SUBPATH var. > + > +TOOL=$(basename "$0") > +HELP=$(cat < +${TOOL} - Tarantool Automatic Bug Reporting Tool > + > +This tool collects all required artefacts (listed below) and packs them into > +a single archive with unified format: > + - /checklist - the plain text file with the list of tarball contents > + - /version - the plain text file containing \`tarantool --version' output > + - /tarantool - the executable binary file produced the core dump > + - /coredump - the core dump file produced by the executable > + - all shared libraries loaded (even via dlopen(3)) at the crash moment. > + > +SYNOPSIS > + > + ${TOOL} [-h] [-c core] [-d dir] [-e executable] [-p procID] [-t datetime] > + > +Supported options are: > + -c COREDUMP Use file COREDUMP as a core dump to examine. 5. I propose to specify somewhere links to documentation how to setup coredump paths and where to find coredumps in a system. For Linux it is core(5) [1]. 1. https://man7.org/linux/man-pages/man5/core.5.html > + > + -d DIRECTORY Create the resulting archive with the artefacts > + within DIRECTORY. > + > + -e TARANTOOL Use file TARANTOOL as the executable file for > + examining with a core dump COREDUMP. If PID is > + specified, the one from /proc/PID/exe is chosen > + (see proc(5) for more info). If TARANTOOL is > + omitted, /usr/bin/tarantool is chosen. > + > + -p PID PID of the dumped process, as seen in the PID > + namespace in which the given process resides > + (see %p in core(5) for more info). This flag > + have to be set when ${TOOL} is used as > + kernel.core_pattern pipeline script. > + > + -t DATETIME Time of dump, expressed as seconds since the > + epoch, 1970-01-01 00:00:00 +0000 (UTC). > + 6. Why users may want to specify datetime manually? We can get datetime in the script with stat:  $stat -c '%Y' core.2715242 1614773664 > + -h Shows this message and exit. > + > +USAGE > + > + - Manual usage. User can simply pack all necessary artefacts by running the > + following command. > + $ /path/to/${TOOL} -c ./core -d /tmp > + > + - Automatic usage. If user faces the failures often, one can set this script > + as a pipe reciever in kernel.core_pattern syntax. > + # sysctl -w kernel.core_pattern="|/absolute/path/to/${TOOL} -d /var/core -p %p -t %t" > + > +HELP > +) > + > +# Parse CLI options. > +OPTIONS=$(getopt -o c:d:e:hp:t: -n "${TOOL}" -- "$@") > +eval set -- "${OPTIONS}" > +while true; do > + case "$1" in > + --) shift; break;; > + -c) COREFILE=$2; shift 2;; > + -d) COREDIR=$2; shift 2;; > + -e) BINARY=$2; shift 2;; > + -p) PID=$2; shift 2;; > + -t) TIME=$2; shift 2;; > + -h) printf "%s\n", "${HELP}"; > + exit 0;; > + *) printf "Invalid option: $1\n%s\n", "${HELP}"; > + exit 1;; > + esac > +done > + > +# Use the default values for the remaining parameters. > +BINARY=${BINARY:-/usr/bin/tarantool} > +COREDIR=${COREDIR:-${PWD}} > +COREFILE=${COREFILE:-} > +PID=${PID:-} > +TIME=${TIME:-$(date +%s)} > + > +# XXX: This section handles the case when the script is used for > +# kernel.core_pattern. If PID is set and there is a directory in > +# procfs with this PID, the script processes the core dumped by > +# this process. If the process exe (or strictly saying its comm) > +# is not 'tarantool' then the coredump is simply saved to the > +# COREDIR; otherwise the dumped core is packed to the tarball. > +if [ -n "${PID}" ] && [ -d /proc/"${PID}" ]; then > + BINARY=$(readlink /proc/"${PID}"/exe) > + CMDNAME=$(sed -z 's/\s$//' /proc/"${PID}"/comm) > + COREFILE=${COREDIR}/${CMDNAME}-core.${PID}.${TIME} > + cat >"${COREFILE}" > + if [ "${CMDNAME}" != 'tarantool' ]; then > + [ -t 1 ] && cat < +/proc/${PID}/comm doesn't equal to 'tarantool', so we assume the > +obtained core is dumped by \`${CMDNAME}' and should be packed in > +a different way. As a result it is simply stored to the file, so > +you can process it on your own. > + > +The file with core dump: ${COREFILE} > +ALIENCOREDUMP > + exit 0; > + fi > +fi > + > +if [ -z "${COREFILE}" ]; then > + [ -t 1 ] && cat < +There is no core dump file passed to ${TOOL}. The artefacts can't > +be collected. If you see this message, check the usage by running > +\`${TOOL} -h': -c option is the obligatory one. > +NOCOREDUMP > + exit 1; > +fi > + > +if file "${COREFILE}" | grep -qv 'core file'; then > + [ -t 1 ] && cat < +Not a core dump: ${COREFILE} > + > +The given COREDUMP file is not a valid core dump (see core(5) for > +more info) or not even an ELF (see elf(5) for more info). If you > +see this message, check the COREDUMP file the following way: > +$ file ${COREFILE} > +NOTACOREDUMP > + exit 1; > +fi > + > +# Check that gdb is installed. > +if ! command -v gdb >/dev/null; then > + [ -t 1 ] && cat < +gdb is not installed, but it is obligatory for collecting the > +loaded shared libraries from the core dump. > + > +You can proceed collecting the artefacts manually later by running > +the following command: > +$ ${TOOL} -e ${BINARY} -c ${COREFILE} > +NOGDB > + exit 1; > +fi > + > +if file "${BINARY}" | grep -qv 'executable'; then > + [ -t 1 ] && cat < +Not an ELF file: ${BINARY} > + > +The given BINARY file is not an ELF (see elf(5) for more info). > +If you see this message, check the BINARY file the following way: > +$ file ${BINARY} > +NOTELF > + exit 1; > +fi > + > +if gdb -batch -n "${BINARY}" -ex 'info symbol tarantool_version' 2>/dev/null | \ > + grep -q 'tarantool_version in section .text' > +then > + # XXX: This is a very ugly hack to implement 'unless' > + # operator in bash for a long pipeline as a conditional. > + : > +else > + [ -t 1 ] && cat < +Not a Tarantool binary: ${BINARY} > + > +The given BINARY file is not a Tarantool executable: there is no a > +signature symbol in the binary file. If you see this message, > +check the BINARY file the following way: > +$ ${BINARY} --help > +NOTARANTOOL > + exit 1; > +fi > + > +# Resolve the host name if possible. > +HOSTNAME=$(hostname 2>/dev/null || echo hostname) > + > +# Proceed with collecting and packing artefacts. > +TMPDIR=$(mktemp -d -p "${COREDIR}") > +TARLIST=${TMPDIR}/tarlist > +VERSION=${TMPDIR}/version > +ARCHIVENAME=${COREDIR}/tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz > + > +# Dump the version to checkout the right commit later. > +${BINARY} --version >"${VERSION}" > + > +# Collect the most important artefacts. > +{ > + echo "${BINARY}" > + echo "${COREFILE}" > + echo "${VERSION}" > +} >>"${TARLIST}" > + > +SEPARATOR1="Shared Object Library" > +SEPARATOR2="Shared library is missing debugging information" > +# XXX: This is kinda "postmortem ldd": the command below dumps the > +# full list of the shared libraries the binary is linked against > +# or those loaded via dlopen at the platform runtime. > +# This is black voodoo magic. Do not touch. You are warned. > +if gdb -batch -n "${BINARY}" -c "${COREFILE}" -ex 'info shared' | \ > + sed -n "/${SEPARATOR1}/,/${SEPARATOR2}/p;/${SEPARATOR2}/q" | \ > + awk '{ print $NF }' | grep '^/' >>"${TARLIST}" > +then > + # XXX: This is a very ugly hack to implement 'unless' > + # operator in bash for a long pipeline as a conditional. > + : > +else > + [ -t 1 ] && cat < +Core dump file is produced by the different Tarantool executable. > + > +Looks like '${COREFILE}' is not generated by \`${BINARY}'. > +If you see this message, please check that the given COREDUMP > +is produced by the specified BINARY. > +There are some temporary artefacts in ${TMPDIR}. > +Remove it manually if you don't need them anymore. > +COREMISMATCH > + exit 1; > +fi > + > +# Pack everything listed in TARLIST file into a tarball. To unify > +# the archive format BINARY, COREFILE, VERSION and TARLIST are > +# renamed while packing. > +tar -czhf "${ARCHIVENAME}" -P -T "${TARLIST}" \ > + --transform="s|${BINARY}|tarantool|" \ > + --transform="s|${COREFILE}|coredump|" \ > + --transform="s|${TARLIST}|checklist|" \ > + --transform="s|${VERSION}|version|" \ > + --add-file="${TARLIST}" > + > +[ -t 1 ] && cat < +The resulting is located here: ${ARCHIVENAME} > + > +If you want to upload it, choose the available resourse 7. Typo: resourse -> resource > +(e.g. http://transfer.sh) and run the following command: > +$ curl -T ${ARCHIVENAME} 8. Typo: resourse-uri -> resource-uri > +FINALIZE > + > +# Cleanup temporary files. > +[ -f "${TARLIST}" ] && rm -f "${TARLIST}" > +[ -f "${VERSION}" ] && rm -f "${VERSION}" > +[ -d "${TMPDIR}" ] && rmdir "${TMPDIR}"