From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id B32BA7030F; Thu, 25 Feb 2021 16:23:56 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B32BA7030F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614259436; bh=pQaW33w6a/K25jySwepsjIaoYHirbMSYOdGuFE0aFMY=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=ZNJ1H2JWUKSw88L5GqUcMAbR3idmwthWTz+C93wQVqyMbm6KX+gwmzt/fHt0RIKBG a/uj7vowDNojJIO7H/wkYa3ml9m6x19A5f7hOoV/EI0iJNERU2Gbw64PmljdVcR4i+ b/mQ4GdwjFSsNws3QQW3ofAfJQrkCsP3O/z0q4K0= Received: from smtpng2.m.smailru.net (smtpng2.m.smailru.net [94.100.179.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 07B427030F for ; Thu, 25 Feb 2021 16:23:56 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 07B427030F Received: by smtpng2.m.smailru.net with esmtpa (envelope-from ) id 1lFGcU-0000eq-TS; Thu, 25 Feb 2021 16:23:55 +0300 To: Alexander Turenko , Sergey Bronnikov Date: Thu, 25 Feb 2021 16:23:52 +0300 Message-Id: <7f444b92bacefea0e3a0b2d2846737d9791626ea.1614256802.git.imun@tarantool.org> X-Mailer: git-send-email 2.25.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD975C3EC174F56692254B0AABE1FB071B229FBFD4D57485274182A05F53808504063D26104D881C187C78C43DBB751EE318C0E9914F4D65B64010AE8E0A6AA88F5 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE79145AB6E9E75F07EEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063779018AFA06A12C538638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95CDDE882590F889B1C8D44C232C7C2E4E4BC6BCD203C830A93A471835C12D1D9774AD6D5ED66289B5259CC434672EE6371117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658378DA827A17800CE71AE4D56B06699BBC9FA2833FD35BB23DF004C9065253843057739F23D657EF2B13377AFFFEAFD26923F8577A6DFFEA7CDDB9BF3B882869D593EC92FD9297F6715571747095F342E857739F23D657EF2BD5E8D9A59859A8B6F82A78844E5C6993089D37D7C0E48F6C5571747095F342E857739F23D657EF2B6825BDBE14D8E7028C9DFF55498CEFB0BD9CCCA9EDD067B1EDA766A37F9254B7 X-C1DE0DAB: 8BD88D57C5CADBC8B2710865C386751094C72BDDC9A8ED5CA3B1A56EE2B804F6B226C914C9968946695E9D90444CEC264DCC8C77FBA9901322D2CEDE4E95CF1BDBE8DEE28BC9005C095FFBCAB1CFE8AAD6CF32B5F8F9D404E07A3523D083E6CB95B6D35713C5808B589120F7DAE46353205367B2BCC23E5B0407C56AD5D25B2CBDAD6C7F3747799A X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D340CB2836B823694494AE7513513229B024E935D3F2BC60FAD96E4B1092508FEA0EFF3863E9E2A392C1D7E09C32AA3244CDD65DE2C5B98770ACBCE7CEF956D389DFE8DA44ABE2443F7927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXgapH+rK1Lrnr+6Ox/uPKNz X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822897E41856FB8B274D01706ED5DA163F2A7C8D0F45F857DBFE9F1EFEE2F478337FB559BB5D741EB964C8C2C849690F8E70A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: [Tarantool-patches] [PATCH] tools: implement toolchain for crash artefacts X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Igor Munkin via Tarantool-patches Reply-To: Igor Munkin Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This patch introduces two scripts to ease crash artefacts collecting and loading for postmortem analysis: * tarabrt.sh - the tool collecting a tarball with the crash artefacts the right way: the coredump with the binary, all loaded shared libs, Tarantool version (this is a separate exercise to get it from the binary built with -O2). Besides, the tarball has a unified layout, so it can be easily processed with the second script: - /coredump - core dump file on the root level - /binary - tarantool executable on the root level - /version - plain text file on the root level with `tarantool --version` output - /checklist - plain text file on the root level with the list of the collected entities - all shared libraries used by the crashed instance - their layout respects the one on the host machine, so them can be easily loaded with the following gdb command: set sysroot $(realpath .) The script can be easily used either manually or via kernel.core_pattern variable. * gdb.sh - the auxiliary script originally written by @Totktonada, but needed to be adjusted to the crash artefacts layout every time. Since there is a unified layout, the original script is enhanced a bit to automatically load the coredump via gdb the right way. Closes #5569 Signed-off-by: Igor Munkin --- Issue: https://github.com/tarantool/tarantool/issues/5569 Branch: https://github.com/tarantool/tarantool/tree/imun/gh-5569-coredump-tooling changelogs/unreleased/tarabrt.md | 3 + tools/gdb.sh | 59 ++++++++ tools/tarabrt.sh | 234 +++++++++++++++++++++++++++++++ 3 files changed, 296 insertions(+) create mode 100644 changelogs/unreleased/tarabrt.md create mode 100755 tools/gdb.sh create mode 100755 tools/tarabrt.sh diff --git a/changelogs/unreleased/tarabrt.md b/changelogs/unreleased/tarabrt.md new file mode 100644 index 000000000..e5e616111 --- /dev/null +++ b/changelogs/unreleased/tarabrt.md @@ -0,0 +1,3 @@ +## feature/tools + +* Introduced tooling for crash artefacts collecting and postmortem analysis (gh-5569). diff --git a/tools/gdb.sh b/tools/gdb.sh new file mode 100755 index 000000000..a58c47cab --- /dev/null +++ b/tools/gdb.sh @@ -0,0 +1,59 @@ +#!/bin/sh +set -eu + +# Check that gdb is installed. +if ! command -v gdb >/dev/null; then + cat < definition +# considering it is located in src/main.cc within Tarantool repo. +SUBPATH=$(gdb -batch -n ./tarantool -ex 'info line main' | \ + grep -oP 'Line \d+ of \"\K.+(?=\/src\/main\.cc\")') + +# Launch gdb and load coredump with all related artefacts. +gdb ./tarantool \ + -ex "set sysroot $(realpath .)" \ + -ex "set substitute-path $SUBPATH sources" \ + -ex 'core coredump' diff --git a/tools/tarabrt.sh b/tools/tarabrt.sh new file mode 100755 index 000000000..3d44803be --- /dev/null +++ b/tools/tarabrt.sh @@ -0,0 +1,234 @@ +#!/bin/sh +set -eu + +TOOL=$(basename "$0") +HELP=$(cat <"${COREFILE}" + if [ "${CMDNAME}" != 'tarantool' ]; then + [ -t 1 ] && cat </dev/null; then + [ -t 1 ] && cat </dev/null | \ + grep -q 'tarantool_version in section .text' +then + # XXX: This is a very ugly hack to implement 'unless' + # operator in bash for a long pipeline as a conditional. + : +else + [ -t 1 ] && cat </dev/null || echo hostname) + +# Proceed with collecting and packing artefacts. +TMPDIR=$(mktemp -d -p "${COREDIR}") +TARLIST=${TMPDIR}/tarlist +VERSION=${TMPDIR}/version +ARCHIVENAME=${COREDIR}/tarantool-core-${PID:-N}-$(date +%Y%m%d%H%M -d @"${TIME}")-${HOSTNAME%%.*}.tar.gz + +# Dump the version to checkout the right commit later. +${BINARY} --version >"${VERSION}" + +# Collect the most important artefacts. +{ + echo "${BINARY}" + echo "${COREFILE}" + echo "${VERSION}" +} >>"${TARLIST}" + +SEPARATOR1="Shared Object Library" +SEPARATOR2="Shared library is missing debugging information" +# XXX: This is kinda "postmortem ldd": the command below dumps the +# full list of the shared libraries the binary is linked against +# or those loaded via dlopen at the platform runtime. +# This is black voodoo magic. Do not touch. You are warned. +if gdb -batch -n "${BINARY}" -c "${COREFILE}" -ex 'info shared' | \ + sed -n "/${SEPARATOR1}/,/${SEPARATOR2}/p;/${SEPARATOR2}/q" | \ + awk '{ print $NF }' | grep '^/' >>"${TARLIST}" +then + # XXX: This is a very ugly hack to implement 'unless' + # operator in bash for a long pipeline as a conditional. + : +else + [ -t 1 ] && cat < +FINALIZE + +# Cleanup temporary files. +[ -f "${TARLIST}" ] && rm -f "${TARLIST}" +[ -f "${VERSION}" ] && rm -f "${VERSION}" +[ -d "${TMPDIR}" ] && rmdir "${TMPDIR}" -- 2.25.0