<HTML><BODY><div><br>Hi Sasha, thanks for the review, I'm going to split and resend patch, I'll take into account your comments.<br> <blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;">Понедельник, 20 апреля 2020, 20:42 +03:00 от Alexander Tikhonov <avtikhon@tarantool.org>:<br> <div id=""><div class="js-helper js-readmsg-msg"><style type="text/css"></style><div><div id="style_15874045351464821656_BODY"><div class="class_1587452063"><div><div>Oleg, thanks for the patch, please check my questions below and minor corrections in the letter after:<br>- as about scripts per benchmarks all changes are correct;<br>- bench-run repository creation moved from dockerfile of the first stage to the second, may be it will be good to notice about it in the commit message;<br>- fetching was specially set to bench-run while you already added ‘--no-cache’ option - fetching will never be used in such situation, due to ‘--no-cache’ option will block the docker from its cache use;<br>- fetching was set to Tarantool sources, but it is not needed because again used ‘--no-cache’ option and the main cause that gitlab-ci fetches the Tarantool sources itself at the very start of each gitlab-ci job.<br> </div><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;">Воскресенье, 19 апреля 2020, 19:28 +03:00 от Олег Пискунов <<a href="/compose?To=o.piskunov@corp.mail.ru">o.piskunov@corp.mail.ru</a>>:<br> <div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><style type="text/css"></style><div><div id="style_15873136980545068336_BODY_mailru_css_attribute_postfix"><div class="class_1587417288_mailru_css_attribute_postfix"><div><div><br>Resurrect bench.tarantool.org:<br>- submit perf results into bench.tarantool.org database</div><div> </div><div>Fix docker files for perf testing:<br>- Disable cashing from docker file creation.<br>- adding python "requests" module needed for publish script.</div><div> </div><div>Closes #4865, #4870<br>---</div><div> </div><div>Github: <a href="https://github.com/tarantool/tarantool/tree/opiskunov/gh-4865-submit-perf-results" rel="noopener noreferrer" target="_blank">https://github.com/tarantool/tarantool/tree/opiskunov/gh-4865-submit-perf-results</a></div></div></div></div></div></div></div></blockquote></div><div>Wrong link.</div><div><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div><div class="class_1587417288_mailru_css_attribute_postfix"><div><div><br>Issue: <a href="https://github.com/tarantool/tarantool/issues/4865" rel="noopener noreferrer" target="_blank">https://github.com/tarantool/tarantool/issues/4865</a><br> <a href="https://github.com/tarantool/tarantool/issues/4870" rel="noopener noreferrer" target="_blank">https://github.com/tarantool/tarantool/issues/4870</a><br>---<br> benchs/cbench/run.sh | 38 ++++++++++++++++++------<br> benchs/linkbench/run.sh | 13 +++++++--<br> benchs/nosqlbench/run.sh | 22 ++++++++------<br> benchs/publication/publish.py | 67 +++++++++++++++++++++++++++++++++++++++++++<br> benchs/sysbench/run.sh | 19 ++++++------<br> benchs/tpcc/run.sh | 21 ++++++++------<br> benchs/ycsb/run.sh | 16 +++++++++--<br> dockerfiles/ubuntu_benchs | 4 +--<br> dockerfiles/ubuntu_tnt | 6 ++++<br> targets.mk | 2 +-<br> 10 files changed, 165 insertions(+), 43 deletions(-)<br> create mode 100755 benchs/publication/publish.py</div><div> </div><div>diff --git a/benchs/cbench/run.sh b/benchs/cbench/run.sh<br>index 4d07972..c4867ad 100755<br>--- a/benchs/cbench/run.sh<br>+++ b/benchs/cbench/run.sh<br>@@ -1,11 +1,11 @@<br> #!/usr/bin/env bash</div><div> </div><div> kill `pidof tarantool`<br>+<br> set -e<br> set -o pipefail</div><div> </div><div> TAR_VER=$(tarantool -v | grep -e "Tarantool" | grep -oP '\s\K\S*')<br>-echo ${TAR_VER} | tee cbench_t_version.txt</div><div> </div><div> killall tarantool 2>/dev/null || true<br> rm -rf 5* 0*<br>@@ -23,14 +23,34 @@ killall tarantool 2>/dev/null || true<br> rm -rf 5* 0*<br> sync<br> echo 3 > /proc/sys/vm/drop_caches<br>-numactl --membind=1 --cpunodebind=1 --physcpubind=11 tarantool /opt/cbench/cbench_runner.lua vinyl write 500 2>&1 | tee cbench_output_memtx_write.txt<br>+numactl --membind=1 --cpunodebind=1 --physcpubind=11 tarantool /opt/cbench/cbench_runner.lua vinyl write 500 2>&1 | tee cbench_output_vinyl_write.txt</div><div> </div><div>-grep "^?tab" cbench_output_memtx.txt | sed "s/.*name=//"| sed "s/¶m=/:/"| sed "s/cb\./cb\.memtx\./"| tee -a cbench_result.txt<br>-grep "^?tab" cbench_output_vinyl_fsync.txt | sed "s/.*name=//"| sed "s/¶m=/:/"| sed "s/cb\./cb\.vinyl\.fsync\./"| tee -a cbench_result.txt<br>-grep "^?tab" cbench_output_memtx_write.txt | sed "s/.*name=//"| sed "s/¶m=/:/"| sed "s/cb\./cb\.vinyl\.write\./"| tee -a cbench_result.txt<br>+grep "^?tab=cbench.tree" cbench_output_memtx.txt | sed "s/.*name=//"| sed "s/¶m=/:/"| sed "s/cb\./cb\.memtx\./"| tee -a cbench-memtx-tree_result.txt<br>+grep "^?tab=cbench.hash" cbench_output_memtx.txt | sed "s/.*name=//"| sed "s/¶m=/:/"| sed "s/cb\./cb\.memtx\./"| tee -a cbench-memtx-hash_result.txt<br>+grep "^?tab" cbench_output_vinyl_fsync.txt | sed "s/.*name=//"| sed "s/¶m=/:/"| sed "s/cb\./cb\.vinyl\.fsync\./"| tee -a cbench-vinyl-fsync_result.txt<br>+grep "^?tab" cbench_output_vinyl_write.txt | sed "s/.*name=//"| sed "s/¶m=/:/"| sed "s/cb\./cb\.vinyl\.write\./"| tee -a cbench-vinyl-write_result.txt</div><div> </div><div>-#mv tarantool-server.log cbench_tarantool_server.log<br>-<br>-echo "RESULTS:"<br>-cat cbench_result.txt<br>+echo ${TAR_VER} | tee cbench-memtx-tree_t_version.txt<br>+echo ${TAR_VER} | tee cbench-memtx-hash_t_version.txt<br>+echo ${TAR_VER} | tee cbench-vinyl-fsync_t_version.txt<br>+echo ${TAR_VER} | tee cbench-vinyl-write_t_version.txt<br>+echo ${TAR_VER} | tee cbench_t_version.txt</div><div> </div><div>+echo "Tarantool TAG:"<br>+cat cbench_t_version.txt<br>+echo "Overall results:"<br>+echo "================"<br>+echo "RESULTS (cbench-memtx-tree_result.txt):"<br>+cat cbench-memtx-tree_result.txt<br>+echo " "<br>+echo "RESULTS (cbench-memtx-hash_result.txt):"<br>+cat cbench-memtx-hash_result.txt<br>+echo " "<br>+echo "RESULTS (cbench-vinyl-fsync_result.txt):"<br>+cat cbench-vinyl-fsync_result.txt<br>+echo " "<br>+echo "RESULTS (cbench-vinyl-write_result.txt):"<br>+cat cbench-vinyl-write_result.txt<br>+echo " "<br>+echo "Publish data to bench database"<br>+/opt/bench-run/benchs/publication/publish.py<br>diff --git a/benchs/linkbench/run.sh b/benchs/linkbench/run.sh<br>index 6738436..e7c77c3 100755<br>--- a/benchs/linkbench/run.sh<br>+++ b/benchs/linkbench/run.sh<br>@@ -6,7 +6,6 @@ set -e<br> set -o pipefail</div><div> </div><div> TAR_VER=$(tarantool -v | grep -e "Tarantool" | grep -oP '\s\K\S*')<br>-echo ${TAR_VER} | tee linkbench_t_version.txt</div><div> </div><div> cd /opt/linkbench && mvn clean package -Dmaven.test.skip=true<br> cd src/tarantool && make<br>@@ -34,4 +33,14 @@ sync<br> echo 3 > /proc/sys/vm/drop_caches<br> $numaopts /opt/linkbench/bin/linkbench -c $cfgfile -r 2>&1 | tee linkbench_output.txt</div><div> </div><div>-grep "REQUEST PHASE COMPLETED" linkbench_output.txt | sed "s/.*second = /linkbench:/" | tee -a linkbench_result.txt<br>+grep "REQUEST PHASE COMPLETED" linkbench_output.txt | sed "s/.*second = /linkbench:/" | tee -a linkbench.ssd_result.txt<br>+echo ${TAR_VER} | tee linkbench.ssd_t_version.txt<br>+<br>+echo "Tarantool TAG:"<br>+cat linkbench.ssd_t_version.txt<br>+echo "Overall results:"<br>+echo "================"<br>+cat linkbench.ssd_result.txt<br>+echo " "<br>+echo "Publish data to bench database"<br>+/opt/bench-run/benchs/publication/publish.py<br>diff --git a/benchs/nosqlbench/run.sh b/benchs/nosqlbench/run.sh<br>index c9491f5..a68248e 100755<br>--- a/benchs/nosqlbench/run.sh<br>+++ b/benchs/nosqlbench/run.sh<br>@@ -6,13 +6,11 @@ if [ "$type" == "" ]; then<br> fi</div><div> </div><div> TAR_VER=$(tarantool -v | grep -e "Tarantool" | grep -oP '\s\K\S*')<br>-echo ${TAR_VER} | tee nosqlbench_t_version.txt</div><div> </div><div> kill `pidof tarantool`</div><div> </div><div>-#set -e<br>-#<br>-#set -o pipefail<br>+set -e<br>+set -o pipefail</div><div> </div><div> killall tarantool 2>/dev/null || true<br> rm -rf 0*.xlog 0*.snap<br>@@ -42,8 +40,14 @@ numactl --membind=1 --cpunodebind=1 --physcpubind=6,7,8,9,10,11 /opt/nosqlbench/<br> #echo "Latest 1000 lines:"<br> #tail -1000 nosqlbench_output.txt</div><div> </div><div>-echo "Getting results"<br>-grep "TOTAL RPS STATISTICS:" nosqlbench_output.txt -A6 | awk -F "|" 'NR > 4 {print $2,":", $4}' > nosqlbench_result.txt<br>-<br>-cat nosqlbench_result.txt<br>-<br>+grep "TOTAL RPS STATISTICS:" nosqlbench_output.txt -A6 | awk -F "|" 'NR > 4 {print $2,":", $4}' > noSQLbench.${type}_result.txt<br>+echo ${TAR_VER} | tee noSQLbench.${type}_t_version.txt<br>+<br>+echo "Tarantool TAG:"<br>+cat noSQLbench.${type}_t_version.txt<br>+echo "Overall results:"<br>+echo "================"<br>+cat noSQLbench.${type}_result.txt<br>+echo " "<br>+echo "Publish data to bench database"<br>+/opt/bench-run/benchs/publication/publish.py<br>diff --git a/benchs/publication/publish.py b/benchs/publication/publish.py<br>new file mode 100755<br>index 0000000..d472fd7<br>--- /dev/null<br>+++ b/benchs/publication/publish.py<br>@@ -0,0 +1,67 @@<br>+#!/usr/bin/env python<br>+import fnmatch<br>+import os<br>+from urllib import urlencode<br>+import requests<br>+<br>+<br>+def parse_bench(filename):<br>+ fileHandle = open(filename)<br>+ lastline = fileHandle.readlines()<br>+ fileHandle.close()<br>+ return lastline<br>+<br>+<br>+def get_version(filename):<br>+ fileHandle = open(filename)<br>+ lastline = fileHandle.readlines()[-1]<br>+ fileHandle.close()<br>+ return lastline.split()[0]<br>+<br>+<br>+def push_to_microb(server, token, name, value, version, tab):<br>+ uri = 'http://%s/push?%s' % (server, urlencode(dict(<br>+ key=token, name=name, param=value,<br>+ v=version, unit='trps', tab=tab<br>+ )))<br>+<br>+ r = requests.get(uri)<br>+ if r.status_code == 200:<br>+ print('Export complete')<br>+ else:<br>+ print('Export error http: %d' % r.status_code)<br>+ print('Export error text: %d' % r.text)<br>+<br>+<br>+def main():<br>+ if "MICROB_WEB_TOKEN" in os.environ and "MICROB_WEB_HOST" in os.environ:<br>+ bench = {}<br>+ res = []<br>+ current_data = {}<br>+ version = ''<br>+ for file in os.listdir('.'):<br>+ if fnmatch.fnmatch(file, '*_result.txt'):<br>+ values = parse_bench(file)<br>+ benchmark = file.split('_')[0]<br>+ version = get_version('{}_t_version.txt'.format(benchmark))<br>+ for value in values:<br>+ test_name = value.split(':')[0]<br>+ test_res = float(value.split(':')[1])<br>+ res.append(test_res)<br>+ push_to_microb(<br>+ os.environ['MICROB_WEB_HOST'],<br>+ os.environ['MICROB_WEB_TOKEN'],<br>+ test_name,<br>+ test_res,<br>+ version,<br>+ benchmark,<br>+ )<br>+ print ("VERSION - ", version)<br>+ else:<br>+ print("MICROB params not specified")<br>+<br>+ return 0<br>+<br>+<br>+if __name__ == '__main__':<br>+ main()<br>diff --git a/benchs/sysbench/run.sh b/benchs/sysbench/run.sh<br>index 6a69d14..ef0af43 100755<br>--- a/benchs/sysbench/run.sh<br>+++ b/benchs/sysbench/run.sh<br>@@ -11,8 +11,6 @@ set -e<br> set -o pipefail</div><div> </div><div> TAR_VER=$(tarantool -v | grep -e "Tarantool" | grep -oP '\s\K\S*')<br>-echo ${TAR_VER} | tee sysbench_t_version.txt<br>-set -e</div><div> </div><div> ARRAY_TESTS=(<br> "oltp_read_only"<br>@@ -44,11 +42,10 @@ opts="--db-driver=${DBMS} --threads=${THREADS}"</div><div> </div><div> export LD_LIBRARY_PATH=/usr/local/lib</div><div> </div><div>-rm -f sysbench_result.txt<br>-set -o pipefail<br>+rm -f Sysbench_result.txt<br> for test in "${ARRAY_TESTS[@]}"; do<br> res=0<br>- tlog=sysbench_${test}_result.txt<br>+ tlog=sysbench_${test}_results.txt<br> rm -f $tlog<br> maxres=0<br> for run in `eval echo {1..$runs}` ; do<br>@@ -69,7 +66,7 @@ for test in "${ARRAY_TESTS[@]}"; do<br> if [[ $tres -gt $maxres ]]; then maxres=$tres ; fi<br> done<br> res=$(($res/$runs))<br>- echo "${test}: $res" >>sysbench_result.txt<br>+ echo "${test}: $res" >>Sysbench_result.txt</div><div> echo "Subtest '$test' results:"<br> echo "==============================="<br>@@ -78,7 +75,13 @@ for test in "${ARRAY_TESTS[@]}"; do<br> printf "Diviations (AVG -> MAX): %.2f" `bc <<< "scale = 4; (1 - $res / $maxres) * 100"` ; echo %<br> done</div><div> </div><div>+echo ${TAR_VER} | tee Sysbench_t_version.txt<br>+<br>+echo "Tarantool TAG:"<br>+cat Sysbench_t_version.txt<br> echo "Overall results:"<br> echo "================"<br>-cat sysbench_result.txt<br>-<br>+cat Sysbench_result.txt<br>+echo " "<br>+echo "Publish data to bench database"<br>+/opt/bench-run/benchs/publication/publish.py<br>diff --git a/benchs/tpcc/run.sh b/benchs/tpcc/run.sh<br>index f1f60cc..76754e5 100755<br>--- a/benchs/tpcc/run.sh<br>+++ b/benchs/tpcc/run.sh<br>@@ -1,15 +1,12 @@<br> #!/usr/bin/env bash<br>+set -e<br>+set -o pipefail</div><div> </div><div> TAR_VER=$(tarantool -v | grep -e "Tarantool" | grep -oP '\s\K\S*')<br>-echo ${TAR_VER} | tee tpcc_t_version.txt<br>-<br>-set -e</div><div> </div><div> if [ ! -n "${TIME}" ]; then TIME=1200; fi<br> if [ ! -n "${WARMUP_TIME}" ]; then WARMUP_TIME=10; fi</div><div> </div><div>-set -o pipefail<br>-<br> killall tarantool tpcc_load 2>/dev/null || true<br> rm -rf 0*.xlog 0*.snap<br> sync<br>@@ -30,10 +27,16 @@ cd /opt/tpcc<br> numactl --membind=1 --cpunodebind=1 --physcpubind=6,7,8,9,10,11 \<br> /opt/tpcc/tpcc_start $tpcc_opts -r10 -l${TIME} -i${TIME} >tpcc_output.txt 2>/dev/null</div><div> </div><div>-echo -n "tpcc:" | tee tpcc_result.txt<br>-cat tpcc_output.txt | grep -e '<TpmC>' | grep -oP '\K[0-9.]*' | tee -a tpcc_result.txt<br>-<br>+echo -n "tpcc:" | tee tpc.c_result.txt<br>+cat tpcc_output.txt | grep -e '<TpmC>' | grep -oP '\K[0-9.]*' | tee -a tpc.c_result.txt<br> cat tpcc_output.txt<br>+echo ${TAR_VER} | tee tpc.c_t_version.txt<br>+<br>+echo "Tarantool TAG:"<br>+cat tpc.c_t_version.txt<br> echo "Overall result:"<br> echo "==============="<br>-cat tpcc_result.txt<br>+cat tpc.c_result.txt<br>+echo " "<br>+echo "Publish data to bench database"<br>+/opt/bench-run/benchs/publication/publish.py<br>diff --git a/benchs/ycsb/run.sh b/benchs/ycsb/run.sh<br>index 5ae1c75..d2cf8af 100755<br>--- a/benchs/ycsb/run.sh<br>+++ b/benchs/ycsb/run.sh<br>@@ -17,7 +17,6 @@ set -e<br> set -o pipefail</div><div> </div><div> TAR_VER=$(tarantool -v | grep -e "Tarantool" | grep -oP '\s\K\S*')<br>-echo ${TAR_VER} | tee ycsb_t_version.txt</div><div> </div><div> ws=/opt/ycsb<br> cd $ws<br>@@ -50,7 +49,18 @@ for l in a b c d e f ; do<br> echo 3 > /proc/sys/vm/drop_caches<br> numactl --membind=1 --cpunodebind=1 --physcpubind=6,7,8,9,10,11 bin/ycsb run tarantool -s -P workloads/workload${l} >${res}.log 2>&1 || cat ${res}.log<br> grep Thro ${res}.log | awk '{ print "Overall result: "$3 }' | tee ${res}.txt<br>- sed "s#Overall result#$l $r#g" ${res}.txt >>${plogs}/results.txt<br>+ sed "s#Overall result#$l $r#g" ${res}.txt >>${plogs}/ycsb.${mode}_result.txt<br> done<br> done<br>-cat ${plogs}/results.txt<br>+<br>+echo ${TAR_VER} | tee ycsb.${mode}_t_version.txt<br>+cp -f ${plogs}/ycsb.${mode}_result.txt .<br>+<br>+echo "Tarantool TAG:"<br>+cat ycsb.${mode}_t_version.txt<br>+echo "Overall results:"<br>+echo "================"<br>+cat ycsb.${mode}_result.txt<br>+echo " "<br>+echo "Publish data to bench database"<br>+/opt/bench-run/benchs/publication/publish.py<br>diff --git a/dockerfiles/ubuntu_benchs b/dockerfiles/ubuntu_benchs<br>index fd9acb4..2c2e717 100644<br>--- a/dockerfiles/ubuntu_benchs<br>+++ b/dockerfiles/ubuntu_benchs<br>@@ -71,5 +71,5 @@ RUN luarocks install \<br> <a href="https://raw.githubusercontent.com/tarantool/gperftools/master/rockspecs/gperftools-scm-1.rockspec" rel="noopener noreferrer" target="_blank">https://raw.githubusercontent.com/tarantool/gperftools/master/rockspecs/gperftools-scm-1.rockspec</a> \<br> --local >build.log 2>&1 || ( cat build.log && false )</div><div> </div><div>-# benchmarks runners<br>-RUN git clone <a href="https://github.com/tarantool/bench-run.git" rel="noopener noreferrer" target="_blank">https://github.com/tarantool/bench-run.git</a> /opt/bench-run<br>+# adding python dependency<br>+RUN pip install requests<br>diff --git a/dockerfiles/ubuntu_tnt b/dockerfiles/ubuntu_tnt<br>index 361e9d6..c1657ff 100644<br>--- a/dockerfiles/ubuntu_tnt<br>+++ b/dockerfiles/ubuntu_tnt<br>@@ -3,11 +3,17 @@ FROM ${image_from}</div><div> </div><div> COPY . /opt/tarantool<br> WORKDIR /opt/tarantool<br>+RUN git fetch --all<br> RUN git submodule update --recursive --init --force<br> RUN ( cmake . -DENABLE_DIST=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo >build.log 2>&1 \<br> && make -j >>build.log 2>&1 && make install >>build.log 2>&1 ) || \<br> ( cat build.log && false )</div><div> </div><div>+# benchmarks runners<br>+RUN git clone <a href="https://github.com/tarantool/bench-run.git" rel="noopener noreferrer" target="_blank">https://github.com/tarantool/bench-run.git</a> /opt/bench-run<br>+WORKDIR /opt/bench-run<br>+RUN git fetch --all<br>+<br> # cbench<br> RUN git clone <a href="https://github.com/tarantool/cbench.git" rel="noopener noreferrer" target="_blank">https://github.com/tarantool/cbench.git</a> /opt/cbench<br> WORKDIR /opt/cbench<br>diff --git a/targets.mk b/targets.mk<br>index 6cf36b6..581b722 100644<br>--- a/targets.mk<br>+++ b/targets.mk<br>@@ -21,7 +21,7 @@ prepare:<br> docker push ${IMAGE_PERF}<br> # build Tarantool and benchmarks with depends on Tarantool sources<br> ${DOCKERFILE_BUILD} --build-arg image_from=${IMAGE_PERF} \<br>- -t ${IMAGE_PERF_BUILT} -f bench-run/dockerfiles/ubuntu_tnt .<br>+ -t ${IMAGE_PERF_BUILT} --no-cache -f bench-run/dockerfiles/ubuntu_tnt .<br> docker push ${IMAGE_PERF_BUILT}</div><div> </div><div> # #####################################################<br>--<br>1.8.3.1</div><div> </div></div></div></div></div></div></div></blockquote> <div> </div><div data-signature-widget="container"><div data-signature-widget="content"><div>--<br>Alexander Tikhonov</div></div></div><div> </div></div></div></div></div></div></div></blockquote> <div> </div><div data-signature-widget="container"><div data-signature-widget="content"><div>--<br>Oleg Piskunov</div></div></div><div> </div></div></BODY></HTML>