* [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci
@ 2020-02-17 14:01 Alexander V. Tikhonov
2020-02-18 21:41 ` Alexander Turenko
0 siblings, 1 reply; 5+ messages in thread
From: Alexander V. Tikhonov @ 2020-02-17 14:01 UTC (permalink / raw)
To: Oleg Piskunov; +Cc: tarantool-patches
Enabled Tarantool performance testing on Gitlab-CI
for release/master branches and "*-perf" named branches.
For this purpose 'perf' and 'cleanup' stages were added
into Gitlab-CI pipeline.
Performance testing support next benchmarks:
- cbench
- linkbench
- nosqlbench (hash and tree Tarantool run modes)
- sysbench
- tpcc
- ycsb (hash and tree Tarantool run modes)
Benchmarks use scripts from repository:
http://gitlab.com/tarantool/bench-run
Perfomance testing uses docker images, built
with docker files from bench-run repository:
- perf/ubuntu-bionic:perf_master
parent image with benchmarks only
- perf_tmp/ubuntu-bionic:perf_<commit_SHA>
child images used for testing Tarantool sources
---
Github: https://github.com/tarantool/tarantool/tree/avtikhon/gitlab-ci-perf
Changes v4:
- moved the rest of Docker infrastructure for perf testing
into the bench-run repository
- renamed calls to performance prepare/run/cleanup stages
Changes v3:
- updated commit message
- merged all make targets for perf testing into single
- set to use variables in gitlab-ci for running perf testings
Changes v2:
- moved performance variables from global setup
to performance template
- updated commit message
- added more comments to sources
.gitlab-ci.yml | 100 +++++++++++++++++++++++++++++++++++++++++++++++++
.gitlab.mk | 19 ++++++++++
2 files changed, 119 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 727b7c94e..1bc53d0fb 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,7 @@
stages:
- test
+ - perf
+ - cleanup
variables:
GITLAB_MAKE: "make -f .gitlab.mk"
@@ -26,6 +28,14 @@ variables:
- merge_requests
- /^.*-full-ci$/
+.perf_only_template: &perf_only_definition
+ only:
+ - master
+ - /^.*-perf$/
+ variables: &perf_vars_definition
+ IMAGE_PERF: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf/ubuntu-bionic:perf_master"
+ IMAGE_PERF_BUILT: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf_tmp/ubuntu-bionic:perf_${CI_COMMIT_SHORT_SHA}"
+
.docker_test_template: &docker_test_definition
image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-stretch:latest"
stage: test
@@ -77,6 +87,15 @@ variables:
after_script:
- ${GITLAB_MAKE} vms_shutdown
+.perf_docker_test_template: &perf_docker_test_definition
+ <<: *perf_only_definition
+ image: ${IMAGE_PERF_BUILT}
+ stage: perf
+ tags:
+ - docker_perf
+ script:
+ - ${GITLAB_MAKE} perf_run
+
# Tests
release:
@@ -168,6 +187,87 @@ freebsd_12_release:
script:
- ${GITLAB_MAKE} vms_test_freebsd
+# ####
+# Perf
+# ####
+
+# Pre-testing part
+
+perf_bootstrap:
+ <<: *perf_only_definition
+ stage: test
+ tags:
+ - perf
+ script:
+ - ${GITLAB_MAKE} perf_prepare
+
+# Testing part
+
+perf_sysbench:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'sysbench'
+
+perf_tpcc:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'tpcc'
+
+perf_ycsb_hash:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'ycsb'
+ ARG: 'hash'
+
+perf_ycsb_tree:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'ycsb'
+ ARG: 'tree'
+
+perf_nosqlbench_hash:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'nosqlbench'
+ ARG: 'hash'
+
+perf_nosqlbench_tree:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'nosqlbench'
+ ARG: 'tree'
+
+perf_cbench:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'cbench'
+
+perf_linkbench_ssd:
+ <<: *perf_docker_test_definition
+ tags:
+ - docker_perf_ssd
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'linkbench'
+
+# Post-testing part
+
+remove_images:
+ <<: *perf_only_definition
+ stage: cleanup
+ when: always
+ tags:
+ - perf
+ script:
+ - ${GITLAB_MAKE} perf_cleanup
+
# Packs
centos_6:
diff --git a/.gitlab.mk b/.gitlab.mk
index 24b18b7f2..b39c5c651 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -78,6 +78,18 @@ docker_bootstrap:
docker push ${DEBIAN_STRETCH_IMAGE}:latest
docker push ${DEBIAN_BUSTER_IMAGE}:latest
+# Clone the benchmarks repository for performance testing
+perf_clone_benchs_repo:
+ git clone https://github.com/tarantool/bench-run.git
+
+# Build images for performance testing
+perf_prepare: perf_clone_benchs_repo
+ make -f bench-run/targets.mk prepare
+
+# Remove temporary performance image from the test host
+perf_cleanup: perf_clone_benchs_repo
+ make -f bench-run/targets.mk cleanup
+
# #################################
# Run tests under a virtual machine
# #################################
@@ -126,3 +138,10 @@ deploy: package
static_build:
docker build --network=host --build-arg RUN_TESTS="${RUN_TESTS}" -f Dockerfile.staticbuild .
+
+# ###################
+# Performance testing
+# ###################
+
+perf_run:
+ /opt/bench-run/benchs/${BENCH}/run.sh ${ARG}
--
2.17.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci
2020-02-17 14:01 [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci Alexander V. Tikhonov
@ 2020-02-18 21:41 ` Alexander Turenko
2020-02-19 5:37 ` Alexander Tikhonov
0 siblings, 1 reply; 5+ messages in thread
From: Alexander Turenko @ 2020-02-18 21:41 UTC (permalink / raw)
To: Alexander V. Tikhonov; +Cc: Oleg Piskunov, tarantool-patches
I still think that details might be better, but okay, I see: you need
some base now to proceed further. Don't want to block it anymore.
I commented the patch below, but didn't perform any changes except a bit
reworded commit message.
Pushed to master. CCed Kirill.
Don't sure how it should look at other release branches:
- Whether something need to be changed for 2.3/2.2?
- perf_only_template should have "2.3" / "2.2" branch instead of
master, that I understood.
- Should IMAGE_PERF be tagged as "perf_2.3" / "perf_2.2" instead of
"perf_master"? Are benchmarks should be adjusted for those versions
and should this lead to such separation base images (or will be
handled at runtime with bench-run scripts?).
- Whether something need to be changed for 1.10?
- At least SQL benchmarks will not work. Should it be handled here or
they will be skipped on bench-run side?
- Same question re IMAGE_PERF as above.
Let's elaborate those questions. After this we can push it downward.
WBR, Alexander Turenko.
> Implement perf testing at gitlab-ci
Changed to: 'gitlab-ci: enable performance testing'.
>
> Enabled Tarantool performance testing on Gitlab-CI
> for release/master branches and "*-perf" named branches.
> For this purpose 'perf' and 'cleanup' stages were added
> into Gitlab-CI pipeline.
>
> Performance testing support next benchmarks:
> - cbench
> - linkbench
> - nosqlbench (hash and tree Tarantool run modes)
> - sysbench
> - tpcc
> - ycsb (hash and tree Tarantool run modes)
>
> Benchmarks use scripts from repository:
> http://gitlab.com/tarantool/bench-run
Dead link. Changed gitlab.com to github.com.
>
> Perfomance testing uses docker images, built
Fixed typo: 'Perfomance'.
> with docker files from bench-run repository:
> - perf/ubuntu-bionic:perf_master
> parent image with benchmarks only
> - perf_tmp/ubuntu-bionic:perf_<commit_SHA>
> child images used for testing Tarantool sources
Formatted a bit (to fit 72 symbols, but not much less).
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gitlab-ci-perf
> +.perf_only_template: &perf_only_definition
> + only:
> + - master
> + - /^.*-perf$/
> + variables: &perf_vars_definition
> + IMAGE_PERF: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf/ubuntu-bionic:perf_master"
> + IMAGE_PERF_BUILT: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf_tmp/ubuntu-bionic:perf_${CI_COMMIT_SHORT_SHA}"
> +
The resulting bench-run API looks strage for me:
* It expects that a caller will set CI_REGISTRY, CI_REGISTRY_USER,
CI_REGISTRY_PASSWORD environment variables, which come from GitLab-CI,
but can be set manually.
* However it does not use CI_REGISTRY, CI_PROJECT_PATH,
CI_COMMIT_SHORT_SHA to choose images name on its own, but expect
IMAGE_PERF and IMAGE_PERF_BUILT from a caller.
* All those variables have prefix CI_*, not, say, BENCH_RUN_*.
> +# Pre-testing part
> +
> +perf_bootstrap:
> + <<: *perf_only_definition
> + stage: test
> + tags:
> + - perf
> + script:
> + - ${GITLAB_MAKE} perf_prepare
There is no reason to use two terms for the same thing: bootstrap and
prepare.
Also I don't see a reason to extract such one-two-liners into a
gitlab.mk.
> +# Post-testing part
> +
> +remove_images:
> + <<: *perf_only_definition
> + stage: cleanup
> + when: always
> + tags:
> + - perf
> + script:
> + - ${GITLAB_MAKE} perf_cleanup
> +
Same as above: there is no reason to name it both 'remove_images' and
'perf_cleanup'.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci
2020-02-18 21:41 ` Alexander Turenko
@ 2020-02-19 5:37 ` Alexander Tikhonov
2020-02-21 12:38 ` Alexander Turenko
0 siblings, 1 reply; 5+ messages in thread
From: Alexander Tikhonov @ 2020-02-19 5:37 UTC (permalink / raw)
To: Alexander Turenko; +Cc: Oleg Piskunov, tarantool-patches
[-- Attachment #1: Type: text/plain, Size: 5048 bytes --]
Alexander, thanks a lot for a deep review, I've answered the questions below.
>Среда, 19 февраля 2020, 0:41 +03:00 от Alexander Turenko <alexander.turenko@tarantool.org>:
>
>I still think that details might be better, but okay, I see: you need
>some base now to proceed further. Don't want to block it anymore.
Right, thanks a lot.
>
>
>I commented the patch below, but didn't perform any changes except a bit
>reworded commit message.
>
>Pushed to master. CCed Kirill.
>
>Don't sure how it should look at other release branches:
>
>- Whether something need to be changed for 2.3/2.2?
> - perf_only_template should have "2.3" / "2.2" branch instead of
> master, that I understood.
Right, that is the only change.
>
> - Should IMAGE_PERF be tagged as "perf_2.3" / "perf_2.2" instead of
> "perf_master"? Are benchmarks should be adjusted for those versions
> and should this lead to such separation base images (or will be
> handled at runtime with bench-run scripts?).
No, there is no any need to have separate IMAGE_PERF images, due to it
has only built benchmarks w/o depends to Tarantool sources.
>
>- Whether something need to be changed for 1.10?
> - At least SQL benchmarks will not work. Should it be handled here or
> they will be skipped on bench-run side?
> - Same question re IMAGE_PERF as above.
Right, it needs special Dockerfile w/o SQL benchmarks, I already have
separate 'avtikhon/gitlab-ci_1.10-perf' branch for it, I'll update it with changes
from the current commit.
>
>
>Let's elaborate those questions. After this we can push it downward.
>
>WBR, Alexander Turenko.
>
>> Implement perf testing at gitlab-ci
>
>Changed to: 'gitlab-ci: enable performance testing'.
Ok.
>
>
>>
>> Enabled Tarantool performance testing on Gitlab-CI
>> for release/master branches and "*-perf" named branches.
>> For this purpose 'perf' and 'cleanup' stages were added
>> into Gitlab-CI pipeline.
>>
>> Performance testing support next benchmarks:
>> - cbench
>> - linkbench
>> - nosqlbench (hash and tree Tarantool run modes)
>> - sysbench
>> - tpcc
>> - ycsb (hash and tree Tarantool run modes)
>>
>> Benchmarks use scripts from repository:
>> http://gitlab.com/tarantool/bench-run
>
>Dead link. Changed gitlab.com to github.com.
Right, thanks.
>
>
>>
>> Perfomance testing uses docker images, built
>
>Fixed typo: 'Perfomance'.
Right, thanks.
>
>> with docker files from bench-run repository:
>> - perf/ubuntu-bionic:perf_master
>> parent image with benchmarks only
>> - perf_tmp/ubuntu-bionic:perf_<commit_SHA>
>> child images used for testing Tarantool sources
>
>Formatted a bit (to fit 72 symbols, but not much less).
Ok.
>
>
>> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gitlab-ci-perf
>
>> +.perf_only_template: &perf_only_definition
>> + only:
>> + - master
>> + - /^.*-perf$/
>> + variables: &perf_vars_definition
>> + IMAGE_PERF: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf/ubuntu-bionic:perf_master"
>> + IMAGE_PERF_BUILT: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf_tmp/ubuntu-bionic:perf_${CI_COMMIT_SHORT_SHA}"
>> +
>
>The resulting bench-run API looks strage for me:
>
>* It expects that a caller will set CI_REGISTRY, CI_REGISTRY_USER,
> CI_REGISTRY_PASSWORD environment variables, which come from GitLab-CI,
> but can be set manually.
These variables are in use for docker registry login at bench-run scripts,
these variables recreating each job run by gitlab-ci.
>
>* However it does not use CI_REGISTRY, CI_PROJECT_PATH,
> CI_COMMIT_SHORT_SHA to choose images name on its own, but expect
> IMAGE_PERF and IMAGE_PERF_BUILT from a caller.
Gitlab-ci initiates in .gitlab-ci.yml jobs running under docker images, so right
here the performance image IMAGE_PERF_BUILT must be set, IMAGE_PERF
is the part of the image IMAGE_PERF_BUILT and it's setup better to have at the
same place as IMAGE_PERF image.
>
>* All those variables have prefix CI_*, not, say, BENCH_RUN_*.
CI_* are gitlab-ci variables and visible to the bench-run make targets - no need
to setup additional variables.
>
>
>> +# Pre-testing part
>> +
>> +perf_bootstrap:
>> + <<: *perf_only_definition
>> + stage: test
>> + tags:
>> + - perf
>> + script:
>> + - ${GITLAB_MAKE} perf_prepare
>
>There is no reason to use two terms for the same thing: bootstrap and
>prepare.
I'm ok with your suggestion, let's do it on the next commit iteration.
>
>
>Also I don't see a reason to extract such one-two-liners into a
>gitlab.mk.
Previously we decided not to use .gitlab-ci.yml for code and to use it from
standalone make files, anyway I'm Ok with your suggestion, let's discuss
it a bit.
>
>> +# Post-testing part
>> +
>> +remove_images:
>> + <<: *perf_only_definition
>> + stage: cleanup
>> + when: always
>> + tags:
>> + - perf
>> + script:
>> + - ${GITLAB_MAKE} perf_cleanup
>> +
>
>Same as above: there is no reason to name it both 'remove_images' and
>'perf_cleanup'.
Ok, the same answer as above.
--
Alexander Tikhonov
[-- Attachment #2: Type: text/html, Size: 9052 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci
2020-02-19 5:37 ` Alexander Tikhonov
@ 2020-02-21 12:38 ` Alexander Turenko
0 siblings, 0 replies; 5+ messages in thread
From: Alexander Turenko @ 2020-02-21 12:38 UTC (permalink / raw)
To: Alexander Tikhonov; +Cc: Oleg Piskunov, tarantool-patches
> >Don't sure how it should look at other release branches:
> >
> >- Whether something need to be changed for 2.3/2.2?
> > - perf_only_template should have "2.3" / "2.2" branch instead of
> > master, that I understood.
> Right, that is the only change.
Pushed to 2.3 and 2.2.
> Right, it needs special Dockerfile w/o SQL benchmarks, I already have
> separate 'avtikhon/gitlab-ci_1.10-perf' branch for it, I'll update it with changes
> from the current commit.
Pushed to 1.10.
^ permalink raw reply [flat|nested] 5+ messages in thread
* [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci
@ 2020-02-17 10:00 Alexander V. Tikhonov
0 siblings, 0 replies; 5+ messages in thread
From: Alexander V. Tikhonov @ 2020-02-17 10:00 UTC (permalink / raw)
To: Oleg Piskunov; +Cc: tarantool-patches
Enabled Tarantool performance testing on Gitlab-CI
for release/master branches and "*-perf" named branches.
For this purpose 'perf' and 'cleanup' stages were added
into Gitlab-CI pipeline.
Performance testing support next benchmarks:
- cbench
- linkbench
- nosqlbench (hash and tree Tarantool run modes)
- sysbench
- tpcc
- ycsb (hash and tree Tarantool run modes)
Benchmarks use scripts from repository:
http://gitlab.com/tarantool/bench-run
Perfomance testing uses docker images, built
with docker files from bench-run repository:
- perf/ubuntu-bionic:perf_master
parent image with benchmarks only
- perf_tmp/ubuntu-bionic:perf_<commit_SHA>
child images used for testing Tarantool sources
---
Changes v4:
- moved the rest of Docker infrastructure for perf testing
into the bench-run repository
- renamed calls to performance prepare/run/cleanup stages
Changes v3:
- updated commit message
- merged all make targets for perf testing into single
- set to use variables in gitlab-ci for running perf testings
Changes v2:
- moved performance variables from global setup
to performance template
- updated commit message
- added more comments to sources
.gitlab-ci.yml | 100 +++++++++++++++++++++++++++++++++++++++++++++++++
.gitlab.mk | 16 ++++++++
2 files changed, 116 insertions(+)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 727b7c94e..1bc53d0fb 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,7 @@
stages:
- test
+ - perf
+ - cleanup
variables:
GITLAB_MAKE: "make -f .gitlab.mk"
@@ -26,6 +28,14 @@ variables:
- merge_requests
- /^.*-full-ci$/
+.perf_only_template: &perf_only_definition
+ only:
+ - master
+ - /^.*-perf$/
+ variables: &perf_vars_definition
+ IMAGE_PERF: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf/ubuntu-bionic:perf_master"
+ IMAGE_PERF_BUILT: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf_tmp/ubuntu-bionic:perf_${CI_COMMIT_SHORT_SHA}"
+
.docker_test_template: &docker_test_definition
image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-stretch:latest"
stage: test
@@ -77,6 +87,15 @@ variables:
after_script:
- ${GITLAB_MAKE} vms_shutdown
+.perf_docker_test_template: &perf_docker_test_definition
+ <<: *perf_only_definition
+ image: ${IMAGE_PERF_BUILT}
+ stage: perf
+ tags:
+ - docker_perf
+ script:
+ - ${GITLAB_MAKE} perf_run
+
# Tests
release:
@@ -168,6 +187,87 @@ freebsd_12_release:
script:
- ${GITLAB_MAKE} vms_test_freebsd
+# ####
+# Perf
+# ####
+
+# Pre-testing part
+
+perf_bootstrap:
+ <<: *perf_only_definition
+ stage: test
+ tags:
+ - perf
+ script:
+ - ${GITLAB_MAKE} perf_prepare
+
+# Testing part
+
+perf_sysbench:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'sysbench'
+
+perf_tpcc:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'tpcc'
+
+perf_ycsb_hash:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'ycsb'
+ ARG: 'hash'
+
+perf_ycsb_tree:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'ycsb'
+ ARG: 'tree'
+
+perf_nosqlbench_hash:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'nosqlbench'
+ ARG: 'hash'
+
+perf_nosqlbench_tree:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'nosqlbench'
+ ARG: 'tree'
+
+perf_cbench:
+ <<: *perf_docker_test_definition
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'cbench'
+
+perf_linkbench_ssd:
+ <<: *perf_docker_test_definition
+ tags:
+ - docker_perf_ssd
+ variables:
+ <<: *perf_vars_definition
+ BENCH: 'linkbench'
+
+# Post-testing part
+
+remove_images:
+ <<: *perf_only_definition
+ stage: cleanup
+ when: always
+ tags:
+ - perf
+ script:
+ - ${GITLAB_MAKE} perf_cleanup
+
# Packs
centos_6:
diff --git a/.gitlab.mk b/.gitlab.mk
index 24b18b7f2..bc37ed7aa 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -78,6 +78,15 @@ docker_bootstrap:
docker push ${DEBIAN_STRETCH_IMAGE}:latest
docker push ${DEBIAN_BUSTER_IMAGE}:latest
+# Build images for performance testing
+perf_prepare:
+ git clone https://github.com/tarantool/bench-run.git
+ make -f bench-run/targets.mk prepare
+
+# Remove temporary performance image from the test host
+perf_cleanup:
+ make -f bench-run/targets.mk cleanup
+
# #################################
# Run tests under a virtual machine
# #################################
@@ -126,3 +135,10 @@ deploy: package
static_build:
docker build --network=host --build-arg RUN_TESTS="${RUN_TESTS}" -f Dockerfile.staticbuild .
+
+# ###################
+# Performance testing
+# ###################
+
+perf_run:
+ /opt/bench-run/benchs/${BENCH}/run.sh ${ARG}
--
2.17.1
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-02-21 12:38 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-17 14:01 [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci Alexander V. Tikhonov
2020-02-18 21:41 ` Alexander Turenko
2020-02-19 5:37 ` Alexander Tikhonov
2020-02-21 12:38 ` Alexander Turenko
-- strict thread matches above, loose matches on Subject: below --
2020-02-17 10:00 Alexander V. Tikhonov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox