[Tarantool-patches] [PATCH 0/4] Integrate Jepsen tests to CI

[Tarantool-patches] [PATCH 0/4] Integrate Jepsen tests to CI

[sergeyb]

From: Sergey Bronnikov <sergeyb@tarantool.org>

Patch series adds Terraform configs to setup test stand in a cloud
infrastructure, script to run Jepsen tests, separate targets to
Makefile.

Issue: https://github.com/tarantool/tarantool/issues/5277
CI: https://gitlab.com/tarantool/tarantool/-/pipelines/190225147

Sergey Bronnikov (4):
  extra: add Terraform config files
  cmake: add targets to run Jepsen tests
  tools: add script to run Jepsen tests
  ci: integrate Jepsen tests to GitLab CI

 .gitlab-ci.yml            | 14 +++++++
 .travis.mk                | 34 ++++++++++++++++-
 CMakeLists.txt            | 27 ++++++++++++++
 extra/tf/main.tf          | 43 ++++++++++++++++++++++
 extra/tf/outputs.tf       | 10 +++++
 extra/tf/provider.tf      | 49 +++++++++++++++++++++++++
 extra/tf/versions.tf      |  8 ++++
 tools/run-jepsen-tests.sh | 77 +++++++++++++++++++++++++++++++++++++++
 8 files changed, 261 insertions(+), 1 deletion(-)
 create mode 100644 extra/tf/main.tf
 create mode 100644 extra/tf/outputs.tf
 create mode 100644 extra/tf/provider.tf
 create mode 100644 extra/tf/versions.tf
 create mode 100755 tools/run-jepsen-tests.sh

-- 
2.25.1

[Tarantool-patches] [PATCH 1/4] extra: add Terraform config files

[sergeyb]

From: Sergey Bronnikov <sergeyb@tarantool.org>

For testing Tarantool with Jepsen we use virtual machines as they provides
better resource isolation in comparison to containers. Jepsen tests may need a
single instance or a set of instances for testing cluster.  To setup virtual
machines we use Terraform [1]. Patch adds a set of configuration files for
Terraform that can create required number of virtual machines in MCS and output
IP addresses to stdout.

Terraform needs some parameters before run. They are:

- id, identificator of a test stand that should be specific for this run, id
also is a part of virtual machine name
- keypair_name, name of keypair used in a cloud, public SSH key of that key pair
will be placed to virtual machine
- instance_count, number of virtual machines in a test stand
- ssh_key, SSH private key, used to access to a virtual machine
- user_name
- password
- tenant_id
- user_domain_id

These parameters can be passed via enviroment variables with TF_VAR_ prefix
(like TF_VAR_id) or via command-line parameters.

To demonstrate full lifecycle of a test stand with Terraform one needs to
perform these commands:

terraform init extra/tf
terraform apply extra/tf
terraform output instance_names
terraform output instance_ips
terraform destroy extra/tf

1. https://www.terraform.io/

Part of #5277
---
 extra/tf/main.tf     | 43 ++++++++++++++++++++++++++++++++++++++
 extra/tf/outputs.tf  | 10 +++++++++
 extra/tf/provider.tf | 49 ++++++++++++++++++++++++++++++++++++++++++++
 extra/tf/versions.tf |  8 ++++++++
 4 files changed, 110 insertions(+)
 create mode 100644 extra/tf/main.tf
 create mode 100644 extra/tf/outputs.tf
 create mode 100644 extra/tf/provider.tf
 create mode 100644 extra/tf/versions.tf

diff --git a/extra/tf/main.tf b/extra/tf/main.tf
new file mode 100644
index 000000000..0b760884e
--- /dev/null
+++ b/extra/tf/main.tf
@@ -0,0 +1,43 @@
+resource "openstack_compute_instance_v2" "instance" {
+  count = var.instance_count
+
+  name = "tarantool_testing-${count.index + 1}-${var.id}"
+
+  image_name = "Ubuntu-18.04-202003"
+
+  image_id = "3f03b393-d45b-455d-bdf5-0ac4399ecf42"
+
+  flavor_name = "Basic-1-2-20"
+
+  key_pair = var.keypair_name
+
+  config_drive = true
+
+  availability_zone = "DP1"
+
+  security_groups = [
+    "default",
+    "allow-ssh",
+    "allow-tarantool"
+  ]
+
+  network {
+    name = "ext-net"
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "sudo hostnamectl set-hostname n${count.index + 1}",
+      "sudo apt-get update"
+    ]
+  }
+
+  connection {
+    host = self.access_ip_v4
+    type = "ssh"
+    user = "ubuntu"
+    timeout = "5m"
+    agent = true
+    private_key = file(var.ssh_key_path)
+  }
+}
diff --git a/extra/tf/outputs.tf b/extra/tf/outputs.tf
new file mode 100644
index 000000000..5690d6b0d
--- /dev/null
+++ b/extra/tf/outputs.tf
@@ -0,0 +1,10 @@
+output "instance_names" {
+  description = "List of names assigned to the instances."
+  value = openstack_compute_instance_v2.instance.*.name
+}
+
+output "instance_ips" {
+  description = "List of public IP addresses assigned to the instances."
+  value = openstack_compute_instance_v2.instance.*.access_ip_v4
+  sensitive = true
+}
diff --git a/extra/tf/provider.tf b/extra/tf/provider.tf
new file mode 100644
index 000000000..cd9488ce5
--- /dev/null
+++ b/extra/tf/provider.tf
@@ -0,0 +1,49 @@
+variable "user_name" {
+  type = string
+}
+
+variable "password" {
+  type = string
+}
+
+variable "tenant_id" {
+  type = string
+}
+
+variable "user_domain_id" {
+  type = string
+}
+
+variable "keypair_name" {
+  type = string
+}
+
+variable "ssh_key_path" {
+  type = string
+}
+
+variable "instance_count" {
+  type = number
+  default = 1
+  description = "Number of instances."
+
+  validation {
+    condition = var.instance_count > 0
+    error_message = "The instance_count value must be greater than zero."
+  }
+}
+
+variable "id" {
+  type = string
+  default = null
+}
+
+provider "openstack" {
+  user_name = var.user_name
+  password = var.password
+  tenant_id = var.tenant_id
+  user_domain_id = var.user_domain_id
+  auth_url = "https://infra.mail.ru/identity/v3/"
+  region = "RegionOne"
+  use_octavia = true
+}
diff --git a/extra/tf/versions.tf b/extra/tf/versions.tf
new file mode 100644
index 000000000..5af3d608b
--- /dev/null
+++ b/extra/tf/versions.tf
@@ -0,0 +1,8 @@
+terraform {
+  required_providers {
+    openstack = {
+      source = "terraform-providers/openstack"
+    }
+  }
+  required_version = ">= 0.13"
+}
-- 
2.25.1

[Tarantool-patches] [PATCH 2/4] cmake: add targets to run Jepsen tests

[sergeyb]

From: Sergey Bronnikov <sergeyb@tarantool.org>

Added targets 'make jepsen-single' and 'make jepsen-cluster'
to run Jepsen tests on a single Tarantool instance and
cluster of Tarantool instances.

Part of #5277
---
 CMakeLists.txt | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 695e80c21..e8850e320 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,9 +17,11 @@ include(TestBigEndian)
 include(CheckFunctionExists)
 include(FindOptionalPackage)
 include(FindPackageMessage)
+include(ExternalProject)
 
 find_program(ECHO echo)
 find_program(CAT cat)
+find_program(BASH bash)
 find_program(GIT git)
 find_program(LD ld)
 find_program(CTAGS ctags)
@@ -162,6 +164,31 @@ add_custom_command(TARGET luacheck
     COMMENT "Perform static analysis of Lua code"
 )
 
+ExternalProject_Add(
+    jepsen-tests
+    GIT_REPOSITORY https://github.com/tarantool/jepsen.tarantool
+    CONFIGURE_COMMAND ""
+    BUILD_COMMAND ""
+    INSTALL_COMMAND ""
+    TEST_COMMAND ""
+)
+
+#
+# Enable 'make jepsen-*' targets.
+#
+
+add_custom_target(jepsen-single DEPENDS jepsen-tests)
+add_custom_command(TARGET jepsen-single
+    COMMAND ${BASH} ${PROJECT_SOURCE_DIR}/tools/run-jepsen-tests.sh ${PROJECT_SOURCE_DIR} ${PROJECT_BINARY_DIR}
+    COMMENT "Running Jepsen tests on a single Tarantool instance"
+)
+
+add_custom_target(jepsen-cluster DEPENDS jepsen-tests)
+add_custom_command(TARGET jepsen-cluster
+    COMMAND ${BASH} ${PROJECT_SOURCE_DIR}/tools/run-jepsen-tests.sh ${PROJECT_SOURCE_DIR} ${PROJECT_BINARY_DIR} 5
+    COMMENT "Running Jepsen tests on a cluster with 5 Tarantool instances"
+)
+
 #
 # Get version
 #
-- 
2.25.1

[Tarantool-patches] [PATCH 3/4] tools: add script to run Jepsen tests

[sergeyb]

From: Sergey Bronnikov <sergeyb@tarantool.org>

Main script that handle creation of set of virtual machines
using Terraform, setup for remote connection, running
Jepsen tests and teardown test environment.

Part of #5277
---
 tools/run-jepsen-tests.sh | 77 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100755 tools/run-jepsen-tests.sh

diff --git a/tools/run-jepsen-tests.sh b/tools/run-jepsen-tests.sh
new file mode 100755
index 000000000..865a8f4b0
--- /dev/null
+++ b/tools/run-jepsen-tests.sh
@@ -0,0 +1,77 @@
+#!/usr/bin/bash
+
+# Script performs setup of test environment using Terraform,
+# runs Jepsen tests and teardown test environment.
+# Script expects following environment variables:
+# 	TF_VAR_user_name
+# 	TF_VAR_password
+# 	TF_VAR_tenant_id
+# 	TF_VAR_user_domain_id
+# 	TF_VAR_keypair_name
+# 	TF_VAR_ssh_key
+# and three arguments: path to a Tarantool project source directory, path to a
+# Tarantool project binary directory and number on instances (optional).
+
+set -Eeo pipefail
+
+PROJECT_SOURCE_DIR=$1
+PROJECT_BINARY_DIR=$2
+INSTANCE_COUNT=$3
+
+TESTS_DIR="$PROJECT_BINARY_DIR/jepsen-tests-prefix/src/jepsen-tests/"
+TERRAFORM_CONFIG="$PROJECT_SOURCE_DIR/extra/tf"
+TERRAFORM_STATE="$PROJECT_BINARY_DIR/terraform.tfstate"
+SSH_KEY_FILENAME="$PROJECT_SOURCE_DIR/build/tf-cloud-init"
+NODES_FILENAME="$PROJECT_SOURCE_DIR/build/nodes"
+
+LEIN_OPTIONS="--nodes-file $NODES_FILENAME --username ubuntu --workload register"
+
+[[ -z ${PROJECT_SOURCE_DIR} ]] && (echo "Please specify path to a project source directory")
+[[ -z ${PROJECT_BINARY_DIR} ]] && (echo "Please specify path to a project binary directory")
+[[ -z ${TF_VAR_ssh_key} ]] && (echo "Please specify TF_VAR_ssh_key env var"; exit 1)
+[[ -z ${TF_VAR_keypair_name} ]] && (echo "Please specify TF_VAR_keypair_name env var"; exit 1)
+
+TERRAFORM_BIN=$(which terraform)
+LEIN_BIN=$(which lein)
+CLOJURE_BIN=$(which clojure)
+
+[[ -z ${TERRAFORM_BIN} ]] && (echo "terraform is not installed"; exit 1)
+[[ -z ${LEIN_BIN} ]] && (echo "lein is not installed"; exit 1)
+[[ -z ${CLOJURE_BIN} ]] && (echo "clojure is not installed"; exit 1)
+
+function cleanup {
+    echo "cleanup"
+    rm -f $NODES_FILENAME $SSH_KEY_FILENAME
+    [[ -e $TERRAFORM_STATE ]] && \
+	terraform destroy -state=$TERRAFORM_STATE -auto-approve $TERRAFORM_CONFIG
+}
+
+trap "{ cleanup; exit 255; }" SIGINT SIGTERM ERR
+
+echo -e "${TF_VAR_ssh_key//_/\\n}" > $SSH_KEY_FILENAME
+chmod 400 $SSH_KEY_FILENAME
+$(pgrep ssh-agent 2>&1 > /dev/null) || eval "$(ssh-agent)"
+ssh-add $SSH_KEY_FILENAME
+
+if [[ -z ${CI_JOB_ID} ]]; then
+    TF_VAR_id=$CI_JOB_ID
+else
+    RANDOM_ID=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13; echo '')
+    TF_VAR_id=TF-$RANDOM_ID
+fi
+
+export TF_VAR_id
+export TF_VAR_ssh_key_path=$SSH_KEY_FILENAME
+[[ -z ${INSTANCE_COUNT} ]] && TF_VAR_instance_count=1 || TF_VAR_instance_count=$INSTANCE_COUNT
+export TF_VAR_instance_count
+
+[[ -e $HOME/.ssh ]] || (mkdir $HOME/.ssh && touch $HOME/.ssh/known_hosts)
+
+terraform init $TERRAFORM_CONFIG
+terraform apply -state=$TERRAFORM_STATE -auto-approve $TERRAFORM_CONFIG
+terraform providers $TERRAFORM_CONFIG
+terraform output -state=$TERRAFORM_STATE instance_names
+terraform output -state=$TERRAFORM_STATE -json instance_ips | jq --raw-output '.[]' > $NODES_FILENAME
+pushd $TESTS_DIR && lein run test $LEIN_OPTIONS
+popd
+cleanup
-- 
2.25.1

[Tarantool-patches] [PATCH 4/4] ci: integrate Jepsen tests to GitLab CI

[sergeyb]

From: Sergey Bronnikov <sergeyb@tarantool.org>

added a new stage with a single job to run Jepsen tests.
Job is not started automatically by default, one need to
trigger it manually. Directory with test results
(logs, graphs, operations history) published to artifacts.

Closes #5277
---
 .gitlab-ci.yml | 14 ++++++++++++++
 .travis.mk     | 34 +++++++++++++++++++++++++++++++++-
 2 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 05d40b013..3afcfc245 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,6 +1,7 @@
 stages:
   - static_analysis
   - test
+  - long_tests
   - perf
   - cleanup
 
@@ -210,6 +211,19 @@ freebsd_12_release:
     - ${GITLAB_MAKE} vms_start
     - ${GITLAB_MAKE} vms_test_freebsd
 
+jepsen:
+  <<: *docker_test_definition
+  script:
+    - ${GITLAB_MAKE} test_jepsen
+  stage: long_tests
+  when: manual
+  tags:
+    - mcs_jepsen_docker
+  artifacts:
+    paths:
+      - jepsen-tests-prefix/src/jepsen-tests/store
+    expire_in: 1 week
+
 # ####
 # Perf
 # ####
diff --git a/.travis.mk b/.travis.mk
index efc05cf05..781b37c15 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -8,6 +8,12 @@ TEST_RUN_EXTRA_PARAMS?=
 MAX_FILES?=65534
 MAX_PROC?=2500
 OOS_SRC_PATH="/source"
+BIN_DIR=/usr/local/bin
+
+CLOJURE_URL="https://download.clojure.org/install/linux-install-1.10.1.561.sh"
+LEIN_URL="https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein"
+TERRAFORM_NAME="terraform_0.13.1_linux_amd64.zip"
+TERRAFORM_URL="https://releases.hashicorp.com/terraform/0.13.1/"$(TERRAFORM_NAME)
 
 all: package
 
@@ -60,7 +66,7 @@ docker_%:
 # commit, so the build requires old dependencies to be installed.
 # See ce623a23416eb192ce70116fd14992e84e7ccbbe ('Enable GitLab CI
 # testing') for more information.
-deps_debian:
+deps_debian: $(BIN_DIR)/clojure $(BIN_DIR)/lein $(BIN_DIR)/terraform
 	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
 		build-essential cmake coreutils sed \
 		libreadline-dev libncurses5-dev libyaml-dev libssl-dev \
@@ -69,6 +75,11 @@ deps_debian:
 		python-msgpack python-yaml python-argparse python-six python-gevent \
 		lcov ruby clang llvm llvm-dev zlib1g-dev autoconf automake libtool
 
+deps_debian_jepsen:
+	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
+		openjdk-8-jre openjdk-8-jre-headless libjna-java gnuplot graphviz \
+		zip unzip openssh-client jq
+
 deps_buster_clang_8: deps_debian
 	echo "deb http://apt.llvm.org/buster/ llvm-toolchain-buster-8 main" > /etc/apt/sources.list.d/clang_8.list
 	echo "deb-src http://apt.llvm.org/buster/ llvm-toolchain-buster-8 main" >> /etc/apt/sources.list.d/clang_8.list
@@ -76,6 +87,19 @@ deps_buster_clang_8: deps_debian
 	apt-get update
 	apt-get install -y clang-8 llvm-8-dev
 
+$(BIN_DIR)/clojure: deps_debian_jepsen
+	curl $(CLOJURE_URL) | sudo bash
+
+$(BIN_DIR)/lein: deps_debian_jepsen
+	curl $(LEIN_URL) > $@
+	chmod a+x $@
+	$@ version
+
+$(BIN_DIR)/terraform: deps_debian_jepsen
+	curl -O $(TERRAFORM_URL)
+	unzip -o $(TERRAFORM_NAME) terraform -d $(dir $@)
+	rm -f $(TERRAFORM_NAME)
+
 # Release
 
 configure_debian:
@@ -232,3 +256,11 @@ test_freebsd_no_deps: build_freebsd
 	cd test && python2.7 test-run.py --force $(TEST_RUN_EXTRA_PARAMS)
 
 test_freebsd: deps_freebsd test_freebsd_no_deps
+
+# ###################
+# Jepsen testing
+# ###################
+
+test_jepsen: deps_debian
+	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON
+	make jepsen-single
-- 
2.25.1

Re: [Tarantool-patches] [PATCH 4/4] ci: integrate Jepsen tests to GitLab CI

[Alexander V. Tikhonov]

Hi Sergey, thanks for the patch, please check my comments below.

On Wed, Sep 16, 2020 at 10:07:24AM +0300, sergeyb@tarantool.org wrote:
> From: Sergey Bronnikov <sergeyb@tarantool.org>
> 
> added a new stage with a single job to run Jepsen tests.
> Job is not started automatically by default, one need to
> trigger it manually. Directory with test results
> (logs, graphs, operations history) published to artifacts.
> 
> Closes #5277
> ---
>  .gitlab-ci.yml | 14 ++++++++++++++
>  .travis.mk     | 34 +++++++++++++++++++++++++++++++++-
>  2 files changed, 47 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 05d40b013..3afcfc245 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,6 +1,7 @@
>  stages:
>    - static_analysis
>    - test
> +  - long_tests
>    - perf
>    - cleanup
>

May be, we may want to merge long_tests and perf stages to some single
common stage ?

> @@ -210,6 +211,19 @@ freebsd_12_release:
>      - ${GITLAB_MAKE} vms_start
>      - ${GITLAB_MAKE} vms_test_freebsd
>  
> +jepsen:
> +  <<: *docker_test_definition
> +  script:
> +    - ${GITLAB_MAKE} test_jepsen
> +  stage: long_tests
> +  when: manual
> +  tags:
> +    - mcs_jepsen_docker
> +  artifacts:
> +    paths:
> +      - jepsen-tests-prefix/src/jepsen-tests/store
> +    expire_in: 1 week
> +

Let's use more time for artifacts store, may be 6 months.

>  # ####
>  # Perf
>  # ####
> diff --git a/.travis.mk b/.travis.mk
> index efc05cf05..781b37c15 100644
> --- a/.travis.mk
> +++ b/.travis.mk
> @@ -8,6 +8,12 @@ TEST_RUN_EXTRA_PARAMS?=
>  MAX_FILES?=65534
>  MAX_PROC?=2500
>  OOS_SRC_PATH="/source"
> +BIN_DIR=/usr/local/bin
> +
> +CLOJURE_URL="https://download.clojure.org/install/linux-install-1.10.1.561.sh"
> +LEIN_URL="https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein"
> +TERRAFORM_NAME="terraform_0.13.1_linux_amd64.zip"
> +TERRAFORM_URL="https://releases.hashicorp.com/terraform/0.13.1/"$(TERRAFORM_NAME)
>  
>  all: package
>  
> @@ -60,7 +66,7 @@ docker_%:
>  # commit, so the build requires old dependencies to be installed.
>  # See ce623a23416eb192ce70116fd14992e84e7ccbbe ('Enable GitLab CI
>  # testing') for more information.
> -deps_debian:
> +deps_debian: $(BIN_DIR)/clojure $(BIN_DIR)/lein $(BIN_DIR)/terraform
>  	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
>  		build-essential cmake coreutils sed \
>  		libreadline-dev libncurses5-dev libyaml-dev libssl-dev \

I don't think we need to add jepsen's new targets to the base one, let's
move it to 'deps_debian_jepsen'.

> @@ -69,6 +75,11 @@ deps_debian:
>  		python-msgpack python-yaml python-argparse python-six python-gevent \
>  		lcov ruby clang llvm llvm-dev zlib1g-dev autoconf automake libtool
>  
> +deps_debian_jepsen:
> +	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
> +		openjdk-8-jre openjdk-8-jre-headless libjna-java gnuplot graphviz \
> +		zip unzip openssh-client jq
> +
>  deps_buster_clang_8: deps_debian
>  	echo "deb http://apt.llvm.org/buster/ llvm-toolchain-buster-8 main" > /etc/apt/sources.list.d/clang_8.list
>  	echo "deb-src http://apt.llvm.org/buster/ llvm-toolchain-buster-8 main" >> /etc/apt/sources.list.d/clang_8.list
> @@ -76,6 +87,19 @@ deps_buster_clang_8: deps_debian
>  	apt-get update
>  	apt-get install -y clang-8 llvm-8-dev
>  
> +$(BIN_DIR)/clojure: deps_debian_jepsen
> +	curl $(CLOJURE_URL) | sudo bash
> +
> +$(BIN_DIR)/lein: deps_debian_jepsen
> +	curl $(LEIN_URL) > $@
> +	chmod a+x $@
> +	$@ version
> +
> +$(BIN_DIR)/terraform: deps_debian_jepsen
> +	curl -O $(TERRAFORM_URL)
> +	unzip -o $(TERRAFORM_NAME) terraform -d $(dir $@)
> +	rm -f $(TERRAFORM_NAME)
> +

Do we really need standalone targets that will not be used in separate?
This file is in makefile style, but not makefile in real, it has
standalone targets that provides some process meanings, like:
 - prepare deps
 - build
 - test

>  # Release
>  
>  configure_debian:
> @@ -232,3 +256,11 @@ test_freebsd_no_deps: build_freebsd
>  	cd test && python2.7 test-run.py --force $(TEST_RUN_EXTRA_PARAMS)
>  
>  test_freebsd: deps_freebsd test_freebsd_no_deps
> +
> +# ###################
> +# Jepsen testing
> +# ###################
> +
> +test_jepsen: deps_debian
> +	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON
> +	make jepsen-single

Let's use deps_debian_jepsen here.

> -- 
> 2.25.1
> 

Re: [Tarantool-patches] [PATCH 4/4] ci: integrate Jepsen tests to GitLab CI

[Alexander Turenko]

> > +jepsen:
> > +  <<: *docker_test_definition
> > +  script:
> > +    - ${GITLAB_MAKE} test_jepsen
> > +  stage: long_tests
> > +  when: manual
> > +  tags:
> > +    - mcs_jepsen_docker
> > +  artifacts:
> > +    paths:
> > +      - jepsen-tests-prefix/src/jepsen-tests/store
> > +    expire_in: 1 week
> > +
> 
> Let's use more time for artifacts store, may be 6 months.

I agree. We sometimes return to bad results after some significant
delay.

> > -deps_debian:
> > +deps_debian: $(BIN_DIR)/clojure $(BIN_DIR)/lein $(BIN_DIR)/terraform
> >  	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
> >  		build-essential cmake coreutils sed \
> >  		libreadline-dev libncurses5-dev libyaml-dev libssl-dev \
> 
> I don't think we need to add jepsen's new targets to the base one, let's
> move it to 'deps_debian_jepsen'.

I agree, it is a bit strange to install jepsen dependencies for all jobs
(when a separate jepsen goal exists at least).

> > +deps_debian_jepsen:
> > +	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
> > +		openjdk-8-jre openjdk-8-jre-headless libjna-java gnuplot graphviz \
> > +		zip unzip openssh-client jq
> > +

> > +$(BIN_DIR)/clojure: deps_debian_jepsen
> > +	curl $(CLOJURE_URL) | sudo bash
> > +
> > +$(BIN_DIR)/lein: deps_debian_jepsen
> > +	curl $(LEIN_URL) > $@
> > +	chmod a+x $@
> > +	$@ version
> > +
> > +$(BIN_DIR)/terraform: deps_debian_jepsen
> > +	curl -O $(TERRAFORM_URL)
> > +	unzip -o $(TERRAFORM_NAME) terraform -d $(dir $@)
> > +	rm -f $(TERRAFORM_NAME)
> > +
> 
> Do we really need standalone targets that will not be used in separate?
> This file is in makefile style, but not makefile in real, it has
> standalone targets that provides some process meanings, like:
>  - prepare deps
>  - build
>  - test

In my understanding it is Makefile anyway and it is usual for a Makefile
to use a file name as a goal (so the goal will be considered as made if
the file exists, if the goal is not marked as .PHONY).

> > +
> > +# ###################
> > +# Jepsen testing
> > +# ###################
> > +
> > +test_jepsen: deps_debian
> > +	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON
> > +	make jepsen-single
> 
> Let's use deps_debian_jepsen here.

I would do it this way, yep.

Re: [Tarantool-patches] [PATCH 4/4] ci: integrate Jepsen tests to GitLab CI

[Sergey Bronnikov]

Hello!

Thanks for review! I made some changes according to your comments

and pushed them to the branch.

CI: https://gitlab.com/tarantool/tarantool/-/pipelines/191621859


On 17.09.2020 18:25, Alexander Turenko wrote:
>>> +jepsen:
>>> +  <<: *docker_test_definition
>>> +  script:
>>> +    - ${GITLAB_MAKE} test_jepsen
>>> +  stage: long_tests
>>> +  when: manual
>>> +  tags:
>>> +    - mcs_jepsen_docker
>>> +  artifacts:
>>> +    paths:
>>> +      - jepsen-tests-prefix/src/jepsen-tests/store
>>> +    expire_in: 1 week
>>> +
>> Let's use more time for artifacts store, may be 6 months.
> I agree. We sometimes return to bad results after some significant
> delay.

increased to 6 months:

--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -222,7 +222,7 @@ jepsen:
    artifacts:
      paths:
        - jepsen-tests-prefix/src/jepsen-tests/store
-    expire_in: 1 week
+    expire_in: 6 month

  # ####
  # Perf

>
>>> -deps_debian:
>>> +deps_debian: $(BIN_DIR)/clojure $(BIN_DIR)/lein $(BIN_DIR)/terraform
>>>   	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
>>>   		build-essential cmake coreutils sed \
>>>   		libreadline-dev libncurses5-dev libyaml-dev libssl-dev \
>> I don't think we need to add jepsen's new targets to the base one, let's
>> move it to 'deps_debian_jepsen'.
> I agree, it is a bit strange to install jepsen dependencies for all jobs
> (when a separate jepsen goal exists at least).
>
Agree. Removed jepsen dependencies for deps_debian target.


>>> +deps_debian_jepsen:
>>> +	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
>>> +		openjdk-8-jre openjdk-8-jre-headless libjna-java gnuplot graphviz \
>>> +		zip unzip openssh-client jq
>>> +
>>> +$(BIN_DIR)/clojure: deps_debian_jepsen
>>> +	curl $(CLOJURE_URL) | sudo bash
>>> +
>>> +$(BIN_DIR)/lein: deps_debian_jepsen
>>> +	curl $(LEIN_URL) > $@
>>> +	chmod a+x $@
>>> +	$@ version
>>> +
>>> +$(BIN_DIR)/terraform: deps_debian_jepsen
>>> +	curl -O $(TERRAFORM_URL)
>>> +	unzip -o $(TERRAFORM_NAME) terraform -d $(dir $@)
>>> +	rm -f $(TERRAFORM_NAME)
>>> +
>> Do we really need standalone targets that will not be used in separate?
>> This file is in makefile style, but not makefile in real, it has
>> standalone targets that provides some process meanings, like:
>>   - prepare deps
>>   - build
>>   - test
> In my understanding it is Makefile anyway and it is usual for a Makefile
> to use a file name as a goal (so the goal will be considered as made if
> the file exists, if the goal is not marked as .PHONY).
>
>>> +
>>> +# ###################
>>> +# Jepsen testing
>>> +# ###################
>>> +
>>> +test_jepsen: deps_debian
>>> +	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON
>>> +	make jepsen-single
>> Let's use deps_debian_jepsen here.

Done.


> I would do it this way, yep.

Re: [Tarantool-patches] [PATCH 4/4] ci: integrate Jepsen tests to GitLab CI

[Sergey Bronnikov]

Hello!

Thanks for review!

Please see my comments inline. I made some changes and pushed them to 
the branch.

CI: https://gitlab.com/tarantool/tarantool/-/pipelines/191621859

On 17.09.2020 10:12, Alexander V. Tikhonov wrote:
> Hi Sergey, thanks for the patch, please check my comments below.
>
> On Wed, Sep 16, 2020 at 10:07:24AM +0300, sergeyb@tarantool.org wrote:
>> From: Sergey Bronnikov <sergeyb@tarantool.org>
>>
>> added a new stage with a single job to run Jepsen tests.
>> Job is not started automatically by default, one need to
>> trigger it manually. Directory with test results
>> (logs, graphs, operations history) published to artifacts.
>>
>> Closes #5277
>> ---
>>   .gitlab-ci.yml | 14 ++++++++++++++
>>   .travis.mk     | 34 +++++++++++++++++++++++++++++++++-
>>   2 files changed, 47 insertions(+), 1 deletion(-)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 05d40b013..3afcfc245 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -1,6 +1,7 @@
>>   stages:
>>     - static_analysis
>>     - test
>> +  - long_tests
>>     - perf
>>     - cleanup
>>
> May be, we may want to merge long_tests and perf stages to some single
> common stage ?

Good idea. I believe it's a topic to discuss and we can combine stages

to a single one in scope of [1]. Right now performance jobs are not 
included to a pipeline by default.

1. https://github.com/tarantool/tarantool/issues/5305

>
>> @@ -210,6 +211,19 @@ freebsd_12_release:
>>       - ${GITLAB_MAKE} vms_start
>>       - ${GITLAB_MAKE} vms_test_freebsd
>>   
>> +jepsen:
>> +  <<: *docker_test_definition
>> +  script:
>> +    - ${GITLAB_MAKE} test_jepsen
>> +  stage: long_tests
>> +  when: manual
>> +  tags:
>> +    - mcs_jepsen_docker
>> +  artifacts:
>> +    paths:
>> +      - jepsen-tests-prefix/src/jepsen-tests/store
>> +    expire_in: 1 week
>> +
> Let's use more time for artifacts store, may be 6 months.
Updated in a branch.
>
>>   # ####
>>   # Perf
>>   # ####
>> diff --git a/.travis.mk b/.travis.mk
>> index efc05cf05..781b37c15 100644
>> --- a/.travis.mk
>> +++ b/.travis.mk
>> @@ -8,6 +8,12 @@ TEST_RUN_EXTRA_PARAMS?=
>>   MAX_FILES?=65534
>>   MAX_PROC?=2500
>>   OOS_SRC_PATH="/source"
>> +BIN_DIR=/usr/local/bin
>> +
>> +CLOJURE_URL="https://download.clojure.org/install/linux-install-1.10.1.561.sh"
>> +LEIN_URL="https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein"
>> +TERRAFORM_NAME="terraform_0.13.1_linux_amd64.zip"
>> +TERRAFORM_URL="https://releases.hashicorp.com/terraform/0.13.1/"$(TERRAFORM_NAME)
>>   
>>   all: package
>>   
>> @@ -60,7 +66,7 @@ docker_%:
>>   # commit, so the build requires old dependencies to be installed.
>>   # See ce623a23416eb192ce70116fd14992e84e7ccbbe ('Enable GitLab CI
>>   # testing') for more information.
>> -deps_debian:
>> +deps_debian: $(BIN_DIR)/clojure $(BIN_DIR)/lein $(BIN_DIR)/terraform
>>   	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
>>   		build-essential cmake coreutils sed \
>>   		libreadline-dev libncurses5-dev libyaml-dev libssl-dev \
> I don't think we need to add jepsen's new targets to the base one, let's
> move it to 'deps_debian_jepsen'.
Splitted targets.
>
>> @@ -69,6 +75,11 @@ deps_debian:
>>   		python-msgpack python-yaml python-argparse python-six python-gevent \
>>   		lcov ruby clang llvm llvm-dev zlib1g-dev autoconf automake libtool
>>   
>> +deps_debian_jepsen:
>> +	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
>> +		openjdk-8-jre openjdk-8-jre-headless libjna-java gnuplot graphviz \
>> +		zip unzip openssh-client jq
>> +
>>   deps_buster_clang_8: deps_debian
>>   	echo "deb http://apt.llvm.org/buster/ llvm-toolchain-buster-8 main" > /etc/apt/sources.list.d/clang_8.list
>>   	echo "deb-src http://apt.llvm.org/buster/ llvm-toolchain-buster-8 main" >> /etc/apt/sources.list.d/clang_8.list
>> @@ -76,6 +87,19 @@ deps_buster_clang_8: deps_debian
>>   	apt-get update
>>   	apt-get install -y clang-8 llvm-8-dev
>>   
>> +$(BIN_DIR)/clojure: deps_debian_jepsen
>> +	curl $(CLOJURE_URL) | sudo bash
>> +
>> +$(BIN_DIR)/lein: deps_debian_jepsen
>> +	curl $(LEIN_URL) > $@
>> +	chmod a+x $@
>> +	$@ version
>> +
>> +$(BIN_DIR)/terraform: deps_debian_jepsen
>> +	curl -O $(TERRAFORM_URL)
>> +	unzip -o $(TERRAFORM_NAME) terraform -d $(dir $@)
>> +	rm -f $(TERRAFORM_NAME)
>> +
> Do we really need standalone targets that will not be used in separate?
> This file is in makefile style, but not makefile in real, it has
> standalone targets that provides some process meanings, like:
>   - prepare deps
>   - build
>   - test

We discussed orally and decided to involve third reviewer

to come to agreement regarding this point. Alexander agreed that this 
set of target is ok.

>
>>   # Release
>>   
>>   configure_debian:
>> @@ -232,3 +256,11 @@ test_freebsd_no_deps: build_freebsd
>>   	cd test && python2.7 test-run.py --force $(TEST_RUN_EXTRA_PARAMS)
>>   
>>   test_freebsd: deps_freebsd test_freebsd_no_deps
>> +
>> +# ###################
>> +# Jepsen testing
>> +# ###################
>> +
>> +test_jepsen: deps_debian
>> +	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON
>> +	make jepsen-single
> Let's use deps_debian_jepsen here.

Done.


>> -- 
>> 2.25.1
>>

Re: [Tarantool-patches] [PATCH 4/4] ci: integrate Jepsen tests to GitLab CI

[Alexander V. Tikhonov]

Hi Sergey, thanks for the updates, patch LGTM.

On Fri, Sep 18, 2020 at 03:28:49PM +0300, Sergey Bronnikov wrote:
> Hello!
> 
> Thanks for review! I made some changes according to your comments
> 
> and pushed them to the branch.
> 
> CI: https://gitlab.com/tarantool/tarantool/-/pipelines/191621859
> 
> 
> On 17.09.2020 18:25, Alexander Turenko wrote:
> > > > +jepsen:
> > > > +  <<: *docker_test_definition
> > > > +  script:
> > > > +    - ${GITLAB_MAKE} test_jepsen
> > > > +  stage: long_tests
> > > > +  when: manual
> > > > +  tags:
> > > > +    - mcs_jepsen_docker
> > > > +  artifacts:
> > > > +    paths:
> > > > +      - jepsen-tests-prefix/src/jepsen-tests/store
> > > > +    expire_in: 1 week
> > > > +
> > > Let's use more time for artifacts store, may be 6 months.
> > I agree. We sometimes return to bad results after some significant
> > delay.
> 
> increased to 6 months:
> 
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -222,7 +222,7 @@ jepsen:
>    artifacts:
>      paths:
>        - jepsen-tests-prefix/src/jepsen-tests/store
> -    expire_in: 1 week
> +    expire_in: 6 month
> 
>  # ####
>  # Perf
> 
> > 
> > > > -deps_debian:
> > > > +deps_debian: $(BIN_DIR)/clojure $(BIN_DIR)/lein $(BIN_DIR)/terraform
> > > >   	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
> > > >   		build-essential cmake coreutils sed \
> > > >   		libreadline-dev libncurses5-dev libyaml-dev libssl-dev \
> > > I don't think we need to add jepsen's new targets to the base one, let's
> > > move it to 'deps_debian_jepsen'.
> > I agree, it is a bit strange to install jepsen dependencies for all jobs
> > (when a separate jepsen goal exists at least).
> > 
> Agree. Removed jepsen dependencies for deps_debian target.
> 
> 
> > > > +deps_debian_jepsen:
> > > > +	apt-get update ${APT_EXTRA_FLAGS} && apt-get install -y -f \
> > > > +		openjdk-8-jre openjdk-8-jre-headless libjna-java gnuplot graphviz \
> > > > +		zip unzip openssh-client jq
> > > > +
> > > > +$(BIN_DIR)/clojure: deps_debian_jepsen
> > > > +	curl $(CLOJURE_URL) | sudo bash
> > > > +
> > > > +$(BIN_DIR)/lein: deps_debian_jepsen
> > > > +	curl $(LEIN_URL) > $@
> > > > +	chmod a+x $@
> > > > +	$@ version
> > > > +
> > > > +$(BIN_DIR)/terraform: deps_debian_jepsen
> > > > +	curl -O $(TERRAFORM_URL)
> > > > +	unzip -o $(TERRAFORM_NAME) terraform -d $(dir $@)
> > > > +	rm -f $(TERRAFORM_NAME)
> > > > +
> > > Do we really need standalone targets that will not be used in separate?
> > > This file is in makefile style, but not makefile in real, it has
> > > standalone targets that provides some process meanings, like:
> > >   - prepare deps
> > >   - build
> > >   - test
> > In my understanding it is Makefile anyway and it is usual for a Makefile
> > to use a file name as a goal (so the goal will be considered as made if
> > the file exists, if the goal is not marked as .PHONY).
> > 
> > > > +
> > > > +# ###################
> > > > +# Jepsen testing
> > > > +# ###################
> > > > +
> > > > +test_jepsen: deps_debian
> > > > +	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON
> > > > +	make jepsen-single
> > > Let's use deps_debian_jepsen here.
> 
> Done.
> 
> 
> > I would do it this way, yep.

Re: [Tarantool-patches] [PATCH 1/4] extra: add Terraform config files

[Alexander V. Tikhonov]

Hi Sergey, patch has only configuration data and enough comments, so
patch LGTM.

On Wed, Sep 16, 2020 at 10:07:19AM +0300, sergeyb@tarantool.org wrote:
> From: Sergey Bronnikov <sergeyb@tarantool.org>
> 
> For testing Tarantool with Jepsen we use virtual machines as they provides
> better resource isolation in comparison to containers. Jepsen tests may need a
> single instance or a set of instances for testing cluster.  To setup virtual
> machines we use Terraform [1]. Patch adds a set of configuration files for
> Terraform that can create required number of virtual machines in MCS and output
> IP addresses to stdout.
> 
> Terraform needs some parameters before run. They are:
> 
> - id, identificator of a test stand that should be specific for this run, id
> also is a part of virtual machine name
> - keypair_name, name of keypair used in a cloud, public SSH key of that key pair
> will be placed to virtual machine
> - instance_count, number of virtual machines in a test stand
> - ssh_key, SSH private key, used to access to a virtual machine
> - user_name
> - password
> - tenant_id
> - user_domain_id
> 
> These parameters can be passed via enviroment variables with TF_VAR_ prefix
> (like TF_VAR_id) or via command-line parameters.
> 
> To demonstrate full lifecycle of a test stand with Terraform one needs to
> perform these commands:
> 
> terraform init extra/tf
> terraform apply extra/tf
> terraform output instance_names
> terraform output instance_ips
> terraform destroy extra/tf
> 
> 1. https://www.terraform.io/
> 
> Part of #5277
> ---
>  extra/tf/main.tf     | 43 ++++++++++++++++++++++++++++++++++++++
>  extra/tf/outputs.tf  | 10 +++++++++
>  extra/tf/provider.tf | 49 ++++++++++++++++++++++++++++++++++++++++++++
>  extra/tf/versions.tf |  8 ++++++++
>  4 files changed, 110 insertions(+)
>  create mode 100644 extra/tf/main.tf
>  create mode 100644 extra/tf/outputs.tf
>  create mode 100644 extra/tf/provider.tf
>  create mode 100644 extra/tf/versions.tf
> 
> diff --git a/extra/tf/main.tf b/extra/tf/main.tf
> new file mode 100644
> index 000000000..0b760884e
> --- /dev/null
> +++ b/extra/tf/main.tf
> @@ -0,0 +1,43 @@
> +resource "openstack_compute_instance_v2" "instance" {
> +  count = var.instance_count
> +
> +  name = "tarantool_testing-${count.index + 1}-${var.id}"
> +
> +  image_name = "Ubuntu-18.04-202003"
> +
> +  image_id = "3f03b393-d45b-455d-bdf5-0ac4399ecf42"
> +
> +  flavor_name = "Basic-1-2-20"
> +
> +  key_pair = var.keypair_name
> +
> +  config_drive = true
> +
> +  availability_zone = "DP1"
> +
> +  security_groups = [
> +    "default",
> +    "allow-ssh",
> +    "allow-tarantool"
> +  ]
> +
> +  network {
> +    name = "ext-net"
> +  }
> +
> +  provisioner "remote-exec" {
> +    inline = [
> +      "sudo hostnamectl set-hostname n${count.index + 1}",
> +      "sudo apt-get update"
> +    ]
> +  }
> +
> +  connection {
> +    host = self.access_ip_v4
> +    type = "ssh"
> +    user = "ubuntu"
> +    timeout = "5m"
> +    agent = true
> +    private_key = file(var.ssh_key_path)
> +  }
> +}
> diff --git a/extra/tf/outputs.tf b/extra/tf/outputs.tf
> new file mode 100644
> index 000000000..5690d6b0d
> --- /dev/null
> +++ b/extra/tf/outputs.tf
> @@ -0,0 +1,10 @@
> +output "instance_names" {
> +  description = "List of names assigned to the instances."
> +  value = openstack_compute_instance_v2.instance.*.name
> +}
> +
> +output "instance_ips" {
> +  description = "List of public IP addresses assigned to the instances."
> +  value = openstack_compute_instance_v2.instance.*.access_ip_v4
> +  sensitive = true
> +}
> diff --git a/extra/tf/provider.tf b/extra/tf/provider.tf
> new file mode 100644
> index 000000000..cd9488ce5
> --- /dev/null
> +++ b/extra/tf/provider.tf
> @@ -0,0 +1,49 @@
> +variable "user_name" {
> +  type = string
> +}
> +
> +variable "password" {
> +  type = string
> +}
> +
> +variable "tenant_id" {
> +  type = string
> +}
> +
> +variable "user_domain_id" {
> +  type = string
> +}
> +
> +variable "keypair_name" {
> +  type = string
> +}
> +
> +variable "ssh_key_path" {
> +  type = string
> +}
> +
> +variable "instance_count" {
> +  type = number
> +  default = 1
> +  description = "Number of instances."
> +
> +  validation {
> +    condition = var.instance_count > 0
> +    error_message = "The instance_count value must be greater than zero."
> +  }
> +}
> +
> +variable "id" {
> +  type = string
> +  default = null
> +}
> +
> +provider "openstack" {
> +  user_name = var.user_name
> +  password = var.password
> +  tenant_id = var.tenant_id
> +  user_domain_id = var.user_domain_id
> +  auth_url = "https://infra.mail.ru/identity/v3/"
> +  region = "RegionOne"
> +  use_octavia = true
> +}
> diff --git a/extra/tf/versions.tf b/extra/tf/versions.tf
> new file mode 100644
> index 000000000..5af3d608b
> --- /dev/null
> +++ b/extra/tf/versions.tf
> @@ -0,0 +1,8 @@
> +terraform {
> +  required_providers {
> +    openstack = {
> +      source = "terraform-providers/openstack"
> +    }
> +  }
> +  required_version = ">= 0.13"
> +}
> -- 
> 2.25.1
> 

Re: [Tarantool-patches] [PATCH 2/4] cmake: add targets to run Jepsen tests

[Alexander V. Tikhonov]

Hi Sergey, thanks for the patch. It has 2 make targets which will be
usable for tests reproduce. Later I hope we'll make some more
instructions on its use, but not now, so patch LGTM.

On Wed, Sep 16, 2020 at 10:07:21AM +0300, sergeyb@tarantool.org wrote:
> From: Sergey Bronnikov <sergeyb@tarantool.org>
> 
> Added targets 'make jepsen-single' and 'make jepsen-cluster'
> to run Jepsen tests on a single Tarantool instance and
> cluster of Tarantool instances.
> 
> Part of #5277
> ---
>  CMakeLists.txt | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> diff --git a/CMakeLists.txt b/CMakeLists.txt
> index 695e80c21..e8850e320 100644
> --- a/CMakeLists.txt
> +++ b/CMakeLists.txt
> @@ -17,9 +17,11 @@ include(TestBigEndian)
>  include(CheckFunctionExists)
>  include(FindOptionalPackage)
>  include(FindPackageMessage)
> +include(ExternalProject)
>  
>  find_program(ECHO echo)
>  find_program(CAT cat)
> +find_program(BASH bash)
>  find_program(GIT git)
>  find_program(LD ld)
>  find_program(CTAGS ctags)
> @@ -162,6 +164,31 @@ add_custom_command(TARGET luacheck
>      COMMENT "Perform static analysis of Lua code"
>  )
>  
> +ExternalProject_Add(
> +    jepsen-tests
> +    GIT_REPOSITORY https://github.com/tarantool/jepsen.tarantool
> +    CONFIGURE_COMMAND ""
> +    BUILD_COMMAND ""
> +    INSTALL_COMMAND ""
> +    TEST_COMMAND ""
> +)
> +
> +#
> +# Enable 'make jepsen-*' targets.
> +#
> +
> +add_custom_target(jepsen-single DEPENDS jepsen-tests)
> +add_custom_command(TARGET jepsen-single
> +    COMMAND ${BASH} ${PROJECT_SOURCE_DIR}/tools/run-jepsen-tests.sh ${PROJECT_SOURCE_DIR} ${PROJECT_BINARY_DIR}
> +    COMMENT "Running Jepsen tests on a single Tarantool instance"
> +)
> +
> +add_custom_target(jepsen-cluster DEPENDS jepsen-tests)
> +add_custom_command(TARGET jepsen-cluster
> +    COMMAND ${BASH} ${PROJECT_SOURCE_DIR}/tools/run-jepsen-tests.sh ${PROJECT_SOURCE_DIR} ${PROJECT_BINARY_DIR} 5
> +    COMMENT "Running Jepsen tests on a cluster with 5 Tarantool instances"
> +)
> +
>  #
>  # Get version
>  #
> -- 
> 2.25.1
> 

Re: [Tarantool-patches] [PATCH 3/4] tools: add script to run Jepsen tests

[Alexander V. Tikhonov]

Hi Sergey, thanks a lot for the patch. IMHO the script is on bash and
its great for the initiating the new feature, because it is really easy
to fix on any needs, so patch LGTM.

On Wed, Sep 16, 2020 at 10:07:22AM +0300, sergeyb@tarantool.org wrote:
> From: Sergey Bronnikov <sergeyb@tarantool.org>
> 
> Main script that handle creation of set of virtual machines
> using Terraform, setup for remote connection, running
> Jepsen tests and teardown test environment.
> 
> Part of #5277
> ---
>  tools/run-jepsen-tests.sh | 77 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 77 insertions(+)
>  create mode 100755 tools/run-jepsen-tests.sh
> 
> diff --git a/tools/run-jepsen-tests.sh b/tools/run-jepsen-tests.sh
> new file mode 100755
> index 000000000..865a8f4b0
> --- /dev/null
> +++ b/tools/run-jepsen-tests.sh
> @@ -0,0 +1,77 @@
> +#!/usr/bin/bash
> +
> +# Script performs setup of test environment using Terraform,
> +# runs Jepsen tests and teardown test environment.
> +# Script expects following environment variables:
> +# 	TF_VAR_user_name
> +# 	TF_VAR_password
> +# 	TF_VAR_tenant_id
> +# 	TF_VAR_user_domain_id
> +# 	TF_VAR_keypair_name
> +# 	TF_VAR_ssh_key
> +# and three arguments: path to a Tarantool project source directory, path to a
> +# Tarantool project binary directory and number on instances (optional).
> +
> +set -Eeo pipefail
> +
> +PROJECT_SOURCE_DIR=$1
> +PROJECT_BINARY_DIR=$2
> +INSTANCE_COUNT=$3
> +
> +TESTS_DIR="$PROJECT_BINARY_DIR/jepsen-tests-prefix/src/jepsen-tests/"
> +TERRAFORM_CONFIG="$PROJECT_SOURCE_DIR/extra/tf"
> +TERRAFORM_STATE="$PROJECT_BINARY_DIR/terraform.tfstate"
> +SSH_KEY_FILENAME="$PROJECT_SOURCE_DIR/build/tf-cloud-init"
> +NODES_FILENAME="$PROJECT_SOURCE_DIR/build/nodes"
> +
> +LEIN_OPTIONS="--nodes-file $NODES_FILENAME --username ubuntu --workload register"
> +
> +[[ -z ${PROJECT_SOURCE_DIR} ]] && (echo "Please specify path to a project source directory")
> +[[ -z ${PROJECT_BINARY_DIR} ]] && (echo "Please specify path to a project binary directory")
> +[[ -z ${TF_VAR_ssh_key} ]] && (echo "Please specify TF_VAR_ssh_key env var"; exit 1)
> +[[ -z ${TF_VAR_keypair_name} ]] && (echo "Please specify TF_VAR_keypair_name env var"; exit 1)
> +
> +TERRAFORM_BIN=$(which terraform)
> +LEIN_BIN=$(which lein)
> +CLOJURE_BIN=$(which clojure)
> +
> +[[ -z ${TERRAFORM_BIN} ]] && (echo "terraform is not installed"; exit 1)
> +[[ -z ${LEIN_BIN} ]] && (echo "lein is not installed"; exit 1)
> +[[ -z ${CLOJURE_BIN} ]] && (echo "clojure is not installed"; exit 1)
> +
> +function cleanup {
> +    echo "cleanup"
> +    rm -f $NODES_FILENAME $SSH_KEY_FILENAME
> +    [[ -e $TERRAFORM_STATE ]] && \
> +	terraform destroy -state=$TERRAFORM_STATE -auto-approve $TERRAFORM_CONFIG
> +}
> +
> +trap "{ cleanup; exit 255; }" SIGINT SIGTERM ERR
> +
> +echo -e "${TF_VAR_ssh_key//_/\\n}" > $SSH_KEY_FILENAME
> +chmod 400 $SSH_KEY_FILENAME
> +$(pgrep ssh-agent 2>&1 > /dev/null) || eval "$(ssh-agent)"
> +ssh-add $SSH_KEY_FILENAME
> +
> +if [[ -z ${CI_JOB_ID} ]]; then
> +    TF_VAR_id=$CI_JOB_ID
> +else
> +    RANDOM_ID=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 13; echo '')
> +    TF_VAR_id=TF-$RANDOM_ID
> +fi
> +
> +export TF_VAR_id
> +export TF_VAR_ssh_key_path=$SSH_KEY_FILENAME
> +[[ -z ${INSTANCE_COUNT} ]] && TF_VAR_instance_count=1 || TF_VAR_instance_count=$INSTANCE_COUNT
> +export TF_VAR_instance_count
> +
> +[[ -e $HOME/.ssh ]] || (mkdir $HOME/.ssh && touch $HOME/.ssh/known_hosts)
> +
> +terraform init $TERRAFORM_CONFIG
> +terraform apply -state=$TERRAFORM_STATE -auto-approve $TERRAFORM_CONFIG
> +terraform providers $TERRAFORM_CONFIG
> +terraform output -state=$TERRAFORM_STATE instance_names
> +terraform output -state=$TERRAFORM_STATE -json instance_ips | jq --raw-output '.[]' > $NODES_FILENAME
> +pushd $TESTS_DIR && lein run test $LEIN_OPTIONS
> +popd
> +cleanup
> -- 
> 2.25.1
> 

Re: [Tarantool-patches] [PATCH 0/4] Integrate Jepsen tests to CI

[Kirill Yukhin]

Hello,

On 16 сен 10:07, sergeyb@tarantool.org wrote:
> From: Sergey Bronnikov <sergeyb@tarantool.org>
> 
> Patch series adds Terraform configs to setup test stand in a cloud
> infrastructure, script to run Jepsen tests, separate targets to
> Makefile.
> 
> Issue: https://github.com/tarantool/tarantool/issues/5277
> CI: https://gitlab.com/tarantool/tarantool/-/pipelines/190225147

In future, please mention branch name in cover letter.

I've checked your patchset into master.

--
Regards, Kirill Yukhin