From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp50.i.mail.ru (smtp50.i.mail.ru [94.100.177.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id CF210469719 for ; Mon, 17 Feb 2020 01:04:00 +0300 (MSK) Date: Mon, 17 Feb 2020 01:04:19 +0300 From: Alexander Turenko Message-ID: <20200216220419.pum3uwjcfkaklnd6@tkn_work_nb> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: Subject: Re: [Tarantool-patches] [PATCH v3] Implement perf testing at gitlab-ci List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Alexander V. Tikhonov" Cc: Oleg Piskunov , tarantool-patches@dev.tarantool.org In order to don't make anybody confused I'll share our agreement: - Proposal (stage 1): - Don't review / change workloads and harness. - Enable automatic runs of benchmarks on long-term branches (per-push). - Save those results to the existing database (one under bench.tarantool.org). - Resurrect bench.tarantool.org: it should show new results. After this we should review workload kinds and sizes, improve visualization, setup alerts and made other enhancement that will make performance tracking / measurements being the useful tool. Since we discussing the first stage now, there is nothing to review. There was the suggestion from me: move all things where we have no agreement to the separate repository (bench-run) to don't do many fixups within tarantool repository in the near future and split the responsibility (QA team is both producer and consumer of performance tracking results). We have no agreement on using docker in performance testing (I'm strongly against, but it is not in my responsibility). So any trails of docker should be within bench-run repository. So here I expect only ./bench-run/prepare.sh and ./bench-run/sysbench.sh calls, nothing more. We can pass docker repository URI and credentials within environment variables (secret ones for credentials) and use it in bench-run. I don't see any problem to do it in this way. Aside of this, I'm against of using gitlab-runner on performance machines, because I don't know how it works. But okay, maybe everything will be fine, however please monitor its behaviour. My objections against using docker in a performance testing are below. Skip them: it is only to say 'I said this!' in the future. Several questions about the patch and bench-run are at end of the email (it is about stage 2, yep, but anyway). WBR, Alexander Turenko. ---- Docker virtualizes network and disk (both root and volumes). Any virtualization level adds complexity: requires more expertise and work to investigate and explain results, may affect results on its own and make them less predictable and stable. On the other hand, it does not give any gains for performance testing. One may say that it freezes userspace, but it may be easily achieved w/o docker: just don't change it. That's all. Okay, this topic is not so easy when the machine where a performance testing performed is not fully controlled: weird processes within an organization does not save us from somebody who will login and update something (strange, yep?). Docker will not save us from this situation: somebody may update docker itself, or kernel, or run something that will affect results that are in fly. The problem is in processes and it should be solved first. One may say that docker does not spoil performance results. Maybe. Maybe not. It is hard to say without deep investigation. While gains are so vague I would not pay my time to look at this direction. This is basically all, but I'll share several questions to show that my point 'adding of a virtualization level requires more expertise' have some ground downward. ---- Whether vm.dirty_ratio will work in the same way for dirty pages of a filesystem within a volume as for an underlying filesystem? Whether it depends on a certain underlying filesystem? Whether it'll use docker's memory size to calculate a dirty pages percent or system-wide one? Whether `sync + drop caches` within a container will affect disc buffers outside of the container (say, one that remains after a previous run within another container)? Whether a unix domain socket that is created within overlay filesystem will behave in the same way as on a real filesystem (in case we'll test iproto via unix socket)? Will fsync() flush data to a real disc or will be catched somewhere within docker? We had related regression [1]. [1]: https://github.com/tarantool/tarantool/issues/3747 ---- Black box testing sucks. We should deeply understand what we're testing, otherwise it will get 'some quality' which never will be good. Performance testing with docker is black box for me. When it is 'tarantool + libc + some libs + kernel' I more or less understand (at least able to inspect) what is going on and I can, say, propose to add / remove / tune workloads to cover specific needs. I can dig into docker, of course, but there are so many things which deserves time more than this activity. ---- I looked here and there around the patch and bench-run and have several questions. Since we agreed to don't review anything around workloads now, it is just questions. Okay to ignore. I don't see any volume / mount parameters. Aren't this means that WAL writes will going to an overlay fs? I guess it may be far from a real disc and may have a separate level of caching. AFAIS, the current way to use docker don't even try to freeze userspace: it uses 'ubuntu:18.04' tag, which is updated from time to time, not, say, 'ubuntu:bionic-20200112'. It also performs 'apt-get update' inside and so userspace will be changed for each rebuilt of the image. We unable to change something inside the image and don't update everything. This way we don't actually control userspace updates. BTW, why Ubuntu is used while all production environments (where performance matters) are on RHEL / CentOS 7? Why dirty cache is not cleaned (`sync`) before flushing clean cache to disc (`echo 3 > /proc/sys/vm/drop_caches`)?