Schedule periodic jobs into separate pipelines to avoid quota limit

[kiwik]

Let's make things better!

Describe the bug

There are 2 periodic pipelines (periodic and periodic-2) to run periodic jobs in OpenLab CI. Because all of jobs in a pipeline will be triggered at same time, these jobs will execute test cases concurrently to allocate resources in a same public cloud tenant, that might reach tenant quota limits to cause tests failing, we should separate periodic jobs into more pipelines to avoid it.

Related Job information

If the bug is related a CI job, put the job information below:

• job name:
• job build log link:
• error log piece:

Steps to reproduce the issue

Describe how to reproduce the issue:

Additional context

Add any other context about the problem here.

[kiwik]

@h00130372 and @liu-sheng, new periodic pipelines plan, please help to review.

Pipeline Name	Trigger Time(UTC-0)	Projects	notes
periodic	0:00 and 12:00	Deprecated, don't remove, keep backward compatible	bc
periodic-0/12	0:00 and 12:00	ansible/ansible apache/spark	new
periodic-2/14	2:00 and 14:00	terraform-providers/terraform-provider-openstack terraform-providers/terraform-provider-huaweicloud	new
periodic-4/16	4:00 and 16:00	gophercloud/gophercloud cloudfoundry/bosh-openstack-cpi-release kubernetes/cloud-provider-openstack huaweicloud/openshift-ansible	new
periodic-6/18	6:00 and 18:00	hashicorp/packer	new
periodic-8/20	8:00 and 20:00	docker/machine	new
periodic-10/22	10:00 and 22:00	manageiq/manageiq-providers-openstack kubernetes-sigs/cluster-api-provider-openstack	new

[liusheng]

@kiwik looks good to me. we'd better to add this table in doc, at least convenient for us. and it is better to add a column to record the numbers of jobs in per pipeline. what do you think ?

[huangtianhua]

@kiwik lgtm, one suggestion: seems all the time is running the period jobs, maybe we should take some time to maintain? and one question: what' s the principle to add new period job to which time?

[kiwik]

@liu-sheng , good idea, I will add the table into OpenLab document https://docs.openlabtesting.org/

[wangxiyuan]

@kiwik lgtm, one suggestion: seems all the time is running the period jobs, maybe we should take some time to maintain? and one question: what' s the principle to add new period job to which time?

++, it's good to add a guide describing how to deal with the new coming projects.

[kiwik]

@h00130372 each job will be triggered twice in one day (day and night), so if we maintain OpenLab and have to restart Zuul, will only impact one time for the job in the pipeline that will be triggered.

what' s the principle to add new period job to which time?

Good question, the basic principle is splitting the jobs have same backend(public cloud account), like: otc, huaweicloud or vexxhost into different pipeline to avoid influencing each other, local deployment, like: openstack, kubernetes is OK in same pipeline, and choose trigger time you like

[kiwik]

@wangxiyuan sure, will add it into document too.

[kiwik]

Now we have applied new periodic pipelines in OpenLab, and put all of periodic jobs into separated time period.

Next step

Move all of periodic job declaration from outside projects into openlab-zuul-jobs/projects.yaml, the main reason is that lots of project have different branch maintaining policy, and there are lots of periodic job declaration in different stable branch of these projects, that always cause some confusions and wrong usage, and it's difficult to plan the testing resource usage amount for OpenLab. So we plan to use openlab-zuul-jobs/projects.yaml as an unified check point to list all of periodic jobs, and OpenLab @theopenlab/dev team will manage the periodic jobs according to periodic-job-and-pipeline-schedule, and control the job count in each periodic pipeline.

We will commit PR in related projects to move periodic jobs one by one, only periodic jobs will be moved back to OpenLab, the PR triggered jobs is not impacted.

Imported projects:

1. gophercloud/gophercloud
2. cloudfoundry/bosh-openstack-cpi-release
3. kubernetes/cloud-provider-openstack

@liu-sheng @h00130372 @mrhillsman this is next step for periodic job refactor