Skip to content

ci: re-ordering Drone CI for optimizing time #9719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Jan 13, 2020

Conversation

sapk
Copy link
Member

@sapk sapk commented Jan 12, 2020

This PR re-organize CI to optimize timing :

  • Extract a new pipeline compliance with validation of code and requirement of minimal version compat.
  • testing pipeline is run directly
  • unit-test step run directly without waiting for gitea binary from build step
  • Run docker dry run after compliance and not testing
    • docker release (master/tag) will still be run after all tests validated
  • Run compliance on arm64 which have a lot more memory than amd mainly for golangci-lint
  • Remove docker dry run on amd64 (doing it on arm64 is enough and build on amd64 are validated in other steps)

This PR also offload mysql8 integration test to arm64 (let some cpu available for other integration on amd64) this need to use the mysql official image inplace of docker official to have an arm64 build https://hub.docker.com/r/mysql/mysql-server. Maybe this part is not really needed and could be reverted.

@sapk
Copy link
Member Author

sapk commented Jan 12, 2020

This is wip since currently I am pretty sure I broke tag/release.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jan 12, 2020
@sapk sapk added pr/wip This PR is not ready for review and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jan 12, 2020
@codecov-io
Copy link

codecov-io commented Jan 12, 2020

Codecov Report

Merging #9719 into master will increase coverage by 0.01%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #9719      +/-   ##
==========================================
+ Coverage   42.27%   42.29%   +0.01%     
==========================================
  Files         597      597              
  Lines       78257    78257              
==========================================
+ Hits        33083    33095      +12     
+ Misses      41121    41114       -7     
+ Partials     4053     4048       -5
Impacted Files Coverage Δ
models/webhook.go 69.39% <0%> (-1.07%) ⬇️
models/error.go 31.05% <0%> (+0.55%) ⬆️
models/gpg_key.go 55.59% <0%> (+0.55%) ⬆️
modules/log/file.go 77.62% <0%> (+2.09%) ⬆️
services/pull/check.go 57.43% <0%> (+4.05%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7823709...1d26650. Read the comment docs.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jan 12, 2020
@sapk
Copy link
Member Author

sapk commented Jan 12, 2020

This need more work but we could save a third of the duration of PR tests

@sapk sapk removed the pr/wip This PR is not ready for review label Jan 12, 2020
@sapk sapk changed the title [wip] ci: try re-ordering for optimizing time ci: try re-ordering for optimizing time Jan 12, 2020
@sapk
Copy link
Member Author

sapk commented Jan 12, 2020

last CI jobs has taken longer globally with pgsql test on arm64 because the integration was started before since the arm64 was idle and launched pipeline on arm64 and the jobs has to wait for a amd64 after (for the 3 running jobs to end).

@sapk sapk changed the title ci: try re-ordering for optimizing time ci: re-ordering Drone CI for optimizing time Jan 12, 2020
@sapk sapk added topic/build PR changes how Gitea is built, i.e. regarding Docker or the Makefile type/refactoring Existing code has been cleaned up. There should be no new functionality. skip-changelog This PR is irrelevant for the (next) changelog, for example bug fixes for unreleased features. and removed type/refactoring Existing code has been cleaned up. There should be no new functionality. labels Jan 12, 2020
@sapk
Copy link
Member Author

sapk commented Jan 12, 2020

I revert the part of running mysql8 on arm64 since it seems to fail. I will test this in a separate PR.

@sapk
Copy link
Member Author

sapk commented Jan 12, 2020

I added changes to try to keep this PR as minimal change for review.
Now builds are under 20 minutes!

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jan 13, 2020
@6543
Copy link
Member

6543 commented Jan 13, 2020

CI works with this PR can we go on with this? @go-gitea/maintainers

@sapk
Copy link
Member Author

sapk commented Jan 13, 2020

For details, this PR will not totally solve the problem with golangci-lint (I have re-started multiple time and it failed one time) but it greatly improve the time of CI for PR (I limited my self on release pipeline to sure to not break them).

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Jan 13, 2020
@sapk sapk merged commit b787aaf into go-gitea:master Jan 13, 2020
@sapk sapk deleted the optimize-ci branch January 13, 2020 10:40
6543 pushed a commit to 6543-forks/gitea that referenced this pull request Jan 19, 2020
* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try offloading mysql8 to arm64

* Revert "ci: try offloading mysql8 to arm64"

This reverts commit c60de5d.

* ci: try offloading pgsql to arm64

* ci: activate ldap on arm64

* ci: test mysql8 in place pgsql arm64

* chore: clean un-needed move

* typo

* ci: revert runnning mysql on arm64

* ci: run compliance on arm

* chore: limit change

* chore: readd maybe need for release fetch-tags

* ci: remove docker-linux-amd64-dry-run

* ci: remove docker-linux-amd64-dry-run

* Revert "ci: remove docker-linux-amd64-dry-run"

This reverts commit 0715f65.

Co-authored-by: zeripath <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
sapk added a commit that referenced this pull request Jan 19, 2020
* ci: use docker image for golangci-lint (#9737)

* ci: re-ordering Drone CI for optimizing time (#9719)

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try re-ordering for optimizing time

* ci: try offloading mysql8 to arm64

* Revert "ci: try offloading mysql8 to arm64"

This reverts commit c60de5d.

* ci: try offloading pgsql to arm64

* ci: activate ldap on arm64

* ci: test mysql8 in place pgsql arm64

* chore: clean un-needed move

* typo

* ci: revert runnning mysql on arm64

* ci: run compliance on arm

* chore: limit change

* chore: readd maybe need for release fetch-tags

* ci: remove docker-linux-amd64-dry-run

* ci: remove docker-linux-amd64-dry-run

* Revert "ci: remove docker-linux-amd64-dry-run"

This reverts commit 0715f65.

Co-authored-by: zeripath <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>

* ci: use new mssql image (#9720)

Co-authored-by: Lunny Xiao <[email protected]>

* ci: run notify on arm64 (#9762)

* ci: run notify on arm64

Free one jobs on amrd64

* Update .drone.yml

* Update .drone.yml

* Update .drone.yml

Based on: https://github.com/appleboy/drone-discord/blob/master/.drone.yml#L339

* improve trigger

Co-authored-by: techknowlogick <[email protected]>

* ci: move some integration tests on arm64 (#9747)

* tests: configure github remaining limit + read token (#9800)

* ci: configure remaining github limmit

* prepend with github since package is common to all migrations

* add RefreshRate

* Update github.go

* add missing space

* go fmt

* Read env variable GITHUB_READ_TOKEN for token

* Update .drone.yml

* ci: simplify tag/release by always running coverage (#9774)

* ci: simplify tag/release by always running coverage

* use mod and vendor for unit test coverage

* remove not needed lfs for unit test

* use arm drone agent for docs (#9776)

* run translations pipeline on arm server (#9865)

* add git-check to Makefile

Co-authored-by: Antoine GIRARD <[email protected]>
Co-authored-by: zeripath <[email protected]>
Co-authored-by: Lunny Xiao <[email protected]>
Co-authored-by: techknowlogick <[email protected]>
@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. skip-changelog This PR is irrelevant for the (next) changelog, for example bug fixes for unreleased features. topic/build PR changes how Gitea is built, i.e. regarding Docker or the Makefile
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants