Skip to content

Commit 10fb2c0

Browse files
mosuemmosuem
authored
Ecosystem testing (#320)
Workflow for testing a package upgrade against the ecosystem. See for example mosuem/i18n#2 (comment) --- - [x] I’ve reviewed the contributor guide and applied the relevant portions to this PR. <details> <summary>Contribution guidelines:</summary><br> - See our [contributor guide](https://github.com/dart-lang/.github/blob/main/CONTRIBUTING.md) for general expectations for PRs. - Larger or significant changes should be discussed in an issue before creating a PR. - Contributions to our repos should follow the [Dart style guide](https://dart.dev/guides/language/effective-dart) and use `dart format`. - Most changes should add an entry to the changelog and may need to [rev the pubspec package version](https://github.com/dart-lang/sdk/blob/main/docs/External-Package-Maintenance.md#making-a-change). - Changes to packages require [corresponding tests](https://github.com/dart-lang/.github/blob/main/CONTRIBUTING.md#Testing). Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback. </details>
1 parent 248b180 commit 10fb2c0

File tree

13 files changed

+702
-0
lines changed

13 files changed

+702
-0
lines changed

‎.github/test_repos/repos.json

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
{
2+
"$schema": "../../pkgs/quest/schema.json",
3+
"https://github.com/mosuem/my_app_old_web": {
4+
"level": "analyze"
5+
},
6+
"https://github.com/mosuem/my_app_new_web": {
7+
"level": "test",
8+
"packages": {
9+
"exclude": "intl4x"
10+
}
11+
}
12+
}

‎.github/workflows/ecosystem_test.yaml

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
name: Ecosystem test
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
repos_file:
7+
description: 'Path to the file containing the list of repository names'
8+
type: string
9+
required: true
10+
local_debug:
11+
description: Whether to use a local version of ecosystem testing - only for debug
12+
default: false
13+
type: boolean
14+
required: false
15+
16+
jobs:
17+
update_and_test:
18+
runs-on: ubuntu-latest
19+
steps:
20+
- name: Checkout repository
21+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
22+
with:
23+
token: ${{ secrets.GITHUB_TOKEN }}
24+
25+
- uses: subosito/flutter-action@44ac965b96f18d999802d4b807e3256d5a3f9fa1
26+
with:
27+
channel: main
28+
29+
- run: echo "${{ toJSON(github.event.pull_request.labels.*.name) }}"
30+
31+
- name: Install firehose
32+
run: dart pub global activate -s path pkgs/quest
33+
if: ${{ inputs.local_debug }}
34+
35+
- run: dart pub global activate -s git https://github.com/dart-lang/ecosystem.git --git-ref main --git-path pkgs/quest
36+
if: ${{ !inputs.local_debug }}
37+
38+
- name: Update package and test
39+
run: |
40+
dart pub global run quest ${{ inputs.repos_file }} ${{ github.repositoryUrl }} ${{ github.head_ref || github.ref_name }} "${{ toJSON(github.event.pull_request.labels.*.name) }}"
41+
env:
42+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
43+
44+
- name: Output issue number
45+
run: |
46+
mkdir -p output
47+
echo ${{ github.event.number }} > output/issueNumber
48+
49+
- name: Find Comment
50+
uses: peter-evans/find-comment@90e9b82d6319a7ae3f32bc0b434db48d6c376e55
51+
id: fc
52+
with:
53+
issue-number: ${{ github.event.number }}
54+
comment-author: github-actions[bot]
55+
body-includes: '## Ecosystem testing'
56+
57+
- name: Write comment id to file
58+
if: ${{ steps.fc.outputs.comment-id != 0 }}
59+
run: echo ${{ steps.fc.outputs.comment-id }} >> output/commentId
60+
61+
- name: Upload markdown
62+
if: success() || failure()
63+
uses: actions/upload-artifact@b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882
64+
with:
65+
name: output
66+
path: output/

‎.github/workflows/ecosystem_test_internal.yaml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
name: Ecosystem test:Internal
2+
3+
on:
4+
pull_request:
5+
branches: [ main ]
6+
types: [opened, synchronize, reopened, labeled, unlabeled]
7+
8+
jobs:
9+
test_ecosystem:
10+
uses: ./.github/workflows/ecosystem_test.yaml
11+
with:
12+
repos_file: .github/test_repos/repos.json
13+
local_debug: true

‎.github/workflows/post_summaries.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ on:
88
workflows:
99
- Publish:Internal
1010
- Health:Internal
11+
- Ecosystem test:Internal
1112
types:
1213
- completed
1314

@@ -45,6 +46,12 @@ jobs:
4546
fs.writeFileSync('${{ github.workspace }}/comment.zip', Buffer.from(download.data));
4647
- run: unzip comment.zip
4748

49+
- name: 'Print the comment'
50+
run: |
51+
if [ -f "./comment.md" ]; then
52+
cat ./comment.md
53+
fi
54+
4855
4956
# Create the comment, or update the existing one, with the markdown
5057
# generated in the Health workflow.

‎.github/workflows/quest.yml

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
name: package:quest
2+
3+
permissions: read-all
4+
5+
on:
6+
pull_request:
7+
branches: [ main ]
8+
paths:
9+
- '.github/workflows/quest.yml'
10+
- 'pkgs/quest/**'
11+
push:
12+
branches: [ main ]
13+
paths:
14+
- '.github/workflows/quest.yml'
15+
- 'pkgs/quest/**'
16+
schedule:
17+
- cron: '0 0 * * 0' # weekly
18+
19+
defaults:
20+
run:
21+
working-directory: pkgs/quest
22+
23+
env:
24+
GH_TOKEN: ${{ github.token }}
25+
26+
jobs:
27+
build:
28+
runs-on: ubuntu-latest
29+
steps:
30+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
31+
- uses: subosito/flutter-action@44ac965b96f18d999802d4b807e3256d5a3f9fa1
32+
with:
33+
channel: main
34+
35+
- run: dart pub get
36+
37+
- run: dart analyze --fatal-infos
38+
39+
- run: dart format --output=none --set-exit-if-changed .
40+
41+
- run: dart test

‎pkgs/quest/.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# https://dart.dev/guides/libraries/private-files
2+
# Created by `dart pub`
3+
.dart_tool/

‎pkgs/quest/LICENSE

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
Copyright 2024, the Dart project authors.
2+
3+
Redistribution and use in source and binary forms, with or without
4+
modification, are permitted provided that the following conditions are
5+
met:
6+
7+
* Redistributions of source code must retain the above copyright
8+
notice, this list of conditions and the following disclaimer.
9+
* Redistributions in binary form must reproduce the above
10+
copyright notice, this list of conditions and the following
11+
disclaimer in the documentation and/or other materials provided
12+
with the distribution.
13+
* Neither the name of Google LLC nor the names of its
14+
contributors may be used to endorse or promote products derived
15+
from this software without specific prior written permission.
16+
17+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
18+
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
19+
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
20+
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
21+
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
22+
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
23+
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
24+
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
25+
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
26+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
27+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎pkgs/quest/README.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Quest: Ecosystem Testing for Dart Packages
2+
Embark your package on a quest of testing against a suite of applications. This helps identify potential breaking changes introduced by package updates, ensuring seamless integration across the ecosystem.
3+
4+
## What does it do?
5+
It checks if your package upgrade would result in failures in the ecosystem. This is achieved by running the following pseudocode:
6+
```dart
7+
for (final app in applicationSuite) {
8+
if (app.dependencies.contains(package)) {
9+
pubGet(app);
10+
analyze(app);
11+
test(app);
12+
13+
upgradePackage(app);
14+
15+
pubGet(app);
16+
analyze(app);
17+
test(app);
18+
}
19+
}
20+
```
21+
22+
## How do I use it?
23+
1. Create a suite of repositories to test against at `.github/test_repos/repos.json`. Follow the schema specified [here](schema.json).
24+
```json
25+
{
26+
"https://github.com/mosuem/my_app_old_web": {
27+
"level": "analyze"
28+
},
29+
"https://github.com/mosuem/my_app_new_web": {
30+
"level": "test",
31+
"packages": {
32+
"exclude": "intl4x"
33+
}
34+
}
35+
}
36+
```
37+
38+
2. Add a workflow file with the following contents:
39+
```yaml
40+
name: Ecosystem test
41+
42+
on:
43+
pull_request:
44+
branches: [ main ]
45+
types: [opened, synchronize, reopened, labeled, unlabeled]
46+
47+
jobs:
48+
test_ecosystem:
49+
uses: dart-lang/ecosystem/.github/workflows/ecosystem_test.yaml@main
50+
with:
51+
repos_file: .github/test_repos/repos.json
52+
```
53+
54+
3. To show the markdown result as a comment, also add a workflow file
55+
```yaml
56+
name: Comment on the pull request
57+
58+
on:
59+
# Trigger this workflow after the Health workflow completes. This workflow will have permissions to
60+
# do things like create comments on the PR, even if the original workflow couldn't.
61+
workflow_run:
62+
workflows:
63+
- Health
64+
- Publish
65+
- Ecosystem test
66+
types:
67+
- completed
68+
69+
jobs:
70+
upload:
71+
uses: dart-lang/ecosystem/.github/workflows/post_summaries.yaml@main
72+
permissions:
73+
pull-requests: write
74+
```
75+
76+
4. Profit!
77+
78+
# Contributing
79+
Contributions are welcome! Please see the [contribution guidelines](../../CONTRIBUTING.md).

‎pkgs/quest/analysis_options.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
include: package:dart_flutter_team_lints/analysis_options.yaml
2+
3+
linter:
4+
rules:
5+
- prefer_final_locals

0 commit comments

Comments
 (0)