CI enable integration test on iOS Simulator #349

sunmou99 · 2021-03-29T16:10:16Z

Script that running game-loop test on simulator: scripts/gha/test_simulator.py
A UI Test App that enable game-loop test: scripts/gha/integration_testing/gameloop

Workflow passed (admob,analytics):
https://github.com/firebase/firebase-cpp-sdk/runs/2241571055?check_suite_focus=true#step:33:28

Many test failed on simulator (due to keychain issue):
https://github.com/firebase/firebase-cpp-sdk/runs/2241582501?check_suite_focus=true#step:33:31

jonsimantov

What is this file? scripts/gha/integration_testing/gameloop.zip

sunmou99 · 2021-03-30T18:30:09Z

What is this file? scripts/gha/integration_testing/gameloop.zip

I built the app & XCUITest, and compressed the test files into a .zip file.
Thus we could skip the build gameloop (app & test) step and save some time.

jonsimantov · 2021-03-31T18:27:29Z

Rather than letting our tests go red, could you add a flag that turns off the simulator tests, so that we can file a ticket to re-enable and fix them?

(I recommend an env setting since we are out of input parameters.)

vimanyu

Minor comments on python code.

scripts/gha/test_simulator.py

vimanyu · 2021-03-31T18:38:51Z

scripts/gha/test_simulator.py

+    logging.info("No test Result")
+    return None
+
+  log_path = os.path.join(result.stdout.strip(), "Documents/GameLoopResults/Results1.json") 


Is the log path guaranteed to be this all the time?

Yes. Or we could iterate the directory to find the result.

vimanyu · 2021-03-31T18:45:27Z

scripts/gha/test_simulator.py

+  """Create a simulator locally. Will wait until this simulator botted."""
+  device_info = ios_device.split("-")
+  device_name = device_info[0]
+  device_os = device_info[1]


This could error if the device name does not have a "-". Maybe check for that first and raise exception if that is the case?
A comment stating an example string that could be passed for ios_device will be helpful to follow any string operations.

I comment here:

firebase-cpp-sdk/scripts/gha/test_simulator.py

Line 34 in a104da7

xcrun simctl list

I checked before, no device name (both Android & iOS) contains "-".
But I noticed that setting device info could be a problem.
My next plan is to expend the test device matrix. And I will figure out how to deal with it then.

Sorry, I meant device_info could be missing a "-". and the token split will fail in that case.

Got it. Currently, we only use the default value, thus it won't be a problem.
But I will keep this in mind when expending the test device matrix.

scripts/gha/test_simulator.py

vimanyu · 2021-03-31T18:52:55Z

scripts/gha/test_simulator.py

+  ios_device = FLAGS.ios_device
+
+  config_path = os.path.join(current_dir, "integration_testing/build_testapps.json")
+  with open(config_path, "r") as config:


nit: file handle variable could be named "config_file" for easier reading. Json data and file handle have the same name currently.

Historically, testing scripts name variable this way.

vimanyu · 2021-03-31T19:14:39Z

scripts/gha/integration_testing/gameloop/gameloopUITests/gameloopUITests.swift

+import XCTest
+
+/// Constants used across the loop launcher and UI test.
+public struct Constants {


Don't know much about swift but noticed the same constants are also defined in "Constants.swift". Could we reuse?

I create this class for reuse.
But sadly, UITest focus on UI, and cannot get access to the app's source code.
AFAIK, the way to reuse this class is to build it into a Framework.

scripts/gha/test_simulator.py

jonsimantov · 2021-03-31T19:18:15Z

What is this file? scripts/gha/integration_testing/gameloop.zip

I built the app & XCUITest, and compressed the test files into a .zip file.
Thus we could skip the build gameloop (app & test) step and save some time.

I don't really like including a prebuilt app in our repo - is this zip file used by the script? Or does the script build the app?

sunmou99 · 2021-03-31T19:24:11Z

What is this file? scripts/gha/integration_testing/gameloop.zip

I built the app & XCUITest, and compressed the test files into a .zip file.
Thus we could skip the build gameloop (app & test) step and save some time.

I don't really like including a prebuilt app in our repo - is this zip file used by the script? Or does the script build the app?

The script uses this zip file

sunmou99 · 2021-03-31T21:22:57Z

Rather than letting our tests go red, could you add a flag that turns off the simulator tests, so that we can file a ticket to re-enable and fix them?

(I recommend an env setting since we are out of input parameters.)

Add an env, and modified the build script to support it.
Also, test device takes 4 inputs. I think is better to address this issue in the following expending test device matrix ticket.

github-actions · 2021-03-31T23:34:43Z

❌ Integration test FAILED

Requested by @sunmou99 on commit 79affe1
Last updated: Wed Mar 31 18:52:52 PDT 2021
View integration test results

Platform	Build failures	Test failures
iOS (built on MacOS)		database

sunmou99 added 4 commits March 29, 2021 03:44

test on simulator script & tool

1e19769

add to workflow

ae802d6

update the script that get device id

7164434

update the script logging

6ebc18e

google-cla bot added the cla: yes label Mar 29, 2021

update the script logging

ef4a0cc

sunmou99 requested a review from jonsimantov March 29, 2021 21:13

sunmou99 force-pushed the feature/test-ios-simulator branch 2 times, most recently from 819410f to ef4a0cc Compare March 29, 2021 22:40

sunmou99 and others added 4 commits March 29, 2021 15:58

Merge branch 'main' into feature/test-ios-simulator

9466cf8

log summary indicate simulator

daf6004

delete extraneous files

d7c6741

update test device

a104da7

jonsimantov reviewed Mar 30, 2021

View reviewed changes

jonsimantov added the tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). label Mar 30, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: full Trigger a FULL set of integration tests (uses expanded test matrix). labels Mar 30, 2021

sunmou99 requested a review from jonsimantov March 30, 2021 19:44

github-actions bot added the tests: failed This PR's integration tests failed. label Mar 30, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Mar 30, 2021

Add support for simulator vs device in test summarizer.

87f8dba

jonsimantov added the tests-requested: quick Trigger a quick set of integration tests. label Mar 31, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: quick Trigger a quick set of integration tests. tests: failed This PR's integration tests failed. labels Mar 31, 2021

vimanyu reviewed Mar 31, 2021

View reviewed changes

github-actions bot added the tests: failed This PR's integration tests failed. label Mar 31, 2021

sunmou99 added 2 commits March 31, 2021 13:46

add test on device,simulator flag, and disable simulator test by default

56eebc6

minor changes based on comments

33326ce

sunmou99 requested a review from vimanyu March 31, 2021 21:11

using env instead of input

068be04

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Mar 31, 2021

sunmou99 added 2 commits March 31, 2021 16:24

build gameloop test app & removing the zip

4cc0f45

enable simulator test

79affe1

sunmou99 added the tests-requested: quick Trigger a quick set of integration tests. label Mar 31, 2021

github-actions bot added tests: in-progress This PR's integration tests are in progress. and removed tests-requested: quick Trigger a quick set of integration tests. tests: failed This PR's integration tests failed. labels Mar 31, 2021

sunmou99 added 2 commits March 31, 2021 17:19

disable simulator test

697022c

remove old comments

d6ecf08

jonsimantov approved these changes Apr 1, 2021

View reviewed changes

sunmou99 enabled auto-merge (squash) April 1, 2021 01:32

github-actions bot added the tests: failed This PR's integration tests failed. label Apr 1, 2021

firebase-workflow-trigger bot removed the tests: in-progress This PR's integration tests are in progress. label Apr 1, 2021

sunmou99 removed the tests: failed This PR's integration tests failed. label Apr 1, 2021

sunmou99 merged commit c74dc47 into main Apr 1, 2021

sunmou99 deleted the feature/test-ios-simulator branch April 1, 2021 01:55

firebase locked and limited conversation to collaborators May 2, 2021

sunmou99 changed the title ~~Add test on iOS simulator~~ CI enable integration test on iOS simulator Feb 25, 2022

sunmou99 changed the title ~~CI enable integration test on iOS simulator~~ CI enable integration test on iOS Simulator Feb 25, 2022

CI enable integration test on iOS Simulator #349

CI enable integration test on iOS Simulator #349

Uh oh!

Conversation

sunmou99 commented Mar 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonsimantov left a comment

Choose a reason for hiding this comment

Uh oh!

sunmou99 commented Mar 30, 2021

Uh oh!

jonsimantov commented Mar 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vimanyu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sunmou99 Mar 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jonsimantov commented Mar 31, 2021

Uh oh!

sunmou99 commented Mar 31, 2021

Uh oh!

sunmou99 commented Mar 31, 2021

Uh oh!

github-actions bot commented Mar 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Integration test FAILED

Uh oh!

Uh oh!

sunmou99 commented Mar 29, 2021 •

edited

Loading

jonsimantov commented Mar 31, 2021 •

edited

Loading

sunmou99 Mar 31, 2021 •

edited

Loading

github-actions bot commented Mar 31, 2021 •

edited

Loading