Add test-backend from Zulip server.

[rht]

test pr works!

[timabbott]

@rht what's the thinking behind running the Zulip server's backend tests here?

[rht]

This is to automate the integration test to check for PRs which require such, such as #18. The caveat is that it will slow down the aggregate CI test time. The other option is to add this commit only for the PRs where this test is needed, so the test is run once only.

The integration test will be required once again to test the stage 2 of Slack data import.

[timabbott]

@derAnfaenger @eeshangarg what are your thoughts on this?

[timabbott]

I think this might actually be a fine way to test for changes here that impact the integration with the backend...

[roberthoenig]

This is really cool, but it brings up the effective testing time from 1 minute to 11 minutes and goes over the magic five test jobs. Ultimately, I think we should merge this, but if it isn't urgent, maybe with a tweak similar to facebook/react#2000 ? Here, Travis checks the diff of the changes before running a suite - this would allow us to run the backend tests iff e.g. zulip/zulip is modified.

[rht]

@derAnfaenger agreed on the test duration and the limitation of the parallel tests ! Should the test be triggered in main only when https://github.com/zulip/python-zulip-api/tree/master/zulip/zulip is patched?

I really need this to test the slack data import PR (this (#908) has been in the roadmap for ages who knows when).

[roberthoenig]

Testing when zulip/zulip is patched would be a god start, yeah. We could then later refine the exact directories which are affected. For slack, I guess you can just follow your own suggestion for now and add this commit to the PR for test purposes?

[roberthoenig]

@rht would it be sufficient in this case to just run tools/test-api?

[rht]

Ahh, right, right, let me check how long the test takes with just tools/test-api.

[roberthoenig]

@rht what's the state of this? We're possibly doing some iterations on the api code in the next days, so it'd be good to have the api tests set up by then. If you're working on other things right now, feel free to tell and I'd be happy to finish this :)

[rht]

The only blocker is that I was testing between various Git clone ssh paths:

1. sed -i "s|[email protected]|python-zulip-api.git@pull/$TRAVIS_PULL_REQUEST|" requirements/dev_lock.txt
2. sed -i "s|zulip/[email protected]|$TRAVIS_PULL_REQUEST_SLUG.git@$TRAVIS_PULL_REQUEST_BRANCH|" requirements/dev_lock.txt
3. sed -i "s|zulip/[email protected]|$TRAVIS_PULL_REQUEST_SLUG.git/$TRAVIS_PULL_REQUEST_BRANCH|" requirements/dev_lock.txt

[rht]

Most likely wrong URL formatting. If you have any idea on what's going on, and unblock this, I'm totally up for this.

[rht]

I will have to update this to use the circleci script.

[timabbott]

We can also mitigate the performance impact here by having the tool only run the subset of backend tests that's related to the subsystem.

[zulipbot]

Heads up @rht, we just merged some commits that conflict with the changes your made in this pull request! You can review this repository's recent commits to see where the conflicts occur. Please rebase your feature branch against the upstream/master branch and resolve your pull request's merge conflicts accordingly.

[timabbott]

@PIG208 I think this would be a good effort to pick up. We probably won't end up reusing much of the exact code here, because we've moved to GitHub Actions since then, but I do think it'd be really nice if we can run the relevant part of the server's test suites against new python-zulip-api commits in CI. The main things I'm imagining are:

• tools/test-api -- this is the most valuable, since it involves actually running the examples in our API documentation.
• tools/test-backend x y z where we pass just the modules like BotStorage that integrate with the API project.

but once we have this setup, we can add other things as well.