ENH/CI: Cache packages installed from conda

[lithomas1]

Currently, I think we are downloading all the packages every time CI is run. This takes a long time(~5-10 min. on azure) and makes CI flakier, since sometimes we can't connect to the Anaconda servers, and sometimes we timeout on Azure. We should cache the packages downloaded and only check for updates everytime the CI is run. I think we're doing some form of this on GHA, but we should also do it on azure.

cc @fangchenli

[fangchenli]

It shouldn't be hard.
https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#pythonanaconda

I'll try cache for the windows build in #42581

[alimcmaster1]

It shouldn't be hard.
https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#pythonanaconda

I'll try cache for the windows build in #42581

Do you still want to work on this? If not I think we can mark as good first issue/contrib welcome as instructions above look comprehensive :)

@fangchenli

[amrutha-mukkatira]

1. Export yaml : conda env export > env.yaml
   and then if there is a section of pip: packages, try to replace them all with Conda versions (editing the YAML). If there isn't a pip section, then it might be the conda clean -p thing. Either way, recreate your environment using the YAML

2)conda env create -f env.yaml -n new_env
This should force redownloading packages missing from the cache, as well as switch to non-PyPI versions of packages, assuming you removed all the pip: packages.

I would expect conda-pack to then work on this rebuilt version of the environment.

[Sara-cos]

Is it still open?

[mroeschke]

I think we do with now with provision-with-micromamba and don't use azure now so closing this out.