Support standalone MPAS runs

[xylar]

We haven't been testing standalone MPAS-O and MPAS-SeaIce runs on a regular basis. As a result, they were no longer working and this merge attempts to remedy that situation.

File names produced by standalone MPAS runs don't necessarily conform to E3SM standards, and are therefore not supported by ncclimo all versions of NCO. (ncclimo in NCO 4.7.3 will include more general file-name parsing so this could be a partial solution to the problem.) The solution here is to create symlinks to the timeSeriesStatsMonthlyOutput files in the analysis output directory where the links have the expected E3SM file names.

Additionally, there are 3 places in MPAS-Analysis where years and months are extracted from file names. Previously, this was done in a lazy way assuming the E3SM file-naming convention. With this merge, file names are instead parsed based on the filename_template attribute in the associated stream.

Finally, the documentation has been updated to give instructions on how to set up analysis members, including output streams, in standalone MPAS runs to they are compatible with MPAS-Analysis. These instructions will need to be updated as new variables are used in analysis task. (A documentation page with a list of authors has also been added.)

[xylar]

Testing

I ran a QU240 MPAS-Ocean test on my laptop from scratch. I followed the same procedure as in the new instructions I've added to the documentation. Once the run had reached 2 years, all analysis worked fine.

I'll note that I had a little trouble producing an MOC mask file that worked with the grid I have, which has a different number of cells from the QU240v3 grid for some reason. Some thought will need to go into how to produce masks in a more automated way once we have a manual workflow for these masks that we're happy with.

I have also verified that analysis of a QU240 E3SM run works fine, as before.

[xylar]

@pwolfram, I'm not sure what the easiest way for you to review this might be. Hopefully, you can run the analysis on your SOMA uniform32km test case and, assuming everything works, that will suffice for testing these changes.

[pwolfram]

@xylar, the SOMA run is on IC at /lustre/scratch2/turquoise/pwolfram/SOMA_Q2metric/uniform32kmMPASAnalysis and uses your instructions at https://github.com/xylar/MPAS-Analysis/blob/support_standalone/docs/mpaso.rst to set up the namelist and streams. I've started it at 5 years (for consistency with the other runs) and it is running now.

[xylar]

@pwolfram, I was able to use e3sm-unified on Grizzly (module load e3sm-unified) to complete the analysis, with the exceptions of:

• MOC -- there's no region mask for this grid and I don't think we want to make one
• Nino34 index -- we don't have a long enough time series yet
• SOSE comparison plots -- I'm uploading the necessary observations right now

I'll try again when the SOSE data are there and when we're further along. So far, it looks like all the necessary data is there.

Note: I'm having trouble with MPAS-Analysis using python/anaconda-2.7-climate. I get HDF5 errors when I use ESMF_RegridWeightGen.

[xylar]

Further note on running at LANL: The latest HDF5 (1.10.1) glitches because some expected locks are not supported. To handle this issue, you must set:

export HDF5_USE_FILE_LOCKING=FALSE

after module load python/anaconda-2.7-climate.

[xylar]

@pwolfram, I was able to run the analysis successfully but I haven't had a chance to look at the results. Here's the config file I used:

[runs]
## options related to the run to be analyzed and reference runs to be
## compared against

# mainRunName is a name that identifies the simulation being analyzed.
mainRunName = SOMA_QU2metric_uniform32km
# preprocessedReferenceRunName is the name of a reference run that has been
# preprocessed to compare against (or None to turn off comparison).  Reference
# runs of this type would have preprocessed results because they were not
# performed with MPAS components (so they cannot be easily ingested by
# MPAS-Analysis)
preprocessedReferenceRunName = None

# config file for a reference run to which this run will be compared.  The
# analysis should have already been run to completion once with this config
# file, so that the relevant MPAS climatologies already exist and have been
# remapped to the comparison grid.  Leave this option commented out if no
# reference run is desired.
# referenceRunConfigFile = /path/to/config/file

[execute]
# options related to executing parallel tasks

# the number of parallel tasks (1 means tasks run in serial, the default)
parallelTaskCount = 4

# Prefix on the commnd line before a parallel task (e.g. 'srun -n 1 python')
# Default is no prefix (run_mpas_analysis is executed directly)
commandPrefix =

# the parallelism mode in ncclimo ("serial" or "bck")
# Set this to "bck" (background parallelism) if running on a machine that can
# handle 12 simultaneous processes, one for each monthly climatology.
ncclimoParallelMode = bck

[input]
## options related to reading in the results to be analyzed

# directory containing model results
baseDirectory = /lustre/scratch2/turquoise/pwolfram/SOMA_Q2metric/uniform32kmMPASAnalysis

oceanNamelistFileName = namelist.ocean
oceanStreamsFileName = streams.ocean

# names of ocean and sea ice meshes (e.g. oEC60to30v3, oQU240v3, oRRS30to10v3, etc.)
mpasMeshName = oSOMA32

# Directory for mapping files (if they have been generated already). If mapping
# files needed by the analysis are not found here, they will be generated and
# placed in the output mappingSubdirectory
mappingDirectory = /turquoise/usr/projects/climate/SHARED_CLIMATE/mpas_analysis/mapping/

[output]
## options related to writing out plots, intermediate cached data sets, logs,
## etc.

# directory where analysis should be written
baseDirectory = /lustre/scratch2/turquoise/xylar/mpas_analysis/SOMA/uniform32km

# a list of analyses to generate.  Valid names can be seen by running:
#   ./run_mpas_analysis --list
# This command also lists tags for each analysis.
# Shortcuts exist to generate (or not generate) several types of analysis.
# These include:
#   'all' -- all analyses will be run
#   'all_<tag>' -- all analysis with a particular tag will be run
#   'all_<component>' -- all analyses from a given component (either 'ocean'
#                        or 'seaIce') will be run
#   'only_<component>', 'only_<tag>' -- all analysis from this component or
#                                       with this tag will be run, and all
#                                       analysis for other components or
#                                       without the tag will be skipped
#   'no_<task_name>' -- skip the given task
#   'no_<component>', 'no_<tag>' -- in analogy to 'all_*', skip all analysis
#                                   tasks from the given compoonent or with
#                                   the given tag.  Do
#                                      ./run_mpas_analysis --list
#                                   to list all task names and their tags
# an equivalent syntax can be used on the command line to override this
# option:
#    ./run_mpas_analysis config.analysis --generate \
#         all,no_ocean,all_timeSeries
# All tasks with tag "landIceCavities" are disabled because this run did not
# include land-ice cavities.
generate = ['all', 'no_landIceCavities', 'no_seaIce', 'no_streamfunctionMOC']

[climatology]
## options related to producing climatologies, typically to compare against
## observations and previous runs

# the first year over which to average climatalogies
startYear = 5
# the last year over which to average climatalogies
endYear = 6

[timeSeries]
## options related to producing time series plots, often to compare against
## observations and previous runs

# start and end years for timeseries analysis. Using out-of-bounds values
#   like start_year = 1 and end_year = 9999 will be clipped to the valid range
#   of years, and is a good way of insuring that all values are used.
startYear = 5
endYear = 6

[index]
## options related to producing nino index.

# start and end years for the nino 3.4 analysis.  Using out-of-bounds values
#   like start_year = 1 and end_year = 9999 will be clipped to the valid range
#   of years, and is a good way of insuring that all values are used.
# For valid statistics, index times should include at least 30 years
startYear = 5
endYear = 6

[oceanObservations]
## options related to ocean observations with which the results will be compared

# directory where ocean observations are stored
baseDirectory = /usr/projects/climate/SHARED_CLIMATE/observations

[oceanPreprocessedReference]
## options related to preprocessed ocean reference run with which the results
## will be compared (e.g. a POP, CESM or ACME v0 run)

# directory where ocean reference simulation results are stored
baseDirectory = /usr/projects/climate/SHARED_CLIMATE/ACMEv0_lowres/B1850C5_ne30_v0.4/ocn/postprocessing

[seaIceObservations]
## options related to sea ice observations with which the results will be
## compared

# directory where sea ice observations are stored
baseDirectory = /usr/projects/climate/SHARED_CLIMATE/observations/SeaIce

[seaIcePreprocessedReference]
## options related to preprocessed sea ice reference run with which the results
## will be compared (e.g. a CICE, CESM or ACME v0 run)

# directory where ocean reference simulation results are stored
baseDirectory = /usr/projects/climate/SHARED_CLIMATE/ACMEv0_lowres/B1850C5_ne30_v0.4/ice/postprocessing

[regions]
## options related to ocean regions used in several analysis modules

# Directory for region mask files
regionMaskDirectory = /turquoise/usr/projects/climate/SHARED_CLIMATE/mpas_analysis/region_masks

[xylar]

Could you try the same, move the results over to the NERSC portal, and post it here? That would be enough testing to allow us to merge this PR, I think.

[xylar]

@darincomeau, can you review this based on your standalone MPAS-SeaIce testing?

[darincomeau]

@xylar sure, no problem.

[darincomeau]

@xylar I successfully ran the analysis on 10 years of an MPAS-Seaice QU120km standalone on LANL IC, and posted the results here:
http://portal.nersc.gov/project/m2833/dcomeau/mpas_analysis_output/html/
Otherwise just a couple typos noted.

[darincomeau]

typo in 'Prediction'

[darincomeau]

same 'Prediction' typo

[darincomeau]

typo in 'output_interval'

[xylar]

@darincomeau, thanks for the review. The typos have been fixed. I just rebased onto the current develop.

@pwolfram, when might you have simulations to use to check this PR?

[pwolfram]

@xylar, as we are discussing offline the check is in progress now...

[pwolfram]

The output is at http://portal.nersc.gov/project/m2833/pwolfram/mpas_analysis_output/html/. As the SOMA case is an idealized spherical cap, I'm not sure how seriously we should take this output. But, I think the key thing is that this demonstrates the opportunity for stand alone analysis to be implemented in MPAS-Analysis.

[pwolfram]

Verified code completed without error.

[pwolfram]

I'm assuming this is from the wiki we used in this process. Thanks for including it here!

[xylar]

These will become pages in the official documentation on ReadTheDocs as soon as this is merged.

[pwolfram]

One minor piece of food for thought: Note, for whatever reason, if there is bad output in the output directory, there can be a crash (happened to me the first time). I had a suspicion this was the case and turned out to be lucky that a fresh output directory fixed the problem. I don't recall if we have a purge on the output directory or not but it might be a good feature to have if we already don't have it in MPAS-Analysis, especially for new users.

[xylar]

Ha, ha!, the analysis does look pretty silly in this case. But presumably as we build up more capabilities, there could be useful plots here. Thanks for testing, @pwolfram.

[xylar]

@pwolfram, yes, just run ./run_mpas_analysis --purge config.<my_config> to first purge the output directory and then run the analysis. I typically run this way.