Fill in xml for observations #335

xylar · 2018-04-17T13:06:52Z

This merge updates an XML database describing the observations used in MPAS-Analysis and makes some changes to how this database is displayed as tables in the documentation.

xylar · 2018-04-17T13:10:15Z

@milenaveneziani, this is a PR off of #333 for filling in the XML files based on the tables on confluence.

What we want here is some basic information about which observations we use, how a user can get them, and how they can be post-processed to be usable in MPAS-Analysis. Essentially, a user should be able to start from nothing, download MPAS-Analysis (or install it from the e3sm anaconda channel), follow these instructions to set up the observations, and run MPAS-Analysis.

xylar · 2018-04-17T13:10:28Z

Just to note: Another issue is the MOC and ice-shelf region files

xylar · 2018-04-17T13:12:45Z

@milenaveneziani, to the degree that you have time, I would appreciate your help filling in these XML files. In particular, when I get to the point where there are instructions on how to download and install the observations, I will need your help on any data sets you know how to re-create.

There are no tasks that relate to it yet.

Also, rename the file (it is not a table) and modify the requested fields for each entry.

xylar · 2018-04-24T15:03:44Z

mpas_analysis/obs/observational_datasets.xml

+    <bibtex>
+    </bibtex>
+    <dataUrls>
+      (missing)


@milenaveneziani, I need help finding this URL. I've wandered around the Aquarius FTP server with no luck. We need to find Aquarius_V3_SSS_Monthly.nc (or its V4 or V5 equivalent).

Apparently this file/data is no longer available, I cannot find it anywhere either. But I read a bit of the SSS data that is available now, and this seems to be interesting:
https://podaac.jpl.nasa.gov/dataset/AQUARIUS_L4_OISSS_IPRC_7DAY_V4?ids=ProcessingLevel&values=*4*&search=Aquarius
It's a blended product where the OI takes into account ARGO data as well, to correct/amend where the satellite data has a high uncertainty/error. It's also higher resolution, 0.5degx0.5deg.
It's only available as a 7-daily product (many 7-day files), so we would have to compute the monthly data ourselves as preprocessing.

Okay, so it sounds like we just leave this as "missing" for the time being and we'll do a whole process down the road where we make a preprocessing script that hopefully can download the data (if needed) and process it into a monthly climatology. Definitely a longer-term project...

the older one was all monthly data between 2011 and 2015 in one single file, and MPAS-Analysis would do the climatology, right?

MPAS-Analysis would do the climatology, right?

Yes, that's right.

xylar · 2018-04-24T15:04:04Z

mpas_analysis/obs/observational_datasets.xml

+      (missing)
+    </dataUrls>
+    <preprocessing>
+      (missing)


@milenaveneziani, I believe there is no preprocessing, correct?

left preprocessing as 'missing', but I added the source link that points to the blended/OI product.

xylar · 2018-04-24T15:04:43Z

mpas_analysis/obs/observational_datasets.xml

+      (http://www.aviso.altimetry.fr/duacs/)"
+    </releasePolicy>
+    <references>
+      (missing)


@milenaveneziani or @maltrud, are either of you aware of an AVISO reference we should be citing here?

The Journal reference is missing and the doi page brings you to:
https://podaac.jpl.nasa.gov/dataset/AVISO_L4_DYN_TOPO_1DEG_1MO

Maybe just citing this pdf as a @misc?
ftp://podaac.jpl.nasa.gov/allData/aviso/L4/dynamic_topo_1deg_1mo/docs/zosTechNote_AVISO_L4_199210-201012.pdf

xylar · 2018-04-24T15:06:55Z

mpas_analysis/obs/observational_datasets.xml

+      ocean
+    </component>
+    <description>
+     (missing)


@maltrud, can you help fill in all the missing info about the Trenberth and Caron MHT data set?

xylar · 2018-04-24T15:07:25Z

mpas_analysis/obs/observational_datasets.xml

+    </component>
+    <description>
+      (missing)
+    </description>


@vanroekel, can you help fill in the missing info about the 2 Nino 3.4 data sets?

xylar · 2018-04-24T15:08:06Z

mpas_analysis/obs/observational_datasets.xml

+      }
+    </bibtex>
+    <dataUrls>
+      (missing, requires registration with NSIDC)


@milenaveneziani, can you help me figure out what the URL would be here? Same for Bootstrap.

hmm, look at this:
https://urs.earthdata.nasa.gov/oauth/authorize?client_id=OeZxXA9CSNdsGKSltLUmIQ&response_type=code&redirect_uri=https%3A%2F%2Fdaacdata.apps.nsidc.org%2Foauth%2Fredirect_uri&state=aHR0cHM6Ly9kYWFjZGF0YS5hcHBzLm5zaWRjLm9yZy9wdWIvREFUQVNFVFMvbnNpZGMwMDUxX2dzZmNfbmFzYXRlYW1fc2VhaWNlLw
looks like a registration is needed now?? I am confused..

Yes, I'm registered but then I just see a huge number of .bin files, one for each month since 1979. There obviously was also a pre-processing script if this is what the input data was...

Yes, the sea-ice data is the one we (myself and @maltrud) pre-processed the most. We went from binary to netcdf files, and from the polar stereographic projection to the regular lon/lat grid. I did the first step, with a matlab script, and Mat did the second step, either with feret or ncremap, now I can't recall.
But I didn't know now a user registration was necessary..

same for ICESat.

Okay, well, we're going to have to redo that stuff in a python script. Wen we do that, we might want to consider staying with the polar stereographic grid like I already do for the Antarctic melt and SOSE plots (i.e. leaving the obs on that grid and just remapping MPAS data to the same grid). But clearly that's way beyond the scope of the current database.

Any script(s) you have around that you want to add to the preprocess_observations directory, feel free. But I think if your scripts only document part of the preprocessing that was performed, it would be good to at least indicate here that there are missing steps.

Added all scripts I have into the preprocessing_obs folder. I also added a note txt file with reminders about how we preprocessed the single seaice data sets.

xylar · 2018-04-24T15:08:25Z

mpas_analysis/obs/observational_datasets.xml

+      (missing, requires registration with NSIDC)
+    </dataUrls>
+    <preprocessing>
+      (missing)


@milenaveneziani, did you have to preprocess these?

xylar · 2018-04-24T15:09:14Z

mpas_analysis/obs/observational_datasets.xml

+      - https://neptune.gsfc.nasa.gov/uploads/files/SH_IceExt_Monthly_1978-2012.txt
+    </dataUrls>
+    <preprocessing>
+      (missing)


@milenaveneziani, do you have a script you used to convert these text files to NetCDF?

yes, it's a simple matlab script.

Okay, might be worth adding for now (under preprocess_observations). We'll convert it to a python script when we want to use it in an automated way down the road.

xylar · 2018-04-24T15:09:43Z

mpas_analysis/obs/observational_datasets.xml

+      }
+    </bibtex>
+    <dataUrls>
+      (missing, requires registration with NSIDC)


@milenaveneziani, can you figure out the URL?

hmm, I realize tracking future changes in the data sets will be a task in itself.. I now see different text files, which also contain ice area for specific subregions of the Arctic/SO. The last column is the total hemispheric average (as before).
Here are the URLs:
https://neptune.gsfc.nasa.gov/uploads/files/NH_IceArea_Monthly_1978-2012.txt
https://neptune.gsfc.nasa.gov/uploads/files/SH_IceArea_Monthly_1978-2012.txt

xylar · 2018-04-24T15:10:27Z

mpas_analysis/obs/observational_datasets.xml

+      - http://psc.apl.uw.edu/wordpress/wp-content/uploads/schweiger/ice_volume/PIOMAS.2sst.monthly.Current.v2.1.txt
+    </dataUrls>
+    <preprocessing>
+      (missing)


@milenaveneziani, did you have a script for processing the text file into a NetCDF file?

yup, matlab again. I'll add it to the preprocessing folder.

milenaveneziani · 2018-04-24T20:34:13Z

mpas_analysis/obs/observational_datasets.xml

+      co-located on the same 1 degree grids.
+    </description>
+    <source>
+      - [AVISO+ website](https://www.aviso.altimetry.fr/en/data/products/sea-surface-height-products/global/madt-h-uv.html)


I would remove this link, since we got the data from the NASA JPL site. Seems confusing to me.

Okay, fine with me. If you make any other changes to this branch, feel free to remove this link at the same time.

milenaveneziani · 2018-04-25T03:01:17Z

mpas_analysis/obs/observational_datasets.xml

+    <releasePolicy>
+      [Acknowledgment:] Hurrell, J. W., J. J. Hack, D. Shea, J. M. Caron, and J. Rosinski,
+      2008: A New Sea Surface Temperature and Sea Ice Boundary Dataset for the Community 
+      Atmosphere Model. Journal of Climate, 21, 5145-5153.


added this here, from email I got from UCAR.

The scripts were added to the repo but not listed in the database.

xylar · 2018-04-25T07:42:15Z

mpas_analysis/obs/observational_datasets.xml

    </bibtex>
    <dataUrls>
-      (missing)
+      (ftp://podaac-ftp.jpl.nasa.gov/allData/aquarius/L4/IPRC/v4/7day/)


@milenaveneziani, I was thinking that these URLs would be for the individual data files, not just a directory. The idea is that the script will eventually use them to download the data. But maybe the better plan is just to hard-code the URL into the scripts themselves. Especially for cases like this where hundreds of files are needed, a list of files is impractical.

oh, there are many 7-daily files in this case..
Also, we need to think of a best way for us to maintain these scripts in the future, since changes to data sets are likely to happen perhaps more often than we would like..

Yes, my thinking is that we would do a completely "fresh" checkout of the analysis from time to time where we would test that all the downloading and preprocessing works as expected. Presumably this would also show us where datasets have disappeared and scripts need to be updated accordingly.

Obviously, this is well beyond the time frame of this PR ;-)

xylar · 2018-04-25T07:46:53Z

@milenaveneziani, I think in the interest of getting a 0.7.5 tag, we should call this good enough and get input from @vanroekel and @maltrud as part of a separate PR. Updating this database will be an ongoing effort in any case. What do you think?

milenaveneziani

I agree. This is good enough for now.

xylar · 2018-04-25T15:55:23Z

Thanks, @milenaveneziani!

xylar added enhancement in progress priority work in progress labels Apr 17, 2018

xylar self-assigned this Apr 17, 2018

xylar requested a review from milenaveneziani April 17, 2018 13:06

This was referenced Apr 17, 2018

Adds documentation observation workflow #333

Merged

Update to v0.7.5 in docs, setup and recipe #334

Merged

Add ARGO preprocessing jupyter notebook

d9b4291

xylar force-pushed the fill_in_obs_xml branch from 1d54517 to 40429ce Compare April 20, 2018 17:27

xylar added 2 commits April 20, 2018 19:35

Fix capitalization ARGO --> Argo

e29260e

Remove landice component

d2ed189

There are no tasks that relate to it yet.

xylar force-pushed the fill_in_obs_xml branch from 40429ce to a476649 Compare April 20, 2018 17:36

Update obs. database with a number of new data sets

0212f86

Also, rename the file (it is not a table) and modify the requested fields for each entry.

xylar force-pushed the fill_in_obs_xml branch from a476649 to 0212f86 Compare April 24, 2018 14:36

xylar commented Apr 24, 2018

View reviewed changes

milenaveneziani reviewed Apr 24, 2018

View reviewed changes

xylar added the help wanted label Apr 24, 2018

Update obs. database. Add Milena's matlab pre-process sea-ice scripts

d3db913

milenaveneziani reviewed Apr 25, 2018

View reviewed changes

Add Matlab scripts to preprocessing steps in database

da66005

The scripts were added to the repo but not listed in the database.

xylar commented Apr 25, 2018

View reviewed changes

Some clean-up to the obs database

9fd3910

xylar removed in progress help wanted work in progress labels Apr 25, 2018

xylar changed the title ~~Work in Progress: Fill in xml for observations~~ Fill in xml for observations Apr 25, 2018

milenaveneziani approved these changes Apr 25, 2018

View reviewed changes

xylar merged commit f2b8487 into MPAS-Dev:develop Apr 25, 2018

xylar deleted the fill_in_obs_xml branch April 25, 2018 15:56

Fill in xml for observations #335

Fill in xml for observations #335

Uh oh!

Conversation

xylar commented Apr 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Apr 17, 2018

Uh oh!

xylar commented Apr 17, 2018

Uh oh!

xylar commented Apr 17, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milenaveneziani Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xylar Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

xylar commented Apr 17, 2018 •

edited

Loading

milenaveneziani Apr 24, 2018 •

edited

Loading

xylar Apr 24, 2018 •

edited

Loading

milenaveneziani Apr 25, 2018 •

edited

Loading