Skip to content

Fill in xml for observations #335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 25, 2018
Merged

Fill in xml for observations #335

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d9b4291
Add ARGO preprocessing jupyter notebook
xylar Apr 20, 2018
e29260e
Fix capitalization ARGO --> Argo
xylar Apr 20, 2018
d2ed189
Remove landice component
xylar Apr 20, 2018
0212f86
Update obs. database with a number of new data sets
xylar Apr 20, 2018
d3db913
Update obs. database. Add Milena's matlab pre-process sea-ice scripts
milenaveneziani Apr 24, 2018
da66005
Add Matlab scripts to preprocessing steps in database
xylar Apr 25, 2018
9fd3910
Some clean-up to the obs database
xylar Apr 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions mpas_analysis/obs/observational_datasets.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,9 @@
(https://climatedataguide.ucar.edu/climate-data/merged-hadley-noaaoi-sea-surface-temperature-sea-ice-concentration-hurrell-et-al-2008)
</source>
<releasePolicy>
Unknown
[Acknowledgment:] Hurrell, J. W., J. J. Hack, D. Shea, J. M. Caron, and J. Rosinski,
2008: A New Sea Surface Temperature and Sea Ice Boundary Dataset for the Community
Atmosphere Model. Journal of Climate, 21, 5145-5153.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added this here, from email I got from UCAR.

</releasePolicy>
<references>
[Hurrell et al. (2008)](https://doi.org/10.1175/2008JCLI2292.1)
Expand Down Expand Up @@ -76,7 +78,8 @@
</description>
<source>
[NASA Aquarius FTP server]
(ftp://podaac-ftp.jpl.nasa.gov/allData/aquarius/L3/mapped/V4/7day_running/SCI/)
(https://podaac.jpl.nasa.gov/dataset/AQUARIUS_L4_OISSS_IPRC_7DAY_V4)

</source>
<releasePolicy>
NASA data are not copyrighted; however, when you publish our data or
Expand All @@ -91,7 +94,7 @@
<bibtex>
</bibtex>
<dataUrls>
(missing)
(ftp://podaac-ftp.jpl.nasa.gov/allData/aquarius/L4/IPRC/v4/7day/)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@milenaveneziani, I was thinking that these URLs would be for the individual data files, not just a directory. The idea is that the script will eventually use them to download the data. But maybe the better plan is just to hard-code the URL into the scripts themselves. Especially for cases like this where hundreds of files are needed, a list of files is impractical.

Copy link
Collaborator

@milenaveneziani milenaveneziani Apr 25, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, there are many 7-daily files in this case..
Also, we need to think of a best way for us to maintain these scripts in the future, since changes to data sets are likely to happen perhaps more often than we would like..

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, my thinking is that we would do a completely "fresh" checkout of the analysis from time to time where we would test that all the downloading and preprocessing works as expected. Presumably this would also show us where datasets have disappeared and scripts need to be updated accordingly.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obviously, this is well beyond the time frame of this PR ;-)

</dataUrls>
<preprocessing>
(missing)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@milenaveneziani, I believe there is no preprocessing, correct?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left preprocessing as 'missing', but I added the source link that points to the blended/OI product.

Expand Down Expand Up @@ -124,7 +127,6 @@
co-located on the same 1 degree grids.
</description>
<source>
- [AVISO+ website](https://www.aviso.altimetry.fr/en/data/products/sea-surface-height-products/global/madt-h-uv.html)
- [NASA JPL AVISO website](https://podaac.jpl.nasa.gov/dataset/AVISO_L4_DYN_TOPO_1DEG_1MO)
</source>
<releasePolicy>
Expand All @@ -136,7 +138,7 @@
(http://www.aviso.altimetry.fr/duacs/)"
</releasePolicy>
<references>
(missing)
[ftp://podaac.jpl.nasa.gov/allData/aviso/L4/dynamic_topo_1deg_1mo/docs/zosTechNote_AVISO_L4_199210-201012.pdf]
</references>
<bibtex>
</bibtex>
Expand Down
73 changes: 73 additions & 0 deletions preprocess_observations/Bootstrap_Antarctic_compute_climo.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
%
% Combined SSM/I-SSMIS and SMMR sea-ice concentration data for both the
% Arctic and Antarctic.
% *** Bootstrap algorithm ***
%
% Source: http://nsidc.org/data/NSIDC-0079
%
clear all;
close all;

homedir = getenv('HOME');

maindir = [homedir '/ACME/observations/obsdir/SeaIce/SSMI/'...
'Bootstrap_NSIDC0079'];
datadir = [maindir '/final-gsfc/south/monthly'];
griddir = [homedir '/ACME/observations/obsdir/SeaIce/ICESat/Antarctic/climo'];
lonlatfile = 'spring_ICESat_gridded_mean_thickness';
months4climo = [12 1 2; % (boreal) winter months
3 4 5; % (boreal) spring months
6 7 8; % (boreal) summer months
9 10 11]; % (boreal) fall months
seasons = {'djf','mam','jja','son'};

M = 332; % Polar Stereographic grid for Antarctica, # of rows
N = 316; % Polar Stereographic grid for Antarctica, # of columns

[err,cwd] = unix('pwd');
% Read in lon,lat:
eval(sprintf('cd %s;',griddir));
eval(sprintf('load %s.txt',lonlatfile));
eval(sprintf('cd %s;',cwd));
eval(sprintf('lat = %s(:,1);',lonlatfile));
eval(sprintf('lon = %s(:,2);',lonlatfile));
x = reshape(lon,M,N);
y = reshape(lat,M,N);

eval(sprintf('cd %s;',datadir));
for is=1:length(seasons),
season = char(seasons(is));

% Get list of files to compute seasonal climatologies from:
filenames = sprintf('bt_????%02d*',months4climo(is,1));
for i=2:size(months4climo,2),
filenames = [filenames sprintf(' bt_????%02d*',months4climo(is,i))];
end
eval(sprintf('[err,filelist] = unix(''/bin/ls %s'');',filenames));
filelist = strsplit(filelist); % convert to cell of strings
filelist = filelist(2:end-1); % remove redudant first and last space chars

% Compute seasonal climatology:
for ifile=1:length(filelist);
infile = char(filelist(ifile));
% Read in binary data:
fid = fopen(infile,'r','l');
aice(ifile,:,:) = fread(fid,[N M],'int16');
fclose(fid);
aice(find(aice>1000)) = nan;
aice(find(aice==0))=nan;
end
aice = squeeze(nanmean(aice,1));
aice = aice/1000;
aice(find(isnan(aice)==1)) = -999;
aice = aice';

% Write to file:
outfile = ['SSMI_Bootstrap_gridded_concentration_SH_' season];
fid = fopen(sprintf('%s/%s.txt',maindir,outfile),'w');
fprintf(fid,'%12.4f %12.4f %12.5f\n',[y(:)'; x(:)'; aice(:)']);
fclose(fid);

clear aice
end
eval(sprintf('cd %s;',cwd));
73 changes: 73 additions & 0 deletions preprocess_observations/Bootstrap_Arctic_compute_climo.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
%
% Combined SSM/I-SSMIS and SMMR sea-ice concentration data for both the
% Arctic and Antarctic.
% *** Bootstrap algorithm ***
%
% Source: http://nsidc.org/data/NSIDC-0079
%
clear all;
close all;

homedir = getenv('HOME');

maindir = [homedir '/ACME/observations/obsdir/SeaIce/SSMI/'...
'Bootstrap_NSIDC0079'];
datadir = [maindir '/final-gsfc/north/monthly'];
griddir = [homedir '/ACME/observations/obsdir/SeaIce/ICESat/Arctic/' ...
'NSIDC0393_GLAS_SI_Freeboard_v01/glas_seaice_grids'];
lonfile = [griddir '/PS25km_north_lon.img'];
latfile = [griddir '/PS25km_north_lat.img'];
months4climo = [ 1 2 3; % (boreal) winter months
4 5 6; % (boreal) spring months
7 8 9; % (boreal) summer months
10 11 12]; % (boreal) fall months
seasons = {'jfm','amj','jas','ond'};

M = 304; % Polar Stereographic grid for the Arctic, # of rows
N = 448; % Polar Stereographic grid for the Arctic, # of columns

% Read in lon,lat:
fid = fopen(lonfile,'r','l');
x = fread(fid,[M N],'single');
fclose(fid);
fid = fopen(latfile,'r','l');
y = fread(fid,[M N],'single');
fclose(fid);

[err,cwd] = unix('pwd');
eval(sprintf('cd %s;',datadir));
for is=1:length(seasons),
season = char(seasons(is));

% Get list of files to compute seasonal climatologies from:
filenames = sprintf('bt_????%02d*',months4climo(is,1));
for i=2:size(months4climo,2),
filenames = [filenames sprintf(' bt_????%02d*',months4climo(is,i))];
end
eval(sprintf('[err,filelist] = unix(''/bin/ls %s'');',filenames));
filelist = strsplit(filelist); % convert to cell of strings
filelist = filelist(2:end-1); % remove redudant first and last space chars

% Compute seasonal climatology:
for ifile=1:length(filelist);
infile = char(filelist(ifile));
% Read in binary data:
fid = fopen(infile,'r','l');
aice(ifile,:,:) = fread(fid,[M N],'int16');
fclose(fid);
aice(find(aice>1000)) = nan;
aice(find(aice==0))=nan;
end
aice = squeeze(nanmean(aice,1));
aice = aice/1000;
aice(find(isnan(aice)==1)) = -999;

% Write to file:
outfile = ['SSMI_Bootstrap_gridded_concentration_NH_' season];
fid = fopen(sprintf('%s/%s.txt',maindir,outfile),'w');
fprintf(fid,'%12.4f %12.4f %12.5f\n',[y(:)'; x(:)'; aice(:)']);
fclose(fid);

clear aice
end
eval(sprintf('cd %s;',cwd));
95 changes: 95 additions & 0 deletions preprocess_observations/ICESat_Arctic_compute_climo.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
%
% ICESat sea-ice thickness data for the Arctic.
%
% Source: http://nsidc.org/data/NSIDC-0393
%
clear all;
close all;

homedir = getenv('HOME');

datadir = [homedir '/ACME/observations/obsdir/SeaIce/ICESat/Arctic/' ...
'NSIDC0393_GLAS_SI_Freeboard_v01/glas_seaice_grids'];
% Infile names (austral spring (Oct-Nov), max Antarctic ice extent;
% austral late summer (Feb-Mar), min Antarctic ice extent):
on_files_th = {'laser2a_thickness_mskd.img','laser3a_thickness_mskd.img',...
'laser3d_thickness_mskd.img','laser3g_thickness_mskd.img',...
'laser3i_thickness_mskd.img','laser3k_thickness_mskd.img'};
fm_files_th = {'laser1_thickness_mskd.img' ,'laser2b_thickness_mskd.img',...
'laser3b_thickness_mskd.img','laser3e_thickness_mskd.img',...
'laser3h_thickness_mskd.img','laser3j_thickness_mskd.img'};
on_files_fb = {'laser2a_freeboard_mskd.img','laser3a_freeboard_mskd.img',...
'laser3d_freeboard_mskd.img','laser3g_freeboard_mskd.img',...
'laser3i_freeboard_mskd.img','laser3k_freeboard_mskd.img'};
fm_files_fb = {'laser1_freeboard_mskd.img' ,'laser2b_freeboard_mskd.img',...
'laser3b_freeboard_mskd.img','laser3e_freeboard_mskd.img',...
'laser3h_freeboard_mskd.img','laser3j_freeboard_mskd.img'};
outfiles = {'ICESat_gridded_mean_thickness_NH_on',...
'ICESat_gridded_mean_thickness_NH_fm'};
lonfile = 'PS25km_north_lon.img';
latfile = 'PS25km_north_lat.img';

M = 304; % Polar Stereographic grid for the Arctic, # of rows
N = 448; % Polar Stereographic grid for the Arctic, # of columns

eval(sprintf('cd %s;',datadir));

% Read in lon,lat:
fid = fopen(lonfile,'r','l');
x = fread(fid,[M N],'single');
fclose(fid);
fid = fopen(latfile,'r','l');
y = fread(fid,[M N],'single');
fclose(fid);

% Compute Oct-Nov climatologies:
for ifile=1:length(on_files_th),
on_file = char(on_files_th(ifile));
fid = fopen(on_file,'r','l');
hi(ifile,:,:) = fread(fid,[M N],'single');
fclose(fid);
end
hi(find(hi<0)) = nan;
hi = squeeze(nanmean(hi,1));
hi(find(isnan(hi)==1)) = -999;
for ifile=1:length(on_files_fb),
on_file = char(on_files_fb(ifile));
fid = fopen(on_file,'r','l');
fb(ifile,:,:) = fread(fid,[M N],'single');
fclose(fid);
end
fb(find(fb<0)) = nan;
fb = squeeze(nanmean(fb,1));
fb(find(isnan(fb)==1)) = -999;
% Write to file:
outfile = char(outfiles(1));
fid = fopen(sprintf('%s.txt',outfile),'w');
fprintf(fid,'%12.4f %12.4f %12.5f %12.5f\n',[y(:)'; x(:)'; fb(:)'; hi(:)']);
fclose(fid);

clear hi fb

% Compute Feb-Mar climatologies:
for ifile=1:length(fm_files_th),
fm_file = char(fm_files_th(ifile));
fid = fopen(fm_file,'r','l');
hi(ifile,:,:) = fread(fid,[M N],'single');
fclose(fid);
end
hi(find(hi<0)) = nan;
hi = squeeze(nanmean(hi,1));
hi(find(isnan(hi)==1)) = -999;
for ifile=1:length(fm_files_fb),
fm_file = char(fm_files_fb(ifile));
fid = fopen(fm_file,'r','l');
fb(ifile,:,:) = fread(fid,[M N],'single');
fclose(fid);
end
fb(find(fb<0)) = nan;
fb = squeeze(nanmean(fb,1));
fb(find(isnan(fb)==1)) = -999;
% Write to file:
outfile = char(outfiles(2));
fid = fopen(sprintf('%s.txt',outfile),'w');
fprintf(fid,'%12.4f %12.4f %12.5f %12.5f\n',[y(:)'; x(:)'; fb(:)'; hi(:)']);
fclose(fid);
96 changes: 96 additions & 0 deletions preprocess_observations/IceaArea_txt2netcdf.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
% Converts txt IceArea files to netcdf files
%
% Data source: http://neptune.gsfc.nasa.gov/csb/index.php?section=59
%

workdir = '/lustre/atlas1/cli115/proj-shared/milena/observations/SeaIce/IceArea_timeseries';
files = {'iceAreaNH','iceAreaSH'};

cwd = pwd;
eval(sprintf('cd %s;',workdir));

for i=1:length(files),
varname = char(files(i));
infile = sprintf('%s_year.txt',varname);
outfile1 = sprintf('%s_year.nc', varname);
outfile2 = sprintf('%s_climo.nc', varname);
varname = [varname '_year'];

% the following loads t (yyyy.yearfraction),icearea (km^2)
eval(sprintf('load %s;',infile));

% First create netcdf file with original data:
%
% create netcdf file
ncid = netcdf.create(outfile1,'clobber');
% define dimension(s)
t_dimid = netcdf.defDim(ncid,'Time',netcdf.getConstant('NC_UNLIMITED'));
strLen_dimid = netcdf.defDim(ncid,'StrLen',64);
% define variables and attributes
t_varid = netcdf.defVar(ncid,'xtime','NC_CHAR',[strLen_dimid,t_dimid]);
%t_varid = netcdf.defVar(ncid,'time','NC_DOUBLE',t_dimid);
icearea_varid = netcdf.defVar(ncid,'IceArea','NC_DOUBLE',t_dimid);
netcdf.putAtt(ncid,t_varid,'long_name','calendar date');
%netcdf.putAtt(ncid,t_varid,'long_name','days since 0001-01-01');
netcdf.putAtt(ncid,t_varid,'format','YYYY-MM-DD_HH:MM:SS character string');
netcdf.putAtt(ncid,icearea_varid,'long_name','SSM/I derived ice area');
netcdf.putAtt(ncid,icearea_varid,'units','km^2');

% leave define mode and enter data mode to write data
netcdf.endDef(ncid);

% add variables
eval(sprintf('t = %s(:,1);',varname));
eval(sprintf('var = %s(:,2);',varname));
% go from year.fraction_of_year to date (string format)
tdate = yf2num(t);
date = datestr(tdate,'yyyy-mm-dd_HH:MM:SS');
ntot = length(t);
%netcdf.putVar(ncid,t_varid,0,ntot,t);
for n=1:ntot,
netcdf.putVar(ncid,t_varid,[0,n-1],[size(date,2),1],date(n,:));
end
netcdf.putVar(ncid,icearea_varid,0,ntot,var);

% close netcdf file
netcdf.close(ncid);

% Then create netcdf file with climatological annual cycle:
%
% create netcdf file
ncid = netcdf.create(outfile2,'clobber');
% define dimension(s)
t_dimid = netcdf.defDim(ncid,'Time',netcdf.getConstant('NC_UNLIMITED'));
strLen_dimid = netcdf.defDim(ncid,'StrLen',64);
% define variables and attributes
t_varid = netcdf.defVar(ncid,'xtime','NC_CHAR',[strLen_dimid,t_dimid]);
%t_varid = netcdf.defVar(ncid,'time','NC_DOUBLE',t_dimid);
icearea_varid = netcdf.defVar(ncid,'IceArea','NC_DOUBLE',t_dimid);
netcdf.putAtt(ncid,t_varid,'long_name','climatological date');
%netcdf.putAtt(ncid,t_varid,'long_name','days since 0001-01-01');
netcdf.putAtt(ncid,t_varid,'format','YYYY-MM-DD_HH:MM:SS character string');
netcdf.putAtt(ncid,icearea_varid,'long_name','SSM/I derived ice area');
netcdf.putAtt(ncid,icearea_varid,'units','km^2');

% leave define mode and enter data mode to write data
netcdf.endDef(ncid);

% add variables
tdatevec = datevec(tdate);
for im=1:12,
indmonth = find(tdatevec(:,2)==im);
var_climo(im) = nanmean(var(indmonth));
end
tdate = datenum(1,1:12,15);
date = datestr(tdate,'yyyy-mm-dd_HH:MM:SS');
%netcdf.putVar(ncid,t_varid,0,ntot,t);
for n=1:12,
netcdf.putVar(ncid,t_varid,[0,n-1],[size(date,2),1],date(n,:));
end
netcdf.putVar(ncid,icearea_varid,0,12,var_climo);

% close netcdf file
netcdf.close(ncid);
end % loop on data files

eval(sprintf('cd %s;',cwd));
Loading