Bug in the calculation of the avg daysSinceStartOfSim in the timeSeriesStats AM

[milenaveneziani]

This issue was discovered while debugging the MPAS-Analysis, and using ACME output data. A close look at the variable timeMonthly_avg_daysSinceStartOfSim revealed that the quantity starts out fine in year 1 (~15 days for month 1, year 1), but then is gradually reduced as the simulation year increases (by a factor that is not clear to me). By the time we get to year 50, a dump of the variable in the 0051-01-01 file gives the following value: 18240.9498811632, which corresponds to 49 years and ~356 days, so we are in December of year 50 rather than in mid January of year 51 as one would expect. This bug is also present in the MPAS-SI timeSeriesStats AM, and again it was found within ACME (not sure if we have the same thing in MPAS standalone).

Since the updated timeSeriesStats AM's now also have xtime_start and xtime_end, this bug will no longer be relevant for the analysis (especially after both MPAS-O and MPAS-SI have identical timeSeriesStats AM's). But it should be investigated nonetheless.

[xylar]

It would be worth investigating if the problem is caused by the analysis member (timeSeriesStats I think?) or if it is present in daysSinceStartOfSim before time averaging. If it's a problem specific to timeSeriesStats, it'll likely have been fixed when MPAS-Seaice pulled in the MPAS-Ocean version of this AM. If it's the former, though, that would be important to fix.

[milenaveneziani]

If it is a problem of timeSeriesStats, then it will not be fixed with the new MPAS-SI AM (I saw the problem in both MPAS-O and MPAS-SI timeSeriesStats results). But it will not be too relevant for the analysis, because we can rely on xtime_start and xtime_end for our time calculations, and those are definitely correct.

[xylar]

@milenaveneziani, Thanks for the clarification. I hadn't understood that the same problem was in both cores. If this is in both MPAS-O and MPAS-SI I suspect this is very lokely to be another manifestation of #1096, which @mark-petersen, @douglasjacobsen and I had looked into a while back but didn't find the cause of.

[akturner]

I attempted to reproduce in MPAS-Seaice in standalone but was unable to. After 34 years the value had not diverged from the expected value.

[milenaveneziani]

I see.. Thanks for checking that @akturner.
The output I was looking at was from ACME simulations..

[xylar]

As with #1096, I suspect this is only showing up in ACME for some reason, not in standalone runs with MPAS components. Very odd!

[jonbob]

It may get fixed with the impending MPAS push to ACME. @pwolfram found that there's an call to ocn_analysis_compute_startup missing in the ACME driver that's present in the standalone case. Any chance that will be responsible?

[xylar]

Any chance that will be responsible?

Sounds quite likely! Thanks for letting us know, @jonbob.

[akturner]

Confirmed a serious divergence in the expected and actual averaged days since start of simulation for beta0. The plot shows this divergence, becoming a whopping 140 days at the end of the 256 year simulation.

[akturner]

Jon rang a test case on wolf to try to determine the cause of the timing issues. The first set of numbers below is from this test on wolf and the second from the beta0 on edison. The second number is the time averaged daysSinceStartOfSim. Each row is a month from the start of the simulation. The drift in the edison data is clearly visible, but curiously is absent from the wolf run.

pn1512828:analysis akt$ more output_wolf.txt
15.500000 15.541667
45.000000 45.020833
74.500000 74.520833
105.000000 105.020833
135.500000 135.520833
166.000000 166.020833
196.500000 196.520833
227.500000 227.520833
pn1512828:analysis akt$ more output.txt
15.500000 15.479167
45.000000 44.933037
74.500000 74.399867
105.000000 104.854169
135.500000 135.317879
166.000000 165.769447
196.500000 196.235890
227.500000 227.194224

[akturner]

And the above observation is true of both MPAS-Ocean and MPAS-Seaice.

[akturner]

Global stats on edison also has the daysSinceStartOfSim in it for beta zero. Taking the 0251-01-01 dated files for time series stats and global stats. 250 years on simulation has happened by that date = 250 * 365 = 91250 days. The global stats file has 31 entries ranging from 91251 to 91280 which seems correct. The time series stats file has 91142.83 which seems too low. This seems to suggest that it was a problem with the implementation of time series stats. Has time series stats changed since beta0 for the ocean?