Skip to content

add UO SRML data reader #589

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wholmgren opened this issue Sep 21, 2018 · 4 comments
Closed

add UO SRML data reader #589

wholmgren opened this issue Sep 21, 2018 · 4 comments
Labels
enhancement io solarfx2 DOE SETO Solar Forecasting 2 / Solar Forecast Arbiter
Milestone

Comments

@wholmgren
Copy link
Member

Add data reader for the University of Oregon Solar Radiation Monitoring Laboratory archive http://solardat.uoregon.edu/

Primarily interested in the 1 minute data sets. Challenges include

  • translating numeric columns to pvlib names
  • stitching together files across months
  • stitching together files when the data format at a station changes
  • determining metadata for a station

This should be a new module under the pvlib.iotools subpackage proposed in #272.

cc @lboeman

@wholmgren wholmgren added enhancement solarfx2 DOE SETO Solar Forecasting 2 / Solar Forecast Arbiter labels Sep 21, 2018
@wholmgren wholmgren added this to the 0.6.1 milestone Sep 21, 2018
@wholmgren wholmgren added the io label Sep 21, 2018
@lboeman
Copy link
Contributor

lboeman commented Sep 26, 2018

A couple more challenges I have discovered:

  • Some files/stations have multiple values for the same variable recorded by different instruments. For instance, the file from Eugene below has 6 ghi measurements and 3 dni (100x and 201x respectively).
  • It seems both -999 and -99.9 exist as fill values. For example, ambient temperature (9303) has a fill value of -99.9 and dhi (3001) has a fill value of -999 in this file: http://solardat.uoregon.edu/download/Archive/EUPO1704.txt

@wholmgren
Copy link
Member Author

We might want to map all 100[1-9] values to ghi_[1-9], all 201[1-9] to dni_[1-9], etc. It would be convenient to also get a simple set of names like ghi and dni but the feasibility of that depends on the failure modes of the data. Maybe median of all values for which the QA flag is ok. So a file with 3 GHI columns labeled 1001, 1002, 1003 would produce a DataFrame with 4 columns: ghi_1, ghi_2, ghi_3, ghi where ghi = data.filter(like='ghi').median()

@cwhanse @adriesse are there recommended algorithms for how to combine multiple measurements of the same irradiance quantities from a single location?

Probably ok to set nans like data[data < -99] = np.nan.

@cwhanse
Copy link
Member

cwhanse commented Sep 26, 2018

depends on the instruments in use. For the same quality/type of instrument, it makes sense to average the measurements (regarding the differences among instruments as random error). It does not make sense to average among different instruments (e.g., photodiodes and thermopile pyranometers).

@wholmgren
Copy link
Member Author

Good point on instrument type. I don't see a simple way to handle this for the UO data set based only on the metadata on their website. Maybe they have more available offline. For now I suggest we move forward with a PR that simply maps the tsv columns like 1001, 1002, 1003 to a DataFrame with columns like ghi_1, ghi_2, ghi_3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement io solarfx2 DOE SETO Solar Forecasting 2 / Solar Forecast Arbiter
Projects
None yet
Development

No branches or pull requests

3 participants