Add map_variables parameter to read_tmy3 function (#1623)

* Added map_variables param in read_tmy3 func

* solved failing tests

* solved failing tests

* solving stickerly test fail

* solving stickerly test fail

* solving stickerly test fail

* added variable_map

* resolving formatting issues

* Update pvlib/iotools/tmy.py

Co-authored-by: Adam R. Jensen <[email protected]>

* Update pvlib/iotools/tmy.py

Co-authored-by: Adam R. Jensen <[email protected]>

* Update pvlib/iotools/tmy.py

Co-authored-by: Adam R. Jensen <[email protected]>

* Update pvlib/iotools/tmy.py

Co-authored-by: Adam R. Jensen <[email protected]>

* Added tests to map_variables and check depreciation warning

* resolving formatting issues

* resolving formatting issues

* Adding more test cases

* updated VARIABLE_MAP

* updates DOIs in tmy.py

* Final changes as requested

* Fix stickler

* Update variables_style_rules.csv

* Add missing map_variables=False

* Add whatsnew deprecation entry

* Apply suggestions from code review

* Add separate if statement for warning

* removed wrong whatsnew entry

* Update pvlib/tests/iotools/test_tmy.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update warning messages according to table

* Update whatsnew

* Set recolumn=None

* Update tests

* Update pvlib/iotools/tmy.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update pvlib/iotools/tmy.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update docs/sphinx/source/whatsnew/v0.9.6.rst

Co-authored-by: Kevin Anderson <[email protected]>

* Implement updated error message logic from kansersolar

* Clean up deprecation warnings

* Fix clearsky.rst

* Update aod variable mapping

* Fix stickler

* Update tmy3 variables names in field,description table

* Update table

* More table stuff

* Table stuff galore

---------

Co-authored-by: Pratham Chauhan <[email protected]>
Co-authored-by: Adam R. Jensen <[email protected]>
Co-authored-by: Kevin Anderson <[email protected]>
Co-authored-by: Kevin Anderson <[email protected]>

Author: 4 people

docs/examples/adr-pvarray/plot_simulate_system.py

@@ -29,10 +29,12 @@
 PVLIB_DIR = pvlib.__path__[0]
 DATA_FILE = os.path.join(PVLIB_DIR, 'data', '723170TYA.CSV')
 
-tmy, metadata = iotools.read_tmy3(DATA_FILE, coerce_year=1990)
+tmy, metadata = iotools.read_tmy3(DATA_FILE, coerce_year=1990,
+                                  map_variables=True)
 
-df = pd.DataFrame({'ghi': tmy['GHI'], 'dhi': tmy['DHI'], 'dni': tmy['DNI'],
-                   'temp_air': tmy['DryBulb'], 'wind_speed': tmy['Wspd'],
+df = pd.DataFrame({'ghi': tmy['ghi'], 'dhi': tmy['dhi'], 'dni': tmy['dni'],
+                   'temp_air': tmy['temp_air'],
+                   'wind_speed': tmy['wind_speed'],
                    })
 
 # %%

docs/examples/irradiance-decomposition/plot_diffuse_fraction.py

@@ -27,7 +27,8 @@
 # of data measured from 1990 to 2010. Therefore we change the timestamps to a
 # common year, 1990.
 DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'
-greensboro, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
+greensboro, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
+                                 map_variables=True)
 
 # Many of the diffuse fraction estimation methods require the "true" zenith, so
 # we calculate the solar positions for the 1990 at Greensboro, NC.
@@ -36,8 +37,8 @@
 solpos = get_solarposition(
     greensboro.index.shift(freq="-30T"), latitude=metadata['latitude'],
     longitude=metadata['longitude'], altitude=metadata['altitude'],
-    pressure=greensboro.Pressure*100,  # convert from millibar to Pa
-    temperature=greensboro.DryBulb)
+    pressure=greensboro.pressure*100,  # convert from millibar to Pa
+    temperature=greensboro.temp_air)
 solpos.index = greensboro.index  # reset index to end of the hour
 
 # %%
@@ -56,10 +57,10 @@
 # an exponential relation with airmass.
 
 out_disc = irradiance.disc(
-    greensboro.GHI, solpos.zenith, greensboro.index, greensboro.Pressure*100)
+    greensboro.ghi, solpos.zenith, greensboro.index, greensboro.pressure*100)
 # use "complete sum" AKA "closure" equations: DHI = GHI - DNI * cos(zenith)
 df_disc = irradiance.complete_irradiance(
-    solar_zenith=solpos.apparent_zenith, ghi=greensboro.GHI, dni=out_disc.dni,
+    solar_zenith=solpos.apparent_zenith, ghi=greensboro.ghi, dni=out_disc.dni,
     dhi=None)
 out_disc = out_disc.rename(columns={'dni': 'dni_disc'})
 out_disc['dhi_disc'] = df_disc.dhi
@@ -72,11 +73,11 @@
 # developed by Richard Perez and Pierre Ineichen in 1992.
 
 dni_dirint = irradiance.dirint(
-    greensboro.GHI, solpos.zenith, greensboro.index, greensboro.Pressure*100,
-    temp_dew=greensboro.DewPoint)
+    greensboro.ghi, solpos.zenith, greensboro.index, greensboro.pressure*100,
+    temp_dew=greensboro.temp_dew)
 # use "complete sum" AKA "closure" equation: DHI = GHI - DNI * cos(zenith)
 df_dirint = irradiance.complete_irradiance(
-    solar_zenith=solpos.apparent_zenith, ghi=greensboro.GHI, dni=dni_dirint,
+    solar_zenith=solpos.apparent_zenith, ghi=greensboro.ghi, dni=dni_dirint,
     dhi=None)
 out_dirint = pd.DataFrame(
     {'dni_dirint': dni_dirint, 'dhi_dirint': df_dirint.dhi},
@@ -91,7 +92,7 @@
 # splits kt into 3 regions: linear for kt <= 0.22, a 4th order polynomial
 # between 0.22 < kt <= 0.8, and a horizontal line for kt > 0.8.
 
-out_erbs = irradiance.erbs(greensboro.GHI, solpos.zenith, greensboro.index)
+out_erbs = irradiance.erbs(greensboro.ghi, solpos.zenith, greensboro.index)
 out_erbs = out_erbs.rename(columns={'dni': 'dni_erbs', 'dhi': 'dhi_erbs'})
 
 # %%
@@ -102,7 +103,7 @@
 # exponential correlation that is continuously differentiable and bounded
 # between zero and one.
 
-out_boland = irradiance.boland(greensboro.GHI, solpos.zenith, greensboro.index)
+out_boland = irradiance.boland(greensboro.ghi, solpos.zenith, greensboro.index)
 out_boland = out_boland.rename(
     columns={'dni': 'dni_boland', 'dhi': 'dhi_boland'})
 
@@ -118,20 +119,20 @@
 # file together to make plotting easier.
 
 dni_renames = {
-    'DNI': 'TMY3', 'dni_disc': 'DISC', 'dni_dirint': 'DIRINT',
+    'dni': 'TMY3', 'dni_disc': 'DISC', 'dni_dirint': 'DIRINT',
     'dni_erbs': 'Erbs', 'dni_boland': 'Boland'}
 dni = [
-    greensboro.DNI, out_disc.dni_disc, out_dirint.dni_dirint,
+    greensboro.dni, out_disc.dni_disc, out_dirint.dni_dirint,
     out_erbs.dni_erbs, out_boland.dni_boland]
 dni = pd.concat(dni, axis=1).rename(columns=dni_renames)
 dhi_renames = {
-    'DHI': 'TMY3', 'dhi_disc': 'DISC', 'dhi_dirint': 'DIRINT',
+    'dhi': 'TMY3', 'dhi_disc': 'DISC', 'dhi_dirint': 'DIRINT',
     'dhi_erbs': 'Erbs', 'dhi_boland': 'Boland'}
 dhi = [
-    greensboro.DHI, out_disc.dhi_disc, out_dirint.dhi_dirint,
+    greensboro.dhi, out_disc.dhi_disc, out_dirint.dhi_dirint,
     out_erbs.dhi_erbs, out_boland.dhi_boland]
 dhi = pd.concat(dhi, axis=1).rename(columns=dhi_renames)
-ghi_kt = pd.concat([greensboro.GHI/1000.0, out_erbs.kt], axis=1)
+ghi_kt = pd.concat([greensboro.ghi/1000.0, out_erbs.kt], axis=1)
 
 # %%
 # Winter

docs/examples/irradiance-transposition/plot_seasonal_tilt.py

@@ -44,12 +44,13 @@ def get_orientation(self, solar_zenith, solar_azimuth):
 # like we expect:
 
 DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'
-tmy, metadata = iotools.read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
+tmy, metadata = iotools.read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
+                                  map_variables=True)
 # shift from TMY3 right-labeled index to left-labeled index:
 tmy.index = tmy.index - pd.Timedelta(hours=1)
 weather = pd.DataFrame({
-    'ghi': tmy['GHI'], 'dhi': tmy['DHI'], 'dni': tmy['DNI'],
-    'temp_air': tmy['DryBulb'], 'wind_speed': tmy['Wspd'],
+    'ghi': tmy['ghi'], 'dhi': tmy['dhi'], 'dni': tmy['dni'],
+    'temp_air': tmy['temp_air'], 'wind_speed': tmy['wind_speed'],
 })
 loc = location.Location.from_tmy(metadata)
 solpos = loc.get_solarposition(weather.index)

docs/examples/irradiance-transposition/plot_transposition_gain.py

@@ -32,7 +32,8 @@
 DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'
 
 # get TMY3 dataset
-tmy, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
+tmy, metadata = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
+                          map_variables=True)
 # TMY3 datasets are right-labeled (AKA "end of interval") which means the last
 # interval of Dec 31, 23:00 to Jan 1 00:00 is labeled Jan 1 00:00. When rolling
 # up hourly irradiance to monthly insolation, a spurious January value is
@@ -60,9 +61,9 @@ def calculate_poa(tmy, solar_position, surface_tilt, surface_azimuth):
     poa = irradiance.get_total_irradiance(
         surface_tilt=surface_tilt,
         surface_azimuth=surface_azimuth,
-        dni=tmy['DNI'],
-        ghi=tmy['GHI'],
-        dhi=tmy['DHI'],
+        dni=tmy['dni'],
+        ghi=tmy['ghi'],
+        dhi=tmy['dhi'],
         solar_zenith=solar_position['apparent_zenith'],
         solar_azimuth=solar_position['azimuth'],
         model='isotropic')
@@ -97,7 +98,7 @@ def calculate_poa(tmy, solar_position, surface_tilt, surface_azimuth):
 df_monthly['SAT-0.4'] = poa_irradiance.resample('m').sum()
 
 # calculate the percent difference from GHI
-ghi_monthly = tmy['GHI'].resample('m').sum()
+ghi_monthly = tmy['ghi'].resample('m').sum()
 df_monthly = 100 * (df_monthly.divide(ghi_monthly, axis=0) - 1)
 
 df_monthly.plot()

docs/examples/soiling/plot_greensboro_kimber_soiling.py

@@ -40,9 +40,10 @@
 DATA_DIR = pathlib.Path(pvlib.__file__).parent / 'data'
 
 # get TMY3 data with rain
-greensboro, _ = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990)
+greensboro, _ = read_tmy3(DATA_DIR / '723170TYA.CSV', coerce_year=1990,
+                          map_variables=True)
 # get the rain data
-greensboro_rain = greensboro.Lprecipdepth
+greensboro_rain = greensboro['Lprecip depth (mm)']
 # calculate soiling with no wash dates and cleaning threshold of 25-mm of rain
 THRESHOLD = 25.0
 soiling_no_wash = kimber(greensboro_rain, cleaning_threshold=THRESHOLD)

docs/sphinx/source/user_guide/clearsky.rst

@@ -216,25 +216,26 @@ wavelengths [Bir80]_, and is implemented in
 
     In [1]: tmy_file = os.path.join(pvlib_data, '703165TY.csv')  # TMY file
 
-    In [1]: tmy_data, tmy_header = read_tmy3(tmy_file, coerce_year=1999)  # read TMY data
+    In [1]: tmy_data, tmy_header = read_tmy3(tmy_file, coerce_year=1999, map_variables=True)
 
     In [1]: tl_historic = clearsky.lookup_linke_turbidity(time=tmy_data.index,
        ...:     latitude=tmy_header['latitude'], longitude=tmy_header['longitude'])
 
     In [1]: solpos = solarposition.get_solarposition(time=tmy_data.index,
        ...:     latitude=tmy_header['latitude'], longitude=tmy_header['longitude'],
-       ...:     altitude=tmy_header['altitude'], pressure=tmy_data['Pressure']*mbars,
-       ...:     temperature=tmy_data['DryBulb'])
+       ...:     altitude=tmy_header['altitude'], pressure=tmy_data['pressure']*mbars,
+       ...:     temperature=tmy_data['temp_air'])
 
     In [1]: am_rel = atmosphere.get_relative_airmass(solpos.apparent_zenith)
 
-    In [1]: am_abs = atmosphere.get_absolute_airmass(am_rel, tmy_data['Pressure']*mbars)
+    In [1]: am_abs = atmosphere.get_absolute_airmass(am_rel, tmy_data['pressure']*mbars)
 
     In [1]: airmass = pd.concat([am_rel, am_abs], axis=1).rename(
        ...:     columns={0: 'airmass_relative', 1: 'airmass_absolute'})
 
     In [1]: tl_calculated = atmosphere.kasten96_lt(
-       ...:     airmass.airmass_absolute, tmy_data['Pwat'], tmy_data['AOD'])
+       ...:     airmass.airmass_absolute, tmy_data['precipitable_water'],
+       ...:     tmy_data['AOD (unitless)'])
 
     In [1]: tl = pd.concat([tl_historic, tl_calculated], axis=1).rename(
        ...:     columns={0:'Historic', 1:'Calculated'})

docs/sphinx/source/user_guide/timetimezones.rst

@@ -272,7 +272,7 @@ Let's first examine how pvlib handles time when it imports a TMY3 file.
     # some gymnastics to find the example file
     pvlib_abspath = os.path.dirname(os.path.abspath(inspect.getfile(pvlib)))
     file_abspath = os.path.join(pvlib_abspath, 'data', '703165TY.csv')
-    tmy3_data, tmy3_metadata = pvlib.iotools.read_tmy3(file_abspath)
+    tmy3_data, tmy3_metadata = pvlib.iotools.read_tmy3(file_abspath, map_variables=True)
 
     tmy3_metadata
 
@@ -287,7 +287,7 @@ print just a few of the rows and columns of the large dataframe.
 
     tmy3_data.index.tz
 
-    tmy3_data.loc[tmy3_data.index[0:3], ['GHI', 'DNI', 'AOD']]
+    tmy3_data.loc[tmy3_data.index[0:3], ['ghi', 'dni', 'AOD (unitless)']]
 
 The :py:func:`~pvlib.iotools.read_tmy2` function also returns a DataFrame
 with a localized DatetimeIndex.

docs/sphinx/source/whatsnew/v0.9.6.rst

@@ -8,9 +8,16 @@ v0.9.6 (Anticipated June 2023)
 Deprecations
 ~~~~~~~~~~~~
 
+* The ``recolumn`` parameter in :py:func:`pvlib.iotools.read_tmy3`, which maps
+  TMY3 column names to nonstandard alternatives, is now deprecated.
+  We encourage using ``map_variables`` (which produces standard pvlib names) instead.
+  (:issue:`1517`, :pull:`1623`)
 
 Enhancements
 ~~~~~~~~~~~~
+* Added ``map_variables`` argument to the :py:func:`pvlib.iotools.read_tmy3` in
+  order to offer the option of mapping column names to standard pvlib names.
+  (:issue:`1517`, :pull:`1623`)
 * Update the URL used in the :py:func:`pvlib.iotools.get_cams` function. The new URL supports load-balancing
   and redirects to the fastest server. (:issue:`1688`, :pull:`1740`)
 * :py:func:`pvlib.iotools.get_psm3` now has a ``url`` parameter to give the user
@@ -49,5 +56,6 @@ Contributors
 * Siddharth Kaul (:ghuser:`k10blogger`)
 * Kshitiz Gupta (:ghuser:`kshitiz305`)
 * Stefan de Lange (:ghuser:`langestefan`)
+* :ghuser:`ooprathamm`
 * Kevin Anderson (:ghuser:`kandersolar`)
 

pvlib/data/variables_style_rules.csv

@@ -7,6 +7,7 @@ dni_extra;direct normal irradiance at top of atmosphere (extraterrestrial)
 dhi;diffuse horizontal irradiance
 bhi;beam/direct horizontal irradiance
 ghi;global horizontal irradiance
+ghi_extra;horizontal irradiance at top of atmosphere (extraterrestrial)
 gri;ground-reflected irradiance
 aoi;angle of incidence between :math:`90\deg` and :math:`90\deg`
 aoi_projection;cos(aoi)
@@ -32,6 +33,8 @@ relative_humidity;relative humidity
 wind_speed;wind speed
 wind_direction;wind direction
 pressure;atmospheric pressure
+albedo;ratio of reflected solar irradiance to global horizontal irradiance, unitless
+precipitable_water;total precipitable water contained in a column of unit cross section from earth to top of atmosphere
 v_mp, i_mp, p_mp;module voltage, current, power at the maximum power point
 v_oc;open circuit module voltage
 i_sc;short circuit module current