Add numpy SPECTRL2 implementation #1062

kandersolar · 2020-09-20T01:23:02Z

~~Closes #xxxx~~
I am familiar with the contributing guidelines
Tests added
Updates entries to docs/sphinx/source/api.rst for API changes.
Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
Pull request is nearly complete and ready for detailed review.
Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

Additional to-do:

Get it to broadcast across vector inputs
remove debugging code

Calculating the spectral distribution for a single timestamp takes under a millisecond on my laptop. I've been comparing against the solar_utils wrapper around the NREL C implementation (thanks @mikofski for making it so easy to use!) and have discovered several discrepancies between the equations in the report, the report code appendix, and the C implementation. It doesn't help that the C implementation uses its own (non-SPA) solar position calculator. By matching the python code to the C implementation for each of the differences, the difference in output spectra is small enough for me to start blaming the accumulated error from the C implementation's single-precision floats:

Discrepancies:

Equation	Report	Appendix	spectrl2_2.c
2-4	1.335	1.335	1.3366
2-11	118.93	118.93	118.3
3-8	To'	Tu'	Tu'
3-5, 3-6, 3-7, 3-1	double-count Cs	single-count Cs	single-count Cs
2-5	kasten1966	kasten1966	kastenyoung1989

kandersolar · 2020-09-20T21:41:58Z

I think this is ready for review whenever folks get a chance. For the sake of ease of testing I've aligned all the inconsistencies (see table in original comment) with spectrl2_2.c but I'm open to alternatives. See https://gist.github.com/kanderso-nrel/0dc7e7ebff7b5a9c13b52f8b48eb71ee for the validation scratchpad I've been using.

I created a new spectrum module for this. Would atmosphere be a better home? Could consider moving the other spectral functions (pvlib.atmosphere.first_solar_spectral_correction, pvlib.pvsystem.sapm_spectral_loss) here if not.

Thanks!

cwhanse · 2020-09-21T16:17:13Z

I created a new spectrum module for this. Would atmosphere be a better home? Could consider moving the other spectral functions (pvlib.atmosphere.first_solar_spectral_correction, pvlib.pvsystem.sapm_spectral_loss) here if not.

I'm in favor the new spectrum.py and, related to #436, eventually moving functions that calculate spectral modifiers from pvsystem.py and atmosphere.py into spectrum.py.

adriesse · 2020-09-24T20:05:13Z

Nice work.

wholmgren

Looks good, but I didn't compare the implementation to the reference.

wholmgren · 2020-12-10T15:55:34Z

pvlib/spectrum.py

+
+    Parameters
+    ----------
+    surface_tilt : float or numpy array


I'm fine with numpy array but worth considering numpy.ndarray for precision.

Do you need to say something about the dimensions of the array arguments?

pvlib/spectrum.py

cwhanse

I'll volunteer to check this against the reference.

Is there a reason to exclude pandas.Series from the input types?

pvlib/spectrum.py

adriesse · 2020-12-10T18:47:38Z

pvlib/spectrum.py

+
+    Parameters
+    ----------
+    surface_tilt : float or numpy array


Do you need to say something about the dimensions of the array arguments?

adriesse · 2020-12-10T18:56:31Z

pvlib/spectrum.py

+    # Is = Ir + Ia + Ig  # Eq 3-1
+    Is = (Ir + Ia + Ig) * Cs  # Eq 3-1
+
+    # calculate spectral irradiance on a tilted surface, Eq 3-18


Could transposition be a separate function? Or could the use be expected to do it using the existing transposition functions?

It could be a separate function, but I'm inclined to implement the complete model.

If users want to do their own transposition, is there any issue with them passing surface_tilt=0 and transposing afterwards (other than the inefficiency of doing an unnecessary transposition in this function)?

kandersolar

Is there a reason to exclude pandas.Series from the input types?

Probably, but I'll have to remind myself what it is. I'd guess to make broadcasting easier. Whatever the reason, re-wrapping in a Series before returning would probably not be an issue.

kandersolar · 2020-12-10T20:35:09Z

pvlib/spectrum.py

+    # Is = Ir + Ia + Ig  # Eq 3-1
+    Is = (Ir + Ia + Ig) * Cs  # Eq 3-1
+
+    # calculate spectral irradiance on a tilted surface, Eq 3-18


It could be a separate function, but I'm inclined to implement the complete model.

If users want to do their own transposition, is there any issue with them passing surface_tilt=0 and transposing afterwards (other than the inefficiency of doing an unnecessary transposition in this function)?

pvlib/spectrum.py

cwhanse · 2020-12-10T20:49:45Z

Is there a reason to exclude pandas.Series from the input types?

Probably, but I'll have to remind myself what it is. I'd guess to make broadcasting easier. Whatever the reason, re-wrapping in a Series before returning would probably not be an issue.

I'm thinking of users who would bring e.g. surface_tilt or apparent_zenith into this function as a Series.

into spectrl2

kandersolar · 2020-12-16T04:11:21Z

Inputs now allowed to be pd.Series. I'm having trouble articulating why, but I'm less enthusiastic about returning pandas objects -- housing the spectra in DataFrames feels like it's misusing DataFrames in some way. Maybe it's because the column names would be floats (one per wavelength). To me it seems cleaner to always return arrays, but I wouldn't mind being overruled.

To-do items:

fix docs build warning: docstring of pvlib.spectrum.spectrl2:49: WARNING: Unexpected indentation.
fix docstring typo "sigle"

pvlib/spectrum.py

adriesse · 2020-12-17T10:28:04Z

This module is already quite big isn't it? What about creating pvlib/spectrum/spectral2.py? or pvlib/spectral/spectral2.py? Then if anyone ever translates smarts they'll know where to put it. :)

adriesse · 2020-12-17T10:34:09Z

Regarding the conversion from Series to arrays, I find that numpy usually plays pretty well with Series, so I would be interested to learn where it causes problems for you. Maybe you don't really need to convert them all?

pvlib/spectrum.py

adriesse · 2020-12-17T09:35:10Z

pvlib/spectrum.py

+    several ways from the original report [1]_.  The report itself also has
+    a few differences between the in-text equations and the code appendix.
+    The list of known differences is shown below.  Note that this
+    implementation follows ``spectrl2_2.c``.


Did you happen to compare with the Excel version from NREL as well?
If there are two distinct variations, we may give the user the choice as we have done for other models.

I didn't compare with the Excel version in any meaningful way. I agree it would be nice to be able to switch between the two (if there is indeed a difference), but I hesitate to add additional complexity to this PR. Future work?

maybe someone at NREL would know if the C code or the Excel version is in some sense authoritative. I agree with deferring any kind of switch.

pvlib/spectrum.py

cwhanse · 2020-12-17T15:23:52Z

This module is already quite big isn't it? What about creating pvlib/spectrum/spectral2.py? or pvlib/spectral/spectral2.py? Then if anyone ever translates smarts they'll know where to put it. :)

I support this, even though a pvlib SMARTS won't appear any time soon.

kandersolar · 2020-12-18T02:29:19Z

Thanks @adriesse for the numpy lesson :) The numpy vs pandas issue is that I want to cleanly produce the outer product of two vectors where one axis indexes time and the other indexes wavelength, for example:

N = 2
airmass = pd.Series(range(N), index=pd.date_range('2019-01-01', freq='h', periods=N))
vapor_coeff = np.array([1, 2, 3])[:, np.newaxis]
airmass * vapor_coeff  # fails, but airmass.values * vapor_coeff works and returns an array with shape (3, 2)

Is there a nice way to get this behavior when one is a Series?

adriesse · 2020-12-18T11:18:46Z

I guess I haven't run into this exact situation often. When one argument is a Series, the pandas.__mul__ function takes over, and it doesn't play along. (Might even warrant raising an issue.)

A clean way to get the desired behavior would be to use np.outer. This would also make the intent entirely obvious, which is not the case when you rely on broadcasting with *. It would require you to review each multiplication to see which one needs it, which is obviously a bit of work. If you need a lot of them, it might indeed encumber the code and be considered "not nice". On the other hand making the code intent obvious is "nice" in another way.

I won't hold anything up though. I just provide my thoughts on what is overall a very good contribution to pvlib!

adriesse · 2020-12-18T11:30:16Z

pvlib/spectrum/spectrl2.py

+    ('water_vapor_absorption', 'float64'),
+    ('ozone_absorption', 'float64'),
+    ('mixed_absorption', 'float64'),
+]))


There seems to be little benefit to putting the constants into a structured array. How about 5 constant column vectors like this?

WAVELENGTH = np.transpose(np.array([[ 300.0, 305..0 etc. ]]))

A side benefit is that one can recognize the use of the constants in the code more easily because they are CAPS.

Of course if you do end up using np.outer the constants can remain row vectors.

cwhanse

Nice work @kanderso-nrel easily traceable to the reference and well-organized.

cwhanse · 2020-12-22T21:50:34Z

pvlib/spectrum/spectrl2.py

+    wavelength = _SPECTRL2_COEFFS['wavelength'][:, np.newaxis]
+    spectrum_et = _SPECTRL2_COEFFS['spectral_irradiance_et'][:, np.newaxis]
+
+    optical_thickness = (


Use pvlib.atmosphere.angstrom_aod_at_lambda, see

pvlib-python/pvlib/atmosphere.py

Line 620 in 6e936f5

def angstrom_aod_at_lambda(aod0, lambda0, alpha=1.14, lambda1=700.0):

cwhanse · 2020-12-22T22:14:28Z

pvlib/spectrum/spectrl2.py

+    ALG = np.log(1 - aerosol_asymmetry_factor)  # Eq 3-14
+    BFS = ALG * (0.0783 + ALG * (-0.3824 - ALG * 0.5874))  # Eq 3-13
+    AFS = ALG * (1.459 + ALG * (0.1595 + ALG * 0.4129))  # Eq 3-12
+    cosZ = cosd(apparent_zenith)


duplicate line

pvlib/spectrum/spectrl2.py

Co-authored-by: Cliff Hansen <[email protected]>

kandersolar · 2020-12-22T22:50:14Z

FYI I updated the comparison notebook using the current HEAD of this branch. No significant changes from the original plots at the top of this thread. https://gist.github.com/kanderso-nrel/0dc7e7ebff7b5a9c13b52f8b48eb71ee

adriesse · 2020-12-29T22:01:44Z

What's a good and convenient source for precipitable_water, ozone, aerosol_turbidity_500nm data? Preferably multiple values per day, USA and/or worldwide.

mikofski · 2020-12-29T23:14:15Z

What's a good and convenient source for precipitable_water, ozone, aerosol_turbidity_500nm data? Preferably multiple values per day, USA and/or worldwide.

ECMWF, try get_ecmwf_macc

adriesse · 2020-12-29T23:20:37Z

I will! There's a lot of stuff in iotools. Where have I been?

wholmgren · 2020-12-29T23:36:24Z

ECMWF, try get_ecmwf_macc

plus angstrom_aod_at_lambda

Are we ready to merge this?

kandersolar · 2020-12-29T23:52:50Z

There seems to be little benefit to putting the constants into a structured array. How about 5 constant column vectors like this?

I never responded to @adriesse's last suggestion -- I had the vague notion that someone might appreciate the constants already being in tabular form so that they can easily export it for another purpose. Other than that, you're right that columns only get used individually, so not much need for a structured array. For this and using np.outer instead of implicit broadcasting, I don't really feel strongly either way and have a mild preference for wrapping this up, so +1 to merge from me. But happy to make continue making changes if anyone else feels differently.

wholmgren · 2021-01-04T20:28:48Z

I liked the structured array when I first read the code, but I also see @adriesse's point. I don't think there's a wrong choice here so I'm going to go ahead and merge. Thanks all!

kandersolar added 9 commits September 19, 2020 18:58

spectrl2 implementation

532d197

sphinx

87b3dfc

package updates

0605e50

add tests

779a047

add gallery example

5b34c3a

stickler

0d5655c

whatsnew

10793e7

more whatsnew

f32663b

here's your newline stickler

83a584e

kandersolar marked this pull request as ready for review September 20, 2020 21:42

cwhanse added this to the 0.8.1 milestone Sep 21, 2020

cwhanse added the enhancement label Sep 21, 2020

kandersolar added 3 commits October 11, 2020 13:23

Merge branch 'master' into spectrl2

80a664c

Merge branch 'master' into spectrl2

e7703b6

merge typo

c0eaaa8

cwhanse assigned cwhanse and unassigned cwhanse Dec 10, 2020

cwhanse self-requested a review December 10, 2020 15:53

wholmgren approved these changes Dec 10, 2020

View reviewed changes

cwhanse reviewed Dec 10, 2020

View reviewed changes

pvlib/spectrum.py Outdated Show resolved Hide resolved

adriesse reviewed Dec 10, 2020

View reviewed changes

kandersolar commented Dec 10, 2020

View reviewed changes

kandersolar added 4 commits December 11, 2020 18:10

et -> dni_extra

18d98b5

use spencer for earth-sun distance; add airmass comment

5db6bd8

bugfix

5505405

Merge branch 'spectrl2' of https://github.com/kanderso-nrel/pvlib-python

f26d6b0

into spectrl2

raise error if not pandas and dayofyear not specified

ff3b981

cwhanse reviewed Dec 16, 2020

View reviewed changes

pvlib/spectrum.py Outdated Show resolved Hide resolved

kandersolar added 2 commits December 16, 2020 09:18

clarify shape of return arrays

06914cf

typo and sphinx warning

53f1199

adriesse reviewed Dec 17, 2020

View reviewed changes

kandersolar added 3 commits December 17, 2020 18:44

create spectrum folder

1039797

changes from review

7ef67c4

remove pdb call

a2a8402

adriesse reviewed Dec 18, 2020

View reviewed changes

wholmgren approved these changes Dec 22, 2020

View reviewed changes

cwhanse approved these changes Dec 22, 2020

View reviewed changes

kandersolar and others added 2 commits December 22, 2020 15:32

Update pvlib/spectrum/spectrl2.py

27406d5

Co-authored-by: Cliff Hansen <[email protected]>

changes from review

9fc9ff4

wholmgren merged commit 84c76f6 into pvlib:master Jan 4, 2021

kandersolar deleted the spectrl2 branch January 4, 2021 20:30

kandersolar mentioned this pull request Dec 29, 2022

Migrate spectral modifier functions to pvlib.spectrum #1628

Merged

7 tasks

mikofski mentioned this pull request May 17, 2025

New default solar position algorithm #2455

Open

Add numpy SPECTRL2 implementation #1062

Add numpy SPECTRL2 implementation #1062

Uh oh!

Conversation

kandersolar commented Sep 20, 2020 • edited by cwhanse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kandersolar commented Sep 20, 2020

Uh oh!

cwhanse commented Sep 21, 2020

Uh oh!

adriesse commented Sep 24, 2020

Uh oh!

wholmgren left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cwhanse left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kandersolar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cwhanse commented Dec 10, 2020

Uh oh!

kandersolar commented Dec 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

adriesse commented Dec 17, 2020

Uh oh!

adriesse commented Dec 17, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cwhanse commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kandersolar commented Dec 18, 2020

Uh oh!

adriesse commented Dec 18, 2020

Uh oh!

adriesse Dec 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cwhanse left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kandersolar commented Sep 20, 2020 •

edited by cwhanse

Loading

kandersolar commented Dec 16, 2020 •

edited

Loading

cwhanse commented Dec 17, 2020 •

edited

Loading

adriesse Dec 18, 2020 •

edited

Loading