Data shift algorithm (#142)

* Add in pvanalytics update (#1)

* v0.1.1 Release (#132)

* change pypi classifier from pre-alpha to beta

* remove unnecessary docs/requirements.txt

* whatsnew v0.1.1

* include 0.1.1 in whatsnew index

* link zenodo in readme

* Added clipping time series example for Sphinx documentation.

* added sphinx documentation + examples for running the clipping mask.

* fixed pep8 formatting errors.

* added a new whatsnew rst file for version 0.1.2

* removed close plot to visualize in sphinx.

* removed trailing whitespace.

* added tight layout for plot sizing.

* Updated the docs based on @kanderso-nrel's recs.

* fixed pep8 warning.

* removed trailing whitespace-pep8 issue.

* Added placeholder scripts for each function for Sphinx documentation.

* added sphinx module for completeness score.

* Update docs/examples/clipping.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/clipping.py

Co-authored-by: Cliff Hansen <[email protected]>

* added each of the python scripts for detecting stale data, interpolated data, and check for daily data completeness.

* updated the interpolated-periods documentation

* cleaning up doc strings.

* updated the naming conventions of the sphinx docs.

* fixed pep8 error on stale data periods docs.

* made updates to the Sphinx docs based on kanderso-nrel's feedback.

* fixed pep8 errors.

* Edited some of the language in the sphinx doc comments.

* Added pv-terms json.

* added the documentation files for hampel, tukey, and zscore outlier detection.

* update the documentation to include separate data files for each of the different issues, to avoid further confusion.

* updated the interpolated data docs to pull the correct csv.

* More docstring cleanup.

* Updated outlier code to use the new outlier csv file.

* updated the outliers routine to handle varying indices.

* Docstring cleaning.

* made updates to the hovertext and the _round edits per @kanderso-nrel's comments

* updated diff to round on docstring per @kanderso-nrel's comment.

* updated the whatsnew doc with the outliers documentation.

* updated the hovertext info.

* updated the routine with the bug fix for whatsnew, and removed the initial graphing.

* added prints to visualize the imported data in example docs.

* Update docs/whatsnew/0.1.2.rst

Co-authored-by: Kevin Anderson <[email protected]>

* added new commenting based on @cwhanse's recommendations.

* fixed improper spelling in comments

* Day night masking sphinx documentation (#139)

* update the day-night masking example.

* update the day-night masking routine.

* added the SERF east data for running the day-night mask examples.

* added the day-night masking routine.

* Added section for comparing day-night mask to PVlib sunrise-sunset times.

* added separate printouts for sunrise and sunset time comparisons.

* added vertical lines for sunrise + sunset in plots

* update the routine to remove hardcoded file name.

* added update to the whatsnew file.

* removed a newline to see if we could get git actions to work.

* Made updates to documentation based on @kanderso-nrel's recommendations.

* Update docs/examples/day-night-masking.py

Co-authored-by: Cliff Hansen <[email protected]>

* Removed default kwargs for pvlib SPA sunrise-sunset function.

* Updating the commenting.

* fixed pep8 line length

Co-authored-by: Perry <[email protected]>
Co-authored-by: Cliff Hansen <[email protected]>

* Irradiance sphinx documentation (#140)

* added initial files for all of the irradiance documentation (need to edit).

* added RMIS example data for irradiance Sphinx documentation.

* added the new qcrad function.

* update the examples for both qcrad functions.

* added qcrad-limits documentation.

* ensured outputs for all irradiance functions in examples.

* added plotting functionality for some of the examples.

* added graphics for all of the irradiance documentation.

* added new line at end of file to stop pep8 failure.

* Clean up of doc strings for irradiance documentation.

* Fixed the docstring PEP8 error.

* Update docs/examples/clearsky-limits-irradiance.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update docs/examples/clearsky-limits-irradiance.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update docs/examples/clearsky-limits-irradiance.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update docs/examples/clearsky-limits-irradiance.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update docs/examples/qcrad-limits-irradiance.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update docs/examples/qcrad-limits-irradiance.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update docs/examples/qcrad-consistency-irradiance.py

Co-authored-by: Kevin Anderson <[email protected]>

* Update docs/examples/qcrad-consistency-irradiance.py

Co-authored-by: Kevin Anderson <[email protected]>

* Removed 'sampled' reference from docstring when describing data

* changed py:func to py:meth in docstring

* Updated the routine to calculate extraterrestrial radition as dni_extra for check_irradiance_limits_qcrad() function.

* Renamed the routine Clearsky Limits for Daily Insolation

* removed pep8 issues

* added the documentation to the whatsnew file.

* Update docs/examples/clearsky-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/clearsky-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/qcrad-consistency-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/qcrad-consistency-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/qcrad-consistency-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/qcrad-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/daily-insolation-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/clearsky-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* added day-night mask to clearsky-limits-irradiance documentation

* removed hardcoded path!

* Update docs/examples/daily-insolation-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/daily-insolation-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/daily-insolation-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* switched the ordering of parameters in ) per @cwhanse's request.

* rearranged the order of inputs for irradiance_consistency_qcrad function in unit test.

* Update docs/examples/qcrad-consistency-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* updated clearsky-limits-irradiance example to comment on Ineichen model performance

* Update docs/examples/qcrad-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/qcrad-consistency-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/examples/daily-insolation-limits-irradiance.py

Co-authored-by: Cliff Hansen <[email protected]>

Co-authored-by: Perry <[email protected]>
Co-authored-by: Kevin Anderson <[email protected]>
Co-authored-by: Cliff Hansen <[email protected]>

Co-authored-by: Kevin Anderson <[email protected]>
Co-authored-by: Perry <[email protected]>
Co-authored-by: Cliff Hansen <[email protected]>

* added back in all of the data shift files.

* Debug pep8 fixes.

* Added data_dir path to conftest.

* pep8 fix.

* renamed pytest fixture.

* added unit test for when ruptures is not available--up test coverage.

* retry to debug the ruptures package load

* fixed my ruptures delete and re-impoort test

* added import error test for ruptures

* more ruptures package unload-reload unit tests

* Update 0.1.2.rst

Updated PR number

* updates based on @kanderso-nrel's review

* renamed variables in the test file for data shifts from time_series to series

* updated the longest segment output to tuple + updated warning in docstring

* Update pvanalytics/quality/data_shifts.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update pvanalytics/quality/data_shifts.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update pvanalytics/quality/data_shifts.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update pvanalytics/quality/data_shifts.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update docs/whatsnew/0.1.2.rst

Co-authored-by: Cliff Hansen <[email protected]>

* Updated PVSC citation.:

* Cleaned up any existing pep8 errors

* Update pvanalytics/quality/data_shifts.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update pvanalytics/quality/data_shifts.py

Co-authored-by: Cliff Hansen <[email protected]>

* Update the docstring

Co-authored-by: Kevin Anderson <[email protected]>
Co-authored-by: Perry <[email protected]>
Co-authored-by: Cliff Hansen <[email protected]>

Author: 4 people

docs/api.rst

@@ -7,6 +7,19 @@ API Reference
 Quality
 =======
 
+Data Shifts
+-----------
+
+Functions for identifying shifts in data values in time series
+and for identifying periods with data shifts. For functions
+that identify shifts in time, see ``quality.time``
+
+.. autosummary::
+   :toctree: generated/
+
+   quality.data_shifts.detect_data_shifts
+   quality.data_shifts.get_longest_shift_segment_dates
+
 Irradiance
 ----------
 

docs/index.rst

@@ -25,6 +25,8 @@ library status.
 - :py:mod:`quality` contains submodules for different kinds of data quality
   checks.
 
+  - :py:mod:`quality.data_shifts` contains quality checks for detecting and 
+    isolating data shifts in PV time series data.
   - :py:mod:`quality.irradiance` contains quality checks for irradiance
     measurements.
   - :py:mod:`quality.weather` contains quality checks for weather data (e.g.

docs/whatsnew/0.1.2.rst

@@ -5,7 +5,9 @@
 
 Enhancements
 ~~~~~~~~~~~~
-
+* Detect data shifts in daily summed time series with 
+   :py:func:`pvanalytics.quality.data_shifts.detect_data_shifts`
+  and :py:func:`pvanalytics.quality.data_shifts.get_longest_shift_segment_dates`. (:pull:`142`)
 
 Bug Fixes
 ~~~~~~~~~