API Documentation
Detector
- class traval.detector.Detector(series, truth=None)[source]
Detector object for applying error detection algorithms to timeseries.
The Detector is used to apply error detection algorithms to a timeseries and optionally contains a ‘truth’ series, to which the error detection result can be compared. An example of a ‘truth’ series is a manually validated timeseries. Custom error detection algorithms can be defined using the RuleSet object.
- Parameters
series (pd.Series or pd.DataFrame) – timeseries to check
truth (pd.Series or pd.DataFrame, optional) – series that represents the ‘truth’, i.e. a benchmark to which the error detection result can be compared, by default None
Examples
Given a timeseries ‘series’ and some ruleset ‘rset’:
>>> d = Detector(series) >>> d.apply_ruleset(rset) >>> d.plot_overview()
See also
traval.RuleSet
object for defining detection algorithms
- static _validate_input_series(series)[source]
Internal method for checking type and dtype of series.
- Parameters
series (object) – timeseries to check, must be pd.Series or pd.DataFrame. Datatype of series or first column of DataFrame must be float.
- Raises
TypeError – if series or dtype of series does not comply
- apply_ruleset(ruleset, compare=True)[source]
Apply RuleSet to series.
- Parameters
ruleset (traval.RuleSet) – RuleSet object containing detection rules
compare (bool or list of int, optional) – if True, compare all results to original series and store in dictionary under comparisons attribute, default is True. If False, do not store comparisons. If list of int, store only those step numbers as comparisons. Note: value of -1 refers to last step for convenience.
See also
traval.RuleSet
object for defining detection algorithms
- confusion_matrix(steps=None, truth=None)[source]
Calculate confusion matrix stats for detection rules.
Note: the calculated statistics per rule contain overlapping counts, i.e. multiple rules can mark the same observatin as suspect.
- Parameters
steps (int, list of int or None, optional) – steps for which to calculate confusion matrix statistics, by default None which uses all steps.
truth (pd.Series or pd.DataFrame, optional) – series representing the “truth”, i.e. a benchmark to which the resulting series is compared. By default None, which uses the stored truth series. Argument is included so a different truth can be passed.
- Returns
df – dataframe containing confusion matrix data, i.e. counts of true positives, false positives, true negatives and false negatives.
- Return type
pd.DataFrame
- get_corrections_dataframe()[source]
Get DataFrame containing corrections.
- Returns
df – DataFrame containing corrections. NaN means value is flagged as suspicious, 0.0 means no correction.
- Return type
pandas.DataFrame
- get_final_result()[source]
Get final timeseries with flagged values set to NaN.
- Returns
series – Timeseries produced by final step in RuleSet with flagged values set to NaN.
- Return type
pandas.Series
- get_results_dataframe()[source]
Get results as DataFrame.
- Returns
df – results with flagged values set to NaN per applied rule.
- Return type
pandas.DataFrame
- plot_overview(mark_suspects=True, **kwargs)[source]
Plot timeseries with flagged values per applied rule.
- Parameters
mark_suspects (bool, optional) – mark suspect values with red X, by default True
- Returns
ax – axes objects
- Return type
list of matplotlib.pyplot.Axes
- set_truth(truth)[source]
Set ‘truth’ series.
Used for comparison with detection result.
- Parameters
truth (pd.Series or pd.DataFrame) – Series or DataFrame containing the “truth”, i.e. a benchmark to compare the detection result to.
- uniqueness(truth=None)[source]
Calculate unique contribution per rule to stats.
Note: the calculated statistics per rule contain an undercount, i.e. when multiple rules mark the same observatin as suspect it is not contained in this result.
- Parameters
steps (int, list of int or None, optional) – steps for which to calculate confusion matrix statistics, by default None which uses all steps.
truth (pd.Series or pd.DataFrame, optional) – series representing the “truth”, i.e. a benchmark to which the resulting series is compared. By default None, which uses the stored truth series. Argument is included so a different truth can be passed.
- Returns
df – dataframe containing confusion matrix data, i.e. unique counts of true positives, false positives, true negatives and false negatives.
- Return type
pd.DataFrame
RuleSet
- class traval.ruleset.RuleSet(name=None)[source]
Create RuleSet object for storing detection rules.
The RuleSet object stores detection rules and other relevant information in a dictionary. The order in which rules are carried out, the functions that parse the timeseries, the extra arguments required by those functions are all stored together.
The detection functions must take a series as the first argument, and return a series with corrections based on the detection rule. In the corrections series invalid values are set to np.nan, and adjustments are defined with a float. No change is defined as 0. Extra keyword arguments for the function can be passed through a kwargs dictionary. These kwargs are also allowed to contain functions. These functions must return some value based on the name of the series.
- Parameters
name (str, optional) – name of the RuleSet, by default None
Examples
Given two detection functions ‘foo’ and ‘bar’:
>>> rset = RuleSet(name="foobar") >>> rset.add_rule("foo", foo, apply_to=0) # add rule 1 >>> rset.add_rule("bar", bar, apply_to=1, kwargs={"n": 2}) # add rule 2 >>> print(rset) # print overview of rules
- add_rule(name, func, apply_to=None, kwargs=None)[source]
Add rule to RuleSet.
- Parameters
name (str) – name of the rule
func (callable) – function that takes series as input and returns a correction series.
apply_to (int or tuple of ints, optional) – series to apply the rule to, by default None, which defaults to the original series. E.g. 0 is the original series, 1 is the result of step 1, etc. If a tuple of ints is passed, the results of those steps are collected and passed to func.
kwargs (dict, optional) – dictionary of additional keyword arguments for func, by default None. Additional arguments can be functions as well, in which case they must return some value based on the name of the series to which the RuleSet will be applied.
- del_rule(name)[source]
Delete rule from RuleSet.
- Parameters
name (str) – name of the rule to delete
- classmethod from_json(fname)[source]
Load RuleSet object from JSON file.
Attempts to load functions in the RuleSet by searching for the function name in traval.rulelib. If the function cannot be found, only the name of the function is preserved. This means a RuleSet with custom functions will not be fully functional when loaded from a JSON file.
- Parameters
fname (str) – filename or path to file
- Returns
RuleSet object
- Return type
See also
to_json
store RuleSet as JSON file (does not support custom functions)
to_pickle
store RuleSet as pickle (supports custom functions)
from_pickle
load RuleSet from pickle file
- classmethod from_pickle(fname)[source]
Load RuleSet object form pickle file.
- Parameters
fname (str) – filename or path to file
- Returns
RuleSet object, including custom functions and parameters
- Return type
- to_dataframe()[source]
Convert RuleSet to pandas.DataFrame.
- Returns
rdf – DataFrame containing all the information from the RuleSet
- Return type
pandas.DataFrame
- to_json(fname, verbose=True)[source]
Write RuleSet to disk as json file.
Note that it is not possible to write custom functions to a JSON file. When writing the JSON only the name of the function is stored. When loading a JSON file, the function name is used to search within traval.rulelib. If the function can be found, it loads that function. A RuleSet making use of functions in the default rulelib.
- Parameters
fname (str) – filename or path to file
verbose (bool, optional) – prints message when operation complete, default is True
See also
from_json
load RuleSet from json file
to_pickle
store RuleSet as pickle (supports custom functions)
from_pickle
load RuleSet from pickle file
- to_pickle(fname, verbose=True)[source]
Write RuleSet to disk as pickle.
- Parameters
fname (str) – filename or path of file
verbose (bool, optional) – prints message when operation complete, default is True
See also
from_pickle
load RuleSet from pickle file
to_json
store RuleSet as json file (does not support custom functions)
from_json
load RuleSet from json file
- update_rule(name, func, apply_to=None, kwargs=None)[source]
Update rule in RuleSet.
- Parameters
name (str) – name of the rule
func (callable) – function that takes series as input and returns a correction series.
apply_to (int or tuple of ints, optional) – series to apply the rule to, by default None, which defaults to the original series. E.g. 0 is the original series, 1 is the result of step 1, etc. If a tuple of ints is passed, the results of those steps are collected and passed to func.
kwargs (dict, optional) – dictionary of additional keyword arguments for func, by default None. Additional arguments can be functions as well, in which case they must return some value based on the name of the series to which the RuleSet will be applied.
- class traval.ruleset.RuleSetEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]
- default(o)[source]
Implement this method in a subclass such that it returns a serializable object for
o
, or calls the base implementation (to raise aTypeError
).For example, to support arbitrary iterators, you could implement default like this:
def default(self, o): try: iterable = iter(o) except TypeError: pass else: return list(iterable) # Let the base class default method raise the TypeError return JSONEncoder.default(self, o)
Rule Library
- traval.rulelib.rule_combine_nan_and(*args)[source]
Combination rule, combine NaN values for any number of timeseries.
Used for combining intermediate results in branching algorithm trees to create one final result, i.e. (s1.isna() AND s2.isna())
- Returns
corrections – a series with same index as the input timeseries containing corrections. Contains NaNs where any of the input series values is NaN.
- Return type
pd.Series
- traval.rulelib.rule_combine_nan_or(*args)[source]
Combination rule, combine NaN values for any number of timeseries.
Used for combining intermediate results in branching algorithm trees to create one final result, i.e. (s1.isna() OR s2.isna())
- Returns
corrections – a series with same index as the input timeseries containing corrections. Contains NaNs where any of the input series values is NaN.
- Return type
pd.Series
- traval.rulelib.rule_diff_outside_of_n_sigma(series, n, max_gap='7D')[source]
Detection rule, calculate diff of series and identify suspect.
observations based on values outside of n * standard deviation of the difference.
- Parameters
series (pd.Series) – timeseries in which suspect values are identified
n (float, optional) – number of standard deviations to use, by default 2
max_gap (str, optional) – only considers observations within this maximum gap between measurements to calculate diff, by default “7D”.
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_diff_ufunc_threshold(series, ufunc, threshold, max_gap='7D')[source]
Detection rule, flag values based on diff, operator and threshold.
Calculate diff of series and identify suspect observations based on comparison with threshold value.
The argument ufunc is a tuple containing a function, e.g. an operator function (i.e. ‘>’, ‘<’, ‘>=’, ‘<=’). These are passed using their named equivalents, e.g. in numpy: np.greater, np.less, np.greater_equal, np.less_equal. This function essentially does the following: ufunc(series, threshold_series). The argument is passed as a tuple to bypass RuleSet logic.
- Parameters
series (pd.Series) – timeseries in which suspect values are identified
ufunc (tuple) – tuple containing ufunc (i.e. (numpy.greater_equal,) ). The function must be callable according to ufunc(series, threshold). The function is passed as a tuple to bypass RuleSet logic.
threshold (float) – value to compare diff of timeseries to
max_gap (str, optional) – only considers observations within this maximum gap between measurements to calculate diff, by default “7D”.
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_flat_signal(series, window, min_obs, std_threshold=0.0075, qbelow=None, qabove=None, hbelow=None, habove=None)[source]
Detection rule, flag values based on dead signal in rolling window.
Flag values when variation in signal within a window falls below a certain threshold value. Optionally provide quantiles below or above which to look for dead/flat signals.
- Parameters
series (pd.Series) – timeseries to analyse
window (int) – number of days in window
min_obs (int) – minimum number of observations in window to calculate standard deviation
std_threshold (float, optional) – standard deviation threshold value, by default 7.5e-3
qbelow (float, optional) – quantile value between 0 and 1, signifying an upper limit. Only search for flat signals below this limit. By default None.
qabove (float, optional) – quantile value between 0 and 1, signifying a lower limit. Only search for flat signals above this limit. By default None.
hbelow (float, optional) – absolute value in units of timeseries signifying an upper limit. Only search for flat signals below this limit. By default None.
habove (float, optional) – absolute value in units of timeseries signifying a lower limit. Only search for flat signals above this limit. By default None.
- Returns
corrections – a series with same index as the input timeseries containing corrections. Contains NaNs where the signal is considered flat or dead.
- Return type
pd.Series
- traval.rulelib.rule_funcdict_to_nan(series, funcdict)[source]
Detection rule, flag values with dictionary of functions.
Use dictionary of functions to identify suspect values and set them to NaN.
- Parameters
series (pd.Series) – timeseries in which suspect values are identified
funcdict (dict) – dictionary with function names as keys and functions/methods as values. Each function is applied to each value in the timeseries using series.apply(func). Suspect values are those where the function evaluates to True.
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values (according to the provided functions) are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_keep_comments(series, keep_comments, comment_series, other_series)[source]
Filter rule, modify timeseries to keep data with certain comments.
This rule was invented to extract timeseries only containing certain types of errors, based on labeled data. For example, to get only erroneous observations caused by sensors above the groundwater level:
series: the raw timeseries
keep_comments: list of comments to keep, e.g. [‘dry sensor’]
comment_series: timeseries containing the comments for erroneous obs
other_series: the validated timeseries where the commented observations were removed (set to NaN).
- Parameters
series (pd.Series) – timeseries to filter
keep_comments (list of str) – list of comments to keep
comment_series (pd.Series) – timeseries containing comments, should have same index as series
other_series (pd.Series) – timeseries containing corrected/adjusted values corresponding to the commmented entries.
- Returns
corrections – timeseries containing NaN values where comment is in keep_comments and 0 otherwise.
- Return type
pd.Series
- traval.rulelib.rule_max_gradient(series, max_step=0.5, max_timestep='1D')[source]
Detection rule, flag values when maximum gradient exceeded.
Set values tot NaN when maximum gradient between two observations is exceeded.
- Parameters
series (pd.Series) – timeseries in which suspect values are identified
max_step (float, optional) – max jump between two observations within given timestep, by default 0.5
timestep (str, optional) – maximum timestep to consider, by default “1D”. The gradient is not calculated for values that lie farther apart.
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_offset_detection(series, threshold=0.15, updown_diff=0.1, max_gap='7D', return_df=False)[source]
Detection rule, detect periods with an offset error.
This rule looks for jumps in both positive and negative direction that are larger than a particular threshold. It then tries to match jumps in upward direction to one in downward direction of a similar size. If this is possible, all observations between two matching but oppposite jumps are set to NaN.
- Parameters
series (pd.Series) – timeseries in which to look for offset errors
threshold (float, optional) – minimum jump to consider as offset error, by default 0.35
updown_diff (float, optional) – the maximum difference between two opposite jumps to consider them matching, by default 0.1
max_gap (str, optional) – only considers observations within this maximum gap between measurements to calculate diff, by default “7D”.
return_df (bool, optional) – return the dataframe containing the potential offsets, by default False
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_other_ufunc_threshold(series, other, ufunc, threshold)[source]
Detection rule, flag values based on other series and threshold.
Set values to Nan based on comparison of another timeseries with a threshold value.
The argument ufunc is a tuple containing an operator function (i.e. ‘>’, ‘<’, ‘>=’, ‘<=’). These are passed using their named equivalents, e.g. in numpy: np.greater, np.less, np.greater_equal, np.less_equal. This function essentially does the following: ufunc(series, threshold_series). The argument is passed as a tuple to bypass RuleSet logic.
- Parameters
series (pd.Series) – timeseries in which suspect values are identified, only used to test if index of other overlaps
other (pd.Series) – other timeseries based on which suspect values are identified
ufunc (tuple) – tuple containing ufunc (i.e. (numpy.greater_equal,) ). The function must be callable according to ufunc(series, threshold). The function is passed as a tuple to bypass RuleSet logic.
threshold (float) – value to compare timeseries to
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_outside_bandwidth(series, lowerbound, upperbound)[source]
Detection rule, set suspect values to NaN if outside bandwidth.
- Parameters
series (pd.Series) – timeseries in which suspect values are identified
lowerbound (pd.Series) – timeseries containing the lower bound, if bound values are less frequent than series, bound is interpolated to series.index
upperbound (pd.Series) – timeseries containing the upper bound, if bound values are less frequent than series, bound is interpolated to series.index
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_outside_n_sigma(series, n=2.0)[source]
Detection rule, set values outside of n * standard deviation to NaN
- Parameters
series (pd.Series) – timeseries in which suspect values are identified
n (float, optional) – number of standard deviations to use, by default 2
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_pastas_outside_pi(series, ml, ci=0.95, min_ci=None, smoothfreq=None, tmin=None, tmax=None, savedir=None, verbose=False)[source]
Detection rule, flag values based on pastas model prediction interval.
Flag suspect outside prediction interval calculated by pastas timeseries model. Uses a pastas.Model and a confidence interval as input.
- Parameters
series (pd.Series) – timeseries to identify suspect observations in
ml (pastas.Model) – timeseries model for series
ci (float, optional) – confidence interval for calculating bandwidth, by default 0.95. Higher confidence interval means that bandwidth is wider and more observations will fall within the bounds.
min_ci (float, optional) – value indicating minimum distance between upper and lower bounds, if ci does not meet this requirement, this value is added to the bounds. This can be used to prevent extremely narrow prediction intervals. Default is None.
smoothfreq (str, optional) – str indicating no. of periods and frequency str (i.e. “1D”) for smoothing upper and lower bounds only used if smoothbounds=True, default is None.
tmin (str or pd.Timestamp, optional) – set tmin for model simulation
tmax (str or pd.Timestamp, optional) – set tmax for model simulation
savedir (str, optional) – save calculated prediction interval to folder as pickle file.
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_shift_to_manual_obs(series, hseries, method='linear', max_dt='1D', reset_dates=None)[source]
Adjustment rule, for shifting timeseries onto manual observations.
Used for shifting timeseries based on sensor observations onto manual verification measurements. By default uses linear interpolation between two manual verification observations.
- Parameters
series (pd.Series) – timeseries to adjust
hseries (pd.Series) – timeseries containing manual observations
method (str, optional) – method to use for interpolating between two manual observations, by default “linear”. Other options are those that are accepted by series.reindex(): ‘bfill’, ‘ffill’, ‘nearest’.
max_dt (str, optional) – maximum amount of time between manual observation and value in series, by default “1D”
reset_dates (list, optional) – list of dates (as str or pd.Timestamp) on which to reset the adjustments to 0.0, by default None. Useful for resetting the adjustments when the sensor is replaced, for example.
- Returns
adjusted_series – timeseries containing adjustments to shift series onto manual observations.
- Return type
pd.Series
- traval.rulelib.rule_spike_detection(series, threshold=0.15, spike_tol=0.15, max_gap='7D')[source]
Detection rule, identify spikes in timeseries and set to NaN.
Spikes are sudden jumps in the value of a timeseries that last 1 timestep. They can be both negative or positive.
- Parameters
series (pd.Series) – timeseries in which suspect values are identified
threshold (float, optional) – the minimum size of the jump to qualify as a spike, by default 0.15
spike_tol (float, optional) – offset between value of timeseries before spike and after spike, by default 0.15. After a spike, the value of the timeseries is usually close to but not identical to the value that preceded the spike. Use this parameter to control how close the value has to be.
max_gap (str, optional) – only considers observations within this maximum gap between measurements to calculate diff, by default “7D”.
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
- traval.rulelib.rule_ufunc_threshold(series, ufunc, threshold, offset=0.0)[source]
Detection rule, flag values based on operator and threshold value.
Set values to Nan based on operator function and threshold value. The argument ufunc is a tuple containing an operator function (i.e. ‘>’, ‘<’, ‘>=’, ‘<=’). These are passed using their named equivalents, e.g. in numpy: np.greater, np.less, np.greater_equal, np.less_equal. This function essentially does the following: ufunc(series, threshold).
- Parameters
series (pd.Series) – timeseries in which suspect values are identified
ufunc (tuple) – tuple containing ufunc (i.e. (numpy.greater_equal,) ). The function must be callable according to ufunc(series, threshold). The function is passed as a tuple to bypass RuleSet logic.
threshold (float or pd.Series) – value or timeseries to compare series with
offset (float, optional) – value that is added to the threshold, e.g. if some extra tolerance is allowable. Default value is 0.0.
- Returns
corrections – a series with same index as the input timeseries containing corrections. Suspect values are set to np.nan.
- Return type
pd.Series
Timeseries Comparison
- class traval.ts_comparison.DateTimeIndexComparison(idx1, idx2)[source]
Helper class for comparing two DateTimeIndexes.
- idx_in_both()[source]
Index members in both DateTimeIndexes.
- Returns
index with entries in both
- Return type
DateTimeIndex
- class traval.ts_comparison.SeriesComparison(s1, s2, names=None, diff_threshold=0.0)[source]
Object for comparing two timeseries.
Comparison yields the following categories:
in_both_identical: in both series and difference <= than diff_threshold
in_both_different: in both series and difference > than diff_threshold
in_s1: only in series #1
in_s2: only in series #2
in_both_nan: NaN in both
- Parameters
s1 (pd.Series or pd.DataFrame) – first series to compare
s2 (pd.Series or pd.DataFrame) – second series to compare
diff_threshold (float, optional) – value beyond which a difference is considered significant, by default 0.0. Two values whose difference is smaller than threshold are considered identical.
- compare_by_comment()[source]
Compare series per comment.
- Returns
comparison – series containing the possible comparison outcomes, but split into categories, one for each unique comment. Comments must be passed via series2.
- Return type
pd.Series
- Raises
ValueError – if no comment series is found
- comparison_series()[source]
Create series that indicates what happend to a value.
Series index is the union of s1 and s2 with a value indicating the status of the comparison:
-1: value is modified
0: value stays the same
1: value only in series 1
2: value only in series 2
-9999: value is NaN in both series
- Returns
s – series containing status of value from comparison
- Return type
pd.Series
- class traval.ts_comparison.SeriesComparisonRelative(s1, truth, base, diff_threshold=0.0)[source]
Object for comparing two timeseries relative to a third timeseries.
Extends the SeriesComparison object to include a comparison between two timeseries and a third base timeseries. This is used for example, when comparing the results of two error detection outcomes to the original raw timeseries.
Comparison yields both the results from SeriesComparison as well as the following categories for the relative comparison to the base timeseries:
kept_in_both: both timeseries and the base timeseries contain values
flagged_in_s1: value is NaN/missing in series #1
flagged_in_s2: value is NaN/missing in series #2
flagged_in_both: value is NaN/missing in both series #1 and series #2
in_all_nan: value is NaN in all timeseries (series #1, #2 and base)
introduced_in_s1: value is NaN/missing in base but has value in series #1
introduced_in_s2: value is NaN/missing in base but has value in series #2
introduced_in_both: value is NaN/missing in base but has value in both timeseries
- Parameters
s1 (pd.Series or pd.DataFrame) – first series to compare
truth (pd.Series or pd.DataFrame) – second series to compare, if a “truth” timeseries is available pass it as the second timeseries. Stored in object as ‘s2’.
base (pd.Series or pd.DataFrame) – timeseries to compare other two series with
diff_threshold (float, optional) – value beyond which a difference is considered significant, by default 0.0. Two values whose difference is smaller than threshold are considered identical.
See also
SeriesComparison
Comparison of two timeseries relative to each other
- compare_to_base_by_comment()[source]
Compare two series to base series per comment.
- Returns
comparison – Series containing the number of observations in each possible comparison category, but split per unique comment. Comments must be provided via ‘truth’ series (series2).
- Return type
pd.Series
- Raises
ValueError – if no comment series is available.
Timeseries Utilities
- traval.ts_utils.bandwidth_moving_avg_n_sigma(series, window, n)[source]
Calculate bandwidth around timeseries based moving average + n * std.
- Parameters
series (pd.Series) – series to calculate bandwidth for
window (int) – number of observations to consider for moving average
n (float) – number of standard deviations from moving average for bandwidth
- Returns
bandwidth – dataframe with 2 columns, with lower and upper bandwidth
- Return type
pd.DataFrame
- traval.ts_utils.create_synthetic_raw_timeseries(raw_series, truth_series, comments)[source]
Create synthetic raw timeseries.
Updates ‘truth_series’ (where values are labelled with a comment) with values from raw_series. Used for removing unlabeled changes between a raw and validated timeseries.
- Parameters
raw_series (pd.Series) – timeseries with raw data
truth_series (pd.Series) – timeseries with validated data
comments (pd.Series) – timeseries with comments. Index must be same as ‘truth_series’. When value does not have a comment it must be an empty string: ‘’.
- Returns
s – synthetic raw timeseries, same as truth_series but updated with raw_series where value has been commented.
- Return type
pd.Series
- traval.ts_utils.diff_with_gap_awareness(series, max_gap='7D')[source]
Get diff of timeseries with a limit on gap between two values.
- Parameters
series (pd.Series) – timeseries to calculate diff for
max_gap (str, optional) – maximum period between two observations for calculating diff, otherwise set value to NaN, by default “7D”
- Returns
diff – timeseries with diff, with NaNs whenever two values are farther apart than max_gap.
- Return type
pd.Series
- traval.ts_utils.interpolate_series_to_new_index(series, new_index)[source]
Interpolate timeseries to new DateTimeIndex.
- Parameters
series (pd.Series) – original series
new_index (DateTimeIndex) – new index to interpolate series to
- Returns
si – new series with new index, with interpolated values
- Return type
pd.Series
- traval.ts_utils.mask_corrections_as_nan(series, mask)[source]
Get corrections series with NaNs where mask == True.
- Parameters
series (pd.Series) – timeseries to provide corrections for
mask (DateTimeIndex or boolean np.array) – DateTimeIndex containing timestamps where value should be set to NaN, or boolean array with same length as series set to True where value should be set to NaN. (Uses pandas .loc[mask] to set values.)
- Returns
c – return corrections series
- Return type
pd.Series
- traval.ts_utils.resample_short_series_to_long_series(short_series, long_series)[source]
Resample a short timeseries to index from a longer timeseries.
First uses ‘ffill’ then ‘bfill’ to fill new series.
- Parameters
short_series (pd.Series) – short timeseries
long_series (pd.Series) – long timeseries
- Returns
new_series – series with index from long_series and data from short_series
- Return type
pd.Series
- traval.ts_utils.spike_finder(series, threshold=0.15, spike_tol=0.15, max_gap='7D')[source]
Find spikes in timeseries.
Spikes are sudden jumps in the value of a timeseries that last 1 timestep. They can be both negative or positive.
- Parameters
series (pd.Series) – timeseries to find spikes in
threshold (float, optional) – the minimum size of the jump to qualify as a spike, by default 0.15
spike_tol (float, optional) – offset between value of timeseries before spike and after spike, by default 0.15. After a spike, the value of the timeseries is usually close to but not identical to the value that preceded the spike. Use this parameter to control how close the value has to be.
max_gap (str, optional) – only considers observations within this maximum gap between measurements to calculate diff, by default “7D”.
- Returns
upspikes, downspikes – pandas DateTimeIndex objects containing timestamps of upward and downward spikes.
- Return type
pandas.DateTimeIndex
- traval.ts_utils.unique_nans_in_series(series, *args)[source]
Get mask where NaNs in series are unique compared to other series.
- Parameters
series (pd.Series) – identify unique NaNs in series
*args – any number of pandas.Series
- Returns
mask – mask with value True where NaN is unique to series
- Return type
pd.Series
Binary Classification
- class traval.binary_classifier.BinaryClassifier(tp, fp, tn, fn)[source]
Class for calculating binary classification statistics.
- property accuracy
Accuracy of binary classification.
ACC = (TP + TN) / (TP + FP + FN + TN)
where - TP : True Positives - TN : True Negatives - FP : False Positives - FN : False Negatives
- confusion_matrix(as_array=False)[source]
Calculate confusion matrix.
Confusion matrix shows the performance of the algorithm given a certain truth. An abstract example of the confusion matrix:
Algorithm ||-------------------| | error | correct |
- ——|---------|———|---------|
- error | TP | FN |
- Truth |---------|———|---------|
- correct | FP | TN |
where: - TP: True Positives = errors correctly detected by algorithm - TN: True Negatives = correct values correctly not flagged by algorithm - FP: False Positives = correct values marked as errors by algorithm - FN: False Negatives = errors not detected by algorithm
- Parameters
as_array (bool, optional) – return data as array instead of DataFrame, by default False
- Returns
data – confusion matrix
- Return type
pd.DataFrame or np.array
- property false_discovery_rate
False discovery rate.
FDR = 1 - PPV = FP / (FP + TP)
where - TP : True Positives - FP : False Positives
- property false_negative_rate
False Negative Rate = (1 - sensitivity).
FNR = FN / (FN + TP)
where - FN : False Negatives - TP : True Positives
- property false_omission_rate
False omission rate.
FOR = 1 - NPV = FN / (TN + FN)
where - TN : True Negatives - FN : False Negatives
- property false_positive_rate
False Positive Rate = (1 - specificity).
FPR = FP / (FP + TN)
where - FP : False Positives - TN : True Negatives
- classmethod from_confusion_matrix(cmat)[source]
Create BinaryClassifier from confusion matrix.
Note
Confusion Matrix must be passed as an np.array or pd.DataFrame corresponding to: [[TP, FN], [FP, TN]], like the one returned by BinaryClassifier.confusion_matrix
- Parameters
cmat (np.array or pd.DataFrame) –
- a 2x2 dataset with structure [[TP, FN],
[FP, TN]]
- Returns
BinaryClassifier object based on values in confusion matrix.
- Return type
See also
BinaryClassifier.confusion_matrix
for explanation (of abbreviations)
- classmethod from_series_comparison_relative(comparison)[source]
Binary Classification object from SeriesComparisonRelative object.
- Parameters
comparison (traval.SeriesComparisonRelative) – object comparing two timeseries with base timeseries
- Returns
object for calculating binary classification statistics
- Return type
- get_all_statistics(use_abbreviations=True)[source]
Get all statistics in pandas.Series.
- Parameters
use_abbreviations (bool, optional) – whether to use abbreviations or full names for index, by default True
- Returns
s – series containing all statistics
- Return type
pandas.Series
- property informedness
Informedness statistic (a.k.a. Youden’s J statistic).
Measure of diagnostic performance, and has a zero value when a diagnostic test gives the same proportion of positive results for groups with and without a condition, i.e the test is useless. A value of 1 indicates that there are no false positives or false negatives, i.e. the test is perfect.
Calculated as:
informedness = specificity + sensitivity - 1.
- property matthews_correlation_coefficient
Matthews correlation coefficient (MCC).
The MCC is in essence a correlation coefficient between the observed and predicted binary classifications; it returns a value between −1 and +1. A coefficient of +1 represents a perfect prediction, 0 no better than random prediction and −1 indicates total disagreement between prediction and observation.
- Returns
phi – the Matthews correlation coefficient
- Return type
float
See also
mcc
convenience method for calculating MCC
- property mcc
Convenience method for calculating Matthews correlation coefficient.
- Returns
phi – the Matthews correlation coefficient
- Return type
float
See also
matthews_correlation_coefficient
more information about the statistic
- property negative_predictive_value
Negative predictive value.
NPV = TN / (TN + FN)
where - TN : True Negatives - FN : False Negatives
- property positive_predictive_value
Positive predictive value (a.k.a. precision).
PPV = TP / (TP + FP)
where - TP : True Positives - FP : False Positives
- property prevalence
Prevalance of true errors in total population.
Prevalence = (TP + FN) / (TP + FP + FN + TN)
where - TP : True Positives - TN : True Negatives - FP : False Positives - FN : False Negatives
- property sensitivity
Sensitivity or True Positive Rate.
Statistic describing ratio of true positives identified, which also says something about the avoidance of false negatives.
Sensitivity = TP / (TP + FN)
where - TP : True Positives - FN : False Negatives
- property specificity
Specificity or True Negative Rate.
Statistic describing ratio of true negatives identified, which also says something about the avoidance of false positives.
Specificity = TN / (TN + FP)
where - TN : True Negatives - FP : False Positives
- property true_negative_rate
True Negative Rate. Synonym for specificity.
See specificity for description.
- property true_positive_rate
True Positive Rate. Synonym for sensitivity.
See sensitiviy for description.
Plots
- class traval.plots.ComparisonPlots(cp)[source]
Mix-in class for plots for comparing timeseries.
- plot_relative_comparison(mark_unique=True, mark_different=True, mark_identical=True, mark_introduced=False, ax=None)[source]
Plot comparison between two timeseries relative to base timeseries.
- Parameters
mark_unique (bool, optional) – mark unique observations with colored X’s, by default True
mark_different (bool, optional) – highlight where series are different in red, by default True
mark_identical (bool, optional) – highlight where series are identical with green, by default True
mark_introduced (bool, optional) – mark observations that are not in the base timeseries with X’s, by default False
ax (axis, optional) – axis to plot on, by default None
- Returns
ax – axis handle
- Return type
axis
- plot_series_comparison(mark_unique=True, mark_different=True, mark_identical=True, ax=None)[source]
Plot comparison between two timeseries.
- Parameters
mark_unique (bool, optional) – mark unique values with colored X’s, by default True
mark_different (bool, optional) – highlight where timeseries differ with red, by default True
mark_identical (bool, optional) – highlight where timeseries are identical with green, by default True
ax (axis, optional) – axis object to plot on, by default None
- Returns
ax – axis object
- Return type
axis
- traval.plots.det_plot(fpr, fnr, labels, ax=None, **kwargs)[source]
Detection Error Tradeoff plot.
Adapted from scikitlearn DetCurveDisplay.
- Parameters
fpr (list or value or array) – false positive rate. If passed as a list loops through each entry and plots it. Otherwise just plots the array or value.
fnr (list or value or array) – false negative rate. If passed as a list loops through each entry and plots it. Otherwise just plots the array or value.
labels (list or str) – label for each fpr/fnr entry.
ax (matplotlib.pyplot.Axes, optional) – axes handle to plot on, by default None, which creates a new figure
- Returns
ax – axes handle
- Return type
matplotlib.pyplot.Axes
- traval.plots.roc_plot(tpr, fpr, labels, colors=None, ax=None, plot_diagonal=True, colorbar_label=None, **kwargs)[source]
Receiver operator characteristic plot.
Plots the false positive rate (x-axis) versus the true positive rate (y-axis). The ‘tpr’ and ‘fpr’ can be passed as: - values: outcome of a single error detection algorithm - arrays: outcomes of error detection algorithm in which a detection
parameter is varied.
lists: for passing multiple results, entries can be values or arrays, as listed above.
- Parameters
tpr (list or value or array) – true positive rate. If passed as a list loops through each entry and plots it. Otherwise just plots the array or value.
fpr (list or value or array) – false positive rate. If passed as a list loops through each entry and plots it. Otherwise just plots the array or value.
labels (list or str) – label for each tpr/fpr entry.
ax (matplotlib.pyplot.Axes, optional) – axes to plot on, default is None, which creates new figure
plot_diagonal (bool, optional) – whether to plot the diagonal (useful for combining multiple ROC plots)
**kwargs – passed to ax.scatter
- Returns
ax – axes instance
- Return type
matplotlib.pyplot.Axes