mir_eval.pattern
Pattern discovery involves the identification of musical patterns (i.e. short fragments or melodic ideas that repeat at least twice) both from audio and symbolic representations. The metrics used to evaluate pattern discovery systems attempt to quantify the ability of the algorithm to not only determine the present patterns in a piece, but also to find all of their occurrences.
- Based on the methods described here:
T. Collins. MIREX task: Discovery of repeated themes & sections. http://www.music-ir.org/mirex/wiki/2013:Discovery_of_Repeated_Themes_&_Sections, 2013.
Conventions
The input format can be automatically generated by calling
mir_eval.io.load_patterns()
. This format is a list of a list of
tuples. The first list collections patterns, each of which is a list of
occurrences, and each occurrence is a list of MIDI onset tuples of
(onset_time, mid_note)
A pattern is a list of occurrences. The first occurrence must be the prototype of that pattern (i.e. the most representative of all the occurrences). An occurrence is a list of tuples containing the onset time and the midi note number.
Metrics
mir_eval.pattern.standard_FPR()
: Strict metric in order to find the possibly transposed patterns of exact length. This is the only metric that considers transposed patterns.mir_eval.pattern.establishment_FPR()
: Evaluates the amount of patterns that were successfully identified by the estimated results, no matter how many occurrences they found. In other words, this metric captures how the algorithm successfully established that a pattern repeated at least twice, and this pattern is also found in the reference annotation.mir_eval.pattern.occurrence_FPR()
: Evaluation of how well an estimation can effectively identify all the occurrences of the found patterns, independently of how many patterns have been discovered. This metric has a threshold parameter that indicates how similar two occurrences must be in order to be considered equal. In MIREX, this evaluation is run twice, with thresholds .75 and .5.mir_eval.pattern.three_layer_FPR()
: Aims to evaluate the general similarity between the reference and the estimations, combining both the establishment of patterns and the retrieval of its occurrences in a single F1 score.mir_eval.pattern.first_n_three_layer_P()
: Computes the three-layer precision for the first N patterns only in order to measure the ability of the algorithm to sort the identified patterns based on their relevance.mir_eval.pattern.first_n_target_proportion_R()
: Computes the target proportion recall for the first N patterns only in order to measure the ability of the algorithm to sort the identified patterns based on their relevance.
- mir_eval.pattern.validate(reference_patterns, estimated_patterns)
Check that the input annotations to a metric look like valid pattern lists, and throws helpful errors if not.
- Parameters:
- reference_patternslist
The reference patterns using the format returned by
mir_eval.io.load_patterns()
- estimated_patternslist
The estimated patterns in the same format
- mir_eval.pattern.standard_FPR(reference_patterns, estimated_patterns, tol=1e-05)
Compute the standard F1 Score, Precision and Recall.
This metric checks if the prototype patterns of the reference match possible translated patterns in the prototype patterns of the estimations. Since the sizes of these prototypes must be equal, this metric is quite restrictive and it tends to be 0 in most of 2013 MIREX results.
- Parameters:
- reference_patternslist
The reference patterns using the format returned by
mir_eval.io.load_patterns()
- estimated_patternslist
The estimated patterns in the same format
- tolfloat
Tolerance level when comparing reference against estimation. Default parameter is the one found in the original matlab code by Tom Collins used for MIREX 2013. (Default value = 1e-5)
- Returns:
- f_measurefloat
The standard F1 Score
- precisionfloat
The standard Precision
- recallfloat
The standard Recall
Examples
>>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt") >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt") >>> F, P, R = mir_eval.pattern.standard_FPR(ref_patterns, est_patterns)
- mir_eval.pattern.establishment_FPR(reference_patterns, estimated_patterns, similarity_metric='cardinality_score')
Compute the establishment F1 Score, Precision and Recall.
- Parameters:
- reference_patternslist
The reference patterns in the format returned by
mir_eval.io.load_patterns()
- estimated_patternslist
The estimated patterns in the same format
- similarity_metricstr
A string representing the metric to be used when computing the similarity matrix. Accepted values:
“cardinality_score”: Count of the intersection between occurrences.
(Default value = “cardinality_score”)
- Returns:
- f_measurefloat
The establishment F1 Score
- precisionfloat
The establishment Precision
- recallfloat
The establishment Recall
Examples
>>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt") >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt") >>> F, P, R = mir_eval.pattern.establishment_FPR(ref_patterns, ... est_patterns)
- mir_eval.pattern.occurrence_FPR(reference_patterns, estimated_patterns, thres=0.75, similarity_metric='cardinality_score')
Compute the occurrence F1 Score, Precision and Recall.
- Parameters:
- reference_patternslist
The reference patterns in the format returned by
mir_eval.io.load_patterns()
- estimated_patternslist
The estimated patterns in the same format
- thresfloat
How similar two occurrences must be in order to be considered equal (Default value = .75)
- similarity_metricstr
A string representing the metric to be used when computing the similarity matrix. Accepted values:
“cardinality_score”: Count of the intersection between occurrences.
(Default value = “cardinality_score”)
- Returns:
- f_measurefloat
The occurrence F1 Score
- precisionfloat
The occurrence Precision
- recallfloat
The occurrence Recall
Examples
>>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt") >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt") >>> F, P, R = mir_eval.pattern.occurrence_FPR(ref_patterns, ... est_patterns)
- mir_eval.pattern.three_layer_FPR(reference_patterns, estimated_patterns)
Three Layer F1 Score, Precision and Recall. As described by Meridith.
- Parameters:
- reference_patternslist
The reference patterns in the format returned by
mir_eval.io.load_patterns()
- estimated_patternslist
The estimated patterns in the same format
- Returns:
- f_measurefloat
The three-layer F1 Score
- precisionfloat
The three-layer Precision
- recallfloat
The three-layer Recall
Examples
>>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt") >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt") >>> F, P, R = mir_eval.pattern.three_layer_FPR(ref_patterns, ... est_patterns)
- mir_eval.pattern.first_n_three_layer_P(reference_patterns, estimated_patterns, n=5)
First n three-layer precision.
This metric is basically the same as the three-layer FPR but it is only applied to the first n estimated patterns, and it only returns the precision. In MIREX and typically, n = 5.
- Parameters:
- reference_patternslist
The reference patterns in the format returned by
mir_eval.io.load_patterns()
- estimated_patternslist
The estimated patterns in the same format
- nint
Number of patterns to consider from the estimated results, in the order they appear in the matrix (Default value = 5)
- Returns:
- precisionfloat
The first n three-layer Precision
Examples
>>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt") >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt") >>> P = mir_eval.pattern.first_n_three_layer_P(ref_patterns, ... est_patterns, n=5)
- mir_eval.pattern.first_n_target_proportion_R(reference_patterns, estimated_patterns, n=5)
First n target proportion establishment recall metric.
This metric is similar is similar to the establishment FPR score, but it only takes into account the first n estimated patterns and it only outputs the Recall value of it.
- Parameters:
- reference_patternslist
The reference patterns in the format returned by
mir_eval.io.load_patterns()
- estimated_patternslist
The estimated patterns in the same format
- nint
Number of patterns to consider from the estimated results, in the order they appear in the matrix. (Default value = 5)
- Returns:
- recallfloat
The first n target proportion Recall.
Examples
>>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt") >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt") >>> R = mir_eval.pattern.first_n_target_proportion_R( ... ref_patterns, est_patterns, n=5)
- mir_eval.pattern.evaluate(ref_patterns, est_patterns, **kwargs)
Load data and perform the evaluation.
- Parameters:
- ref_patternslist
The reference patterns in the format returned by
mir_eval.io.load_patterns()
- est_patternslist
The estimated patterns in the same format
- **kwargs
Additional keyword arguments which will be passed to the appropriate metric or preprocessing functions.
- Returns:
- scoresdict
Dictionary of scores, where the key is the metric name (str) and the value is the (float) score achieved.
Examples
>>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt") >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt") >>> scores = mir_eval.pattern.evaluate(ref_patterns, est_patterns)