I just realised I forgot to blog a paper! I’m second author, so I suppose it’s OK…promise you won’t be mad? A preprint of the paper, entitled “Understanding Effects of Subjectivity in measuring Chord Estimation Accuracy”, can be downloaded from my Publications page.
The main idea behind this paper is to investigate subjectivity in chord transcriptions and estimations. In the former, we were interested in finding out how consistent a set of musical experts annotated the chords to a given song. Crucially, if this agreement is less than 100%, then we cannot hope to ever design a ‘perfect’ chord estimation algorithm. Furthermore, if the maximum agreement between experts is, say, 95%, then any algorithm which scores higher than this must be modelling the nuances of a particular/set of particular annotators, which we define as the annotators’ subjectivity. We see no scientific gain for doing this, and so 95% (in this case) upper-bounds automatic chord estimation performance.
To study this, we asked 5 experts (including myself!) to annotate a set of 20 songs. We then measured each annotator’s estimation against the consensus annotation, which we assume to be a less subjective truth, and to also converge to the ‘true’ annotations as the number of experts tends to infinity. The results, interestingly, show that the most skilled annotator (not me, unfortunately; I’m annotator A.3) was able to score 90% against the consensus, indicating an upper bound of 90% on automatic chord estimation algorithms.
Next, we moved on to study subjectivity in automatic methods. Here, we find that the best systems are already able to achieve accuracies close to that of trained humans. Also, we derive a Sequence Crowd Learning algorithm which is able to obtain an accurate consensus annotation from a set of examples. This type of post-processing/bootstrapping has been explored in speech recognition (see ROVER, Recognizer Output Voting Error Reduction) but also in recent work in beat tracking. I also have something in the works for this…stay tuned!
“To assess the performance of an automatic chord estimation system, reference annotations are indispensable. However, owing to the complexity of music and the sometimes ambiguous harmonic structure of polyphonic music, chord annotations are inherently subjective, and as a result any derived accuracy estimates will be subjective as well. In this paper we investigate the extent of the confounding effect of subjectivity in reference annotations. Our results show that this effect is important, and they affect different types of automatic chord estimation systems in different ways. Our results have implications for research on automatic chord estimation, but also on other fields that evaluate performance by comparing against human provided annotations that are confounded by subjectivity.”