— Helen Walker (@helenwalker7) March 26, 2017
I’m intrigued by the notion that Keyhole purports to be able to deliver information about sentiment. As a former English teacher, I was inherently sceptical about this: I used to spend much time with my students discussing connotations and the slipperiness of words…So, this was interesting:
‘The accuracy of a sentiment analysis system is, in principle, how well it agrees with human judgments. This is usually measured by precision and recall. However, according to research human raters typically agree 79% of the time.
Thus, a 70% accurate program is doing nearly as well as humans, even though such accuracy may not sound impressive. If a program were “right” 100% of the time, humans would still disagree with it about 20% of the time, since they disagree that much about any answer . More sophisticated measures can be applied, but evaluation of sentiment analysis systems remains a complex matter. For sentiment analysis tasks returning a scale rather than a binary judgement, correlation is a better measure than precision because it takes into account how close the predicted value is to the target value.