I have been doing a lot work in the field of effort estimation in conjunction with probability of detecton (pd) and probability of false alarm (pf). The previous work in the field has focused heavily on pd and pf and practically ignored effort. In fact, in a previous paper the authors reported very high pds and acceptable pfs and ignored the fact that if those detectors where to be used the person checking the application would have to look at almost all of the code.
A quick break down of what these three metrics are.
- pd - The number of things a detector detected right over the total number of right things in the set.
- pf - The number of things a detector said were right when they weren't over the total number of wrong things.
- effort - Special metric for my current work. It is the number of lines of code a detecter can make a user look at over the total lines of code in the data base. This is used on some specific data files I am using.
- detector - In the case of this research, it is simply a rule in the form of a conjunction of disjunctions.
- ie: Age >= 10 OR age <= 5 AND Height > 5'10"
The score of a given rule is simply the Euclidean distance from (1,0,0) the rule is. The vector here is (pd,pf,effort). The major problem I have been running into lately is that it seems there is some fundamental tie in the data between pd, pf, and effort. That is they are correllated in such a way that maximizing one while minimizing the other two causes some trade-offs that are not worth it. I very often see rules with a score (0.5, 0.5, 0.5 ) for instance. While it gets a fairly high score (which is the only thing Which cares about) it really does not help us at all.
Anyway, this is the current work that I have been working on for a couple of months right now, to no avail.