Most common effectiveness measures for information retrieval systems
are based on the assumptions of binary relevance (either a document is
relevant to a given query or it is not) and binary retrieval (either a
document is retrieved or it is not). In this article, these
assumptions are questioned, and a new measure named ADM (average
distance measure) is proposed, discussed from a conceptual point of
view, and experimentally validated on Text Retrieval Conference (TREC)
data. Both conceptual analysis and experimental evidence demonstrate
ADM's adequacy in measuring the effectiveness of information retrieval
systems. Some potential problems about precision and recall are also
highlighted and discussed.