Proceedings of the 2001 AMIA Symposium, pp 319-23.
(226kb)
The global weight of a term is greater for the less
frequent terms. This is reasonable because the presence of a term that
occurred in most of the documents would really tell one very little about
a document. On the other hand, a term that occurred in only 100
documents of one million would be very helpful in limiting the set
of documents of interest. A word that occurred in only 10 documents is
likely to be even more informative and will receive an even higher weight.
The local weight of a term is the measure of its
importance in a particular document. Generally, the more frequent a term
is within a document, the more important it is in representing the content
of that document. However, this relationship is saturating, i.e., as
the frequency continues to go up, the importance of the word increases less
rapidly and finally comes to a finite limit. In addition, we do not want a longer
document to be considered more important just because it is longer; therefore,
a length correction is applied. This local weight computation is based
on the Poisson distribution and the formula can be found in:
(226kb)
The similarity between two documents is computed
by adding up the weights (local wt1 × local wt2 × global wt) of all of the
terms the two documents have in common. This provides an indication of
how related two documents are. The resultant score is an example of a vector
score. Vector scoring was originated by Gerard Salton and has a long history
in text retrieval. The interested reader is referred to Salton, Automatic
Text Processing, Reading, MA: Addison-Wesley, 1989 for further information
on this topic. Our approach differs from other approaches in the
way we calculate the local weights for the individual terms. Once the similarity
score of a document in relation to each of the other documents in the database
has been computed, that document's neighbors are identified as the most
similar (highest scoring) documents found. These closely related documents
are pre-computed for each document in PubMed so that when you select
Related Articles, the system has only to retrieve this list. This
enables a fast response time for such queries.
| Last Modified: March 16, 2004 | ii-public | ||||||||||||||||||||||
|
|||||||||||||||||||||||
|
Lister Hill National Center for Biomedical Communications |
|
U.S. National Library of Medicine |
|
National Institutes of Health | ||||||||||||||||||
|
Department of Health and Human Services | ||||||||||||||||||||||
|
|||||||||||||||||||||||