Material Detail

Applying Syntactic Similarity Algorithms for Enterprise Information Management

Applying Syntactic Similarity Algorithms for Enterprise Information Management

This video was recorded at 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Paris 2009. For implementing content management solutions and enabling new applications associated with data retention, regulatory compliance, and litigation issues, enterprises need to develop advanced analytics to uncover relationships among the documents, e.g., content similarity, provenance, and clustering. In this paper, we evaluate the performance of four syntactic similarity algorithms. Three algorithms are based on Broder's ``shingling'' technique while the fourth algorithm employs a more recent approach, ``content-based chunking''. For our experiments, we use a specially designed corpus of documents that includes a set of ``similar'' documents with a controlled number of modifications. Our performance study reveals that the similarity metric of all four algorithms is highly sensitive to settings of the algorithms' parameters: sliding window size and fingerprint sampling frequency. We identify a useful range of these parameters for achieving good practical results, and compare the performance of the four algorithms in a controlled environment. We validate our results by applying these algorithms to finding near-duplicates in two large collections of HP technical support documents.


  • User Rating
  • Comments
  • Learning Exercises
  • Bookmark Collections
  • Course ePortfolios
  • Accessibility Info

More about this material


Log in to participate in the discussions or sign up if you are not already a MERLOT member.