TextSimilarity is pure PHP library for detecting duplicate news articles using morphological analysis and entity-weighted scoring.
Read more about TextSimilarity:
- Morphological text processing via phpMorphy
- Named entity extraction (names, locations, organizations, abbreviations)
- Weighted scoring system with configurable coefficients
- File-based word cache for performance
- English and Russian language support
TextSimilarity is installed via Composer. To add a dependency to TextSimilarity in your project,
Run the following to use the latest stable version
composer require deevee15/text-similarity-php- PHP 8.1 or above
- phpMorphy library
use PHPTextSimilarity\TextSimilarity;
$result = TextSimilarity::compare(
'en',
'First article`s text...',
'Second article`s text...',
['first' => "Article's title", 'second' => "Article's title"]
);The TextSimilarity library divides all words from compared texts and article titles into entities (proper names, common nouns, locations, abbreviations, organizations), converts them to the nominative case, then retains only the matching ones, assigns points based on the matches, and multiplies them by the importance coefficients specified in src/Config/WeightConfig.php.
Here is the link to the demo website
Apache 2.0