퍼지 문자열 비교

퍼지 문자열 비교

---

내가 완성하려고 노력하는 것은 파일을 읽고 원래 문장에 따라 각 문장을 비교하는 프로그램입니다. 원본과 완벽하게 일치하는 문장은 1 점을 받고 정반대 문장은 0 점을받습니다. 다른 모든 퍼지 문장은 1과 0 사이의 점수를받습니다.

Python 3에서이 작업을 완료하는 데 사용할 작업이 무엇인지 잘 모르겠습니다.

텍스트 1이 원본이고 앞의 다른 문자열이 비교 인 샘플 텍스트를 포함했습니다.

텍스트 : 샘플

텍스트 1 : 어둡고 폭풍우가 치는 밤이었습니다. 나는 빨간 의자에 혼자 앉아 있었다. 나는 고양이 세 마리가 있었기 때문에 완전히 혼자가 아니었다.

텍스트 20 : 어둡고 폭풍우가 치는 밤이었습니다. 나는 진홍색 의자에 혼자 앉아 있었다. 나는 3 마리의 고양이가 있었기 때문에 완전히 혼자가 아니었다. // 높은 점수를 받아야하지만 1 점은 아니다

텍스트 21 : 어둡고 격렬한 밤이었습니다. 나는 진홍색 카테 드라에 혼자 앉아 있었다. 나는 3 마리의 고양이가 있었기 때문에 완전히 혼자가 아니었다. // 텍스트 20보다 낮은 점수를 받아야한다

텍스트 22 : 나는 진홍색 카테 드라에 혼자 앉아 있었다. 나는 세 마리의 고양이가 있었기 때문에 완전히 혼자가 아니 었습니다. 어둡고 격렬한 밤이었다. // 점수는 텍스트 21보다 낮지 만 0은 안됩니다.

텍스트 24 : 어둡고 폭풍우가 치는 밤이었습니다. 나는 혼자가 아니었다. 나는 빨간 의자에 앉아 있지 않았다. 나는 고양이 세 마리가 있었다. // 0 점!

---

라는 패키지가 fuzzywuzzy있습니다. pip를 통해 설치 :

pip install fuzzywuzzy

간단한 사용법 :

>>> from fuzzywuzzy import fuzz
>>> fuzz.ratio("this is a test", "this is a test!")
    96

패키지는 difflib. 왜 그것을 사용하지 않습니까? 조금 더 간단한 것 외에도 실제로 더 강력하게 만드는 여러 가지 일치 방법 (예 : 토큰 순서 무감각, 부분 문자열 일치)이 있습니다. process.extract기능이 특히 유용합니다 : 집합에서 가장 일치하는 문자열과 비율을 찾을 수 있습니다. Readme에서 :

부분 비율

>>> fuzz.partial_ratio("this is a test", "this is a test!")
    100

토큰 정렬 비율

>>> fuzz.ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
    90
>>> fuzz.token_sort_ratio("fuzzy wuzzy was a bear", "wuzzy fuzzy was a bear")
    100

토큰 세트 비율

>>> fuzz.token_sort_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")
    84
>>> fuzz.token_set_ratio("fuzzy was a bear", "fuzzy fuzzy was a bear")
    100

방법

>>> choices = ["Atlanta Falcons", "New York Jets", "New York Giants", "Dallas Cowboys"]
>>> process.extract("new york jets", choices, limit=2)
    [('New York Jets', 100), ('New York Giants', 78)]
>>> process.extractOne("cowboys", choices)
    ("Dallas Cowboys", 90)

---

표준 라이브러리 (라고 함 difflib)에는 문자열을 비교하고 유사성에 따라 점수를 반환 할 수 있는 모듈 이 있습니다. SequenceMatcher클래스는 이후에 무엇을해야한다.

편집 : 파이썬 프롬프트의 작은 예 :

>>> from difflib import SequenceMatcher as SM
>>> s1 = ' It was a dark and stormy night. I was all alone sitting on a red chair. I was not completely alone as I had three cats.'
>>> s2 = ' It was a murky and stormy night. I was all alone sitting on a crimson chair. I was not completely alone as I had three felines.'
>>> SM(None, s1, s2).ratio()
0.9112903225806451

HTH!

---

fuzzyset인덱싱과 검색 모두에서 fuzzywuzzy( difflib) 보다 훨씬 빠릅니다 .

from fuzzyset import FuzzySet
corpus = """It was a murky and stormy night. I was all alone sitting on a crimson chair. I was not completely alone as I had three felines
    It was a murky and tempestuous night. I was all alone sitting on a crimson cathedra. I was not completely alone as I had three felines
    I was all alone sitting on a crimson cathedra. I was not completely alone as I had three felines. It was a murky and tempestuous night.
    It was a dark and stormy night. I was not alone. I was not sitting on a red chair. I had three cats."""
corpus = [line.lstrip() for line in corpus.split("\n")]
fs = FuzzySet(corpus)
query = "It was a dark and stormy night. I was all alone sitting on a red chair. I was not completely alone as I had three cats."
fs.get(query)
# [(0.873015873015873, 'It was a murky and stormy night. I was all alone sitting on a crimson chair. I was not completely alone as I had three felines')]

Warning: Be careful not to mix unicode and bytes in your fuzzyset.

---

The task is called Paraphrase Identification which is an active area of research in Natural Language Processing. I have linked several state of the art papers many of which you can find open source code on GitHub for.

Note that all the answered question assume that there is some string/surface similarity between the two sentences while in reality two sentences with little string similarity can be semantically similar.

If you're interested in that kind of similarity you can use Skip-Thoughts. Install the software according to the GitHub guides and go to paraphrase detection section in readme:

import skipthoughts
model = skipthoughts.load_model()
vectors = skipthoughts.encode(model, X_sentences)

This converts your sentences (X_sentences) to vectors. Later you can find the similarity of two vectors by:

similarity = 1 - scipy.spatial.distance.cosine(vectors[0], vectors[1])

where we are assuming vector[0] and vector1 are the corresponding vector to X_sentences[0], X_sentences1 which you wanted to find their scores.

There are other models to convert a sentence to a vector which you can find here.

Once you convert your sentences into vectors the similarity is just a matter of finding the Cosine similarity between those vectors.

참고URL : https://stackoverflow.com/questions/10383044/fuzzy-string-comparison