Module tomotopy.label

tomotopy.label 서브모듈은 자동 토픽 라벨링 기법을 제공합니다. 아래에 나온 코드처럼 간단한 작업을 통해 토픽 모델의 결과에 이름을 붙일 수 있습니다. 그 결과는 코드 하단에 첨부되어 있습니다.

::

import tomotopy as tp

corpus = tp.utils.Corpus(tokenizer=tp.utils.SimpleTokenizer(), stopwords=['.'])
# data_feeder yields a tuple of (raw string, user data) or a str (raw string)
corpus.process(open(input_file, encoding='utf-8'))

# make LDA model and train
mdl = tp.LDAModel(k=20, min_cf=10, min_df=5, corpus=corpus)
mdl.train(0)
print('Num docs:', len(mdl.docs), ', Vocab size:', len(mdl.used_vocabs), ', Num words:', mdl.num_words)
print('Removed top words:', mdl.removed_top_words)
for i in range(0, 1000, 10):
    mdl.train(10)
    print('Iteration: {}\tLog-likelihood: {}'.format(i, mdl.ll_per_word))

# extract candidates for auto topic labeling
extractor = tp.label.PMIExtractor(min_cf=10, min_df=5, max_len=5, max_cand=10000)
cands = extractor.extract(mdl)

# ranking the candidates of labels for a specific topic
labeler = tp.label.FoRelevance(mdl, cands, min_df=5, smoothing=1e-2, mu=0.25)
for k in range(mdl.k):
    print("== Topic #{} ==".format(k))
    print("Labels:", ', '.join(label for label, score in labeler.get_topic_labels(k, top_n=5)))
    for word, prob in mdl.get_topic_words(k, top_n=10):
        print(word, prob, sep='\t')
    print()

# Example of Results
# -----------------
# == Topic #13 ==
# Labels: american basebal, american actress, lawyer politician, race car driver, brown american
# american        0.061747949570417404
# english 0.02476435713469982
# player  0.02357063814997673
# politician      0.020087148994207382
# footbal 0.016364915296435356
# author  0.014303036034107208
# actor   0.01202411763370037
# french  0.009745198301970959
# academ  0.009701790288090706
# produc  0.008822779171168804
# 
# == Topic #16 ==
# Labels: lunar, saturn, orbit moon, nasa report, orbit around
# apollo  0.03052366152405739
# star    0.017564402893185616
# mission 0.015656694769859314
# earth   0.01532777864485979
# lunar   0.015130429528653622
# moon    0.013683202676475048
# orbit   0.011315013282001019
# crew    0.01092031504958868
# space   0.010821640491485596
# nasa    0.009999352507293224

Classes

class Candidate (*args, **kwargs)

인스턴스 변수

var cf

var df

var name

var score

var words

class FoRelevance (topic_model, cands, min_df=5, smoothing=0.01, mu=0.25, window_size=-1, workers=0)

Expand source code

class FoRelevance(_LabelFoRelevance):
    def __init__(self, topic_model, cands, min_df=5, smoothing=0.01, mu=0.25, window_size=-1, workers=0):
        '''.. versionadded:: 0.6.0

This type provides an implementation of First-order Relevance for topic labeling based on the following papers:

> * Mei, Q., Shen, X., & Zhai, C. (2007, August). Automatic labeling of multinomial topic models. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 490-499).

Parameters
----------
topic_model
    an instance of topic model to label topics
cands : Iterable[tomotopy.label.Candidate]
    a list of candidates to be used as topic labels
min_df : int
    minimum document frequency of collocations. Collocations with a smaller document frequency than `min_df` are excluded from the candidates.
    Set this value large if the corpus is big
smoothing : float
    a small value greater than 0 for Laplace smoothing
mu : float
    a discriminative coefficient. Candidates with high score on a specific topic and with low score on other topics get a higher final score when this value is larger.
window_size : int
    .. versionadded:: 0.10.0
    
    size of the sliding window for calculating co-occurrence. If `window_size=-1`, it uses the whole document, instead of the sliding windows.
    If your documents are long, it is recommended to set this value to 50 ~ 100, not -1.
workers : int
    an integer indicating the number of workers to perform samplings. 
    If `workers` is 0, the number of cores in the system will be used.
'''
        super().__init__(topic_model, cands, min_df, smoothing, mu, window_size, workers)
    
    def get_topic_labels(self, k, top_n=10) -> List[Tuple[str, float]]:
        '''Return the top-n label candidates for the topic `k`

Parameters
----------
k : int
    an integer indicating a topic
top_n : int
    the number of labels
'''
        return super().get_topic_labels(k, top_n)

추가된 버전: 0.6.0

First-order Relevance를 이용한 토픽 라벨링 기법을 제공합니다. 이 구현체는 다음 논문에 기초하고 있습니다:

• Mei, Q., Shen, X., & Zhai, C. (2007, August). Automatic labeling of multinomial topic models. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 490-499).

파라미터

topic_model

토픽명을 붙일 토픽 모델의 인스턴스

cands : Iterable[Candidate]

토픽명으로 사용될 후보들의 리스트

min_df : int

사용하려는 후보의 최소 문헌 빈도. 연어가 등장하는 문헌 수가 min_df보다 작은 경우 선택에서 제외됩니다. 분석하려는 코퍼스가 클 경우 이 값을 키우십시오.

smoothing : float

라플라스 평활화에 사용될 0보다 큰 작은 실수

mu : float

변별성 계수. 이 계수가 클 때, 특정 토픽에 대해서만 높은 점수를 가지고 나머지 토픽에 대해서는 낮은 점수를 가진 후보가 더 높은 최종 점수를 받습니다.

window_size : int

추가된 버전: 0.10.0

동시출현 빈도를 계산하기 위한 슬라이딩 윈도우의 크기. -1로 설정시 슬라이딩 윈도우를 사용하지 않고, 문헌 전체를 활용해 동시출현 빈도를 계산합니다. 분석에 사용하는 문헌들의 길이가 길다면, 이 값을 -1이 아닌 50 ~ 100 정도로 설정하는 걸 권장합니다.

workers : int

깁스 샘플링을 수행하는 데에 사용할 스레드의 개수입니다. 만약 이 값을 0으로 설정할 경우 시스템 내의 가용한 모든 코어가 사용됩니다.

추가된 버전: 0.6.0

This type provides an implementation of First-order Relevance for topic labeling based on the following papers:

• Mei, Q., Shen, X., & Zhai, C. (2007, August). Automatic labeling of multinomial topic models. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 490-499).

파라미터

topic_model

an instance of topic model to label topics

cands : Iterable[Candidate]

a list of candidates to be used as topic labels

min_df : int

minimum document frequency of collocations. Collocations with a smaller document frequency than min_df are excluded from the candidates. Set this value large if the corpus is big

smoothing : float

a small value greater than 0 for Laplace smoothing

mu : float

a discriminative coefficient. Candidates with high score on a specific topic and with low score on other topics get a higher final score when this value is larger.

window_size : int

추가된 버전: 0.10.0

size of the sliding window for calculating co-occurrence. If window_size=-1, it uses the whole document, instead of the sliding windows. If your documents are long, it is recommended to set this value to 50 ~ 100, not -1.

workers : int

an integer indicating the number of workers to perform samplings. If workers is 0, the number of cores in the system will be used.

메소드

def get_topic_labels(self, k, top_n=10) ‑> List[Tuple[str, float]]

Expand source code

    def get_topic_labels(self, k, top_n=10) -> List[Tuple[str, float]]:
        '''Return the top-n label candidates for the topic `k`

Parameters
----------
k : int
    an integer indicating a topic
top_n : int
    the number of labels
'''
        return super().get_topic_labels(k, top_n)

토픽 k에 해당하는 레이블 후보 상위 n개를 반환합니다.

파라미터

k : int

토픽을 지정하는 정수

top_n : int

토픽 레이블의 개수

class PMIExtractor (min_cf=10, min_df=5, min_len=1, max_len=5, max_cand=5000, normalized=False)

Expand source code

class PMIExtractor(_LabelPMIExtractor):
    def __init__(self, min_cf=10, min_df=5, min_len=1, max_len=5, max_cand=5000, normalized=False):
        '''.. versionadded:: 0.6.0

`PMIExtractor` exploits multivariate pointwise mutual information to extract collocations. 
It finds a string of words that often co-occur statistically.

Parameters
----------
min_cf : int
    minimum collection frequency of collocations. Collocations with a smaller collection frequency than `min_cf` are excluded from the candidates.
    Set this value large if the corpus is big
min_df : int
    minimum document frequency of collocations. Collocations with a smaller document frequency than `min_df` are excluded from the candidates.
    Set this value large if the corpus is big
min_len : int
    .. versionadded:: 0.10.0
    
    minimum length of collocations. `min_len=1` means that it extracts not only collocations but also all single words.
    The number of single words is excluded in counting `max_cand`.
max_len : int
    maximum length of collocations
max_cand : int
    maximum number of candidates to extract
'''
        super().__init__(min_cf, min_df, min_len, max_len, max_cand, normalized)

    def extract(self, topic_model) -> List:
        '''Return the list of `tomotopy.label.Candidate`s extracted from `topic_model`

Parameters
----------
topic_model
    an instance of topic model with documents to extract candidates
'''
        return super().extract(topic_model)

추가된 버전: 0.6.0

PMIExtractor는 다변수 점별 상호정보량을 활용해 연어를 추출합니다. 이는 통계적으로 자주 함께 등장하는 단어열을 찾아줍니다.

파라미터

min_cf : int

추출하려는 후보의 최소 장서 빈도. 문헌 내 등장하는 빈도수가 min_cf보다 작은 연어는 후보에서 제외됩니다. 분석하려는 코퍼스가 클 경우 이 값을 키우십시오.

min_df : int

추출하려는 후보의 최소 문헌 빈도. 연어가 등장하는 문헌 수가 min_df보다 작은 경우 후보에서 제외됩니다. 분석하려는 코퍼스가 클 경우 이 값을 키우십시오.

min_len : int

추가된 버전: 0.10.0

분석하려는 연어의 최소 길이. 1로 설정시 단일 단어들도 모두 추출합니다. 단일 단어들은 max_cand 개수 계산에서 제외됩니다.

max_len : int

분석하려는 연어의 최대 길이

max_cand : int

추출하려는 후보의 최대 개수

추가된 버전: 0.6.0

PMIExtractor exploits multivariate pointwise mutual information to extract collocations. It finds a string of words that often co-occur statistically.

파라미터

min_cf : int

minimum collection frequency of collocations. Collocations with a smaller collection frequency than min_cf are excluded from the candidates. Set this value large if the corpus is big

min_df : int

minimum document frequency of collocations. Collocations with a smaller document frequency than min_df are excluded from the candidates. Set this value large if the corpus is big

min_len : int

추가된 버전: 0.10.0

minimum length of collocations. min_len=1 means that it extracts not only collocations but also all single words. The number of single words is excluded in counting max_cand.

max_len : int

maximum length of collocations

max_cand : int

maximum number of candidates to extract

메소드

def extract(self, topic_model) ‑> List

Expand source code

    def extract(self, topic_model) -> List:
        '''Return the list of `tomotopy.label.Candidate`s extracted from `topic_model`

Parameters
----------
topic_model
    an instance of topic model with documents to extract candidates
'''
        return super().extract(topic_model)

topic_model로부터 추출된 토픽 레이블 후보인 Candidate의 리스트를 반환합니다.

파라미터

topic_model

후보를 추출할 문헌들을 가지고 있는 토픽 모델의 인스턴스