随机字符串识别 Gibberish Detection

 

一个简单的方案是:

1. 通过大量正常文本, 统计 "字->字" 条件概率方阵,

　　(例子中使用了26个英文字母+空格, 共27个基础字符, 构造的概率转移方阵为27*27)

 

2. 对于需要判断的文本, 将其拆解为顺序的2-gram词组, 对于每个词组, 查表得到第一个字符后出现第二个字符的概率,

 

3. 求字符串中所有2-gram词组的条件概率平均值, 如果低于一定阈值, 则说明该字符串字符间的次序性与常规概率分布有偏, 可能是随机生成的.

　　(例子中的阈值使用了 "(0类的最小概率 + 1类的最大概率) / 2" )

 

 

该方案并未更进一步计算多跳稳态矩阵, 与其说是MC, 其实为简单的条件概率 :)

 

 

#coding=utf8

import math
import pickle

accepted_chars = 'abcdefghijklmnopqrstuvwxyz '

pos = dict([(char, idx) for idx, char in enumerate(accepted_chars)])

def normalize(line):
    """ Return only the subset of chars from accepted_chars.
    This helps keep the  model relatively small by ignoring punctuation,
    infrequenty symbols, etc. """
    return [c.lower() for c in line if c.lower() in accepted_chars]

def ngram(n, l):
    """ Return all n grams from l after normalizing """
    filtered = normalize(l)
    for start in range(0, len(filtered) - n + 1):
        yield ''.join(filtered[start:start + n])

def train():
    """ Write a simple model as a pickle file """
    k = len(accepted_chars)
    # Assume we have seen 10 of each character pair.  This acts as a kind of
    # prior or smoothing factor.  This way, if we see a character transition
    # live that we've never observed in the past, we won't assume the entire
    # string has 0 probability.
    counts = [[10 for i in range(k)] for i in range(k)]

    # Count transitions from big text file, taken
    # from http://norvig.com/spell-correct.html
    # 统计常规大文本段落中,词词出现的次数
    for line in open('big.txt'):
        for a, b in ngram(2, line):
            counts[pos[a]][pos[b]] += 1

    # Normalize the counts so that they become log probabilities.
    # We use log probabilities rather than straight probabilities to avoid
    # numeric underflow issues with long texts.
    # This contains a justification:
    # http://squarecog.wordpress.com/2009/01/10/dealing-with-underflow-in-joint-probability-calculations/
    # 按首词进行概率归一化
    for i, row in enumerate(counts):
        s = float(sum(row))
        for j in range(len(row)):
            row[j] = math.log(row[j] / s)

    # Find the probability of generating a few arbitrarily choosen good and
    # bad phrases.
    # 分别计算good/bad类平均转移概率,以确定阈值
    good_probs = [avg_transition_prob(l, counts) for l in open('good.txt')]
    bad_probs = [avg_transition_prob(l, counts) for l in open('bad.txt')]

    # Assert that we actually are capable of detecting the junk.
    assert min(good_probs) > max(bad_probs)

    # And pick a threshold halfway between the worst good and best bad inputs.
    thresh = (min(good_probs) + max(bad_probs)) / 2
    pickle.dump({'mat': counts, 'thresh': thresh}, open('gib_model.pki', 'wb'))

def avg_transition_prob(line, log_prob_mat):
    """ Return the average transition prob from l through log_prob_mat. """
    log_prob = 0.0
    transition_ct = 0
    for a, b in ngram(2, line):
        log_prob += log_prob_mat[pos[a]][pos[b]]
        transition_ct += 1
    # The exponentiation translates from log probs to probs.
    return math.exp(log_prob / (transition_ct or 1))


if __name__ == '__main__':
    train()

 

 

参考:

https://github.com/rrenaud/Gibberish-Detector

用机器学习识别随机生成的C&C域名
乱语识别