编译Ansj之Solr插件

　　Ansj是一个比较优秀的中文分词组件，具体情况就不在本文介绍了。ansj作者在其官方代码中，提供了对lucene接口的支持。如果用在Solr下，还需要简单的扩展一下。

1、基于maven管理

ansj是基于maven进行开发管理的。我们首先修改一下其pom.xml，具体如下所示：

<PRoject xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><parent><groupId>org.ansj</groupId><artifactId>MavenAccount-aggregator</artifactId><version>0.0.1</version><relativePath>../pom.xml</relativePath></parent><artifactId>ansj_lucene4_plug</artifactId><version>2.0.2</version><packaging>jar</packaging><name>ansj_lucene4_plug</name> <properties>        <solr.version>4.8.0</solr.version>    </properties><dependencies><dependency><groupId>org.ansj</groupId><artifactId>ansj_seg</artifactId><version>2.0.5</version><classifier>min</classifier><scope>provided</scope></dependency><dependency><groupId>org.apache.lucene</groupId><artifactId>lucene-core</artifactId><version>${solr.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.lucene</groupId><artifactId>lucene-highlighter</artifactId><version>${solr.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.lucene</groupId><artifactId>lucene-queries</artifactId><version>${solr.version}</version><scope>provided</scope></dependency><dependency><groupId>org.apache.lucene</groupId><artifactId>lucene-queryparser</artifactId><version>${solr.version}</version><scope>provided</scope></dependency>   <dependency><groupId>org.apache.solr</groupId><artifactId>solr-dataimporthandler</artifactId><version>${solr.version}</version><scope>provided</scope>  </dependency><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.4</version><scope>test</scope></dependency></dependencies></project>

　　其中，代码依赖的配置项：<scope>provided</scope> 表示只用于代码编译阶段。依赖关系整理好以后，写一个TokenizerFactory类，用于solr中配置使用，代码如下：

package org.ansj.solr;import java.io.BufferedReader;import java.io.File;import java.io.FileInputStream;import java.io.FileNotFoundException;import java.io.IOException;import java.io.InputStreamReader;import java.io.Reader;import java.util.HashSet;import java.util.Map;import java.util.Set;import org.ansj.lucene.util.AnsjTokenizer;import org.ansj.splitWord.analysis.IndexAnalysis;import org.ansj.splitWord.analysis.ToAnalysis;import org.apache.lucene.analysis.Tokenizer;import org.apache.lucene.analysis.util.TokenizerFactory;import org.apache.lucene.util.AttributeSource.AttributeFactory;public class AnsjTokenizerFactory extends TokenizerFactory{    boolean pstemming;    boolean isQuery;    private String stopwordsDir;    public Set<String> filter;      public AnsjTokenizerFactory(Map<String, String> args) {        super(args);        assureMatchVersion();        isQuery = getBoolean(args, "isQuery", true);        pstemming = getBoolean(args, "pstemming", false);        stopwordsDir = get(args,"words");        addStopwords(stopwordsDir);    }    //add stopwords list to filter    private void addStopwords(String dir) {        if (dir == null){            System.out.println("no stopwords dir");            return;        }        //read stoplist        System.out.println("stopwords: " + dir);        filter = new HashSet<String>();        File file = new File(dir);         InputStreamReader reader;        try {            reader = new InputStreamReader(new FileInputStream(file),"UTF-8");            BufferedReader br = new BufferedReader(reader);             String word = br.readLine();              while (word != null) {                filter.add(word);                word = br.readLine();             }          } catch (FileNotFoundException e) {            System.out.println("No stopword file found");        } catch (IOException e) {            System.out.println("stopword file io exception");        }          }    @Override    public Tokenizer create(AttributeFactory factory, Reader input) {        if(isQuery == true){            //query            return new AnsjTokenizer(new ToAnalysis(new BufferedReader(input)), input, filter, pstemming);        } else {            //index            return new AnsjTokenizer(new IndexAnalysis(new BufferedReader(input)), input, filter, pstemming);        }    }       }

　　pstemming 参数是ansj需要的参数。

　　isQuery 是用于判断是查询还是索引，一般搜索index阶段分词比较细，查询的分词比较粗。

2、编译jar包。

代码结构如下：

　　编写mavn编译命令：mvn install -DskipTests=true# 忽略单元测试编译。

执行编译：

[INFO] Scanning for projects...[INFO]                                                                         [INFO] ------------------------------------------------------------------------[INFO] Building ansj_lucene4_plug 2.0.2[INFO] ------------------------------------------------------------------------[INFO] [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ ansj_lucene4_plug ---[INFO] Deleting R:/ansj-seg/ansj_seg/plug/ansj_lucene4_plug/target[INFO] [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ ansj_lucene4_plug ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] skip non existing resourceDirectory R:/ansj-seg/ansj_seg/plug/ansj_lucene4_plug/src/main/resources[INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ ansj_lucene4_plug ---[INFO] Compiling 5 source files to R:/ansj-seg/ansj_seg/plug/ansj_lucene4_plug/target/classes[INFO] [INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ ansj_lucene4_plug ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] skip non existing resourceDirectory R:/ansj-seg/ansj_seg/plug/ansj_lucene4_plug/src/test/resources[INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ ansj_lucene4_plug ---[INFO] Compiling 3 source files to R:/ansj-seg/ansj_seg/plug/ansj_lucene4_plug/target/test-classes[INFO] [INFO] --- maven-surefire-plugin:2.7.1:test (default-test) @ ansj_lucene4_plug ---[INFO] Tests are skipped.[INFO] [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ ansj_lucene4_plug ---[INFO] Building jar: R:/ansj-seg/ansj_seg/plug/ansj_lucene4_plug/target/ansj_lucene4_plug-2.0.2.jar[INFO] [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ ansj_lucene4_plug ---[INFO] Installing R:/ansj-seg/ansj_seg/plug/ansj_lucene4_plug/target/ansj_lucene4_plug-2.0.2.jar to C:/Users/GCZX-016/.m2/repository/org/ansj/ansj_lucene4_plug/2.0.2/ansj_lucene4_plug-2.0.2.jar[INFO] Installing R:/ansj-seg/ansj_seg/plug/ansj_lucene4_plug/pom.xml to C:/Users/GCZX-016/.m2/repository/org/ansj/ansj_lucene4_plug/2.0.2/ansj_lucene4_plug-2.0.2.pom[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 8.149s[INFO] Finished at: Tue May 05 15:29:19 CST 2015[INFO] Final Memory: 27M/245M[INFO] ------------------------------------------------------------------------