io.bdrc.lucene.bo

Class TibWordTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

io.bdrc.lucene.bo.TibWordTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public final class TibWordTokenizer
extends org.apache.lucene.analysis.Tokenizer

A maximal-matching word tokenizer for Tibetan that uses a Trie.

Takes a syllable at a time and returns the longest sequence of syllable that form a word within the Trie.
isTibLetter(int) is used to distinguish clusters of letters forming syllables and u\0F0B(tsheg) to distinguish syllables within a word.
- Unknown syllables are tokenized as separate words.
- All the punctuation is discarded from the produced tokens, including the tsheg that usually follows "ང".

Due to its design, this tokenizer doesn't deal with contextual ambiguities.
For example, if both དོན and དོན་གྲུབ exist in the Trie, དོན་གྲུབ will be returned every time the sequence དོན + གྲུབ is found.
The sentence སེམས་ཅན་གྱི་དོན་གྲུབ་པར་ཤོག will be tokenized into "སེམས་ཅན + གྱི + དོན་གྲུབ + པར + ཤོག" (སེམས་ཅན + གྱི + དོན + གྲུབ་པར + ཤོག expected). Derived from Lucene 6.4.1 analysis.util.CharTokenizer

Author:

Élie Roux, Drupchen

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor and Description
TibWordTokenizer() Constructs a TibWordTokenizer using a default lexicon file (here "resource/output/total_lexicon.txt")
TibWordTokenizer(String trieFile)
TibWordTokenizer(io.bdrc.lucene.stemmer.Trie trie) Constructs a TibWordTokenizer using a given trie

Method Summary

Modifier and Type	Method and Description
void	end()
boolean	incrementToken()
boolean	isTibLetter(int c) Finds whether the given character is a Tibetan letter or not.
protected int	normalize(int c) Called on each token character to normalize it before it is added to the token.
void	reset()
void	setDebug(boolean debug)
void	setLemmatize(boolean lemmatize)

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset, setReader

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

TibWordTokenizer

public TibWordTokenizer()
                 throws IOException

Constructs a TibWordTokenizer using a default lexicon file (here "resource/output/total_lexicon.txt")

Throws:

IOException - the file containing the lexicon cannot be read

TibWordTokenizer

public TibWordTokenizer(io.bdrc.lucene.stemmer.Trie trie)

Constructs a TibWordTokenizer using a given trie

Parameters:

trie - built with BuildCompiledTrie.java

TibWordTokenizer

public TibWordTokenizer(String trieFile)
                 throws FileNotFoundException,
                        IOException

Throws:

FileNotFoundException

IOException

Method Detail

normalize

protected int normalize(int c)

Called on each token character to normalize it before it is added to the token. The default implementation does nothing. Subclasses may use this to, e.g., lowercase tokens.

Parameters:

c - the character to normalize

Returns:

the normalized character

incrementToken

public final boolean incrementToken()
                             throws IOException

Specified by:

incrementToken in class org.apache.lucene.analysis.TokenStream

Throws:

IOException

isTibLetter

public final boolean isTibLetter(int c)

Finds whether the given character is a Tibetan letter or not.

Parameters:

c - a unicode code-point

Returns:

true if c in the specified range; false otherwise

end

public final void end()
               throws IOException

Overrides:

end in class org.apache.lucene.analysis.TokenStream

Throws:

IOException

reset

public void reset()
           throws IOException

Overrides:

reset in class org.apache.lucene.analysis.Tokenizer

Throws:

IOException

setLemmatize

public final void setLemmatize(boolean lemmatize)

setDebug

public final void setDebug(boolean debug)