io.bdrc.lucene.bo

Class TibSyllableTokenizer

java.lang.Object

org.apache.lucene.util.AttributeSource

org.apache.lucene.analysis.TokenStream

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.util.CharTokenizer

io.bdrc.lucene.bo.TibSyllableTokenizer

All Implemented Interfaces:

Closeable, AutoCloseable

public final class TibSyllableTokenizer
extends org.apache.lucene.analysis.util.CharTokenizer

A TibSyllableTokenizer divides text between sequences of Tibetan Letter and/or Digit characters and sequences of all other characters - typically some sort of white space but other punctuation and characters from other language code-pages are not considered as constituents of tokens for the purpose of search and indexing.

Adjacent sequences of Tibetan Letter and/or Digit characters form tokens.

Derived from Lucene 6.4.1 analysis.core.WhitespaceTokenizer.java

Author:

Chris Tomlinson, Hélios Hildt

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

org.apache.lucene.util.AttributeSource.State

Field Summary

Fields inherited from class org.apache.lucene.analysis.Tokenizer

input

Fields inherited from class org.apache.lucene.analysis.TokenStream

DEFAULT_TOKEN_ATTRIBUTE_FACTORY

Constructor Summary

Constructors

Constructor and Description
TibSyllableTokenizer() Construct a new TibSyllableTokenizer.

Method Summary

Modifier and Type	Method and Description
protected boolean	isTibLetterOrDigit(int c)
protected boolean	isTokenChar(int c) Collects only characters which satisfy isTibetanLetterOrDigit()

Methods inherited from class org.apache.lucene.analysis.util.CharTokenizer

end, fromSeparatorCharPredicate, fromSeparatorCharPredicate, fromSeparatorCharPredicate, fromSeparatorCharPredicate, fromTokenCharPredicate, fromTokenCharPredicate, fromTokenCharPredicate, fromTokenCharPredicate, incrementToken, normalize, reset

Methods inherited from class org.apache.lucene.analysis.Tokenizer

close, correctOffset, setReader

Methods inherited from class org.apache.lucene.util.AttributeSource

addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

TibSyllableTokenizer

public TibSyllableTokenizer()

Construct a new TibSyllableTokenizer.

Method Detail

isTibLetterOrDigit

protected boolean isTibLetterOrDigit(int c)

isTokenChar

protected boolean isTokenChar(int c)

Collects only characters which satisfy isTibetanLetterOrDigit()

Specified by:

isTokenChar in class org.apache.lucene.analysis.util.CharTokenizer