public interface Tokenizer
Modifier and Type | Method and Description |
---|---|
java.lang.String |
getErrorDescription()
If hasErrors returns true, returns a description of the error
encountered.
|
Token |
getNextToken()
Returns the next token.
|
boolean |
hasErrors()
Returns true if there were errors while reading tokens.
|
boolean |
hasMoreTokens()
Returns true if there are more tokens, false otherwise.
|
boolean |
isBreak()
Determines if the current token should start a new sentence.
|
void |
setInputReader(java.io.Reader reader)
Sets the input reader.
|
void |
setInputText(java.lang.String textToTokenize)
Sets the text to be tokenized by this tokenizer.
|
void |
setPostpunctuationSymbols(java.lang.String symbols)
Sets the postpunctuation symbols of this Tokenizer to the given
symbols.
|
void |
setPrepunctuationSymbols(java.lang.String symbols)
Sets the prepunctuation symbols of this Tokenizer to the given
symbols.
|
void |
setSingleCharSymbols(java.lang.String symbols)
Sets the single character symbols of this Tokenizer to the given
symbols.
|
void |
setWhitespaceSymbols(java.lang.String symbols)
Sets the whitespace symbols of this Tokenizer to the given
symbols.
|
void setInputText(java.lang.String textToTokenize)
textToTokenize
- the text to tokenizevoid setInputReader(java.io.Reader reader)
reader
- the input sourceToken getNextToken()
boolean hasMoreTokens()
boolean hasErrors()
java.lang.String getErrorDescription()
void setWhitespaceSymbols(java.lang.String symbols)
symbols
- the whitespace symbolsvoid setSingleCharSymbols(java.lang.String symbols)
symbols
- the single character symbolsvoid setPrepunctuationSymbols(java.lang.String symbols)
symbols
- the prepunctuation symbolsvoid setPostpunctuationSymbols(java.lang.String symbols)
symbols
- the postpunctuation symbolsboolean isBreak()