org.apache.apex.malhar.flume.storage

Class HDFSStorage

java.lang.Object

org.apache.apex.malhar.flume.storage.HDFSStorage

All Implemented Interfaces:

com.datatorrent.api.Component<com.datatorrent.api.Context>, Storage, org.apache.flume.conf.Configurable

public class HDFSStorage
extends Object
implements Storage, org.apache.flume.conf.Configurable, com.datatorrent.api.Component<com.datatorrent.api.Context>

HDFSStorage is developed to store and retrieve the data from HDFS

The properties that can be set on HDFSStorage are:
baseDir - The base directory where the data is going to be stored
restore - This is used to restore the application from previous failure
blockSize - The maximum size of the each file to created.

Since:

0.9.3

Nested Class Summary

Nested classes/interfaces inherited from interface com.datatorrent.api.Component

com.datatorrent.api.Component.ComponentComplementPair<COMPONENT extends com.datatorrent.api.Component<?>,COMPLEMENT>

Field Summary

Fields

Modifier and Type	Field and Description
static String	BASE_DIR_KEY
static String	BLOCK_SIZE_MULTIPLE
static String	BLOCKSIZE
static int	DEFAULT_BLOCK_SIZE
static String	NUMBER_RETRY
static String	RESTORE_KEY

Fields inherited from interface org.apache.apex.malhar.flume.storage.Storage

ID

Fields inherited from interface com.datatorrent.api.Component

CONCAT_SEPARATOR, SPLIT_SEPARATOR

Constructor Summary

Constructors

Constructor and Description
HDFSStorage()

Method Summary

Methods

Modifier and Type	Method and Description
void	clean(byte[] identifier) This is used to clean up the files identified by identifier
void	configure(org.apache.flume.Context ctx) This stores the Identifier information identified in the last store function call
void	flush() This flushes the data from stream
String	getBaseDir()
long	getBlockSize()
int	getBlockSizeMultiple()
String	getId()
boolean	isRestore()
byte[]	retrieve(byte[] identifier) This returns the data bytes for the current identifier and the identifier for next data bytes.
byte[]	retrieveNext() This returns data bytes and the identifier for the next data bytes.
void	setBaseDir(String baseDir)
void	setBlockSize(long blockSize)
void	setBlockSizeMultiple(int blockSizeMultiple)
void	setId(String id)
void	setRestore(boolean restore)
void	setup(com.datatorrent.api.Context context)
byte[]	store(com.datatorrent.netlet.util.Slice slice) This stores the bytes and returns the unique identifier to retrieve these bytes
void	teardown()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BLOCK_SIZE

public static final int DEFAULT_BLOCK_SIZE

See Also:

Constant Field Values

BASE_DIR_KEY

public static final String BASE_DIR_KEY

See Also:

Constant Field Values

RESTORE_KEY

public static final String RESTORE_KEY

See Also:

Constant Field Values

BLOCKSIZE

public static final String BLOCKSIZE

See Also:

Constant Field Values

BLOCK_SIZE_MULTIPLE

public static final String BLOCK_SIZE_MULTIPLE

See Also:

Constant Field Values

NUMBER_RETRY

public static final String NUMBER_RETRY

See Also:

Constant Field Values

Constructor Detail

HDFSStorage

public HDFSStorage()

Method Detail

configure

public void configure(org.apache.flume.Context ctx)

This stores the Identifier information identified in the last store function call

Specified by:

configure in interface org.apache.flume.conf.Configurable

Parameters:

ctx -

store

public byte[] store(com.datatorrent.netlet.util.Slice slice)

Description copied from interface: Storage

This stores the bytes and returns the unique identifier to retrieve these bytes

Specified by:

store in interface Storage

Returns:

retrieve

public byte[] retrieve(byte[] identifier)

Description copied from interface: Storage

This returns the data bytes for the current identifier and the identifier for next data bytes.
The first eight bytes contain the identifier and the remaining bytes contain the data

Specified by:

retrieve in interface Storage

Returns:

retrieveNext

public byte[] retrieveNext()

Description copied from interface: Storage

This returns data bytes and the identifier for the next data bytes. The identifier for current data bytes is based on the retrieve method call and number of retrieveNext method calls after retrieve method call.
The first eight bytes contain the identifier and the remaining bytes contain the data

Specified by:

retrieveNext in interface Storage

Returns:

clean

public void clean(byte[] identifier)

Description copied from interface: Storage

This is used to clean up the files identified by identifier

Specified by:

clean in interface Storage

flush

public void flush()

Description copied from interface: Storage

This flushes the data from stream

Specified by:

flush in interface Storage

getBlockSizeMultiple

public int getBlockSizeMultiple()

setBlockSizeMultiple

public void setBlockSizeMultiple(int blockSizeMultiple)

getBaseDir

public String getBaseDir()

Returns:

the baseDir

setBaseDir

public void setBaseDir(String baseDir)

Parameters:

baseDir - the baseDir to set

getId

public String getId()

Returns:

the id

setId

public void setId(String id)

Parameters:

id - the id to set

getBlockSize

public long getBlockSize()

Returns:

the blockSize

setBlockSize

public void setBlockSize(long blockSize)

Parameters:

blockSize - the blockSize to set

isRestore

public boolean isRestore()

Returns:

the restore

setRestore

public void setRestore(boolean restore)

Parameters:

restore - the restore to set

setup

public void setup(com.datatorrent.api.Context context)

Specified by:

setup in interface com.datatorrent.api.Component<com.datatorrent.api.Context>

teardown

public void teardown()

Specified by:

teardown in interface com.datatorrent.api.Component<com.datatorrent.api.Context>