org.apache.apex.malhar.stream.api

Interface ApexStream<T>

Type Parameters:

T -

All Known Subinterfaces:

WindowedStream<T>

All Known Implementing Classes:

ApexStreamImpl

public interface ApexStream<T>

The stream interface to build a DAG

Since:

3.4.0

Method Summary

Methods

Modifier and Type	Method and Description
<O,STREAM extends ApexStream<O>> STREAM	addOperator(com.datatorrent.api.Operator op, com.datatorrent.api.Operator.InputPort<T> inputPort, com.datatorrent.api.Operator.OutputPort<O> outputPort) Extend the dag by adding one operator
<O,STREAM extends ApexStream<O>> STREAM	addOperator(String opName, com.datatorrent.api.Operator op, com.datatorrent.api.Operator.InputPort<T> inputPort, com.datatorrent.api.Operator.OutputPort<O> outputPort) Extend the dag by adding one Operator
<STREAM extends ApexStream<Integer>> STREAM	count() Count of all tuples
<STREAM extends ApexStream<Map<Object,Integer>>> STREAM	countByKey() Count tuples by the key If the input is KeyedTuple it will get the key from getKey method from the tuple If not, use the tuple itself as a key
<STREAM extends ApexStream<Map<Object,Integer>>> STREAM	countByKey(int key) Count tuples by the indexed key
com.datatorrent.api.DAG	createDag() Create dag from stream
<STREAM extends ApexStream<T>> STREAM	filter(Function.FilterFunction<T> filter) Filter transformation Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction
<STREAM extends ApexStream<T>> STREAM	filter(String name, Function.FilterFunction<T> filter) Filter transformation Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction
<O,STREAM extends ApexStream<O>> STREAM	flatMap(Function.FlatMapFunction<T,O> flatten) Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O
<O,STREAM extends ApexStream<O>> STREAM	flatMap(String name, Function.FlatMapFunction<T,O> flatten) Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O
<O,STREAM extends ApexStream<O>> STREAM	fold(O initialValue, Function.FoldFunction<T,O> fold) Fold transformation Add an operator to the DAG which merge tuple T to accumulated result tuple O
<O,STREAM extends ApexStream<O>> STREAM	fold(String name, O initialValue, Function.FoldFunction<T,O> fold) Fold transformation Add an operator to the DAG which merge tuple T to accumulated result tuple O
<O,STREAM extends ApexStream<O>> STREAM	map(Function.MapFunction<T,O> mapFunction) Simple map transformation Add an operator to the DAG which convert tuple T to tuple O
<O,STREAM extends ApexStream<O>> STREAM	map(String name, Function.MapFunction<T,O> mapFunction) Simple map transformation Add an operator to the DAG which convert tuple T to tuple O
void	populateDag(com.datatorrent.api.DAG dag) Populate existing dag
<STREAM extends ApexStream<T>> STREAM	print() Add a stdout console output operator
<STREAM extends ApexStream<T>> STREAM	printErr() Add a stderr console output operator
<STREAM extends ApexStream<T>> STREAM	reduce(Function.ReduceFunction<T> reduce) Reduce transformation Add an operator to the DAG which merge tuple t1, t2 to new tuple
<STREAM extends ApexStream<T>> STREAM	reduce(String name, Function.ReduceFunction<T> reduce) Reduce transformation Add an operator to the DAG which merge tuple t1, t2 to new tuple
void	run() Submit the application to cluster
void	runEmbedded(boolean async, long duration, Callable<Boolean> exitCondition) Run the stream application in local mode In Async mode, the method would return immediately and the dag would run for "duration" milliseconds In Sync mode, the method would block "duration" milliseconds to run the dag If duration is negative the dag would run forever until being killed
<STREAM extends ApexStream<T>> STREAM	setGlobalAttribute(com.datatorrent.api.Attribute attribute, Object value) Set attributes at the DAG level
<STREAM extends ApexStream<T>> STREAM	union(ApexStream<T>... others) Union multiple stream into one
<STREAM extends ApexStream<T>> STREAM	with(com.datatorrent.api.Attribute attribute, Object value) Set the attribute value If it is DAG attribute, it will be applied to the whole DAG If it is Operator attribute, it will be applied to last connected operator If it is InputPort attribute, it will be applied to the input port of the last connected stream If it is OutputPort attribute, it will be applied to the output port of the last connected stream If it is both InputPort&OutputPort attribute, it will be applied to last connected stream
<STREAM extends ApexStream<T>> STREAM	with(com.datatorrent.api.DAG.Locality locality) Set the locality
<STREAM extends ApexStream<T>> STREAM	with(String propName, Object value) Set the property value of the last connected operator

Method Detail

map

<O,STREAM extends ApexStream<O>> STREAM map(Function.MapFunction<T,O> mapFunction)

Simple map transformation
Add an operator to the DAG which convert tuple T to tuple O

Type Parameters:

O - Type of the output

Parameters:

mapFunction - map function

Returns:

new stream of type O

map

<O,STREAM extends ApexStream<O>> STREAM map(String name,
                                          Function.MapFunction<T,O> mapFunction)

Simple map transformation
Add an operator to the DAG which convert tuple T to tuple O

Type Parameters:

O - Type of the output

Parameters:

name - operator name

mapFunction - map function

Returns:

new stream of type O

flatMap

<O,STREAM extends ApexStream<O>> STREAM flatMap(Function.FlatMapFunction<T,O> flatten)

Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O

Type Parameters:

O - Type of the output

Parameters:

flatten - flat map

Returns:

new stream of type O

flatMap

<O,STREAM extends ApexStream<O>> STREAM flatMap(String name,
                                              Function.FlatMapFunction<T,O> flatten)

Flat map transformation
Add an operator to the DAG which convert tuple T to a collection of tuple O

Type Parameters:

O - Type of the output

Parameters:

name - operator name

flatten -

Returns:

new stream of type O

filter

<STREAM extends ApexStream<T>> STREAM filter(Function.FilterFunction<T> filter)

Filter transformation
Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction

Parameters:

filter - filter function

Returns:

new stream of same type

filter

<STREAM extends ApexStream<T>> STREAM filter(String name,
                                           Function.FilterFunction<T> filter)

Filter transformation
Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction

Parameters:

name - operator name

filter - filter function

Returns:

new stream of same type

reduce

<STREAM extends ApexStream<T>> STREAM reduce(Function.ReduceFunction<T> reduce)

Reduce transformation
Add an operator to the DAG which merge tuple t1, t2 to new tuple

Parameters:

reduce - reduce function

Returns:

new stream of same type

reduce

<STREAM extends ApexStream<T>> STREAM reduce(String name,
                                           Function.ReduceFunction<T> reduce)

Reduce transformation
Add an operator to the DAG which merge tuple t1, t2 to new tuple

Parameters:

name - operator name

reduce - reduce function

Returns:

new stream of same type

fold

<O,STREAM extends ApexStream<O>> STREAM fold(O initialValue,
                                           Function.FoldFunction<T,O> fold)

Fold transformation
Add an operator to the DAG which merge tuple T to accumulated result tuple O

Type Parameters:

O - Result type

Parameters:

initialValue - initial result value

fold - fold function

Returns:

new stream of type O

fold

<O,STREAM extends ApexStream<O>> STREAM fold(String name,
                                           O initialValue,
                                           Function.FoldFunction<T,O> fold)

Fold transformation
Add an operator to the DAG which merge tuple T to accumulated result tuple O

Type Parameters:

O - Result type

Parameters:

name - name of the operator

initialValue - initial result value

fold - fold function

Returns:

new stream of type O

count

<STREAM extends ApexStream<Integer>> STREAM count()

Count of all tuples

Returns:

new stream of Integer

countByKey

<STREAM extends ApexStream<Map<Object,Integer>>> STREAM countByKey()

Count tuples by the key
If the input is KeyedTuple it will get the key from getKey method from the tuple
If not, use the tuple itself as a key

Returns:

new stream of Map

countByKey

<STREAM extends ApexStream<Map<Object,Integer>>> STREAM countByKey(int key)

Count tuples by the indexed key

Parameters:

key - the index of the field in the tuple that are used as key

Returns:

new stream of Map

addOperator

<O,STREAM extends ApexStream<O>> STREAM addOperator(com.datatorrent.api.Operator op,
                                                  com.datatorrent.api.Operator.InputPort<T> inputPort,
                                                  com.datatorrent.api.Operator.OutputPort<O> outputPort)

Extend the dag by adding one operator

Type Parameters:

O - type of the output

Parameters:

op - Operator added to the stream

inputPort - InputPort of the operator that is connected to last exposed OutputPort in the stream

outputPort - OutputPort of the operator will be connected to next operator

Returns:

new stream of type O

addOperator

<O,STREAM extends ApexStream<O>> STREAM addOperator(String opName,
                                                  com.datatorrent.api.Operator op,
                                                  com.datatorrent.api.Operator.InputPort<T> inputPort,
                                                  com.datatorrent.api.Operator.OutputPort<O> outputPort)

Extend the dag by adding one Operator

Type Parameters:

O - type of the output

Parameters:

opName - Operator name

op - Operator added to the stream

inputPort - InputPort of the operator that is connected to last exposed OutputPort in the stream

outputPort - OutputPort of the operator will be connected to next operator

Returns:

new stream of type O

union

<STREAM extends ApexStream<T>> STREAM union(ApexStream<T>... others)

Union multiple stream into one

Parameters:

others - other streams

Returns:

new stream of same type

print

<STREAM extends ApexStream<T>> STREAM print()

Add a stdout console output operator

Returns:

stream itself

printErr

<STREAM extends ApexStream<T>> STREAM printErr()

Add a stderr console output operator

Returns:

stream itself

with

<STREAM extends ApexStream<T>> STREAM with(com.datatorrent.api.Attribute attribute,
                                         Object value)

Set the attribute value
If it is DAG attribute, it will be applied to the whole DAG
If it is Operator attribute, it will be applied to last connected operator
If it is InputPort attribute, it will be applied to the input port of the last connected stream
If it is OutputPort attribute, it will be applied to the output port of the last connected stream
If it is both InputPort&OutputPort attribute, it will be applied to last connected stream

Parameters:

attribute - Attribute

value - value of the attribute

Returns:

stream itself

setGlobalAttribute

<STREAM extends ApexStream<T>> STREAM setGlobalAttribute(com.datatorrent.api.Attribute attribute,
                                                       Object value)

Set attributes at the DAG level

Parameters:

attribute - Attribute

value - value of the attribute

Returns:

stream itself

with

<STREAM extends ApexStream<T>> STREAM with(com.datatorrent.api.DAG.Locality locality)

Set the locality

Parameters:

locality - DAG.Locality

Returns:

stream itself

with

<STREAM extends ApexStream<T>> STREAM with(String propName,
                                         Object value)

Set the property value of the last connected operator

Parameters:

propName - property name

value - value of the property

Returns:

stream itself

createDag

com.datatorrent.api.DAG createDag()

Create dag from stream

Returns:

DAG

populateDag

void populateDag(com.datatorrent.api.DAG dag)

Populate existing dag

Parameters:

dag - DAG

runEmbedded

void runEmbedded(boolean async,
               long duration,
               Callable<Boolean> exitCondition)

Run the stream application in local mode In Async mode, the method would return immediately and the dag would run for "duration" milliseconds In Sync mode, the method would block "duration" milliseconds to run the dag If duration is negative the dag would run forever until being killed

Parameters:

async - true if run in Async mode false if run in sync mode

run

void run()

Submit the application to cluster