org.apache.apex.malhar.stream.api.impl

Class ApexStreamImpl<T>

java.lang.Object

org.apache.apex.malhar.stream.api.impl.ApexStreamImpl<T>

All Implemented Interfaces:

ApexStream<T>

public class ApexStreamImpl<T>
extends Object
implements ApexStream<T>

Default stream implementation for ApexStream interface. It creates the dag(execution plan) from stream api

Since:

3.4.0

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ApexStreamImpl.Brick<T> The extension point of the stream

Constructor Summary

Constructors

Constructor and Description
ApexStreamImpl()
ApexStreamImpl(ApexStream<T> apexStream)
ApexStreamImpl(DagMeta graph)
ApexStreamImpl(DagMeta graph, ApexStreamImpl.Brick<T> lastBrick)

Method Summary

Methods

Modifier and Type	Method and Description
<O,STREAM extends ApexStream<O>> STREAM	addOperator(com.datatorrent.api.Operator op, com.datatorrent.api.Operator.InputPort<T> inputPort, com.datatorrent.api.Operator.OutputPort<O> outputPort) Extend the dag by adding one operator
<O,STREAM extends ApexStream<O>> STREAM	addOperator(String opName, com.datatorrent.api.Operator op, com.datatorrent.api.Operator.InputPort<T> inputPort, com.datatorrent.api.Operator.OutputPort<O> outputPort) Extend the dag by adding one Operator
<STREAM extends ApexStream<Integer>> STREAM	count() Count of all tuples
<STREAM extends ApexStream<Map<Object,Integer>>> STREAM	countByKey() Count tuples by the key If the input is KeyedTuple it will get the key from getKey method from the tuple If not, use the tuple itself as a key
<STREAM extends ApexStream<Map<Object,Integer>>> STREAM	countByKey(int key) Count tuples by the indexed key
com.datatorrent.api.DAG	createDag() Create dag from stream
<STREAM extends ApexStream<T>> STREAM	filter(Function.FilterFunction<T> filter) Filter transformation Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction
<STREAM extends ApexStream<T>> STREAM	filter(String name, Function.FilterFunction<T> filter) Filter transformation Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction
<O,STREAM extends ApexStream<O>> STREAM	flatMap(Function.FlatMapFunction<T,O> flatten) Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O
<O,STREAM extends ApexStream<O>> STREAM	flatMap(String name, Function.FlatMapFunction<T,O> flatten) Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O
<O,STREAM extends ApexStream<O>> STREAM	fold(O initialValue, Function.FoldFunction<T,O> fold) Fold transformation Add an operator to the DAG which merge tuple T to accumulated result tuple O
<O,STREAM extends ApexStream<O>> STREAM	fold(String name, O initialValue, Function.FoldFunction<T,O> fold) Fold transformation Add an operator to the DAG which merge tuple T to accumulated result tuple O
ApexStreamImpl.Brick<T>	getLastBrick()
<O,STREAM extends ApexStream<O>> STREAM	map(Function.MapFunction<T,O> mf) Simple map transformation Add an operator to the DAG which convert tuple T to tuple O
<O,STREAM extends ApexStream<O>> STREAM	map(String name, Function.MapFunction<T,O> mf) Simple map transformation Add an operator to the DAG which convert tuple T to tuple O
void	populateDag(com.datatorrent.api.DAG dag) Populate existing dag
ApexStreamImpl<T>	print() Add a stdout console output operator
ApexStream<T>	printErr() Add a stderr console output operator
<STREAM extends ApexStream<T>> STREAM	reduce(Function.ReduceFunction<T> reduce) Reduce transformation Add an operator to the DAG which merge tuple t1, t2 to new tuple
<STREAM extends ApexStream<T>> STREAM	reduce(String name, Function.ReduceFunction<T> reduce) Reduce transformation Add an operator to the DAG which merge tuple t1, t2 to new tuple
void	run() Submit the application to cluster
void	runEmbedded(boolean async, long duration, Callable<Boolean> exitCondition) Run the stream application in local mode In Async mode, the method would return immediately and the dag would run for "duration" milliseconds In Sync mode, the method would block "duration" milliseconds to run the dag If duration is negative the dag would run forever until being killed
ApexStream<T>	setGlobalAttribute(com.datatorrent.api.Attribute attribute, Object value) Set attributes at the DAG level
void	setLastBrick(ApexStreamImpl.Brick<T> lastBrick)
<STREAM extends ApexStream<T>> STREAM	union(ApexStream<T>... others) Union multiple stream into one
ApexStream<T>	with(com.datatorrent.api.Attribute attribute, Object value) Set the attribute value If it is DAG attribute, it will be applied to the whole DAG If it is Operator attribute, it will be applied to last connected operator If it is InputPort attribute, it will be applied to the input port of the last connected stream If it is OutputPort attribute, it will be applied to the output port of the last connected stream If it is both InputPort&OutputPort attribute, it will be applied to last connected stream
ApexStream<T>	with(com.datatorrent.api.DAG.Locality locality) Set the locality
ApexStream<T>	with(String propName, Object value) Set the property value of the last connected operator

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ApexStreamImpl

public ApexStreamImpl()

ApexStreamImpl

public ApexStreamImpl(ApexStream<T> apexStream)

ApexStreamImpl

public ApexStreamImpl(DagMeta graph)

ApexStreamImpl

public ApexStreamImpl(DagMeta graph,
              ApexStreamImpl.Brick<T> lastBrick)

Method Detail

getLastBrick

public ApexStreamImpl.Brick<T> getLastBrick()

setLastBrick

public void setLastBrick(ApexStreamImpl.Brick<T> lastBrick)

map

public <O,STREAM extends ApexStream<O>> STREAM map(Function.MapFunction<T,O> mf)

Description copied from interface: ApexStream

Simple map transformation
Add an operator to the DAG which convert tuple T to tuple O

Specified by:

map in interface ApexStream<T>

Type Parameters:

O - Type of the output

Parameters:

mf - map function

Returns:

new stream of type O

map

public <O,STREAM extends ApexStream<O>> STREAM map(String name,
                                          Function.MapFunction<T,O> mf)

Description copied from interface: ApexStream

Simple map transformation
Add an operator to the DAG which convert tuple T to tuple O

Specified by:

map in interface ApexStream<T>

Type Parameters:

O - Type of the output

Parameters:

name - operator name

mf - map function

Returns:

new stream of type O

flatMap

public <O,STREAM extends ApexStream<O>> STREAM flatMap(Function.FlatMapFunction<T,O> flatten)

Description copied from interface: ApexStream

Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O

Specified by:

flatMap in interface ApexStream<T>

Type Parameters:

O - Type of the output

Parameters:

flatten - flat map

Returns:

new stream of type O

flatMap

public <O,STREAM extends ApexStream<O>> STREAM flatMap(String name,
                                              Function.FlatMapFunction<T,O> flatten)

Description copied from interface: ApexStream

Flat map transformation
Add an operator to the DAG which convert tuple T to a collection of tuple O

Specified by:

flatMap in interface ApexStream<T>

Type Parameters:

O - Type of the output

Parameters:

name - operator name

Returns:

new stream of type O

filter

public <STREAM extends ApexStream<T>> STREAM filter(Function.FilterFunction<T> filter)

Description copied from interface: ApexStream

Filter transformation
Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction

Specified by:

filter in interface ApexStream<T>

Parameters:

filter - filter function

Returns:

new stream of same type

filter

public <STREAM extends ApexStream<T>> STREAM filter(String name,
                                           Function.FilterFunction<T> filter)

Description copied from interface: ApexStream

Filter transformation
Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction

Specified by:

filter in interface ApexStream<T>

Parameters:

name - operator name

filter - filter function

Returns:

new stream of same type

reduce

public <STREAM extends ApexStream<T>> STREAM reduce(Function.ReduceFunction<T> reduce)

Description copied from interface: ApexStream

Reduce transformation
Add an operator to the DAG which merge tuple t1, t2 to new tuple

Specified by:

reduce in interface ApexStream<T>

Parameters:

reduce - reduce function

Returns:

new stream of same type

reduce

public <STREAM extends ApexStream<T>> STREAM reduce(String name,
                                           Function.ReduceFunction<T> reduce)

Description copied from interface: ApexStream

Reduce transformation
Add an operator to the DAG which merge tuple t1, t2 to new tuple

Specified by:

reduce in interface ApexStream<T>

Parameters:

name - operator name

reduce - reduce function

Returns:

new stream of same type

fold

public <O,STREAM extends ApexStream<O>> STREAM fold(O initialValue,
                                           Function.FoldFunction<T,O> fold)

Description copied from interface: ApexStream

Fold transformation
Add an operator to the DAG which merge tuple T to accumulated result tuple O

Specified by:

fold in interface ApexStream<T>

Type Parameters:

O - Result type

Parameters:

initialValue - initial result value

fold - fold function

Returns:

new stream of type O

fold

public <O,STREAM extends ApexStream<O>> STREAM fold(String name,
                                           O initialValue,
                                           Function.FoldFunction<T,O> fold)

Description copied from interface: ApexStream

Fold transformation
Add an operator to the DAG which merge tuple T to accumulated result tuple O

Specified by:

fold in interface ApexStream<T>

Type Parameters:

O - Result type

Parameters:

name - name of the operator

initialValue - initial result value

fold - fold function

Returns:

new stream of type O

count

public <STREAM extends ApexStream<Integer>> STREAM count()

Description copied from interface: ApexStream

Count of all tuples

Specified by:

count in interface ApexStream<T>

Returns:

new stream of Integer

countByKey

public <STREAM extends ApexStream<Map<Object,Integer>>> STREAM countByKey(int key)

Description copied from interface: ApexStream

Count tuples by the indexed key

Specified by:

countByKey in interface ApexStream<T>

Parameters:

key - the index of the field in the tuple that are used as key

Returns:

new stream of Map

countByKey

public <STREAM extends ApexStream<Map<Object,Integer>>> STREAM countByKey()

Description copied from interface: ApexStream

Count tuples by the key
If the input is KeyedTuple it will get the key from getKey method from the tuple
If not, use the tuple itself as a key

Specified by:

countByKey in interface ApexStream<T>

Returns:

new stream of Map

addOperator

public <O,STREAM extends ApexStream<O>> STREAM addOperator(com.datatorrent.api.Operator op,
                                                  com.datatorrent.api.Operator.InputPort<T> inputPort,
                                                  com.datatorrent.api.Operator.OutputPort<O> outputPort)

Description copied from interface: ApexStream

Extend the dag by adding one operator

Specified by:

addOperator in interface ApexStream<T>

Type Parameters:

O - type of the output

Parameters:

op - Operator added to the stream

inputPort - InputPort of the operator that is connected to last exposed OutputPort in the stream

outputPort - OutputPort of the operator will be connected to next operator

Returns:

new stream of type O

addOperator

public <O,STREAM extends ApexStream<O>> STREAM addOperator(String opName,
                                                  com.datatorrent.api.Operator op,
                                                  com.datatorrent.api.Operator.InputPort<T> inputPort,
                                                  com.datatorrent.api.Operator.OutputPort<O> outputPort)

Description copied from interface: ApexStream

Extend the dag by adding one Operator

Specified by:

addOperator in interface ApexStream<T>

Type Parameters:

O - type of the output

Parameters:

opName - Operator name

op - Operator added to the stream

inputPort - InputPort of the operator that is connected to last exposed OutputPort in the stream

outputPort - OutputPort of the operator will be connected to next operator

Returns:

new stream of type O

union

public <STREAM extends ApexStream<T>> STREAM union(ApexStream<T>... others)

Description copied from interface: ApexStream

Union multiple stream into one

Specified by:

union in interface ApexStream<T>

Parameters:

others - other streams

Returns:

new stream of same type

print

public ApexStreamImpl<T> print()

Description copied from interface: ApexStream

Add a stdout console output operator

Specified by:

print in interface ApexStream<T>

Returns:

stream itself

printErr

public ApexStream<T> printErr()

Description copied from interface: ApexStream

Add a stderr console output operator

Specified by:

printErr in interface ApexStream<T>

Returns:

stream itself

with

public ApexStream<T> with(com.datatorrent.api.Attribute attribute,
                 Object value)

Description copied from interface: ApexStream

Set the attribute value
If it is DAG attribute, it will be applied to the whole DAG
If it is Operator attribute, it will be applied to last connected operator
If it is InputPort attribute, it will be applied to the input port of the last connected stream
If it is OutputPort attribute, it will be applied to the output port of the last connected stream
If it is both InputPort&OutputPort attribute, it will be applied to last connected stream

Specified by:

with in interface ApexStream<T>

Parameters:

attribute - Attribute

value - value of the attribute

Returns:

stream itself

setGlobalAttribute

public ApexStream<T> setGlobalAttribute(com.datatorrent.api.Attribute attribute,
                               Object value)

Description copied from interface: ApexStream

Set attributes at the DAG level

Specified by:

setGlobalAttribute in interface ApexStream<T>

Parameters:

attribute - Attribute

value - value of the attribute

Returns:

stream itself

with

public ApexStream<T> with(com.datatorrent.api.DAG.Locality locality)

Description copied from interface: ApexStream

Set the locality

Specified by:

with in interface ApexStream<T>

Parameters:

locality - DAG.Locality

Returns:

stream itself

with

public ApexStream<T> with(String propName,
                 Object value)

Description copied from interface: ApexStream

Set the property value of the last connected operator

Specified by:

with in interface ApexStream<T>

Parameters:

propName - property name

value - value of the property

Returns:

stream itself

createDag

public com.datatorrent.api.DAG createDag()

Description copied from interface: ApexStream

Create dag from stream

Specified by:

createDag in interface ApexStream<T>

Returns:

DAG

populateDag

public void populateDag(com.datatorrent.api.DAG dag)

Description copied from interface: ApexStream

Populate existing dag

Specified by:

populateDag in interface ApexStream<T>

Parameters:

dag - DAG

runEmbedded

public void runEmbedded(boolean async,
               long duration,
               Callable<Boolean> exitCondition)

Description copied from interface: ApexStream

Run the stream application in local mode In Async mode, the method would return immediately and the dag would run for "duration" milliseconds In Sync mode, the method would block "duration" milliseconds to run the dag If duration is negative the dag would run forever until being killed

Specified by:

runEmbedded in interface ApexStream<T>

Parameters:

async - true if run in Async mode false if run in sync mode

run

public void run()

Description copied from interface: ApexStream

Submit the application to cluster

Specified by:

run in interface ApexStream<T>