org.apache.apex.malhar.stream.api.impl

Class ApexStreamImpl<T>

java.lang.Object

org.apache.apex.malhar.stream.api.impl.ApexStreamImpl<T>

All Implemented Interfaces:

ApexStream<T>

Direct Known Subclasses:

ApexWindowedStreamImpl

@InterfaceStability.Evolving
public class ApexStreamImpl<T>
extends Object
implements ApexStream<T>

Default stream implementation for ApexStream interface.
It creates the dag(execution plan) from stream api
The dag won't be constructed until populateDag(DAG) is called

Since:

3.4.0

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ApexStreamImpl.Brick<T> The extension point of the stream

Field Summary

Fields

Modifier and Type	Field and Description
protected DagMeta	graph Graph behind the stream
protected ApexStreamImpl.Brick<T>	lastBrick Right now the stream only support single extension point You can have multiple downstream operators connect to this single extension point though

Constructor Summary

Constructors

Constructor and Description
ApexStreamImpl()
ApexStreamImpl(ApexStreamImpl<T> apexStream)
ApexStreamImpl(DagMeta graph)
ApexStreamImpl(DagMeta graph, ApexStreamImpl.Brick<T> lastBrick)

Method Summary

Methods

Modifier and Type	Method and Description
<O,INSTREAM extends ApexStream<T>,OUTSTREAM extends ApexStream<O>> OUTSTREAM	addCompositeStreams(CompositeStreamTransform<INSTREAM,OUTSTREAM> compositeStreamTransform) Extend the dag by adding one CompositeStreamTransform
<O,STREAM extends ApexStream<O>> STREAM	addOperator(com.datatorrent.api.Operator op, com.datatorrent.api.Operator.InputPort<T> inputPort, com.datatorrent.api.Operator.OutputPort<O> outputPort, Option... opts) Extend the dag by adding one operator
<STREAM extends ApexStream<Map.Entry<Object,Integer>>> STREAM	countByElement()
com.datatorrent.api.DAG	createDag() Create dag from stream
<O,STREAM extends ApexStream<O>> STREAM	endWith(com.datatorrent.api.Operator op, com.datatorrent.api.Operator.InputPort<T> inputPort, Option... opts) Extend the dag by adding one end operator
<STREAM extends ApexStream<T>> STREAM	filter(Function.FilterFunction<T> filter, Option... opts) Filter transformation Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction
<O,STREAM extends ApexStream<O>> STREAM	flatMap(Function.FlatMapFunction<T,O> flatten, Option... opts) Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O
ApexStreamImpl.Brick<T>	getLastBrick()
<O,STREAM extends ApexStream<O>> STREAM	map(Function.MapFunction<T,O> mf, Option... opts) Simple map transformation Add an operator to the DAG which convert tuple T to tuple O
protected <O> ApexStream<O>	newStream(DagMeta graph, ApexStreamImpl.Brick<O> newBrick)
void	populateDag(com.datatorrent.api.DAG dag) Populate existing dag
ApexStreamImpl<T>	print() Add a stdout console output operator
ApexStreamImpl<T>	print(Option... opts) Add a stdout console output operator
ApexStream<T>	printErr() Add a stderr console output operator
void	run() Submit the application to cluster
void	runEmbedded(boolean async, long duration, Callable<Boolean> exitCondition) Run the stream application in local mode In Async mode, the method would return immediately and the dag would run for "duration" milliseconds In Sync mode, the method would block "duration" milliseconds to run the dag If duration is negative the dag would run forever until being killed
ApexStream<T>	setGlobalAttribute(com.datatorrent.api.Attribute attribute, Object value) Set attributes at the DAG level
void	setLastBrick(ApexStreamImpl.Brick<T> lastBrick)
<STREAM extends ApexStream<T>> STREAM	union(ApexStream<T>... others) Union multiple stream into one
WindowedStream<T>	window(WindowOption option) Chunk tuples into Windows Window Transform are defined in WindowedStream
WindowedStream<T>	window(WindowOption windowOption, TriggerOption triggerOption) Chunk tuple into windows with window option and trigger option
WindowedStream<T>	window(WindowOption windowOption, TriggerOption triggerOption, org.joda.time.Duration allowLateness) Chunk tuple into windows with window option and trigger option and allowed lateness
ApexStream<T>	with(com.datatorrent.api.Attribute attribute, Object value) Set the attribute value If it is DAG attribute, it will be applied to the whole DAG If it is Operator attribute, it will be applied to last connected operator If it is InputPort attribute, it will be applied to the input port of the last connected stream If it is OutputPort attribute, it will be applied to the output port of the last connected stream If it is both InputPort&OutputPort attribute, it will be applied to last connected stream
ApexStream<T>	with(com.datatorrent.api.DAG.Locality locality) Set the locality
ApexStream<T>	with(String propName, Object value) Set the property value of the last connected operator

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

graph

protected DagMeta graph

Graph behind the stream

lastBrick

protected ApexStreamImpl.Brick<T> lastBrick

Right now the stream only support single extension point You can have multiple downstream operators connect to this single extension point though

Constructor Detail

ApexStreamImpl

public ApexStreamImpl()

ApexStreamImpl

public ApexStreamImpl(ApexStreamImpl<T> apexStream)

ApexStreamImpl

public ApexStreamImpl(DagMeta graph)

ApexStreamImpl

public ApexStreamImpl(DagMeta graph,
              ApexStreamImpl.Brick<T> lastBrick)

Method Detail

getLastBrick

public ApexStreamImpl.Brick<T> getLastBrick()

setLastBrick

public void setLastBrick(ApexStreamImpl.Brick<T> lastBrick)

map

public <O,STREAM extends ApexStream<O>> STREAM map(Function.MapFunction<T,O> mf,
                                          Option... opts)

Description copied from interface: ApexStream

Simple map transformation
Add an operator to the DAG which convert tuple T to tuple O

Specified by:

map in interface ApexStream<T>

Type Parameters:

O - Type of the output

Parameters:

mf - map function

Returns:

new stream of type O

flatMap

public <O,STREAM extends ApexStream<O>> STREAM flatMap(Function.FlatMapFunction<T,O> flatten,
                                              Option... opts)

Description copied from interface: ApexStream

Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O

Specified by:

flatMap in interface ApexStream<T>

Type Parameters:

O - Type of the output

Parameters:

flatten - flat map

Returns:

new stream of type O

filter

public <STREAM extends ApexStream<T>> STREAM filter(Function.FilterFunction<T> filter,
                                           Option... opts)

Description copied from interface: ApexStream

Filter transformation
Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction

Specified by:

filter in interface ApexStream<T>

Parameters:

filter - filter function

Returns:

new stream of same type

countByElement

public <STREAM extends ApexStream<Map.Entry<Object,Integer>>> STREAM countByElement()

endWith

public <O,STREAM extends ApexStream<O>> STREAM endWith(com.datatorrent.api.Operator op,
                                              com.datatorrent.api.Operator.InputPort<T> inputPort,
                                              Option... opts)

Description copied from interface: ApexStream

Extend the dag by adding one end operator

Specified by:

endWith in interface ApexStream<T>

Type Parameters:

O - type of the output

Parameters:

op - Operator added to the stream

inputPort - InputPort of the operator that is connected to last exposed OutputPort in the stream

Returns:

new stream of type O

addOperator

public <O,STREAM extends ApexStream<O>> STREAM addOperator(com.datatorrent.api.Operator op,
                                                  com.datatorrent.api.Operator.InputPort<T> inputPort,
                                                  com.datatorrent.api.Operator.OutputPort<O> outputPort,
                                                  Option... opts)

Description copied from interface: ApexStream

Extend the dag by adding one operator

Specified by:

addOperator in interface ApexStream<T>

Type Parameters:

O - type of the output

Parameters:

op - Operator added to the stream

inputPort - InputPort of the operator that is connected to last exposed OutputPort in the stream

outputPort - OutputPort of the operator will be connected to next operator

Returns:

new stream of type O

addCompositeStreams

public <O,INSTREAM extends ApexStream<T>,OUTSTREAM extends ApexStream<O>> OUTSTREAM addCompositeStreams(CompositeStreamTransform<INSTREAM,OUTSTREAM> compositeStreamTransform)

Description copied from interface: ApexStream

Extend the dag by adding one CompositeStreamTransform

Specified by:

addCompositeStreams in interface ApexStream<T>

Parameters:

compositeStreamTransform - Composite Streams and Transforms

Returns:

new stream of type O

union

public <STREAM extends ApexStream<T>> STREAM union(ApexStream<T>... others)

Description copied from interface: ApexStream

Union multiple stream into one

Specified by:

union in interface ApexStream<T>

Parameters:

others - other streams

Returns:

new stream of same type

print

public ApexStreamImpl<T> print(Option... opts)

Description copied from interface: ApexStream

Add a stdout console output operator

Specified by:

print in interface ApexStream<T>

Returns:

stream itself

print

public ApexStreamImpl<T> print()

Description copied from interface: ApexStream

Add a stdout console output operator

Specified by:

print in interface ApexStream<T>

Returns:

stream itself

printErr

public ApexStream<T> printErr()

Description copied from interface: ApexStream

Add a stderr console output operator

Specified by:

printErr in interface ApexStream<T>

Returns:

stream itself

with

public ApexStream<T> with(com.datatorrent.api.Attribute attribute,
                 Object value)

Description copied from interface: ApexStream

Set the attribute value
If it is DAG attribute, it will be applied to the whole DAG
If it is Operator attribute, it will be applied to last connected operator
If it is InputPort attribute, it will be applied to the input port of the last connected stream
If it is OutputPort attribute, it will be applied to the output port of the last connected stream
If it is both InputPort&OutputPort attribute, it will be applied to last connected stream

Specified by:

with in interface ApexStream<T>

Parameters:

attribute - Attribute

value - value of the attribute

Returns:

stream itself

setGlobalAttribute

public ApexStream<T> setGlobalAttribute(com.datatorrent.api.Attribute attribute,
                               Object value)

Description copied from interface: ApexStream

Set attributes at the DAG level

Specified by:

setGlobalAttribute in interface ApexStream<T>

Parameters:

attribute - Attribute

value - value of the attribute

Returns:

stream itself

with

public ApexStream<T> with(com.datatorrent.api.DAG.Locality locality)

Description copied from interface: ApexStream

Set the locality

Specified by:

with in interface ApexStream<T>

Parameters:

locality - DAG.Locality

Returns:

stream itself

with

public ApexStream<T> with(String propName,
                 Object value)

Description copied from interface: ApexStream

Set the property value of the last connected operator

Specified by:

with in interface ApexStream<T>

Parameters:

propName - property name

value - value of the property

Returns:

stream itself

createDag

public com.datatorrent.api.DAG createDag()

Description copied from interface: ApexStream

Create dag from stream

Specified by:

createDag in interface ApexStream<T>

Returns:

DAG

populateDag

public void populateDag(com.datatorrent.api.DAG dag)

Description copied from interface: ApexStream

Populate existing dag

Specified by:

populateDag in interface ApexStream<T>

Parameters:

dag - DAG

runEmbedded

public void runEmbedded(boolean async,
               long duration,
               Callable<Boolean> exitCondition)

Description copied from interface: ApexStream

Run the stream application in local mode In Async mode, the method would return immediately and the dag would run for "duration" milliseconds In Sync mode, the method would block "duration" milliseconds to run the dag If duration is negative the dag would run forever until being killed

Specified by:

runEmbedded in interface ApexStream<T>

Parameters:

async - true if run in Async mode false if run in sync mode

run

public void run()

Description copied from interface: ApexStream

Submit the application to cluster

Specified by:

run in interface ApexStream<T>

window

public WindowedStream<T> window(WindowOption option)

Description copied from interface: ApexStream

Chunk tuples into Windows Window Transform are defined in WindowedStream

Specified by:

window in interface ApexStream<T>

Returns:

window

public WindowedStream<T> window(WindowOption windowOption,
                       TriggerOption triggerOption)

Description copied from interface: ApexStream

Chunk tuple into windows with window option and trigger option

Specified by:

window in interface ApexStream<T>

Returns:

window

public WindowedStream<T> window(WindowOption windowOption,
                       TriggerOption triggerOption,
                       org.joda.time.Duration allowLateness)

Description copied from interface: ApexStream

Chunk tuple into windows with window option and trigger option and allowed lateness

Specified by:

window in interface ApexStream<T>

Returns:

newStream

protected <O> ApexStream<O> newStream(DagMeta graph,
                          ApexStreamImpl.Brick<O> newBrick)