org.apache.apex.malhar.stream.api

Interface ApexStream<T>

Type Parameters:

T -

All Known Subinterfaces:

WindowedStream<T>

All Known Implementing Classes:

ApexStreamImpl, ApexWindowedStreamImpl

@InterfaceStability.Evolving
public interface ApexStream<T>

The stream interface to build a DAG

Since:

3.4.0

Method Summary

Methods

Modifier and Type	Method and Description
<O,INSTREAM extends ApexStream<T>,OUTSTREAM extends ApexStream<O>> OUTSTREAM	addCompositeStreams(CompositeStreamTransform<INSTREAM,OUTSTREAM> compositeStreamTransform) Extend the dag by adding one CompositeStreamTransform
<O,STREAM extends ApexStream<O>> STREAM	addOperator(com.datatorrent.api.Operator op, com.datatorrent.api.Operator.InputPort<T> inputPort, com.datatorrent.api.Operator.OutputPort<O> outputPort, Option... opts) Extend the dag by adding one operator
com.datatorrent.api.DAG	createDag() Create dag from stream
<O,STREAM extends ApexStream<O>> STREAM	endWith(com.datatorrent.api.Operator op, com.datatorrent.api.Operator.InputPort<T> inputPort, Option... opts) Extend the dag by adding one end operator
<STREAM extends ApexStream<T>> STREAM	filter(Function.FilterFunction<T> filter, Option... opts) Filter transformation Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction
<O,STREAM extends ApexStream<O>> STREAM	flatMap(Function.FlatMapFunction<T,O> flatten, Option... opts) Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O
<O,STREAM extends ApexStream<O>> STREAM	map(Function.MapFunction<T,O> mapFunction, Option... opts) Simple map transformation Add an operator to the DAG which convert tuple T to tuple O
void	populateDag(com.datatorrent.api.DAG dag) Populate existing dag
<STREAM extends ApexStream<T>> STREAM	print() Add a stdout console output operator
<STREAM extends ApexStream<T>> STREAM	print(Option... opts) Add a stdout console output operator
<STREAM extends ApexStream<T>> STREAM	printErr() Add a stderr console output operator
void	run() Submit the application to cluster
void	runEmbedded(boolean async, long duration, Callable<Boolean> exitCondition) Run the stream application in local mode In Async mode, the method would return immediately and the dag would run for "duration" milliseconds In Sync mode, the method would block "duration" milliseconds to run the dag If duration is negative the dag would run forever until being killed
<STREAM extends ApexStream<T>> STREAM	setGlobalAttribute(com.datatorrent.api.Attribute attribute, Object value) Set attributes at the DAG level
<STREAM extends ApexStream<T>> STREAM	union(ApexStream<T>... others) Union multiple stream into one
WindowedStream<T>	window(WindowOption windowOption) Chunk tuples into Windows Window Transform are defined in WindowedStream
WindowedStream<T>	window(WindowOption windowOption, TriggerOption triggerOption) Chunk tuple into windows with window option and trigger option
WindowedStream<T>	window(WindowOption windowOption, TriggerOption triggerOption, org.joda.time.Duration allowLateness) Chunk tuple into windows with window option and trigger option and allowed lateness
<STREAM extends ApexStream<T>> STREAM	with(com.datatorrent.api.Attribute attribute, Object value) Set the attribute value If it is DAG attribute, it will be applied to the whole DAG If it is Operator attribute, it will be applied to last connected operator If it is InputPort attribute, it will be applied to the input port of the last connected stream If it is OutputPort attribute, it will be applied to the output port of the last connected stream If it is both InputPort&OutputPort attribute, it will be applied to last connected stream
<STREAM extends ApexStream<T>> STREAM	with(com.datatorrent.api.DAG.Locality locality) Set the locality
<STREAM extends ApexStream<T>> STREAM	with(String propName, Object value) Set the property value of the last connected operator

Method Detail

map

<O,STREAM extends ApexStream<O>> STREAM map(Function.MapFunction<T,O> mapFunction,
                                          Option... opts)

Simple map transformation
Add an operator to the DAG which convert tuple T to tuple O

Type Parameters:

O - Type of the output

Parameters:

mapFunction - map function

Returns:

new stream of type O

flatMap

<O,STREAM extends ApexStream<O>> STREAM flatMap(Function.FlatMapFunction<T,O> flatten,
                                              Option... opts)

Flat map transformation Add an operator to the DAG which convert tuple T to a collection of tuple O

Type Parameters:

O - Type of the output

Parameters:

flatten - flat map

Returns:

new stream of type O

filter

<STREAM extends ApexStream<T>> STREAM filter(Function.FilterFunction<T> filter,
                                           Option... opts)

Filter transformation
Add an operator to the DAG which filter out tuple T that cannot satisfy the FilterFunction

Parameters:

filter - filter function

Returns:

new stream of same type

addOperator

<O,STREAM extends ApexStream<O>> STREAM addOperator(com.datatorrent.api.Operator op,
                                                  com.datatorrent.api.Operator.InputPort<T> inputPort,
                                                  com.datatorrent.api.Operator.OutputPort<O> outputPort,
                                                  Option... opts)

Extend the dag by adding one operator

Type Parameters:

O - type of the output

Parameters:

op - Operator added to the stream

inputPort - InputPort of the operator that is connected to last exposed OutputPort in the stream

outputPort - OutputPort of the operator will be connected to next operator

Returns:

new stream of type O

endWith

<O,STREAM extends ApexStream<O>> STREAM endWith(com.datatorrent.api.Operator op,
                                              com.datatorrent.api.Operator.InputPort<T> inputPort,
                                              Option... opts)

Extend the dag by adding one end operator

Type Parameters:

O - type of the output

Parameters:

op - Operator added to the stream

inputPort - InputPort of the operator that is connected to last exposed OutputPort in the stream

Returns:

new stream of type O

addCompositeStreams

<O,INSTREAM extends ApexStream<T>,OUTSTREAM extends ApexStream<O>> OUTSTREAM addCompositeStreams(CompositeStreamTransform<INSTREAM,OUTSTREAM> compositeStreamTransform)

Extend the dag by adding one CompositeStreamTransform

Parameters:

compositeStreamTransform - Composite Streams and Transforms

Returns:

new stream of type O

union

<STREAM extends ApexStream<T>> STREAM union(ApexStream<T>... others)

Union multiple stream into one

Parameters:

others - other streams

Returns:

new stream of same type

print

<STREAM extends ApexStream<T>> STREAM print(Option... opts)

Add a stdout console output operator

Returns:

stream itself

print

<STREAM extends ApexStream<T>> STREAM print()

Add a stdout console output operator

Returns:

stream itself

printErr

<STREAM extends ApexStream<T>> STREAM printErr()

Add a stderr console output operator

Returns:

stream itself

with

<STREAM extends ApexStream<T>> STREAM with(com.datatorrent.api.Attribute attribute,
                                         Object value)

Set the attribute value
If it is DAG attribute, it will be applied to the whole DAG
If it is Operator attribute, it will be applied to last connected operator
If it is InputPort attribute, it will be applied to the input port of the last connected stream
If it is OutputPort attribute, it will be applied to the output port of the last connected stream
If it is both InputPort&OutputPort attribute, it will be applied to last connected stream

Parameters:

attribute - Attribute

value - value of the attribute

Returns:

stream itself

setGlobalAttribute

<STREAM extends ApexStream<T>> STREAM setGlobalAttribute(com.datatorrent.api.Attribute attribute,
                                                       Object value)

Set attributes at the DAG level

Parameters:

attribute - Attribute

value - value of the attribute

Returns:

stream itself

with

<STREAM extends ApexStream<T>> STREAM with(com.datatorrent.api.DAG.Locality locality)

Set the locality

Parameters:

locality - DAG.Locality

Returns:

stream itself

with

<STREAM extends ApexStream<T>> STREAM with(String propName,
                                         Object value)

Set the property value of the last connected operator

Parameters:

propName - property name

value - value of the property

Returns:

stream itself

createDag

com.datatorrent.api.DAG createDag()

Create dag from stream

Returns:

DAG

populateDag

void populateDag(com.datatorrent.api.DAG dag)

Populate existing dag

Parameters:

dag - DAG

runEmbedded

void runEmbedded(boolean async,
               long duration,
               Callable<Boolean> exitCondition)

Run the stream application in local mode In Async mode, the method would return immediately and the dag would run for "duration" milliseconds In Sync mode, the method would block "duration" milliseconds to run the dag If duration is negative the dag would run forever until being killed

Parameters:

async - true if run in Async mode false if run in sync mode

run

void run()

Submit the application to cluster

window

WindowedStream<T> window(WindowOption windowOption)

Chunk tuples into Windows Window Transform are defined in WindowedStream

Parameters:

windowOption -

Returns:

window

WindowedStream<T> window(WindowOption windowOption,
                       TriggerOption triggerOption)

Chunk tuple into windows with window option and trigger option

Parameters:

windowOption -

triggerOption -

Returns:

window

WindowedStream<T> window(WindowOption windowOption,
                       TriggerOption triggerOption,
                       org.joda.time.Duration allowLateness)

Chunk tuple into windows with window option and trigger option and allowed lateness

Parameters:

windowOption -

triggerOption -

allowLateness -

Returns: