org.apache.apex.malhar.stream.api

Interface WindowedStream<T>

Type Parameters:

T - Output tuple type

All Superinterfaces:

ApexStream<T>

All Known Implementing Classes:

ApexWindowedStreamImpl

@InterfaceStability.Evolving
public interface WindowedStream<T>
extends ApexStream<T>

A stream with windowed transformation

Transformation types:

• Combine
• Group
• Keyed Combine
• Keyed Group
• Join
• CoGroup

Features supported with windowed transformation

• Watermark - Ingestion time watermark / logical tuple watermark
• Early Triggers - How frequent to emit real-time partial result
• Late Triggers - When to emit updated result with tuple comes after watermark
• Customizable Trigger Behaviour - What to do when fires a trigger
• Spool window state - In-Memory window state can be spooled to disk if it is full
• 3 different accumulation models: ignore, accumulation, accumulation + delta
• Window support: Non-Mergeable window(fix window, sliding window), Mergeable window(session window) base on 3 different tuple time
• Different tuple time support: event time, system time, ingestion time

Since:

3.4.0

Method Summary

Methods

Modifier and Type	Method and Description
<O,ACCU,STREAM extends WindowedStream<Tuple.WindowedTuple<O>>> STREAM	accumulate(Accumulation<T,ACCU,O> accumulation, Option... opts) Add WindowedOperatorImpl with specified Accumulation Accumulate tuples by some key within the window definition in this stream Also give a name to the accumulation
<K,V,O,ACCU,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,O>>>> STREAM	accumulateByKey(Accumulation<V,ACCU,O> accumulation, Function.ToKeyValue<T,K,V> convertToKeyVal, Option... opts) Add KeyedWindowedOperatorImpl with specified Accumulation Accumulate tuples by some key within the window definition in this stream Also give a name to the accumulation
<STREAM extends WindowedStream<Tuple.WindowedTuple<Long>>> STREAM	count(Option... opts) Count of all tuples
<K,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,Long>>>> STREAM	countByKey(Function.ToKeyValue<T,K,Long> convertToKeyValue, Option... opts) Count tuples by the key
<O,STREAM extends WindowedStream<Tuple.WindowedTuple<O>>> STREAM	fold(FoldFn<T,O> fold, Option... opts) Add WindowedOperatorImpl with specified FoldFn Fold transformation
<K,V,O,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,O>>>> STREAM	foldByKey(FoldFn<V,O> fold, Function.ToKeyValue<T,K,V> convertToKeyVal, Option... opts) Add KeyedWindowedOperatorImpl with specified FoldFn Fold transformation by key
<STREAM extends WindowedStream<Iterable<T>>> STREAM	group() Return tuples for each window
<O,K,STREAM extends WindowedStream<KeyValPair<K,Iterable<O>>>> STREAM	groupByKey(Function.ToKeyValue<T,K,O> convertToKeyVal, Option... opts) Return tuples for each key for each window
<STREAM extends WindowedStream<Tuple.WindowedTuple<T>>> STREAM	reduce(ReduceFn<T> reduce, Option... opts) Add WindowedOperatorImpl with specified ReduceFn Do reduce transformation
<K,V,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,V>>>> STREAM	reduceByKey(ReduceFn<V> reduce, Function.ToKeyValue<T,K,V> convertToKeyVal, Option... opts) Add KeyedWindowedOperatorImpl with specified ReduceFn Reduce transformation by selected key Add an operator to the DAG which merge tuple t1, t2 to new tuple by key
<STREAM extends WindowedStream<T>> STREAM	resetAllowedLateness(org.joda.time.Duration allowedLateness) Reset the allowedLateness settings for next transforms
<STREAM extends WindowedStream<T>> STREAM	resetTrigger(TriggerOption triggerOption) Reset the trigger settings for next transforms
<STREAM extends WindowedStream<Tuple.WindowedTuple<List<T>>>> STREAM	top(int N, Option... opts) Return top N tuples of all tuples in the window
<K,V,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,List<V>>>>> STREAM	topByKey(int N, Function.ToKeyValue<T,K,V> convertToKeyVal, Option... opts) Return top N tuples by the selected key

Methods inherited from interface org.apache.apex.malhar.stream.api.ApexStream

addCompositeStreams, addOperator, createDag, endWith, filter, flatMap, map, populateDag, print, print, printErr, run, runEmbedded, setGlobalAttribute, union, window, window, window, with, with, with

Method Detail

count

<STREAM extends WindowedStream<Tuple.WindowedTuple<Long>>> STREAM count(Option... opts)

Count of all tuples

Returns:

new stream of Integer

countByKey

<K,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,Long>>>> STREAM countByKey(Function.ToKeyValue<T,K,Long> convertToKeyValue,
                                                                                           Option... opts)

Count tuples by the key

Parameters:

convertToKeyValue - The function convert plain tuple to k,v pair

Returns:

new stream of Key Value Pair

topByKey

<K,V,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,List<V>>>>> STREAM topByKey(int N,
                                                                                              Function.ToKeyValue<T,K,V> convertToKeyVal,
                                                                                              Option... opts)

Return top N tuples by the selected key

Parameters:

N - how many tuples you want to keep

convertToKeyVal - The function convert plain tuple to k,v pair

Returns:

new stream of Key and top N tuple of the key

top

<STREAM extends WindowedStream<Tuple.WindowedTuple<List<T>>>> STREAM top(int N,
                                                                       Option... opts)

Return top N tuples of all tuples in the window

Parameters:

N -

Returns:

new stream of topN

accumulateByKey

<K,V,O,ACCU,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,O>>>> STREAM accumulateByKey(Accumulation<V,ACCU,O> accumulation,
                                                                                                      Function.ToKeyValue<T,K,V> convertToKeyVal,
                                                                                                      Option... opts)

Add KeyedWindowedOperatorImpl with specified Accumulation
Accumulate tuples by some key within the window definition in this stream Also give a name to the accumulation

Type Parameters:

K - The type of the key used to group tuples

V - The type of value you want to do accumulation on

O - The output type for each given key that you want to accumulate the value to

ACCU - The type of accumulation you want to keep (it can be in memory or on disk)

STREAM - return type

Parameters:

accumulation - Accumulation function you want to do

convertToKeyVal - The function convert plain tuple to k,v pair

Returns:

accumulate

<O,ACCU,STREAM extends WindowedStream<Tuple.WindowedTuple<O>>> STREAM accumulate(Accumulation<T,ACCU,O> accumulation,
                                                                               Option... opts)

Add WindowedOperatorImpl with specified Accumulation
Accumulate tuples by some key within the window definition in this stream Also give a name to the accumulation

Type Parameters:

O - The output type that you want to accumulate the value to

ACCU - The type of accumulation you want to keep (it can be in memory or on disk)

STREAM - return type

Parameters:

accumulation - Accumulation function you want to do

Returns:

reduce

<STREAM extends WindowedStream<Tuple.WindowedTuple<T>>> STREAM reduce(ReduceFn<T> reduce,
                                                                    Option... opts)

Add WindowedOperatorImpl with specified ReduceFn
Do reduce transformation

Type Parameters:

STREAM - return type

Parameters:

reduce - reduce function

Returns:

new stream of same type

reduceByKey

<K,V,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,V>>>> STREAM reduceByKey(ReduceFn<V> reduce,
                                                                                           Function.ToKeyValue<T,K,V> convertToKeyVal,
                                                                                           Option... opts)

Add KeyedWindowedOperatorImpl with specified ReduceFn
Reduce transformation by selected key
Add an operator to the DAG which merge tuple t1, t2 to new tuple by key

Type Parameters:

K - The type of key you want to group tuples by

V - The type of value extract from tuple T

STREAM - return type

Parameters:

reduce - reduce function

convertToKeyVal - The function convert plain tuple to k,v pair

Returns:

new stream of key value pair

fold

<O,STREAM extends WindowedStream<Tuple.WindowedTuple<O>>> STREAM fold(FoldFn<T,O> fold,
                                                                    Option... opts)

Add WindowedOperatorImpl with specified FoldFn
Fold transformation

Type Parameters:

O - output type of fold function

STREAM - return type

Parameters:

fold - fold function

Returns:

foldByKey

<K,V,O,STREAM extends WindowedStream<Tuple.WindowedTuple<KeyValPair<K,O>>>> STREAM foldByKey(FoldFn<V,O> fold,
                                                                                           Function.ToKeyValue<T,K,V> convertToKeyVal,
                                                                                           Option... opts)

Add KeyedWindowedOperatorImpl with specified FoldFn
Fold transformation by key

Type Parameters:

O - Result type

Parameters:

fold - fold function

Returns:

new stream of type O

groupByKey

<O,K,STREAM extends WindowedStream<KeyValPair<K,Iterable<O>>>> STREAM groupByKey(Function.ToKeyValue<T,K,O> convertToKeyVal,
                                                                               Option... opts)

Return tuples for each key for each window

Type Parameters:

O -

K -

STREAM -

Returns:

group

<STREAM extends WindowedStream<Iterable<T>>> STREAM group()

Return tuples for each window

Type Parameters:

STREAM -

Returns:

resetTrigger

<STREAM extends WindowedStream<T>> STREAM resetTrigger(TriggerOption triggerOption)

Reset the trigger settings for next transforms

Type Parameters:

STREAM -

Parameters:

triggerOption -

resetAllowedLateness

<STREAM extends WindowedStream<T>> STREAM resetAllowedLateness(org.joda.time.Duration allowedLateness)

Reset the allowedLateness settings for next transforms

Type Parameters:

STREAM -

Parameters:

allowedLateness -