co.cask.cdap.etl.batch

Class TransformExecutorFactory<T>

java.lang.Object

co.cask.cdap.etl.batch.TransformExecutorFactory<T>

Type Parameters:

T - the type of input for the created transform executors

Direct Known Subclasses:

MapReduceTransformExecutorFactory

public abstract class TransformExecutorFactory<T>
extends Object

Helps create TransformExecutors.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	isMapPhase
protected Metrics	metrics
protected co.cask.cdap.api.data.schema.Schema	outputSchema
protected Map<String,Map<String,co.cask.cdap.api.data.schema.Schema>>	perStageInputSchemas
protected PipelinePluginInstantiator	pluginInstantiator

Constructor Summary

Constructors

Constructor and Description
TransformExecutorFactory(JobContext hadoopContext, PipelinePluginInstantiator pluginInstantiator, Metrics metrics, String sourceStageName, MacroEvaluator macroEvaluator)

Method Summary

Methods

Modifier and Type	Method and Description
TransformExecutor<T>	create(PipelinePhase pipeline) Create a transform executor for the specified pipeline.
protected abstract BatchRuntimeContext	createRuntimeContext(String stageName)
protected <T extends Transformation & StageLifecycle<BatchRuntimeContext>> Transformation	getInitializedTransformation(String stageName) Instantiates and initializes the plugin for the stage.
protected static <IN,OUT> TrackedTransform<IN,OUT>	getTrackedAggregateStep(Transformation<IN,OUT> transform, StageMetrics stageMetrics)
protected static <IN,OUT> TrackedTransform<IN,OUT>	getTrackedEmitKeyStep(Transformation<IN,OUT> transform, StageMetrics stageMetrics)
protected static <IN,OUT> TrackedTransform<IN,OUT>	getTrackedMergeStep(Transformation<IN,OUT> transform, StageMetrics stageMetrics)
protected TrackedTransform	getTransformation(String pluginType, String stageName)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

perStageInputSchemas

protected final Map<String,Map<String,co.cask.cdap.api.data.schema.Schema>> perStageInputSchemas

pluginInstantiator

protected final PipelinePluginInstantiator pluginInstantiator

metrics

protected final Metrics metrics

outputSchema

protected co.cask.cdap.api.data.schema.Schema outputSchema

isMapPhase

protected boolean isMapPhase

Constructor Detail

TransformExecutorFactory

public TransformExecutorFactory(JobContext hadoopContext,
                        PipelinePluginInstantiator pluginInstantiator,
                        Metrics metrics,
                        @Nullable
                        String sourceStageName,
                        MacroEvaluator macroEvaluator)

Method Detail

createRuntimeContext

protected abstract BatchRuntimeContext createRuntimeContext(String stageName)

getTransformation

protected TrackedTransform getTransformation(String pluginType,
                                 String stageName)
                                      throws Exception

Throws:

Exception

create

public TransformExecutor<T> create(PipelinePhase pipeline)
                            throws Exception

Create a transform executor for the specified pipeline. Will instantiate and initialize all sources, transforms, and sinks in the pipeline.

Parameters:

pipeline - the pipeline to create a transform executor for

Returns:

executor for the pipeline

Throws:

InstantiationException - if there was an error instantiating a plugin

Exception - if there was an error initializing a plugin

getInitializedTransformation

protected <T extends Transformation & StageLifecycle<BatchRuntimeContext>> Transformation getInitializedTransformation(String stageName)
                                               throws Exception

Instantiates and initializes the plugin for the stage.

Parameters:

stageName - the stage name.

Returns:

the initialized Transformation

Throws:

InstantiationException - if the plugin for the stage could not be instantiated

Exception - if there was a problem initializing the plugin

getTrackedEmitKeyStep

protected static <IN,OUT> TrackedTransform<IN,OUT> getTrackedEmitKeyStep(Transformation<IN,OUT> transform,
                                                      StageMetrics stageMetrics)

getTrackedAggregateStep

protected static <IN,OUT> TrackedTransform<IN,OUT> getTrackedAggregateStep(Transformation<IN,OUT> transform,
                                                        StageMetrics stageMetrics)

getTrackedMergeStep

protected static <IN,OUT> TrackedTransform<IN,OUT> getTrackedMergeStep(Transformation<IN,OUT> transform,
                                                    StageMetrics stageMetrics)