co.cask.cdap.etl.api.batch

Class SparkCompute<IN,OUT>

java.lang.Object

co.cask.cdap.etl.api.batch.SparkCompute<IN,OUT>

Type Parameters:

IN - Type of input object

OUT - Type of output object

All Implemented Interfaces:

PipelineConfigurable, Serializable

@Beta
public abstract class SparkCompute<IN,OUT>
extends Object
implements PipelineConfigurable, Serializable

Spark Compute stage.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
SparkCompute()

Method Summary

Methods

Modifier and Type	Method and Description
void	configurePipeline(PipelineConfigurer pipelineConfigurer) Configure a pipeline.
void	initialize(SparkExecutionPluginContext context) Initialize the plugin.
abstract org.apache.spark.api.java.JavaRDD<OUT>	transform(SparkExecutionPluginContext context, org.apache.spark.api.java.JavaRDD<IN> input) Transform the input and return the output to be sent to the next stage in the pipeline.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PLUGIN_TYPE

public static final String PLUGIN_TYPE

See Also:

Constant Field Values

Constructor Detail

SparkCompute

public SparkCompute()

Method Detail

configurePipeline

public void configurePipeline(PipelineConfigurer pipelineConfigurer)
                       throws IllegalArgumentException

Configure a pipeline.

Specified by:

configurePipeline in interface PipelineConfigurable

Parameters:

pipelineConfigurer - the configurer used to add required datasets and streams

Throws:

IllegalArgumentException - if the given config is invalid

initialize

public void initialize(SparkExecutionPluginContext context)
                throws Exception

Initialize the plugin. Will be called before any calls to transform(SparkExecutionPluginContext, JavaRDD) are made.

Parameters:

context - SparkExecutionPluginContext for this job

Throws:

Exception - if there is an error initializing

transform

public abstract org.apache.spark.api.java.JavaRDD<OUT> transform(SparkExecutionPluginContext context,
                                               org.apache.spark.api.java.JavaRDD<IN> input)
                                                          throws Exception

Transform the input and return the output to be sent to the next stage in the pipeline.

Parameters:

context - SparkExecutionPluginContext for this job

input - input data to be transformed

Throws:

Exception - if there is an error during this method invocation