co.cask.cdap.etl.api.batch

Class SparkSink<IN>

java.lang.Object

co.cask.cdap.etl.api.batch.BatchConfigurable<SparkPluginContext>

co.cask.cdap.etl.api.batch.SparkSink<IN>

Type Parameters:

IN - The type of input record to the SparkSink.

All Implemented Interfaces:

PipelineConfigurable, SubmitterLifecycle<SparkPluginContext>, Serializable

@Beta
public abstract class SparkSink<IN>
extends BatchConfigurable<SparkPluginContext>
implements Serializable

SparkSink composes a final, optional stage of a Batch ETL Pipeline. In addition to configuring the Batch run, it can also perform RDD operations on the key value pairs provided by the Batch run. run(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext, org.apache.spark.api.java.JavaRDD<IN>) method is called inside the Batch Run while BatchConfigurable.prepareRun(T) and BatchConfigurable.onRunFinish(boolean, T) methods are called on the client side, which launches the Batch run, before the Batch run starts and after it finishes respectively.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
SparkSink()

Method Summary

Modifier and Type	Method and Description
abstract void	run(SparkExecutionPluginContext context, org.apache.spark.api.java.JavaRDD<IN> input) User Spark job which will be executed and is responsible for persisting any data.

Methods inherited from class co.cask.cdap.etl.api.batch.BatchConfigurable

configurePipeline, onRunFinish, prepareRun

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PLUGIN_TYPE

public static final String PLUGIN_TYPE

See Also:

Constant Field Values

Constructor Detail

SparkSink

public SparkSink()

Method Detail

run

public abstract void run(SparkExecutionPluginContext context,
                         org.apache.spark.api.java.JavaRDD<IN> input)
                  throws Exception

User Spark job which will be executed and is responsible for persisting any data.

Parameters:

context - SparkExecutionPluginContext for this job

input - the input from previous stages of the Batch run.

Throws:

Exception