co.cask.cdap.etl.api.batch

Interface SparkPluginContext

All Superinterfaces:

BatchContext, DatasetContext, LookupProvider, ServiceDiscoverer, StageContext, TransformContext

@Beta
public interface SparkPluginContext
extends BatchContext

Context passed to spark plugin types during prepare run phase.

Method Summary

Modifier and Type	Method and Description
void	setSparkConf(org.apache.spark.SparkConf sparkConf) Sets a SparkConf to be used for the Spark execution.

Methods inherited from interface co.cask.cdap.etl.api.batch.BatchContext

createDataset, datasetExists, getArguments, getRuntimeArguments, setRuntimeArgument

Methods inherited from interface co.cask.cdap.api.data.DatasetContext

discardDataset, getDataset, getDataset, getDataset, getDataset, releaseDataset

Methods inherited from interface co.cask.cdap.etl.api.StageContext

getInputSchema, getInputSchemas, getLogicalStartTime, getMetrics, getNamespace, getOutputPortSchemas, getOutputSchema, getPipelineName, getPluginProperties, getPluginProperties, getStageName, loadPluginClass, newPluginInstance

Methods inherited from interface co.cask.cdap.api.ServiceDiscoverer

getServiceURL, getServiceURL

Methods inherited from interface co.cask.cdap.etl.api.LookupProvider

provide

Method Detail

setSparkConf

void setSparkConf(org.apache.spark.SparkConf sparkConf)

Sets a SparkConf to be used for the Spark execution. If your configuration will not change between pipeline runs, use PipelineConfigurer.setPipelineProperties(java.util.Map<java.lang.String, java.lang.String>) instead. This method should only be used when you need different configuration settings for each run. Due to limitations in Spark Streaming, this method cannot be used in realtime data pipelines. Calling this method will throw an UnsupportedOperationException in realtime pipelines.