co.cask.cdap.etl.api.batch

Interface SparkExecutionPluginContext

All Superinterfaces:

DatasetContext, LookupProvider, ServiceDiscoverer, StageContext, TransformContext

@Beta
public interface SparkExecutionPluginContext
extends DatasetContext, TransformContext

Context passed to spark plugin types.

Method Summary

Modifier and Type	Method and Description
SparkInterpreter	createSparkInterpreter() Creates a new instance of SparkInterpreter for Scala code compilation and interpretation.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName) Creates a JavaPairRDD from the given Dataset.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName, Map<String,String> arguments) Creates a JavaPairRDD from the given Dataset with the given set of Dataset arguments.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName, Map<String,String> arguments, Iterable<? extends Split> splits) Creates a JavaPairRDD from the given Dataset with the given set of Dataset arguments and custom list of Splits.
org.apache.spark.api.java.JavaRDD<StreamEvent>	fromStream(String streamName) Creates a JavaRDD that represents all events from the given stream.
<V> org.apache.spark.api.java.JavaPairRDD<Long,V>	fromStream(String streamName, Class<V> valueType) Creates a JavaPairRDD that represents all events from the given stream.
org.apache.spark.api.java.JavaRDD<StreamEvent>	fromStream(String streamName, long startTime, long endTime) Creates a JavaRDD that represents events from the given stream in the given time range.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromStream(String streamName, long startTime, long endTime, Class<? extends co.cask.cdap.api.stream.StreamEventDecoder<K,V>> decoderClass, Class<K> keyType, Class<V> valueType) Creates a JavaPairRDD that represents events from the given stream in the given time range.
<V> org.apache.spark.api.java.JavaPairRDD<Long,V>	fromStream(String streamName, long startTime, long endTime, Class<V> valueType) Creates a JavaPairRDD that represents events from the given stream in the given time range.
long	getLogicalStartTime() Returns the logical start time of the Batch Job.
PluginContext	getPluginContext() Returns a Serializable PluginContext which can be used to request for plugins instances.
Map<String,String>	getRuntimeArguments() Returns runtime arguments of the Batch Job.
org.apache.spark.api.java.JavaSparkContext	getSparkContext() Returns the JavaSparkContext used during the execution.
<K,V> void	saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd, String datasetName) Saves the given JavaPairRDD to the given Dataset.
<K,V> void	saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd, String datasetName, Map<String,String> arguments) Saves the given JavaPairRDD to the given Dataset with the given set of Dataset arguments.

Methods inherited from interface co.cask.cdap.api.data.DatasetContext

discardDataset, getDataset, getDataset, getDataset, getDataset, releaseDataset

Methods inherited from interface co.cask.cdap.etl.api.StageContext

getArguments, getInputSchema, getInputSchemas, getMetrics, getNamespace, getOutputPortSchemas, getOutputSchema, getPipelineName, getPluginProperties, getPluginProperties, getStageName, loadPluginClass, newPluginInstance

Methods inherited from interface co.cask.cdap.api.ServiceDiscoverer

getServiceURL, getServiceURL

Methods inherited from interface co.cask.cdap.etl.api.LookupProvider

provide

Method Detail

getLogicalStartTime

long getLogicalStartTime()

Returns the logical start time of the Batch Job. Logical start time is the time when this Batch job is supposed to start if this job is started by the scheduler. Otherwise it would be the current time when the job runs.

Specified by:

getLogicalStartTime in interface StageContext

Returns:

Time in milliseconds since epoch time (00:00:00 January 1, 1970 UTC).

getRuntimeArguments

Map<String,String> getRuntimeArguments()

Returns runtime arguments of the Batch Job.

Returns:

runtime arguments of the Batch Job.

fromDataset

<K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName)

Creates a JavaPairRDD from the given Dataset.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the Dataset

Returns:

A new JavaPairRDD instance that reads from the given Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

fromDataset

<K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName,
                                                             Map<String,String> arguments)

Creates a JavaPairRDD from the given Dataset with the given set of Dataset arguments.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the Dataset

arguments - arguments for the Dataset

Returns:

A new JavaPairRDD instance that reads from the given Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

fromDataset

<K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName,
                                                             Map<String,String> arguments,
                                                             @Nullable
                                                             Iterable<? extends Split> splits)

Creates a JavaPairRDD from the given Dataset with the given set of Dataset arguments and custom list of Splits. Each Split will create a Partition in the JavaPairRDD.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the Dataset

arguments - arguments for the Dataset

splits - list of Split or null to use the default splits provided by the Dataset

Returns:

A new JavaPairRDD instance that reads from the given Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

fromStream

org.apache.spark.api.java.JavaRDD<StreamEvent> fromStream(String streamName)

Creates a JavaRDD that represents all events from the given stream.

Parameters:

streamName - name of the stream

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

org.apache.spark.api.java.JavaRDD<StreamEvent> fromStream(String streamName,
                                                          long startTime,
                                                          long endTime)

Creates a JavaRDD that represents events from the given stream in the given time range.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive)

endTime - the ending time of the streams to be read in milliseconds (exclusive)

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

<V> org.apache.spark.api.java.JavaPairRDD<Long,V> fromStream(String streamName,
                                                             Class<V> valueType)

Creates a JavaPairRDD that represents all events from the given stream. The key in the resulting JavaPairRDD is the event timestamp. The stream body will be decoded as the give value type. Currently it supports Text, String and ByteWritable.

Parameters:

streamName - name of the stream

valueType - type of the stream body to decode to

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

<V> org.apache.spark.api.java.JavaPairRDD<Long,V> fromStream(String streamName,
                                                             long startTime,
                                                             long endTime,
                                                             Class<V> valueType)

Creates a JavaPairRDD that represents events from the given stream in the given time range. The key in the resulting JavaPairRDD is the event timestamp. The stream body will be decoded as the give value type. Currently it supports Text, String and ByteWritable.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive)

endTime - the ending time of the streams to be read in milliseconds (exclusive)

valueType - type of the stream body to decode to

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

<K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromStream(String streamName,
                                                            long startTime,
                                                            long endTime,
                                                            Class<? extends co.cask.cdap.api.stream.StreamEventDecoder<K,V>> decoderClass,
                                                            Class<K> keyType,
                                                            Class<V> valueType)

Creates a JavaPairRDD that represents events from the given stream in the given time range. Each steam event will be decoded by an instance of the given StreamEventDecoder class.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive)

endTime - the ending time of the streams to be read in milliseconds (exclusive)

decoderClass - the StreamEventDecoder for decoding StreamEvent

keyType - the type of the decoded key

valueType - the type of the decoded value

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

saveAsDataset

<K,V> void saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd,
                         String datasetName)

Saves the given JavaPairRDD to the given Dataset.

Parameters:

rdd - the JavaPairRDD to be saved

datasetName - name of the Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

saveAsDataset

<K,V> void saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd,
                         String datasetName,
                         Map<String,String> arguments)

Saves the given JavaPairRDD to the given Dataset with the given set of Dataset arguments.

Parameters:

rdd - the JavaPairRDD to be saved

datasetName - name of the Dataset

arguments - arguments for the Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

getSparkContext

org.apache.spark.api.java.JavaSparkContext getSparkContext()

Returns the JavaSparkContext used during the execution.

Returns:

the Spark Context

getPluginContext

PluginContext getPluginContext()

Returns a Serializable PluginContext which can be used to request for plugins instances. The instance returned can also be used in Spark program's closures.

Returns:

A Serializable PluginContext.

createSparkInterpreter

SparkInterpreter createSparkInterpreter()
                                 throws IOException

Creates a new instance of SparkInterpreter for Scala code compilation and interpretation.

Returns:

a new instance of SparkInterpreter

Throws:

IOException - if failed to create a local directory for storing the compiled class files