co.cask.cdap.api.spark

Class JavaSparkExecutionContext

java.lang.Object

co.cask.cdap.api.spark.JavaSparkExecutionContext

All Implemented Interfaces:

RuntimeContext, Transactional, WorkflowInfoProvider

@Beta
public abstract class JavaSparkExecutionContext
extends Object
implements RuntimeContext, Transactional, WorkflowInfoProvider

Spark program execution context. User Spark program can interact with CDAP through this context.

Constructor Summary

Constructors

Constructor and Description
JavaSparkExecutionContext()

Method Summary

Methods

Modifier and Type	Method and Description
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName) Creates a JavaPairRDD from the given Dataset.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName, Map<String,String> arguments) Creates a JavaPairRDD from the given Dataset with the given set of Dataset arguments.
abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName, Map<String,String> arguments, Iterable<? extends Split> splits) Creates a JavaPairRDD from the given Dataset with the given set of Dataset arguments and custom list of Splits.
org.apache.spark.api.java.JavaRDD<StreamEvent>	fromStream(String streamName) Creates a JavaRDD that represents all events from the given stream.
<V> org.apache.spark.api.java.JavaPairRDD<Long,V>	fromStream(String streamName, Class<V> valueType) Creates a JavaPairRDD that represents all events from the given stream.
<T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>>	fromStream(String streamName, co.cask.cdap.api.data.format.FormatSpecification formatSpec, Class<T> dataType) Creates a JavaPairRDD that represents all events from the given stream.
abstract <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>>	fromStream(String streamName, co.cask.cdap.api.data.format.FormatSpecification formatSpec, long startTime, long endTime, Class<T> dataType) Creates a JavaPairRDD that represents data from the given stream for events in the given time range.
abstract org.apache.spark.api.java.JavaRDD<StreamEvent>	fromStream(String streamName, long startTime, long endTime) Creates a JavaRDD that represents events from the given stream in the given time range.
abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromStream(String streamName, long startTime, long endTime, Class<? extends co.cask.cdap.api.stream.StreamEventDecoder<K,V>> decoderClass, Class<K> keyType, Class<V> valueType) Creates a JavaPairRDD that represents events from the given stream in the given time range.
abstract <V> org.apache.spark.api.java.JavaPairRDD<Long,V>	fromStream(String streamName, long startTime, long endTime, Class<V> valueType) Creates a JavaPairRDD that represents events from the given stream in the given time range.
abstract TaskLocalizationContext	getLocalizationContext() Returns a Serializable TaskLocalizationContext which can be used to retrieve files localized to task containers.
abstract long	getLogicalStartTime() Returns the logical start time of this Spark job.
abstract Metrics	getMetrics() Returns a Serializable Metrics which can be used to emit custom metrics from user's Spark program.
abstract PluginContext	getPluginContext() Returns a Serializable PluginContext which can be used to request for plugins instances.
abstract ServiceDiscoverer	getServiceDiscoverer() Returns a Serializable ServiceDiscoverer for Service Discovery in Spark Program which can be passed in Spark program's closures.
abstract SparkSpecification	getSpecification()
<K,V> void	saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd, String datasetName) Saves the given JavaPairRDD to the given Dataset.
abstract <K,V> void	saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd, String datasetName, Map<String,String> arguments) Saves the given JavaPairRDD to the given Dataset with the given set of Dataset arguments.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface co.cask.cdap.api.RuntimeContext

getAdmin, getApplicationSpecification, getNamespace, getRunId, getRuntimeArguments

Methods inherited from interface co.cask.cdap.api.Transactional

execute

Methods inherited from interface co.cask.cdap.api.workflow.WorkflowInfoProvider

getWorkflowInfo, getWorkflowToken

Constructor Detail

JavaSparkExecutionContext

public JavaSparkExecutionContext()

Method Detail

getSpecification

public abstract SparkSpecification getSpecification()

Returns:

The specification used to configure this Spark job instance.

getLogicalStartTime

public abstract long getLogicalStartTime()

Returns the logical start time of this Spark job. Logical start time is the time when this Spark job is supposed to start if this job is started by the scheduler. Otherwise it would be the current time when the job runs.

Returns:

Time in milliseconds since epoch time (00:00:00 January 1, 1970 UTC).

getServiceDiscoverer

public abstract ServiceDiscoverer getServiceDiscoverer()

Returns a Serializable ServiceDiscoverer for Service Discovery in Spark Program which can be passed in Spark program's closures.

Returns:

A Serializable ServiceDiscoverer

getMetrics

public abstract Metrics getMetrics()

Returns a Serializable Metrics which can be used to emit custom metrics from user's Spark program. This can also be passed in Spark program's closures and workers can emit their own metrics

Returns:

Serializable Metrics for Spark programs

getPluginContext

public abstract PluginContext getPluginContext()

Returns a Serializable PluginContext which can be used to request for plugins instances. The instance returned can also be used in Spark program's closures.

Returns:

A Serializable PluginContext.

getLocalizationContext

public abstract TaskLocalizationContext getLocalizationContext()

Returns a Serializable TaskLocalizationContext which can be used to retrieve files localized to task containers. The instance returned can also be used in Spark program's closures.

Returns:

the TaskLocalizationContext for the Spark program

fromDataset

public <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName)

Creates a JavaPairRDD from the given Dataset.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the Dataset

Returns:

A new JavaPairRDD instance that reads from the given Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

fromDataset

public <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName,
                                                           Map<String,String> arguments)

Creates a JavaPairRDD from the given Dataset with the given set of Dataset arguments.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the Dataset

arguments - arguments for the Dataset

Returns:

A new JavaPairRDD instance that reads from the given Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

fromDataset

public abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName,
                                                           Map<String,String> arguments,
                                                           @Nullable
                                                           Iterable<? extends Split> splits)

Creates a JavaPairRDD from the given Dataset with the given set of Dataset arguments and custom list of Splits. Each Split will create a Partition in the JavaPairRDD.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the Dataset

arguments - arguments for the Dataset

splits - list of Split or null to use the default splits provided by the Dataset

Returns:

A new JavaPairRDD instance that reads from the given Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

fromStream

public org.apache.spark.api.java.JavaRDD<StreamEvent> fromStream(String streamName)

Creates a JavaRDD that represents all events from the given stream.

Parameters:

streamName - name of the stream

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public abstract org.apache.spark.api.java.JavaRDD<StreamEvent> fromStream(String streamName,
                                                        long startTime,
                                                        long endTime)

Creates a JavaRDD that represents events from the given stream in the given time range.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public <V> org.apache.spark.api.java.JavaPairRDD<Long,V> fromStream(String streamName,
                                                           Class<V> valueType)

Creates a JavaPairRDD that represents all events from the given stream. The key in the resulting JavaPairRDD is the event timestamp. The stream body will be decoded as the give value type. Currently it supports Text, String and ByteWritable.

Parameters:

streamName - name of the stream

valueType - type of the stream body to decode to

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public abstract <V> org.apache.spark.api.java.JavaPairRDD<Long,V> fromStream(String streamName,
                                                           long startTime,
                                                           long endTime,
                                                           Class<V> valueType)

Creates a JavaPairRDD that represents events from the given stream in the given time range. The key in the resulting JavaPairRDD is the event timestamp. The stream body will be decoded as the give value type. Currently it supports Text, String and ByteWritable.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

valueType - type of the stream body to decode to

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromStream(String streamName,
                                                          long startTime,
                                                          long endTime,
                                                          Class<? extends co.cask.cdap.api.stream.StreamEventDecoder<K,V>> decoderClass,
                                                          Class<K> keyType,
                                                          Class<V> valueType)

Creates a JavaPairRDD that represents events from the given stream in the given time range. Each steam event will be decoded by an instance of the given StreamEventDecoder class.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

decoderClass - the StreamEventDecoder for decoding StreamEvent

keyType - the type of the decoded key

valueType - the type of the decoded value

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>> fromStream(String streamName,
                                                                                                           co.cask.cdap.api.data.format.FormatSpecification formatSpec,
                                                                                                           Class<T> dataType)

Creates a JavaPairRDD that represents all events from the given stream. The first entry in the pair is a Long, representing the event timestamp, while the second entry is a GenericStreamEventData, which contains data decoded from the stream event body base on the given FormatSpecification.

Type Parameters:

T - value type

Parameters:

streamName - name of the stream

formatSpec - the FormatSpecification describing the format in the stream

Returns:

a new JavaPairRDD instance that reads from the given stream.

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public abstract <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>> fromStream(String streamName,
                                                                                                           co.cask.cdap.api.data.format.FormatSpecification formatSpec,
                                                                                                           long startTime,
                                                                                                           long endTime,
                                                                                                           Class<T> dataType)

Creates a JavaPairRDD that represents data from the given stream for events in the given time range. The first entry in the pair is a Long, representing the event timestamp, while the second entry is a GenericStreamEventData, which contains data decoded from the stream event body base on the given FormatSpecification.

Type Parameters:

T - value type

Parameters:

streamName - name of the stream

formatSpec - the FormatSpecification describing the format in the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

Returns:

a new JavaPairRDD instance that reads from the given stream.

Throws:

DatasetInstantiationException - if the Stream doesn't exist

saveAsDataset

public <K,V> void saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd,
                       String datasetName)

Saves the given JavaPairRDD to the given Dataset.

Parameters:

rdd - the JavaPairRDD to be saved

datasetName - name of the Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

saveAsDataset

public abstract <K,V> void saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd,
                       String datasetName,
                       Map<String,String> arguments)

Saves the given JavaPairRDD to the given Dataset with the given set of Dataset arguments.

Parameters:

rdd - the JavaPairRDD to be saved

datasetName - name of the Dataset

arguments - arguments for the Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist