co.cask.cdap.api.spark

Class JavaSparkExecutionContextBase

java.lang.Object

co.cask.cdap.api.spark.JavaSparkExecutionContextBase

All Implemented Interfaces:

MetadataReader, MetadataWriter, RuntimeContext, SchedulableProgramContext, SecureStore, Transactional, WorkflowInfoProvider

Direct Known Subclasses:

JavaSparkExecutionContext

@Beta
public abstract class JavaSparkExecutionContextBase
extends Object
implements SchedulableProgramContext, RuntimeContext, Transactional, WorkflowInfoProvider, SecureStore, MetadataReader, MetadataWriter

Spark program execution context. User Spark program can interact with CDAP through this context.

Constructor Summary

Constructors

Constructor and Description
JavaSparkExecutionContextBase()

Method Summary

Modifier and Type	Method and Description
abstract co.cask.cdap.api.spark.dynamic.SparkInterpreter	createInterpreter() Creates a new instance of SparkInterpreter for Scala code compilation and interpretation.
abstract void	execute(int timeoutInSeconds, TxRunnable runnable) Transactions with a custom timeout are not supported in Spark.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName) Creates a JavaPairRDD from the given Dataset.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName, Map<String,String> arguments) Creates a JavaPairRDD from the given Dataset with the given set of dataset arguments.
abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String datasetName, Map<String,String> arguments, Iterable<? extends Split> splits) Creates a JavaPairRDD from the given Dataset with the given set of dataset arguments and custom list of Splits.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String namespace, String datasetName) Creates a JavaPairRDD from the given Dataset.
<K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String namespace, String datasetName, Map<String,String> arguments) Creates a JavaPairRDD from the given Dataset with the given set of dataset arguments.
abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromDataset(String namespace, String datasetName, Map<String,String> arguments, Iterable<? extends Split> splits) Creates a JavaPairRDD from the given Dataset with the given set of dataset arguments and custom list of Splits.
org.apache.spark.api.java.JavaRDD<StreamEvent>	fromStream(String streamName) Creates a JavaRDD that represents all events from the given stream.
<V> org.apache.spark.api.java.JavaPairRDD<Long,V>	fromStream(String streamName, Class<V> valueType) Creates a JavaPairRDD that represents all events from the given stream.
<T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>>	fromStream(String streamName, co.cask.cdap.api.data.format.FormatSpecification formatSpec, Class<T> dataType) Creates a JavaPairRDD that represents all events from the given stream.
abstract <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>>	fromStream(String streamName, co.cask.cdap.api.data.format.FormatSpecification formatSpec, long startTime, long endTime, Class<T> dataType) Creates a JavaPairRDD that represents data from the given stream for events in the given time range.
abstract org.apache.spark.api.java.JavaRDD<StreamEvent>	fromStream(String streamName, long startTime, long endTime) Creates a JavaRDD that represents events from the given stream in the given time range.
abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromStream(String streamName, long startTime, long endTime, Class<? extends co.cask.cdap.api.stream.StreamEventDecoder<K,V>> decoderClass, Class<K> keyType, Class<V> valueType) Creates a JavaPairRDD that represents events from the given stream in the given time range.
abstract <V> org.apache.spark.api.java.JavaPairRDD<Long,V>	fromStream(String streamName, long startTime, long endTime, Class<V> valueType) Creates a JavaPairRDD that represents events from the given stream in the given time range.
org.apache.spark.api.java.JavaRDD<StreamEvent>	fromStream(String namespace, String streamName) Creates a JavaRDD that represents all events from the given stream.
<V> org.apache.spark.api.java.JavaPairRDD<Long,V>	fromStream(String namespace, String streamName, Class<V> valueType) Creates a JavaPairRDD that represents all events from the given stream.
<T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>>	fromStream(String namespace, String streamName, co.cask.cdap.api.data.format.FormatSpecification formatSpec, Class<T> dataType) Creates a JavaPairRDD that represents all events from the given stream.
abstract <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>>	fromStream(String namespace, String streamName, co.cask.cdap.api.data.format.FormatSpecification formatSpec, long startTime, long endTime, Class<T> dataType) Creates a JavaPairRDD that represents data from the given stream for events in the given time range.
abstract org.apache.spark.api.java.JavaRDD<StreamEvent>	fromStream(String namespace, String streamName, long startTime, long endTime) Creates a JavaRDD that represents events from the given stream in the given time range.
abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V>	fromStream(String namespace, String streamName, long startTime, long endTime, Class<? extends co.cask.cdap.api.stream.StreamEventDecoder<K,V>> decoderClass, Class<K> keyType, Class<V> valueType) Creates a JavaPairRDD that represents events from the given stream in the given time range.
abstract <V> org.apache.spark.api.java.JavaPairRDD<Long,V>	fromStream(String namespace, String streamName, long startTime, long endTime, Class<V> valueType) Creates a JavaPairRDD that represents events from the given stream in the given time range.
abstract TaskLocalizationContext	getLocalizationContext() Returns a Serializable TaskLocalizationContext which can be used to retrieve files localized to task containers.
abstract long	getLogicalStartTime() Returns the logical start time of this Spark job.
abstract MessagingContext	getMessagingContext() Returns a MessagingContext which can be used to interact with the transactional messaging system.
abstract Metrics	getMetrics() Returns a Serializable Metrics which can be used to emit custom metrics from user's Spark program.
abstract PluginContext	getPluginContext() Returns a Serializable PluginContext which can be used to request for plugins instances.
abstract SecureStore	getSecureStore() Returns a Serializable SecureStore which can be used to request for plugins instances.
abstract ServiceDiscoverer	getServiceDiscoverer() Returns a Serializable ServiceDiscoverer for Service Discovery in Spark Program which can be passed in Spark program's closures.
abstract SparkExecutionContext	getSparkExecutionContext() Returns the underlying SparkExecutionContext used by this object.
abstract SparkSpecification	getSpecification()
<K,V> void	saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd, String datasetName) Saves the given JavaPairRDD to the given Dataset.
abstract <K,V> void	saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd, String datasetName, Map<String,String> arguments) Saves the given JavaPairRDD to the given Dataset with the given set of Dataset arguments.
<K,V> void	saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd, String namespace, String datasetName) Saves the given JavaPairRDD to the given Dataset.
abstract <K,V> void	saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd, String namespace, String datasetName, Map<String,String> arguments) Saves the given JavaPairRDD to the given Dataset with the given set of dataset arguments.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface co.cask.cdap.api.SchedulableProgramContext

getTriggeringScheduleInfo

Methods inherited from interface co.cask.cdap.api.RuntimeContext

getAdmin, getApplicationSpecification, getClusterName, getDataTracer, getNamespace, getRunId, getRuntimeArguments

Methods inherited from interface co.cask.cdap.api.Transactional

execute

Methods inherited from interface co.cask.cdap.api.workflow.WorkflowInfoProvider

getWorkflowInfo, getWorkflowToken

Methods inherited from interface co.cask.cdap.api.security.store.SecureStore

getSecureData, listSecureData

Methods inherited from interface co.cask.cdap.api.metadata.MetadataReader

getMetadata, getMetadata

Methods inherited from interface co.cask.cdap.api.metadata.MetadataWriter

addProperties, addTags, addTags, removeMetadata, removeProperties, removeProperties, removeTags, removeTags

Constructor Detail

JavaSparkExecutionContextBase

public JavaSparkExecutionContextBase()

Method Detail

getSpecification

public abstract SparkSpecification getSpecification()

Returns:

The specification used to configure this Spark job instance.

getLogicalStartTime

public abstract long getLogicalStartTime()

Returns the logical start time of this Spark job. Logical start time is the time when this Spark job is supposed to start if this job is started by the scheduler. Otherwise it would be the current time when the job runs.

Returns:

Time in milliseconds since epoch time (00:00:00 January 1, 1970 UTC).

getServiceDiscoverer

public abstract ServiceDiscoverer getServiceDiscoverer()

Returns a Serializable ServiceDiscoverer for Service Discovery in Spark Program which can be passed in Spark program's closures.

Returns:

A Serializable ServiceDiscoverer

getMetrics

public abstract Metrics getMetrics()

Returns a Serializable Metrics which can be used to emit custom metrics from user's Spark program. This can also be passed in Spark program's closures and workers can emit their own metrics

Returns:

Serializable Metrics for Spark programs

getPluginContext

public abstract PluginContext getPluginContext()

Returns a Serializable PluginContext which can be used to request for plugins instances. The instance returned can also be used in Spark program's closures.

Returns:

A Serializable PluginContext.

getSecureStore

public abstract SecureStore getSecureStore()

Returns a Serializable SecureStore which can be used to request for plugins instances. The instance returned can also be used in Spark program's closures.

Returns:

A Serializable SecureStore.

getMessagingContext

public abstract MessagingContext getMessagingContext()

Returns a MessagingContext which can be used to interact with the transactional messaging system. Currently the returned instance can only be used in the Spark driver process.

Returns:

A MessagingContext

getLocalizationContext

public abstract TaskLocalizationContext getLocalizationContext()

Returns a Serializable TaskLocalizationContext which can be used to retrieve files localized to task containers. The instance returned can also be used in Spark program's closures.

Returns:

the TaskLocalizationContext for the Spark program

fromDataset

public <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName)

Creates a JavaPairRDD from the given Dataset.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the dataset

Returns:

A new JavaPairRDD instance that reads from the given dataset

Throws:

DatasetInstantiationException - if the dataset doesn't exist

fromDataset

public <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String namespace,
                                                                    String datasetName)

Creates a JavaPairRDD from the given Dataset.

Type Parameters:

K - key type

V - value type

Parameters:

namespace - namespace in which the dataset exists

datasetName - name of the dataset

Returns:

A new JavaPairRDD instance that reads from the given dataset

Throws:

DatasetInstantiationException - if the dataset doesn't exist

fromDataset

public <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName,
                                                                    Map<String,String> arguments)

Creates a JavaPairRDD from the given Dataset with the given set of dataset arguments.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the dataset

arguments - arguments for the dataset

Returns:

A new JavaPairRDD instance that reads from the given dataset

Throws:

DatasetInstantiationException - if the dataset doesn't exist

fromDataset

public <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String namespace,
                                                                    String datasetName,
                                                                    Map<String,String> arguments)

Creates a JavaPairRDD from the given Dataset with the given set of dataset arguments.

Type Parameters:

K - key type

V - value type

Parameters:

namespace - namespace in which the dataset exists

datasetName - name of the dataset

arguments - arguments for the dataset

Returns:

A new JavaPairRDD instance that reads from the given dataset

Throws:

DatasetInstantiationException - if the dataset doesn't exist

fromDataset

public abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String datasetName,
                                                                             Map<String,String> arguments,
                                                                             @Nullable
                                                                             Iterable<? extends Split> splits)

Creates a JavaPairRDD from the given Dataset with the given set of dataset arguments and custom list of Splits. Each Split will create a Partition in the JavaPairRDD.

Type Parameters:

K - key type

V - value type

Parameters:

datasetName - name of the dataset

arguments - arguments for the dataset

splits - list of Split or null to use the default splits provided by the dataset

Returns:

A new JavaPairRDD instance that reads from the given dataset

Throws:

DatasetInstantiationException - if the dataset doesn't exist

fromDataset

public abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromDataset(String namespace,
                                                                             String datasetName,
                                                                             Map<String,String> arguments,
                                                                             @Nullable
                                                                             Iterable<? extends Split> splits)

Creates a JavaPairRDD from the given Dataset with the given set of dataset arguments and custom list of Splits. Each Split will create a Partition in the JavaPairRDD.

Type Parameters:

K - key type

V - value type

Parameters:

namespace - namespace in which the dataset exists

datasetName - name of the dataset

arguments - arguments for the dataset

splits - list of Split or null to use the default splits provided by the dataset

Returns:

A new JavaPairRDD instance that reads from the given dataset

Throws:

DatasetInstantiationException - if the dataset doesn't exist

fromStream

public org.apache.spark.api.java.JavaRDD<StreamEvent> fromStream(String streamName)

Creates a JavaRDD that represents all events from the given stream.

Parameters:

streamName - name of the stream

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public org.apache.spark.api.java.JavaRDD<StreamEvent> fromStream(String namespace,
                                                                 String streamName)

Creates a JavaRDD that represents all events from the given stream.

Parameters:

namespace - namespace in which the stream exists

streamName - name of the stream

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public abstract org.apache.spark.api.java.JavaRDD<StreamEvent> fromStream(String streamName,
                                                                          long startTime,
                                                                          long endTime)

Creates a JavaRDD that represents events from the given stream in the given time range.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public abstract org.apache.spark.api.java.JavaRDD<StreamEvent> fromStream(String namespace,
                                                                          String streamName,
                                                                          long startTime,
                                                                          long endTime)

Creates a JavaRDD that represents events from the given stream in the given time range.

Parameters:

namespace - namespace in which the stream exists

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public <V> org.apache.spark.api.java.JavaPairRDD<Long,V> fromStream(String streamName,
                                                                    Class<V> valueType)

Creates a JavaPairRDD that represents all events from the given stream. The key in the resulting JavaPairRDD is the event timestamp. The stream body will be decoded as the give value type. Currently it supports Text, String and ByteWritable.

Parameters:

streamName - name of the stream

valueType - type of the stream body to decode to

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public <V> org.apache.spark.api.java.JavaPairRDD<Long,V> fromStream(String namespace,
                                                                    String streamName,
                                                                    Class<V> valueType)

Creates a JavaPairRDD that represents all events from the given stream. The key in the resulting JavaPairRDD is the event timestamp. The stream body will be decoded as the give value type. Currently it supports Text, String and ByteWritable.

Parameters:

namespace - namespace in which the stream exists

streamName - name of the stream

valueType - type of the stream body to decode to

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public abstract <V> org.apache.spark.api.java.JavaPairRDD<Long,V> fromStream(String streamName,
                                                                             long startTime,
                                                                             long endTime,
                                                                             Class<V> valueType)

Creates a JavaPairRDD that represents events from the given stream in the given time range. The key in the resulting JavaPairRDD is the event timestamp. The stream body will be decoded as the give value type. Currently it supports Text, String and ByteWritable.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

valueType - type of the stream body to decode to

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public abstract <V> org.apache.spark.api.java.JavaPairRDD<Long,V> fromStream(String namespace,
                                                                             String streamName,
                                                                             long startTime,
                                                                             long endTime,
                                                                             Class<V> valueType)

Creates a JavaPairRDD that represents events from the given stream in the given time range. The key in the resulting JavaPairRDD is the event timestamp. The stream body will be decoded as the give value type. Currently it supports Text, String and ByteWritable.

Parameters:

namespace - namespace in which the stream exists

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

valueType - type of the stream body to decode to

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromStream(String streamName,
                                                                            long startTime,
                                                                            long endTime,
                                                                            Class<? extends co.cask.cdap.api.stream.StreamEventDecoder<K,V>> decoderClass,
                                                                            Class<K> keyType,
                                                                            Class<V> valueType)

Creates a JavaPairRDD that represents events from the given stream in the given time range. Each steam event will be decoded by an instance of the given StreamEventDecoder class.

Parameters:

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

decoderClass - the StreamEventDecoder for decoding StreamEvent

keyType - the type of the decoded key

valueType - the type of the decoded value

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public abstract <K,V> org.apache.spark.api.java.JavaPairRDD<K,V> fromStream(String namespace,
                                                                            String streamName,
                                                                            long startTime,
                                                                            long endTime,
                                                                            Class<? extends co.cask.cdap.api.stream.StreamEventDecoder<K,V>> decoderClass,
                                                                            Class<K> keyType,
                                                                            Class<V> valueType)

Creates a JavaPairRDD that represents events from the given stream in the given time range. Each steam event will be decoded by an instance of the given StreamEventDecoder class.

Parameters:

namespace - namespace in which the stream exists

streamName - name of the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

decoderClass - the StreamEventDecoder for decoding StreamEvent

keyType - the type of the decoded key

valueType - the type of the decoded value

Returns:

A new JavaRDD instance that reads from the given stream

Throws:

DatasetInstantiationException - if the stream doesn't exist

fromStream

public <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>> fromStream(String streamName,
                                                                                                                    co.cask.cdap.api.data.format.FormatSpecification formatSpec,
                                                                                                                    Class<T> dataType)

Creates a JavaPairRDD that represents all events from the given stream. The first entry in the pair is a Long, representing the event timestamp, while the second entry is a GenericStreamEventData, which contains data decoded from the stream event body base on the given FormatSpecification.

Type Parameters:

T - value type

Parameters:

streamName - name of the stream

formatSpec - the FormatSpecification describing the format in the stream

Returns:

a new JavaPairRDD instance that reads from the given stream.

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>> fromStream(String namespace,
                                                                                                                    String streamName,
                                                                                                                    co.cask.cdap.api.data.format.FormatSpecification formatSpec,
                                                                                                                    Class<T> dataType)

Creates a JavaPairRDD that represents all events from the given stream. The first entry in the pair is a Long, representing the event timestamp, while the second entry is a GenericStreamEventData, which contains data decoded from the stream event body base on the given FormatSpecification.

Type Parameters:

T - value type

Parameters:

namespace - namespace in which the stream exists

streamName - name of the stream

formatSpec - the FormatSpecification describing the format in the stream

Returns:

a new JavaPairRDD instance that reads from the given stream.

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public abstract <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>> fromStream(String streamName,
                                                                                                                             co.cask.cdap.api.data.format.FormatSpecification formatSpec,
                                                                                                                             long startTime,
                                                                                                                             long endTime,
                                                                                                                             Class<T> dataType)

Creates a JavaPairRDD that represents data from the given stream for events in the given time range. The first entry in the pair is a Long, representing the event timestamp, while the second entry is a GenericStreamEventData, which contains data decoded from the stream event body base on the given FormatSpecification.

Type Parameters:

T - value type

Parameters:

streamName - name of the stream

formatSpec - the FormatSpecification describing the format in the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

Returns:

a new JavaPairRDD instance that reads from the given stream.

Throws:

DatasetInstantiationException - if the Stream doesn't exist

fromStream

public abstract <T> org.apache.spark.api.java.JavaPairRDD<Long,co.cask.cdap.api.stream.GenericStreamEventData<T>> fromStream(String namespace,
                                                                                                                             String streamName,
                                                                                                                             co.cask.cdap.api.data.format.FormatSpecification formatSpec,
                                                                                                                             long startTime,
                                                                                                                             long endTime,
                                                                                                                             Class<T> dataType)

Creates a JavaPairRDD that represents data from the given stream for events in the given time range. The first entry in the pair is a Long, representing the event timestamp, while the second entry is a GenericStreamEventData, which contains data decoded from the stream event body base on the given FormatSpecification.

Type Parameters:

T - value type

Parameters:

namespace - namespace in which the stream exists

streamName - name of the stream

formatSpec - the FormatSpecification describing the format in the stream

startTime - the starting time of the stream to be read in milliseconds (inclusive); passing in 0 means start reading from the first event available in the stream.

endTime - the ending time of the streams to be read in milliseconds (exclusive); passing in Long.MAX_VALUE means read up to latest event available in the stream.

Returns:

a new JavaPairRDD instance that reads from the given stream.

Throws:

DatasetInstantiationException - if the Stream doesn't exist

saveAsDataset

public <K,V> void saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd,
                                String datasetName)

Saves the given JavaPairRDD to the given Dataset.

Parameters:

rdd - the JavaPairRDD to be saved

datasetName - name of the Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

saveAsDataset

public <K,V> void saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd,
                                String namespace,
                                String datasetName)

Saves the given JavaPairRDD to the given Dataset.

Parameters:

rdd - the JavaPairRDD to be saved

namespace - the namespace in which the specified dataset is to be saved

datasetName - name of the Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

saveAsDataset

public abstract <K,V> void saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd,
                                         String datasetName,
                                         Map<String,String> arguments)

Saves the given JavaPairRDD to the given Dataset with the given set of Dataset arguments.

Parameters:

rdd - the JavaPairRDD to be saved

datasetName - name of the dataset

arguments - arguments for the dataset

Throws:

DatasetInstantiationException - if the dataset doesn't exist

saveAsDataset

public abstract <K,V> void saveAsDataset(org.apache.spark.api.java.JavaPairRDD<K,V> rdd,
                                         String namespace,
                                         String datasetName,
                                         Map<String,String> arguments)

Saves the given JavaPairRDD to the given Dataset with the given set of dataset arguments.

Parameters:

rdd - the JavaPairRDD to be saved

namespace - the namespace in which the specified Dataset is to be saved

datasetName - name of the Dataset

arguments - arguments for the Dataset

Throws:

DatasetInstantiationException - if the Dataset doesn't exist

execute

public abstract void execute(int timeoutInSeconds,
                             TxRunnable runnable)
                      throws org.apache.tephra.TransactionFailureException

Transactions with a custom timeout are not supported in Spark.

Specified by:

execute in interface Transactional

Throws:

org.apache.tephra.TransactionFailureException - always

createInterpreter

public abstract co.cask.cdap.api.spark.dynamic.SparkInterpreter createInterpreter()
                                                                           throws IOException

Creates a new instance of SparkInterpreter for Scala code compilation and interpretation.

Returns:

a new instance of SparkInterpreter

Throws:

IOException - if failed to create a local directory for storing the compiled class files

getSparkExecutionContext

public abstract SparkExecutionContext getSparkExecutionContext()

Returns the underlying SparkExecutionContext used by this object.