co.cask.hydrator.plugin.spark.dynamic

Class ScalaSparkCompute

java.lang.Object

co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

co.cask.hydrator.plugin.spark.dynamic.ScalaSparkCompute

All Implemented Interfaces:

PipelineConfigurable, Serializable

public class ScalaSparkCompute
extends SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

A SparkCompute that takes any scala code and executes it.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ScalaSparkCompute.Config Configuration object for the plugin

Field Summary

Fields inherited from class co.cask.cdap.etl.api.batch.SparkCompute

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
ScalaSparkCompute(ScalaSparkCompute.Config config)

Method Summary

Methods

Modifier and Type	Method and Description
void	configurePipeline(PipelineConfigurer pipelineConfigurer)
void	initialize(SparkExecutionPluginContext context)
org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord>	transform(SparkExecutionPluginContext sparkExecutionPluginContext, org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> javaRDD)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ScalaSparkCompute

public ScalaSparkCompute(ScalaSparkCompute.Config config)

Method Detail

configurePipeline

public void configurePipeline(PipelineConfigurer pipelineConfigurer)
                       throws IllegalArgumentException

Specified by:

configurePipeline in interface PipelineConfigurable

Overrides:

configurePipeline in class SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

IllegalArgumentException

initialize

public void initialize(SparkExecutionPluginContext context)
                throws Exception

Overrides:

initialize in class SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception

transform

public org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> transform(SparkExecutionPluginContext sparkExecutionPluginContext,
                                                                                         org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> javaRDD)
                                                                                           throws Exception

Specified by:

transform in class SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception