co.cask.hydrator.plugin.spark

Class SkipGramTrainer

java.lang.Object

co.cask.cdap.etl.api.batch.BatchConfigurable<co.cask.cdap.etl.api.batch.SparkPluginContext>

co.cask.cdap.etl.api.batch.SparkSink<co.cask.cdap.api.data.format.StructuredRecord>

co.cask.hydrator.plugin.spark.SkipGramTrainer

All Implemented Interfaces:

co.cask.cdap.etl.api.PipelineConfigurable, Serializable

public class SkipGramTrainer
extends co.cask.cdap.etl.api.batch.SparkSink<co.cask.cdap.api.data.format.StructuredRecord>

Spark Sink plugin that trains a model using SkipGram (Spark's Word2Vec). Saves this model to a FileSet.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SkipGramTrainer.SkipGramTrainerConfig Configuration for SkipGramTrainer.

Field Summary

Fields

Modifier and Type	Field and Description
static String	PLUGIN_NAME

Fields inherited from class co.cask.cdap.etl.api.batch.SparkSink

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
SkipGramTrainer()

Method Summary

Methods

Modifier and Type	Method and Description
void	configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)
void	prepareRun(co.cask.cdap.etl.api.batch.SparkPluginContext sparkPluginContext)
void	run(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context, org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> input)

Methods inherited from class co.cask.cdap.etl.api.batch.BatchConfigurable

onRunFinish

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PLUGIN_NAME

public static final String PLUGIN_NAME

See Also:

Constant Field Values

Constructor Detail

SkipGramTrainer

public SkipGramTrainer()

Method Detail

configurePipeline

public void configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)

Specified by:

configurePipeline in interface co.cask.cdap.etl.api.PipelineConfigurable

Overrides:

configurePipeline in class co.cask.cdap.etl.api.batch.BatchConfigurable<co.cask.cdap.etl.api.batch.SparkPluginContext>

run

public void run(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context,
       org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> input)
         throws Exception

Specified by:

run in class co.cask.cdap.etl.api.batch.SparkSink<co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception

prepareRun

public void prepareRun(co.cask.cdap.etl.api.batch.SparkPluginContext sparkPluginContext)
                throws Exception

Specified by:

prepareRun in class co.cask.cdap.etl.api.batch.BatchConfigurable<co.cask.cdap.etl.api.batch.SparkPluginContext>

Throws:

Exception