co.cask.hydrator.plugin.spark

Class SkipGramFeatureGenerator

java.lang.Object

co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

co.cask.hydrator.plugin.spark.SkipGramFeatureGenerator

All Implemented Interfaces:

co.cask.cdap.etl.api.PipelineConfigurable, Serializable

public class SkipGramFeatureGenerator
extends co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Class to generate text based features using SkipGram model (Spark's Word2Vec).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SkipGramFeatureGenerator.FeatureGeneratorConfig Config class for SkipGramFeatureGenerator.

Field Summary

Fields

Modifier and Type	Field and Description
static String	PLUGIN_NAME

Fields inherited from class co.cask.cdap.etl.api.batch.SparkCompute

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
SkipGramFeatureGenerator(SkipGramFeatureGenerator.FeatureGeneratorConfig config)

Method Summary

Methods

Modifier and Type	Method and Description
void	configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)
void	initialize(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context)
org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord>	transform(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context, org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> input)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PLUGIN_NAME

public static final String PLUGIN_NAME

See Also:

Constant Field Values

Constructor Detail

SkipGramFeatureGenerator

public SkipGramFeatureGenerator(SkipGramFeatureGenerator.FeatureGeneratorConfig config)

Method Detail

configurePipeline

public void configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)
                       throws IllegalArgumentException

Specified by:

configurePipeline in interface co.cask.cdap.etl.api.PipelineConfigurable

Overrides:

configurePipeline in class co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

IllegalArgumentException

initialize

public void initialize(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context)
                throws Exception

Overrides:

initialize in class co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception

transform

public org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> transform(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context,
                                                                                         org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> input)
                                                                                           throws Exception

Specified by:

transform in class co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception