co.cask.hydrator.plugin.spark

Class SparkMLTrainer

java.lang.Object

co.cask.cdap.etl.api.batch.BatchConfigurable<co.cask.cdap.etl.api.batch.SparkPluginContext>

co.cask.cdap.etl.api.batch.SparkSink<co.cask.cdap.api.data.format.StructuredRecord>

co.cask.hydrator.plugin.spark.SparkMLTrainer

All Implemented Interfaces:

co.cask.cdap.etl.api.PipelineConfigurable, Serializable

Direct Known Subclasses:

DecisionTreeTrainer, LogisticRegressionTrainer

public abstract class SparkMLTrainer
extends co.cask.cdap.etl.api.batch.SparkSink<co.cask.cdap.api.data.format.StructuredRecord>

Common class for Spark Trainers. Contains common Spark ML Trainer properties, configuartion ,validation and feature extraction methods to be used in trainers.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	SparkMLTrainer.MLTrainerConfig Config class for Trainers.

Field Summary

Fields inherited from class co.cask.cdap.etl.api.batch.SparkSink

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
SparkMLTrainer(SparkMLTrainer.MLTrainerConfig config)

Method Summary

Methods

Modifier and Type	Method and Description
void	configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)
void	prepareRun(co.cask.cdap.etl.api.batch.SparkPluginContext context)
void	run(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context, org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> input)
abstract void	trainModel(org.apache.spark.SparkContext context, co.cask.cdap.api.data.schema.Schema inputSchema, org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> trainingData, String outputPath)

Methods inherited from class co.cask.cdap.etl.api.batch.BatchConfigurable

onRunFinish

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkMLTrainer

public SparkMLTrainer(SparkMLTrainer.MLTrainerConfig config)

Method Detail

configurePipeline

public void configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)

Specified by:

configurePipeline in interface co.cask.cdap.etl.api.PipelineConfigurable

Overrides:

configurePipeline in class co.cask.cdap.etl.api.batch.BatchConfigurable<co.cask.cdap.etl.api.batch.SparkPluginContext>

prepareRun

public void prepareRun(co.cask.cdap.etl.api.batch.SparkPluginContext context)
                throws Exception

Specified by:

prepareRun in class co.cask.cdap.etl.api.batch.BatchConfigurable<co.cask.cdap.etl.api.batch.SparkPluginContext>

Throws:

Exception

run

public void run(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context,
       org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> input)
         throws Exception

Specified by:

run in class co.cask.cdap.etl.api.batch.SparkSink<co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception

trainModel

public abstract void trainModel(org.apache.spark.SparkContext context,
              co.cask.cdap.api.data.schema.Schema inputSchema,
              org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> trainingData,
              String outputPath)