co.cask.hydrator.plugin.spark

Class SparkMLPredictor

java.lang.Object

co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

co.cask.hydrator.plugin.spark.SparkMLPredictor

All Implemented Interfaces:

co.cask.cdap.etl.api.PipelineConfigurable, Serializable

Direct Known Subclasses:

DecisionTreePredictor, LogisticRegressionClassifier

public abstract class SparkMLPredictor
extends co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Common class for Spark Predictors/Classifiers. Contains common Spark ML Classifiers properties, configuartion, validation and feature extraction methods to be used in predictors/classifiers.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	SparkMLPredictor.MLPredictorConfig Config class for Predictors/Classifiers.

Field Summary

Fields inherited from class co.cask.cdap.etl.api.batch.SparkCompute

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
SparkMLPredictor(SparkMLPredictor.MLPredictorConfig config)

Method Summary

Methods

Modifier and Type	Method and Description
void	configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)
abstract void	initialize(org.apache.spark.SparkContext context, String modelPath)
void	initialize(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context)
abstract double	predict(org.apache.spark.mllib.linalg.Vector vector)
org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord>	transform(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context, org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> input)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkMLPredictor

public SparkMLPredictor(SparkMLPredictor.MLPredictorConfig config)

Method Detail

configurePipeline

public void configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)
                       throws IllegalArgumentException

Specified by:

configurePipeline in interface co.cask.cdap.etl.api.PipelineConfigurable

Overrides:

configurePipeline in class co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

IllegalArgumentException

initialize

public void initialize(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context)
                throws Exception

Overrides:

initialize in class co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception

initialize

public abstract void initialize(org.apache.spark.SparkContext context,
              String modelPath)

predict

public abstract double predict(org.apache.spark.mllib.linalg.Vector vector)

transform

public org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> transform(co.cask.cdap.etl.api.batch.SparkExecutionPluginContext context,
                                                                                         org.apache.spark.api.java.JavaRDD<co.cask.cdap.api.data.format.StructuredRecord> input)
                                                                                           throws Exception

Specified by:

transform in class co.cask.cdap.etl.api.batch.SparkCompute<co.cask.cdap.api.data.format.StructuredRecord,co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception