co.cask.cdap.etl.batch.connector

Class ConnectorSource

java.lang.Object

co.cask.cdap.etl.api.batch.BatchConfigurable<BatchSourceContext>

co.cask.cdap.etl.api.batch.BatchSource<LongWritable,Text,KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>>

co.cask.cdap.etl.batch.connector.ConnectorSource

All Implemented Interfaces:

Destroyable, PipelineConfigurable, StageLifecycle<BatchRuntimeContext>, Transformation<KeyValue<LongWritable,Text>,KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>>

public class ConnectorSource
extends BatchSource<LongWritable,Text,KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>>

Internal batch source used as a connector between pipeline phases. Though this extends BatchSource, this will not be instantiated through the plugin framework, but will be created explicitly through the application. The batch connector is just a PartitionedFileSet, where a partition is the name of a phase that wrote to it. This way, multiple phases can have the same local PartitionedFileSet as a sink, and the source will read data from all partitions. This is because we don't want this to show up as a plugin that users can select and use, and also because it uses features not exposed in the etl api (local workflow datasets). TODO: improve storage format. It is currently a json of the record but that is obviously not ideal

Field Summary

Fields inherited from class co.cask.cdap.etl.api.batch.BatchSource

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
ConnectorSource(String datasetName, co.cask.cdap.api.data.schema.Schema schema)

Method Summary

Methods

Modifier and Type	Method and Description
void	configure(WorkflowConfigurer workflowConfigurer)
void	prepareRun(BatchSourceContext context)
void	transform(KeyValue<LongWritable,Text> input, Emitter<KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>> emitter)

Methods inherited from class co.cask.cdap.etl.api.batch.BatchSource

destroy, initialize

Methods inherited from class co.cask.cdap.etl.api.batch.BatchConfigurable

configurePipeline, onRunFinish

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConnectorSource

public ConnectorSource(String datasetName,
               @Nullable
               co.cask.cdap.api.data.schema.Schema schema)

Method Detail

configure

public void configure(WorkflowConfigurer workflowConfigurer)

prepareRun

public void prepareRun(BatchSourceContext context)
                throws Exception

Specified by:

prepareRun in class BatchConfigurable<BatchSourceContext>

Throws:

Exception

transform

public void transform(KeyValue<LongWritable,Text> input,
             Emitter<KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>> emitter)
               throws Exception

Specified by:

transform in interface Transformation<KeyValue<LongWritable,Text>,KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>>

Overrides:

transform in class BatchSource<LongWritable,Text,KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>>

Throws:

Exception