co.cask.cdap.etl.batch.connector

Class ConnectorSink

java.lang.Object

co.cask.cdap.etl.api.batch.BatchConfigurable<BatchSinkContext>

co.cask.cdap.etl.api.batch.BatchSink<KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>,NullWritable,Text>

co.cask.cdap.etl.batch.connector.ConnectorSink

All Implemented Interfaces:

Destroyable, PipelineConfigurable, StageLifecycle<BatchRuntimeContext>, Transformation<KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>,KeyValue<NullWritable,Text>>

public class ConnectorSink
extends BatchSink<KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>,NullWritable,Text>

Internal batch sink used as a connector between pipeline phases. Though this extends BatchSink, this will not be instantiated through the plugin framework, but will be created explicitly through the application. The batch connector is just a PartitionedFileSet, where a partition is the name of a phase that wrote to it. This way, multiple phases can have the same local PartitionedFileSet as a sink, and the source will read data from all partitions. This is because we don't want this to show up as a plugin that users can select and use, and also because it uses features not exposed in the etl api (local workflow datasets). TODO: improve storage format. It is currently a json of the record but that is obviously not ideal

Field Summary

Fields inherited from class co.cask.cdap.etl.api.batch.BatchSink

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
ConnectorSink(String datasetName, String phaseName)

Method Summary

Methods

Modifier and Type	Method and Description
void	prepareRun(BatchSinkContext context)
void	transform(KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord> input, Emitter<KeyValue<NullWritable,Text>> emitter)

Methods inherited from class co.cask.cdap.etl.api.batch.BatchSink

destroy, initialize

Methods inherited from class co.cask.cdap.etl.api.batch.BatchConfigurable

configurePipeline, onRunFinish

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConnectorSink

public ConnectorSink(String datasetName,
             String phaseName)

Method Detail

prepareRun

public void prepareRun(BatchSinkContext context)
                throws Exception

Specified by:

prepareRun in class BatchConfigurable<BatchSinkContext>

Throws:

Exception

transform

public void transform(KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord> input,
             Emitter<KeyValue<NullWritable,Text>> emitter)
               throws Exception

Specified by:

transform in interface Transformation<KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>,KeyValue<NullWritable,Text>>

Overrides:

transform in class BatchSink<KeyValue<String,co.cask.cdap.api.data.format.StructuredRecord>,NullWritable,Text>

Throws:

Exception