co.cask.hydrator.plugin.spark

Class FileStreamingSource

java.lang.Object

co.cask.cdap.etl.api.streaming.StreamingSource<T>

co.cask.hydrator.plugin.spark.ReferenceStreamingSource<co.cask.cdap.api.data.format.StructuredRecord>

co.cask.hydrator.plugin.spark.FileStreamingSource

All Implemented Interfaces:

co.cask.cdap.etl.api.PipelineConfigurable, Serializable

public class FileStreamingSource
extends ReferenceStreamingSource<co.cask.cdap.api.data.format.StructuredRecord>

Source that monitors a directory and reads files from it.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileStreamingSource.Conf Configuration for the source.

Field Summary

Fields inherited from class co.cask.cdap.etl.api.streaming.StreamingSource

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
FileStreamingSource(FileStreamingSource.Conf conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)
org.apache.spark.streaming.api.java.JavaDStream<co.cask.cdap.api.data.format.StructuredRecord>	getStream(co.cask.cdap.etl.api.streaming.StreamingContext context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileStreamingSource

public FileStreamingSource(FileStreamingSource.Conf conf)

Method Detail

configurePipeline

public void configurePipeline(co.cask.cdap.etl.api.PipelineConfigurer pipelineConfigurer)
                       throws IllegalArgumentException

Specified by:

configurePipeline in interface co.cask.cdap.etl.api.PipelineConfigurable

Overrides:

configurePipeline in class ReferenceStreamingSource<co.cask.cdap.api.data.format.StructuredRecord>

Throws:

IllegalArgumentException

getStream

public org.apache.spark.streaming.api.java.JavaDStream<co.cask.cdap.api.data.format.StructuredRecord> getStream(co.cask.cdap.etl.api.streaming.StreamingContext context)
                                                                                                         throws Exception

Specified by:

getStream in class co.cask.cdap.etl.api.streaming.StreamingSource<co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception