co.cask.hydrator.plugin.spark

Class FileStreamingSource

java.lang.Object

co.cask.cdap.etl.api.streaming.StreamingSource<T>

co.cask.hydrator.plugin.spark.ReferenceStreamingSource<co.cask.cdap.api.data.format.StructuredRecord>

co.cask.hydrator.plugin.spark.FileStreamingSource

All Implemented Interfaces:

PipelineConfigurable, Serializable

public class FileStreamingSource
extends ReferenceStreamingSource<co.cask.cdap.api.data.format.StructuredRecord>

Source that monitors a directory and reads files from it.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	FileStreamingSource.Conf Configuration for the source.

Field Summary

Fields inherited from class co.cask.cdap.etl.api.streaming.StreamingSource

PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
FileStreamingSource(FileStreamingSource.Conf conf)

Method Summary

Modifier and Type	Method and Description
void	configurePipeline(PipelineConfigurer pipelineConfigurer)
org.apache.spark.streaming.api.java.JavaDStream<co.cask.cdap.api.data.format.StructuredRecord>	getStream(StreamingContext context)

Methods inherited from class co.cask.cdap.etl.api.streaming.StreamingSource

getRequiredExecutors

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileStreamingSource

public FileStreamingSource(FileStreamingSource.Conf conf)

Method Detail

configurePipeline

public void configurePipeline(PipelineConfigurer pipelineConfigurer)
                       throws IllegalArgumentException

Specified by:

configurePipeline in interface PipelineConfigurable

Overrides:

configurePipeline in class ReferenceStreamingSource<co.cask.cdap.api.data.format.StructuredRecord>

Throws:

IllegalArgumentException

getStream

public org.apache.spark.streaming.api.java.JavaDStream<co.cask.cdap.api.data.format.StructuredRecord> getStream(StreamingContext context)
                                                                                                         throws Exception

Specified by:

getStream in class StreamingSource<co.cask.cdap.api.data.format.StructuredRecord>

Throws:

Exception