co.cask.hydrator.plugin.spark

Class KafkaConfig

java.lang.Object

co.cask.cdap.api.Config

co.cask.cdap.api.plugin.PluginConfig

co.cask.hydrator.common.ReferencePluginConfig

co.cask.hydrator.plugin.spark.KafkaConfig

All Implemented Interfaces:

Serializable

public class KafkaConfig
extends ReferencePluginConfig
implements Serializable

Conf for Kafka streaming source.

See Also:

Serialized Form

Field Summary

Fields inherited from class co.cask.hydrator.common.ReferencePluginConfig

referenceName

Constructor Summary

Constructors

Constructor and Description
KafkaConfig()
KafkaConfig(String referenceName, String brokers, String topic, String partitions, String initialPartitionOffsets, Long defaultInitialOffset, String schema, String format, String timeField, String keyField, String partitionField, String offsetField)
KafkaConfig(String referenceName, String brokers, String topic, String schema, String format, String timeField, String keyField, String partitionField, String offsetField)

Method Summary

Methods

Modifier and Type	Method and Description
Map<String,Integer>	getBrokerMap()
String	getBrokers()
String	getFormat()
Map<kafka.common.TopicAndPartition,Long>	getInitialPartitionOffsets(Set<Integer> partitionsToRead) Get the initial partition offsets for the specified partitions.
String	getKeyField()
co.cask.cdap.api.data.schema.Schema	getMessageSchema()
String	getOffsetField()
String	getPartitionField()
Set<Integer>	getPartitions()
co.cask.cdap.api.data.schema.Schema	getSchema()
String	getTimeField()
String	getTopic()
void	validate()

Methods inherited from class co.cask.cdap.api.plugin.PluginConfig

containsMacro, getProperties

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KafkaConfig

public KafkaConfig()

KafkaConfig

public KafkaConfig(String referenceName,
           String brokers,
           String topic,
           String schema,
           String format,
           String timeField,
           String keyField,
           String partitionField,
           String offsetField)

KafkaConfig

public KafkaConfig(String referenceName,
           String brokers,
           String topic,
           String partitions,
           String initialPartitionOffsets,
           Long defaultInitialOffset,
           String schema,
           String format,
           String timeField,
           String keyField,
           String partitionField,
           String offsetField)

Method Detail

getTopic

public String getTopic()

getBrokers

public String getBrokers()

getTimeField

@Nullable
public String getTimeField()

getKeyField

@Nullable
public String getKeyField()

getPartitionField

@Nullable
public String getPartitionField()

getOffsetField

@Nullable
public String getOffsetField()

getFormat

@Nullable
public String getFormat()

getSchema

public co.cask.cdap.api.data.schema.Schema getSchema()

getMessageSchema

public co.cask.cdap.api.data.schema.Schema getMessageSchema()

getInitialPartitionOffsets

public Map<kafka.common.TopicAndPartition,Long> getInitialPartitionOffsets(Set<Integer> partitionsToRead)

Get the initial partition offsets for the specified partitions. If an initial offset is specified in the initialPartitionOffsets property, that value will be used. Otherwise, the defaultInitialOffset will be used.

Parameters:

partitionsToRead - the partitions to read

Returns:

initial partition offsets.

getBrokerMap

public Map<String,Integer> getBrokerMap()

Returns:

broker host to broker port mapping.

getPartitions

public Set<Integer> getPartitions()

Returns:

set of partitions to read from. Returns an empty list if no partitions were specified.

validate

public void validate()