kafka.bridge.hadoop
Class KafkaOutputFormat<K,V>

java.lang.Object
  org.apache.hadoop.mapreduce.OutputFormat<K,V>
      kafka.bridge.hadoop.KafkaOutputFormat<K,V>

public class KafkaOutputFormat<K,V>
extends org.apache.hadoop.mapreduce.OutputFormat<K,V>

Field Summary

static java.lang.String	KAFKA_CONFIG_PREFIX
static int	KAFKA_QUEUE_BYTES Bytes to buffer before the OutputFormat does a send (i.e., the amortization window): We set the default to a million bytes so that the server will not reject the batch of messages with a MessageSizeTooLargeException.
static java.lang.String	KAFKA_URL

Constructor Summary

KafkaOutputFormat()

Method Summary

void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jobContext)
org.apache.hadoop.mapreduce.OutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
static org.apache.hadoop.fs.Path	getOutputPath(org.apache.hadoop.mapreduce.JobContext job)
org.apache.hadoop.mapreduce.RecordWriter<K,V>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
static void	setOutputPath(org.apache.hadoop.mapreduce.Job job, org.apache.hadoop.fs.Path outputUrl)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KAFKA_URL

public static final java.lang.String KAFKA_URL

See Also:

Constant Field Values

KAFKA_QUEUE_BYTES

public static final int KAFKA_QUEUE_BYTES

Bytes to buffer before the OutputFormat does a send (i.e., the amortization window): We set the default to a million bytes so that the server will not reject the batch of messages with a MessageSizeTooLargeException. The actual size will be smaller after compression.

See Also:

Constant Field Values

KAFKA_CONFIG_PREFIX

public static final java.lang.String KAFKA_CONFIG_PREFIX

See Also:

Constant Field Values

Constructor Detail

KafkaOutputFormat

public KafkaOutputFormat()

Method Detail

setOutputPath

public static void setOutputPath(org.apache.hadoop.mapreduce.Job job,
                                 org.apache.hadoop.fs.Path outputUrl)

getOutputPath

public static org.apache.hadoop.fs.Path getOutputPath(org.apache.hadoop.mapreduce.JobContext job)

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jobContext)
                      throws java.io.IOException,
                             java.lang.InterruptedException

Specified by:

checkOutputSpecs in class org.apache.hadoop.mapreduce.OutputFormat<K,V>

Throws:

java.io.IOException

java.lang.InterruptedException

getOutputCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
                                                               throws java.io.IOException,
                                                                      java.lang.InterruptedException

Specified by:

getOutputCommitter in class org.apache.hadoop.mapreduce.OutputFormat<K,V>

Throws:

java.io.IOException

java.lang.InterruptedException

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<K,V> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                              throws java.io.IOException,
                                                                     java.lang.InterruptedException

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.OutputFormat<K,V>

Throws:

java.io.IOException

java.lang.InterruptedException