org.apache.beam.examples.cookbook

Class DeDupExample

java.lang.Object

org.apache.beam.examples.cookbook.DeDupExample

public class DeDupExample
extends Object

This example uses as input Shakespeare's plays as plaintext files, and will remove any duplicate lines across all the files. (The output does not preserve any input order).

Concepts: the RemoveDuplicates transform, and how to wire transforms together. Demonstrates TextIO.Read/ RemoveDuplicates/TextIO.Write.

To execute this pipeline locally, specify a local output file or output prefix on GCS: --output=[YOUR_LOCAL_FILE | gs://YOUR_OUTPUT_PREFIX]

To change the runner, specify:

   --runner=YOUR_SELECTED_RUNNER
 
 

See examples/java/README.md for instructions about how to configure different runners.

The input defaults to gs://apache-beam-samples/shakespeare/* and can be overridden with --input.

Constructor Summary

Constructors

Constructor and Description
DeDupExample()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeDupExample

public DeDupExample()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception