org.apache.beam.examples.complete.game

Class LeaderBoard

java.lang.Object

org.apache.beam.examples.complete.game.UserScore

org.apache.beam.examples.complete.game.HourlyTeamScore

org.apache.beam.examples.complete.game.LeaderBoard

Direct Known Subclasses:

GameStats

public class LeaderBoard
extends HourlyTeamScore

This class is the third in a series of four pipelines that tell a story in a 'gaming' domain, following UserScore and HourlyTeamScore. Concepts include: processing unbounded data using fixed windows; use of custom timestamps and event-time processing; generation of early/speculative results; using .accumulatingFiredPanes() to do cumulative processing of late- arriving data.

This pipeline processes an unbounded stream of 'game events'. The calculation of the team scores uses fixed windowing based on event time (the time of the game play event), not processing time (the time that an event is processed by the pipeline). The pipeline calculates the sum of scores per team, for each window. By default, the team scores are calculated using one-hour windows.

In contrast-- to demo another windowing option-- the user scores are calculated using a global window, which periodically (every ten minutes) emits cumulative user score sums.

In contrast to the previous pipelines in the series, which used static, finite input data, here we're using an unbounded data source, which lets us provide speculative results, and allows handling of late data, at much lower latency. We can use the early/speculative results to keep a 'leaderboard' updated in near-realtime. Our handling of late data lets us generate correct results, e.g. for 'team prizes'. We're now outputing window results as they're calculated, giving us much lower latency than with the previous batch examples.

Run Injector to generate pubsub data for this pipeline. The Injector documentation provides more detail on how to do this.

To execute this pipeline using the Dataflow service, specify the pipeline configuration like this:

   --project=YOUR_PROJECT_ID
   --tempLocation=gs://YOUR_TEMP_DIRECTORY
   --runner=BlockingDataflowRunner
   --dataset=YOUR-DATASET
   --topic=projects/YOUR-PROJECT/topics/YOUR-TOPIC
 
 

where the BigQuery dataset you specify must already exist. The PubSub topic you specify should be the same topic to which the Injector is publishing.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.examples.complete.game.UserScore

UserScore.ExtractAndSumScore

Constructor Summary

Constructors

Constructor and Description
LeaderBoard()

Method Summary

Modifier and Type	Method and Description
protected static Map<String,WriteToBigQuery.FieldInfo<KV<String,Integer>>>	configureGlobalWindowBigQueryWrite() Create a map of information that describes how to write pipeline output to BigQuery.
protected static Map<String,WriteToBigQuery.FieldInfo<KV<String,Integer>>>	configureWindowedTableWrite() Create a map of information that describes how to write pipeline output to BigQuery.
static void	main(String[] args)

Methods inherited from class org.apache.beam.examples.complete.game.UserScore

configureBigQueryWrite

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LeaderBoard

public LeaderBoard()

Method Detail

configureWindowedTableWrite

protected static Map<String,WriteToBigQuery.FieldInfo<KV<String,Integer>>> configureWindowedTableWrite()

Create a map of information that describes how to write pipeline output to BigQuery. This map is used to write team score sums and includes event timing information.

configureGlobalWindowBigQueryWrite

protected static Map<String,WriteToBigQuery.FieldInfo<KV<String,Integer>>> configureGlobalWindowBigQueryWrite()

Create a map of information that describes how to write pipeline output to BigQuery. This map is used to write user score sums.

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception