org.apache.carbondata.hadoop

Class CarbonInputFormat<T>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

org.apache.carbondata.hadoop.CarbonInputFormat<T>

public class CarbonInputFormat<T>
extends org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

Carbon Input format class representing one carbon table

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Field Summary

Fields

Modifier and Type	Field and Description
static String	INPUT_SEGMENT_NUMBERS

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

INPUT_DIR, INPUT_DIR_RECURSIVE, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor and Description
CarbonInputFormat()

Method Summary

Modifier and Type	Method and Description
protected long	computeSplitSize(long blockSize, long minSize, long maxSize)
org.apache.hadoop.mapreduce.RecordReader<Void,T>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
protected int	getBlockIndex(org.apache.hadoop.fs.BlockLocation[] blkLocations, long offset)
static CarbonTable	getCarbonTable(org.apache.hadoop.conf.Configuration configuration)
org.apache.hadoop.fs.PathFilter	getDataFileFilter(org.apache.hadoop.mapreduce.JobContext job)
FilterResolverIntf	getResolvedFilter(org.apache.hadoop.conf.Configuration configuration, Expression filterExpression) Configurations FileInputFormat.INPUT_DIR, CarbonInputFormat.INPUT_SEGMENT_NUMBERS are used to get table path to read.
long	getRowCount(org.apache.hadoop.mapreduce.JobContext job) get total number of rows.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext job) Configurations FileInputFormat.INPUT_DIR are used to get table path to read.
static CarbonTablePath	getTablePath(org.apache.hadoop.conf.Configuration configuration)
static CarbonTableIdentifier	getTableToAccess(org.apache.hadoop.conf.Configuration configuration) Get CarbonTableIdentifier from job configuration
protected boolean	isSplitable(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path filename)
protected List<org.apache.hadoop.fs.FileStatus>	listStatus(org.apache.hadoop.mapreduce.JobContext job)
static void	setCarbonReadSupport(Class<? extends CarbonReadSupport> readSupportClass, org.apache.hadoop.conf.Configuration configuration)
static void	setCarbonTable(org.apache.hadoop.conf.Configuration configuration, CarbonTable carbonTable) It is optional, if user does not set then it reads from store
static void	setColumnProjection(CarbonProjection projection, org.apache.hadoop.conf.Configuration configuration)
static void	setFilterPredicates(org.apache.hadoop.conf.Configuration configuration, Expression filterExpression) It sets unresolved filter expression.
static void	setFilterPredicates(org.apache.hadoop.conf.Configuration configuration, FilterResolverIntf filterExpression) It sets the resolved filter expression
static void	setTableToAccess(org.apache.hadoop.conf.Configuration configuration, CarbonTableIdentifier tableIdentifier)

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INPUT_SEGMENT_NUMBERS

public static final String INPUT_SEGMENT_NUMBERS

See Also:

Constant Field Values

Constructor Detail

CarbonInputFormat

public CarbonInputFormat()

Method Detail

setTableToAccess

public static void setTableToAccess(org.apache.hadoop.conf.Configuration configuration,
                                    CarbonTableIdentifier tableIdentifier)

getTableToAccess

public static CarbonTableIdentifier getTableToAccess(org.apache.hadoop.conf.Configuration configuration)

Get CarbonTableIdentifier from job configuration

setCarbonTable

public static void setCarbonTable(org.apache.hadoop.conf.Configuration configuration,
                                  CarbonTable carbonTable)
                           throws IOException

It is optional, if user does not set then it reads from store

Parameters:

configuration -

carbonTable -

Throws:

IOException

getCarbonTable

public static CarbonTable getCarbonTable(org.apache.hadoop.conf.Configuration configuration)
                                  throws IOException

Throws:

IOException

setFilterPredicates

public static void setFilterPredicates(org.apache.hadoop.conf.Configuration configuration,
                                       Expression filterExpression)

It sets unresolved filter expression.

Parameters:

configuration -

filterExpression -

setFilterPredicates

public static void setFilterPredicates(org.apache.hadoop.conf.Configuration configuration,
                                       FilterResolverIntf filterExpression)

It sets the resolved filter expression

Parameters:

configuration -

filterExpression -

setColumnProjection

public static void setColumnProjection(CarbonProjection projection,
                                       org.apache.hadoop.conf.Configuration configuration)

setCarbonReadSupport

public static void setCarbonReadSupport(Class<? extends CarbonReadSupport> readSupportClass,
                                        org.apache.hadoop.conf.Configuration configuration)

getTablePath

public static CarbonTablePath getTablePath(org.apache.hadoop.conf.Configuration configuration)
                                    throws IOException

Throws:

IOException

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext job)
                                                       throws IOException

Configurations FileInputFormat.INPUT_DIR are used to get table path to read.

Overrides:

getSplits in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

Parameters:

job -

Returns:

List list of CarbonInputSplit

Throws:

IOException

getRowCount

public long getRowCount(org.apache.hadoop.mapreduce.JobContext job)
                 throws IOException,
                        IndexBuilderException

get total number of rows. Same as count(*)

Throws:

IOException

IndexBuilderException

getResolvedFilter

public FilterResolverIntf getResolvedFilter(org.apache.hadoop.conf.Configuration configuration,
                                            Expression filterExpression)
                                     throws IOException,
                                            IndexBuilderException,
                                            QueryExecutionException

Configurations FileInputFormat.INPUT_DIR, CarbonInputFormat.INPUT_SEGMENT_NUMBERS are used to get table path to read.

Returns:

Throws:

IOException

IndexBuilderException

QueryExecutionException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<Void,T> createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                                                                           org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
                                                                    throws IOException,
                                                                           InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<Void,T>

Throws:

IOException

InterruptedException

computeSplitSize

protected long computeSplitSize(long blockSize,
                                long minSize,
                                long maxSize)

Overrides:

computeSplitSize in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

getBlockIndex

protected int getBlockIndex(org.apache.hadoop.fs.BlockLocation[] blkLocations,
                            long offset)

Overrides:

getBlockIndex in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

listStatus

protected List<org.apache.hadoop.fs.FileStatus> listStatus(org.apache.hadoop.mapreduce.JobContext job)
                                                    throws IOException

Overrides:

listStatus in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

Throws:

IOException

isSplitable

protected boolean isSplitable(org.apache.hadoop.mapreduce.JobContext context,
                              org.apache.hadoop.fs.Path filename)

Overrides:

isSplitable in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<Void,T>

getDataFileFilter

public org.apache.hadoop.fs.PathFilter getDataFileFilter(org.apache.hadoop.mapreduce.JobContext job)

Parameters:

job -

Returns:

the PathFilter for Fact Files.