org.apache.carbondata.sdk.file

Class CarbonWriterBuilder

java.lang.Object

org.apache.carbondata.sdk.file.CarbonWriterBuilder

@InterfaceAudience.User
 @InterfaceStability.Unstable
public class CarbonWriterBuilder
extends Object

Builder for CarbonWriter

Constructor Summary

Constructors

Constructor and Description
CarbonWriterBuilder()

Method Summary

Modifier and Type	Method and Description
CarbonWriter	build() Build a CarbonWriter This writer is not thread safe, use withThreadSafe() configuration in multi thread environment
org.apache.carbondata.processing.loading.model.CarbonLoadModel	buildLoadModel(Schema carbonSchema)
CarbonWriterBuilder	enableLocalDictionary(boolean enableLocalDictionary)
CarbonWriterBuilder	invertedIndexFor(String[] invertedIndexColumns) sets the list of columns for which inverted index needs to generated
CarbonWriterBuilder	localDictionaryThreshold(int localDictionaryThreshold)
CarbonWriterBuilder	outputPath(String path) Sets the output path of the writer builder
CarbonWriterBuilder	sortBy(String[] sortColumns) sets the list of columns that needs to be in sorted order
CarbonWriterBuilder	taskNo(long taskNo) sets the taskNo for the writer.
CarbonWriterBuilder	uniqueIdentifier(long timestamp) to set the timestamp in the carbondata and carbonindex index files
CarbonWriterBuilder	withAvroInput(org.apache.avro.Schema avroSchema) to build a CarbonWriter, which accepts Avro object
CarbonWriterBuilder	withBlockletSize(int blockletSize) To set the blocklet size of CarbonData file
CarbonWriterBuilder	withBlockSize(int blockSize) To set the carbondata file size in MB between 1MB-2048MB
CarbonWriterBuilder	withCsvInput(Schema schema) to build a CarbonWriter, which accepts row in CSV format
CarbonWriterBuilder	withCsvInput(String jsonSchema) to build a CarbonWriter, which accepts row in CSV format
CarbonWriterBuilder	withHadoopConf(org.apache.hadoop.conf.Configuration conf) To support hadoop configuration
CarbonWriterBuilder	withHadoopConf(String key, String value) Updates the hadoop configuration with the given key value
CarbonWriterBuilder	withJsonInput(Schema carbonSchema) to build a CarbonWriter, which accepts Json object
CarbonWriterBuilder	withLoadOption(String key, String value) To support the load options for sdk writer
CarbonWriterBuilder	withLoadOptions(Map<String,String> options) To support the load options for sdk writer
CarbonWriterBuilder	withTableProperties(Map<String,String> options) To support the table properties for sdk writer
CarbonWriterBuilder	withTableProperty(String key, String value) To support the table properties for sdk writer
CarbonWriterBuilder	withThreadSafe(short numOfThreads) To make sdk writer thread safe.
CarbonWriterBuilder	writtenBy(String appName)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CarbonWriterBuilder

public CarbonWriterBuilder()

Method Detail

outputPath

public CarbonWriterBuilder outputPath(String path)

Sets the output path of the writer builder

Parameters:

path - is the absolute path where output files are written This method must be called when building CarbonWriterBuilder

Returns:

updated CarbonWriterBuilder

sortBy

public CarbonWriterBuilder sortBy(String[] sortColumns)

sets the list of columns that needs to be in sorted order

Parameters:

sortColumns - is a string array of columns that needs to be sorted. If it is null or by default all dimensions are selected for sorting If it is empty array, no columns are sorted

Returns:

updated CarbonWriterBuilder

invertedIndexFor

public CarbonWriterBuilder invertedIndexFor(String[] invertedIndexColumns)

sets the list of columns for which inverted index needs to generated

Parameters:

invertedIndexColumns - is a string array of columns for which inverted index needs to generated. If it is null or an empty array, inverted index will be generated for none of the columns

Returns:

updated CarbonWriterBuilder

taskNo

public CarbonWriterBuilder taskNo(long taskNo)

sets the taskNo for the writer. SDKs concurrently running will set taskNo in order to avoid conflicts in file's name during write.

Parameters:

taskNo - is the TaskNo user wants to specify. by default it is system time in nano seconds.

Returns:

updated CarbonWriterBuilder

uniqueIdentifier

public CarbonWriterBuilder uniqueIdentifier(long timestamp)

to set the timestamp in the carbondata and carbonindex index files

Parameters:

timestamp - is a timestamp to be used in the carbondata and carbonindex index files. By default set to zero.

Returns:

updated CarbonWriterBuilder

withLoadOptions

public CarbonWriterBuilder withLoadOptions(Map<String,String> options)

To support the load options for sdk writer

Parameters:

options - key,value pair of load options. supported keys values are a. bad_records_logger_enable -- true (write into separate logs), false b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT c. bad_record_path -- path d. dateformat -- same as JAVA SimpleDateFormat e. timestampformat -- same as JAVA SimpleDateFormat f. complex_delimiter_level_1 -- value to Split the complexTypeData g. complex_delimiter_level_2 -- value to Split the nested complexTypeData h. quotechar i. escapechar Default values are as follows. a. bad_records_logger_enable -- "false" b. bad_records_action -- "FAIL" c. bad_record_path -- "" d. dateformat -- "" , uses from carbon.properties file e. timestampformat -- "", uses from carbon.properties file f. complex_delimiter_level_1 -- "\001" g. complex_delimiter_level_2 -- "\002" h. quotechar -- "\"" i. escapechar -- "\\"

Returns:

updated CarbonWriterBuilder

withLoadOption

public CarbonWriterBuilder withLoadOption(String key,
                                          String value)

To support the load options for sdk writer

Parameters:

key - the key of load option

value - the value of load option

Returns:

updated CarbonWriterBuilder object

withTableProperties

public CarbonWriterBuilder withTableProperties(Map<String,String> options)

To support the table properties for sdk writer

Parameters:

options - key,value pair of create table properties. supported keys values are a. table_blocksize -- [1-2048] values in MB. Default value is 1024 b. table_blocklet_size -- values in MB. Default value is 64 MB c. local_dictionary_threshold -- positive value, default is 10000 d. local_dictionary_enable -- true / false. Default is false e. sort_columns -- comma separated column. "c1,c2". Default all dimensions are sorted. If empty string "" is passed. No columns are sorted j. sort_scope -- "local_sort", "no_sort", "batch_sort". default value is "local_sort" k. long_string_columns -- comma separated string columns which are more than 32k length. default value is null. l. inverted_index -- comma separated string columns for which inverted index needs to be generated

Returns:

updated CarbonWriterBuilder

withTableProperty

public CarbonWriterBuilder withTableProperty(String key,
                                             String value)

To support the table properties for sdk writer

Parameters:

key - property key

value - property value

Returns:

CarbonWriterBuilder object

withThreadSafe

public CarbonWriterBuilder withThreadSafe(short numOfThreads)

To make sdk writer thread safe.

Parameters:

numOfThreads - should number of threads in which writer is called in multi-thread scenario default sdk writer is not thread safe. can use one writer instance in one thread only.

Returns:

updated CarbonWriterBuilder

withHadoopConf

public CarbonWriterBuilder withHadoopConf(org.apache.hadoop.conf.Configuration conf)

To support hadoop configuration

Parameters:

conf - hadoop configuration support, can set s3a AK,SK,end point and other conf with this

Returns:

updated CarbonWriterBuilder

withHadoopConf

public CarbonWriterBuilder withHadoopConf(String key,
                                          String value)

Updates the hadoop configuration with the given key value

Parameters:

key - key word

value - value

Returns:

this object

withBlockSize

public CarbonWriterBuilder withBlockSize(int blockSize)

To set the carbondata file size in MB between 1MB-2048MB

Parameters:

blockSize - is size in MB between 1MB to 2048 MB default value is 1024 MB

Returns:

updated CarbonWriterBuilder

localDictionaryThreshold

public CarbonWriterBuilder localDictionaryThreshold(int localDictionaryThreshold)

Parameters:

localDictionaryThreshold - is localDictionaryThreshold, default is 10000

Returns:

updated CarbonWriterBuilder

writtenBy

public CarbonWriterBuilder writtenBy(String appName)

Parameters:

appName - appName which is writing the carbondata files

Returns:

enableLocalDictionary

public CarbonWriterBuilder enableLocalDictionary(boolean enableLocalDictionary)

Parameters:

enableLocalDictionary - enable local dictionary, default is false

Returns:

updated CarbonWriterBuilder

withBlockletSize

public CarbonWriterBuilder withBlockletSize(int blockletSize)

To set the blocklet size of CarbonData file

Parameters:

blockletSize - is blocklet size in MB default value is 64 MB

Returns:

updated CarbonWriterBuilder

withCsvInput

public CarbonWriterBuilder withCsvInput(Schema schema)

to build a CarbonWriter, which accepts row in CSV format

Parameters:

schema - carbon Schema object {org.apache.carbondata.sdk.file.Schema}

Returns:

CarbonWriterBuilder

withCsvInput

public CarbonWriterBuilder withCsvInput(String jsonSchema)

to build a CarbonWriter, which accepts row in CSV format

Parameters:

jsonSchema - json Schema string

Returns:

CarbonWriterBuilder

withAvroInput

public CarbonWriterBuilder withAvroInput(org.apache.avro.Schema avroSchema)

to build a CarbonWriter, which accepts Avro object

Parameters:

avroSchema - avro Schema object {org.apache.avro.Schema}

Returns:

CarbonWriterBuilder

withJsonInput

public CarbonWriterBuilder withJsonInput(Schema carbonSchema)

to build a CarbonWriter, which accepts Json object

Parameters:

carbonSchema - carbon Schema object

Returns:

CarbonWriterBuilder

build

public CarbonWriter build()
                   throws IOException,
                          org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException

Build a CarbonWriter This writer is not thread safe, use withThreadSafe() configuration in multi thread environment

Returns:

CarbonWriter {AvroCarbonWriter/CSVCarbonWriter/JsonCarbonWriter based on Input Type }

Throws:

IOException

org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException

buildLoadModel

public org.apache.carbondata.processing.loading.model.CarbonLoadModel buildLoadModel(Schema carbonSchema)
                                                                              throws IOException,
                                                                                     org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException

Throws:

IOException

org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException