co.cask.cdap.api.spark.sql

Class DataFrames

java.lang.Object

co.cask.cdap.api.spark.sql.DataFrames

public final class DataFrames
extends Object

Utility class for conversions between DataType and Schema.

Method Summary

Modifier and Type	Method and Description
static co.cask.cdap.api.data.format.StructuredRecord	fromRow(org.apache.spark.sql.Row row, co.cask.cdap.api.data.schema.Schema schema) Creates a StructuredRecord from the data in the given Row.
static <T extends org.apache.spark.sql.types.DataType> T	toDataType(co.cask.cdap.api.data.schema.Schema schema) Converts a Schema to Spark DataType.
static <T extends org.apache.spark.sql.types.DataType> T	toDataType(co.cask.cdap.api.data.schema.Schema schema, scala.Function1<co.cask.cdap.api.data.schema.Schema,org.apache.spark.sql.types.DataType> unionSelector) Converts a Schema to Spark DataType.
static org.apache.spark.sql.Row	toRow(co.cask.cdap.api.data.format.StructuredRecord record, org.apache.spark.sql.types.StructType structType) Creates a Row object that represents data in the given StructuredRecord.
static co.cask.cdap.api.data.schema.Schema	toSchema(org.apache.spark.sql.types.DataType dataType) Converts a Spark DataType to a Schema object.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

toDataType

public static <T extends org.apache.spark.sql.types.DataType> T toDataType(co.cask.cdap.api.data.schema.Schema schema)

Converts a Schema to Spark DataType.

Parameters:

schema - the schema to convert

Returns:

The corresponding DataType

toDataType

public static <T extends org.apache.spark.sql.types.DataType> T toDataType(co.cask.cdap.api.data.schema.Schema schema,
                                                                           scala.Function1<co.cask.cdap.api.data.schema.Schema,org.apache.spark.sql.types.DataType> unionSelector)

Converts a Schema to Spark DataType.

Parameters:

schema - the schema to convert

unionSelector - the function to pick which schema to use when a Schema.Type#UNION type schema is encountered

Returns:

The corresponding DataType

toSchema

public static co.cask.cdap.api.data.schema.Schema toSchema(org.apache.spark.sql.types.DataType dataType)

Converts a Spark DataType to a Schema object.

Parameters:

dataType - the data type to convert from

Returns:

The corresponding Schema

toRow

public static org.apache.spark.sql.Row toRow(co.cask.cdap.api.data.format.StructuredRecord record,
                                             org.apache.spark.sql.types.StructType structType)

Creates a Row object that represents data in the given StructuredRecord.

Parameters:

record - contains the record data

structType - a StructType representing the data type in the resulting Row.

Returns:

a new Row instance

fromRow

public static co.cask.cdap.api.data.format.StructuredRecord fromRow(org.apache.spark.sql.Row row,
                                                                    co.cask.cdap.api.data.schema.Schema schema)

Creates a StructuredRecord from the data in the given Row.

Parameters:

row - contains the record data

schema - the Schema of the resulting StructuredRecord.

Returns:

a new StructuredRecord instance