Скачать или смотреть Python || Pyspark || Tamil || Column() Function || Dataframe

Python || Pyspark || Tamil || Column() Function || Dataframe

Column
pyspark column class represents a single column in a DataFrame
pyspark.sql.colum
class provides several functions to work with DataFrame
Manipulate the column values
Evaluate the boolean expression to filter rows
Retrieve a value or part of a value from a Data Frame column
one of the simplest ways to create a column class object is by using pyspark lit() sql function.

Example
from pyspark.sql.function import lit
col1 = lit("abcd")
print type(col1))


from pyspark.sql import SparkSession
from pyspark.sql.types import *
from pyspark.sql import dataframe
from pyspark.sql.functions import col
from pyspark.sql.functions import *

spark = SparkSession \
.builder \
.appName("Simple") \
.master("local[*]") \
.getOrCreate()

student_list_tup = [(10, 'Ria', '1000'), (11, 'Suma', '2000'), (12, 'kavi', '3000')]
schema = (StructType()
.add(field='Stu_id', data_type=IntegerType())
.add(field='Stu_name', data_type=StringType())
.add(field='Fees', data_type=StringType())
)
df_created = spark.createDataFrame(student_list_tup, schema)
df1 = df_created.withColumn("Country", lit('India'))
print(type(lit('Country')))
df1.show()

df1.select(col('Country')).show()
df1.select(df1.Country).show()
df1.select(df1['Country']).show()
df1.printSchema()

data = [('Ria', {'hair': 'black', 'eye': 'brown'}),
('Suma', {'hair': 'brown', 'eye': 'black'}),
('Kavi', {'hair': 'brown', 'eye': 'black'})]
schema = ['name', 'properties']
df2 = spark.createDataFrame(data, schema)
df2.show(truncate=False)
df2.printSchema()
df3 = df2.withColumn('Gender', lit('Female'))
print(type(lit('Gender')))
df3.show(truncate=False)
df3.select(df3.Gender).show(truncate=False)
df3.printSchema()
df3.select(col('properties.hair')).show()
df3.select(df3['properties.hair']).show()
df3.select(df3.properties.hair).show()