Answer by ValaravausBlack for Inferring Spark DataType from string literals
If you have String Literals written as DataType name, ie. "StringType" , "IntegerType" -use this function -def StrtoDatatype(str: String): org.apache.spark.sql.types.DataType = { val m =...
View ArticleAnswer by alno for Inferring Spark DataType from string literals
Yes, of course Spark has magic you need.In Spark 2.x it's CatalystSqlParser object, defined here.For example:import...
View ArticleAnswer by Sachin Tyagi for Inferring Spark DataType from string literals
Spark/Scala probably already have a helper/util method that will do this for me.You're right. Spark already has its own schema and data type inference code that it uses to infer the schema from...
View ArticleAnswer by gsamaras for Inferring Spark DataType from string literals
From scala, it doesn't seem you can do what you wish magically, check for instance this example:import com.scalakata._@instrument class Playground { val x = 5 def f[T](v: T) = v f(x) val y = "boolean"...
View ArticleInferring Spark DataType from string literals
I am trying to write a Scala function that can infer Spark DataTypes based on a provided input string:/** * Example: * ======== * toSparkType("string") => StringType * toSparkType("boolean") =>...
View Article
