How do you read and write CSV files using the dotnet driver for Apache Spark?
I have a runnable example here:
https://github.com/GoEddie/dotnet-spark-examples
Specifcally:
https://github.com/GoEddie/dotnet-spark-examples/tree/master/examples/split-csv
Let's take a walkthrough of the demo:
Console.WriteLine("Hello Spark!"); var spark = SparkSession .Builder() .GetOrCreate(); We start with the obligatory “Hello World!", then we create a new SparkSession.
//Read a single CSV file var source = spark .Read() .Option("header", true) .Option("inferShchema", true) .Option("ignoreLeadingWhiteSpace", true) .Option("ignoreTrailingWhiteSpace", true) .