How do we test ETL pipelines? Part one unit tests
Why do we bother testing? Testing isn’t an easy thing to define, we all know we should do it, when something goes wrong in production people shout and ask...
2019-07-25
Ed Elliott
spark-dotnet examples - reading and writing csv files
How do you read and write CSV files using the dotnet driver for Apache Spark?
I have a runnable example here:
https://github.com/GoEddie/dotnet-spark-examples
Specifcally:
https://github.com/GoEddie/dotnet-spark-examples/tree/master/examples/split-csv
Let’s take a walkthrough of the demo:
Console.WriteLine("Hello Spark!"); var spark...
2019-07-15
31 reads
spark-dotnet examples - reading and writing csv files
How do you read and write CSV files using the dotnet driver for Apache Spark?
I have a runnable example here:
https://github.com/GoEddie/dotnet-spark-examples
Specifcally:
https://github.com/GoEddie/dotnet-spark-examples/tree/master/examples/split-csv
Let's take a walkthrough of the demo:
Console.WriteLine("Hello Spark!"); var spark...
2019-07-15
1 reads
spark-dotnet how does user .net code run spark code in a java vm
Apache Spark is written in scala, scala compiles to Java and runs inside a Java virtual machine. The spark-dotnet driver runs dotnet code and calls spark functionality, so how...
2019-07-14
7 reads
spark-dotnet how does user .net code run spark code in a java vm
Apache Spark is written in scala, scala compiles to Java and runs inside a Java virtual machine. The spark-dotnet driver runs dotnet code and calls spark functionality, so how...
2019-07-14
2 reads
Spark and dotnet in a single docker container
I really like the new dotnet driver for Spark because I think it makes spark more accesable to devs who might not know pythpn or scala.
If you want to...
2019-07-11
14 reads
Spark and dotnet in a single docker container
I really like the new dotnet driver for Spark because I think it makes spark more accesable to devs who might not know pythpn or scala.
If you want to...
2019-07-11
Where do you do your database development? Hopefully not production
Here are three scenarios, if you work with SQL Server either as a provider of database environments (DBA) or a consumer of database environments for your application (developer) then...
2019-07-05
3 reads
Where do you do your database development? Hopefully not production
Here are three scenarios, if you work with SQL Server either as a provider of database environments (DBA) or a consumer of database environments for your application (developer) then...
2019-07-05
1 reads
As a DBA, how do I offload some of my work?
I don’t have time for this “I am a DBA, I am busy, too busy. Developers keep pushing changes to production without me reviewing the code, and now I...
2019-06-13
267 reads
Blogs
Simple Talks Episode 9–Data masking and subsetting
By Steve Jones
The episode on data masking and subetting is out. You can see it here:...
T-SQL Tuesday #180: Avoid Perfect for Good Enough
By Brian Kelley
I'm listening to Effortless by Greg McKeon (link to author's page) through Audible.com. He...
Book Review – Humanizing Data Strategy by Tiankai Feng
This book was making its rounds on social media, and the concept seems interesting...
Forums
SQL server help (converting rows to columns)
I am using the following sql to convert rows into columns for each pid,...
Ingesting Data From REST API endpoints: Data Engineering with Fabric
Comments posted to this topic are about the item Ingesting Data From REST API...
Separate Reads
Comments posted to this topic are about the item Separate Reads
Question of the Day
The Hash Join I
What are the two inputs called to a hash join operation in SQL Server? (choose 2)
See possible answers