How do we test ETL pipelines? Part one unit tests
Why do we bother testing? Testing isn’t an easy thing to define, we all know we should do it, when something goes wrong in production people shout and ask...
2019-07-25
43 reads
Ed Elliott's Sql Developer Blog
How do we test ETL pipelines? Part one unit tests
Why do we bother testing? Testing isn’t an easy thing to define, we all know we should do it, when something goes wrong in production people shout and ask...
2019-07-25
spark-dotnet examples - reading and writing csv files
How do you read and write CSV files using the dotnet driver for Apache Spark?
I have a runnable example here:
https://github.com/GoEddie/dotnet-spark-examples
Specifcally:
https://github.com/GoEddie/dotnet-spark-examples/tree/master/examples/split-csv
Let’s take a walkthrough of the demo:
Console.WriteLine("Hello Spark!"); var spark...
2019-07-15
31 reads
spark-dotnet examples - reading and writing csv files
How do you read and write CSV files using the dotnet driver for Apache Spark?
I have a runnable example here:
https://github.com/GoEddie/dotnet-spark-examples
Specifcally:
https://github.com/GoEddie/dotnet-spark-examples/tree/master/examples/split-csv
Let's take a walkthrough of the demo:
Console.WriteLine("Hello Spark!"); var spark...
2019-07-15
1 reads
spark-dotnet how does user .net code run spark code in a java vm
Apache Spark is written in scala, scala compiles to Java and runs inside a Java virtual machine. The spark-dotnet driver runs dotnet code and calls spark functionality, so how...
2019-07-14
7 reads
spark-dotnet how does user .net code run spark code in a java vm
Apache Spark is written in scala, scala compiles to Java and runs inside a Java virtual machine. The spark-dotnet driver runs dotnet code and calls spark functionality, so how...
2019-07-14
2 reads
Spark and dotnet in a single docker container
I really like the new dotnet driver for Spark because I think it makes spark more accesable to devs who might not know pythpn or scala.
If you want to...
2019-07-11
14 reads
Spark and dotnet in a single docker container
I really like the new dotnet driver for Spark because I think it makes spark more accesable to devs who might not know pythpn or scala.
If you want to...
2019-07-11
Where do you do your database development? Hopefully not production
Here are three scenarios, if you work with SQL Server either as a provider of database environments (DBA) or a consumer of database environments for your application (developer) then...
2019-07-05
3 reads
Where do you do your database development? Hopefully not production
Here are three scenarios, if you work with SQL Server either as a provider of database environments (DBA) or a consumer of database environments for your application (developer) then...
2019-07-05
1 reads
Blogs
Spark Connect Dotnet November 2024 Where are we?
By Ed Elliott
All Spark Connect Posts Introduction There have been quite a few changes in the last...
A New Word: Ochisia
By Steve Jones
ochisia – n. the fear that the role you once occupied in someone’s life...
Create a Numbers Table in Power Query
This is a quick blog post, mainly so I have the code available if...
Forums
Writer methods in PowerQuery (LOL)
I'm using PowerQuery in Excel on my desktop. (Yeah, I know you can do...
Need help with GROUP BY...
Here's my data... I am trying to return total days in status for each...
Need help with GROUP BY...
Here's my data... I am trying to return total days in status for each...
Question of the Day
The Density Vector
When building statistics, there is the concept of density that refers to the duplicates in a table. How is this calculated?
See possible answers