Spark Connect Dotnet November 2024 Where are we?
All Spark Connect Posts
Introduction There have been quite a few changes in the last couple of months and I just wanted to give a quick update on the current...
2024-11-16
11 reads
Ed Elliott
Spark Connect Dotnet Variant Data Type
All Spark Connect Posts
I recently published the latest version of the Spark Connect Dotnet library which includes support for the new Variant data type in Apache Spark 4.0 here....
2024-10-12
11 reads
Delta Lake from .NET in a Spark Connect gRPC world
All Spark Connect Posts
Code
What to do? At some point we will want to do something with delta lake and so I wanted to explore the options. Before we do...
2024-02-17
3 reads
Apache Spark from PHP - it is not just a .NET thing
All Spark Connect Posts
Code
Goal of this post I wanted to explore what the Spark Connect API looked like from other languages, I am not a php developer - I...
2024-02-10
59 reads
Implementing functions and more fun in Spark Connect using gRPC and .NET
All Spark Connect Posts
Code
Goal of this post The goal of this post is to look at creating a SparkSession and a DataFrame that will wrap the Range relation and...
2024-02-10
33 reads
Moving towards the DataFrame API using the Spark Connect gRPC API in .NET
All Spark Connect Posts
Code
Goal of this post So there are two goals of this post, the first is to take a look at Apache Arrow and how we can...
2024-02-05
70 reads
Exploring the Spark Connect gRPC API more
All Spark Connect Posts
Code
Goal of this post In this post we will continue looking at the gRPC API and the AnalyzePlan method which takes a plan and analyzes it....
2024-02-01
95 reads
Using Spark Connect from .NET
Introductory Ramble Spark Connect In July 2022, at the Data and AI summit, Apache Spark announced “Spark Connect,” which was a way of connecting to Apache Spark using the...
2024-01-31 (first published: 2024-01-17)
353 reads
Exploring the Spark Connect gRPC API
All Spark Connect Posts
Code
Goal of this post In the first two posts, we looked at how to run some Spark code, firstly against a local Spark Connect server and...
2024-01-26
254 reads
Using Spark Connect from .NET to run Spark jobs on Databricks
All Spark Connect Posts
Goal of this post This post aims to show how we can create a .NET application, deploy it to Databricks, and then run a Databricks job...
2024-01-24 (first published: 2024-01-23)
63 reads
Blogs
Focus on Core Skills
By Brian Kelley
Core skills depends on the position, but the point is that ensuring you have...
A New Word: Zverism
By Steve Jones
zverism – n. the wish that people could suspend their civility and indulge in...
SQL Server Source Control on a $0 Budget
By Kevin3NF
The Source Control Dilemma Every DBA has been there. Trying to keep track of...
Forums
Is GenAI Coming Faster Than You Think?
Comments posted to this topic are about the item Is GenAI Coming Faster Than...
Local Storage Options
Comments posted to this topic are about the item Local Storage Options
DeepSeek: What is new with this AI technology?
Comments posted to this topic are about the item DeepSeek: What is new with...
Question of the Day
Local Storage Options
I'm setting up a SQL Server 2019 instance and we are planning on using SMB storage for our database files. However, the file share isn't ready, so the idea is to use the \127.0.0.1dbfile as the location to start and then move these files to the remote server. Can I do this?
See possible answers