Blob Storage automated downloads

Question

Blob Storage automated downloads

Brandie Tarvin

SSC Guru

Points: 173080
More actions
November 22, 2024 at 7:44 pm

#4487916

Dipping my toes into the waters of Azure and of course before I get past a few introductory videos, I'm tasked with automated a download of files from Azure Blob Storage to a server SAN drive.
I've downloaded Azure Storage Explorer and now I'm ready to power-learn, but I have no idea where to start. Any thoughts or reference links to where I can go?
The first thing I found was AZCopy, but as that's yet another tool to download and not currently approved by the corporate software deities, I need another path. Also, I can't figure out how to automate that one either.
Brandie Tarvin, MCITP Database AdministratorLiveJournal Blog: http://brandietarvin.livejournal.com/[/url]On LinkedIn!, Google+, and Twitter.Freelance Writer: ShadowrunLatchkeys: Nevermore, Latchkeys: The Bootleg War, and Latchkeys: Roscoes in the Night are now available on Nook and Kindle.

Viewing 8 posts - 1 through 7 (of 7 total)

You must be logged in to reply to this topic. Login to reply

Steve Jones - SSC Editor SSC Guru Points: 736494 More actions · Answer 1

Microsoft has created and supports the Az powershell module. If you r company doesn't approve of this, that's strange. A lot of what works well with Azure is PoSh or CLI based tools from MS. Azure Storage Explorer is interactive, and you can learn more here:

For AZCopy, this might help:

# Define the path to AzCopy executable
$azCopyPath = "C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy\AzCopy.exe"

# Define the source and destination
$source = "https://<your-storage-account-name>.blob.core.windows.net/<your-container-name>/<your-blob-name>"
$destination = "d:\downloads\<your-blob-name>"

# Define the SAS token (if required)
$sasToken = "<your-sas-token>"

# Construct the AzCopy command
$azCopyCommand = "$azCopyPath copy '$source?$sasToken' '$destination' --recursive"

# Execute the AzCopy command
Invoke-Expression $azCopyCommand

Grant Fritchey SSC Guru Points: 398909 More actions · Answer 2

Just know that moving data OUT of Azure costs money.

"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt

Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning

Scott Mildenberger Ten Centuries Points: 1356 More actions · Answer 3

I would look at Logic Apps, they are an easy way to script tasks. You will have to setup networking to allow it to on-premise resources are setup an on-premise data gateway which runs on an on-premise server and allows Azure apps to get to on-premise items. For moving data another option is Azure Data Factory - think similar functionality as SSIS.

Brandie Tarvin SSC Guru Points: 173080 More actions · Answer 4

Thanks, Everyone. Sorry for the slow response. I've been slammed with last moment of the year work.

Steve, even Microsoft stuff has to go through a testing / peep process here. I'll poke around and see if anyone has done that work yet or if it's going to fall on me. I just know that the tool isn't widely available for just anyone from our internal software controls.

I appreciate the input from all of you and will see what I can do.

Brandie Tarvin, MCITP Database AdministratorLiveJournal Blog: http://brandietarvin.livejournal.com/[/url]On LinkedIn!, Google+, and Twitter.Freelance Writer: ShadowrunLatchkeys: Nevermore, Latchkeys: The Bootleg War, and Latchkeys: Roscoes in the Night are now available on Nook and Kindle.

Brandie Tarvin SSC Guru Points: 173080 More actions · Answer 5

Grant Fritchey wrote:

Just know that moving data OUT of Azure costs money.

I thought just LOOKING at Azure cost money! @=)

Brandie Tarvin, MCITP Database AdministratorLiveJournal Blog: http://brandietarvin.livejournal.com/[/url]On LinkedIn!, Google+, and Twitter.Freelance Writer: ShadowrunLatchkeys: Nevermore, Latchkeys: The Bootleg War, and Latchkeys: Roscoes in the Night are now available on Nook and Kindle.

Grant Fritchey SSC Guru Points: 398909 More actions · Answer 6

Brandie Tarvin wrote:

Grant Fritchey wrote:
Just know that moving data OUT of Azure costs money.
I thought just LOOKING at Azure cost money! @=)

Oh, it's not that bad. They'll let you look. Just get your wallet out before you touch.

"The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood"
- Theodore Roosevelt

Author of:
SQL Server Execution Plans
SQL Server Query Performance Tuning

Jeff Moden SSC Guru Points: 1004409 More actions · Answer 7

Brandie Tarvin wrote:

...even Microsoft stuff has to go through a testing / peep process here.

From the looks of things, MS hasn't done a good job on QA since 2017.

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)