make script more robust

Question

make script more robust

Bruin

SSCrazy

Points: 2058
More actions
April 8, 2022 at 1:25 pm
Go to Answer
#4012703
This script works, but I would like to add a few more things to the process, and need some help.
1) Search multiple folders
2) Search for multiple strings patterns (error,failed)
3) use a dynamic error log filename instead of hard coded in the export-csv
4) Email results.
```
Get-ChildItem `
        -Path "\\ftpsrv\logs\" | `
    Select-String -pattern "failed" | `
    Select-Object -Property Path,LineNumber,Line | `
    Export-CSV "C:\ftp_in\archive\errors.csv" -Append
```

Viewing 15 posts - 1 through 15 (of 20 total)

You must be logged in to reply to this topic. Login to reply

Mr. Brian Gale SSC-Insane Points: 24800 More actions · Answer 1

For requests 3 and 4, I would say google those. 3 is EASY as that is just a parameter. 4 is pretty easy too as you just need to call out the email functions in Powershell.

For number 1 and 2 it is a bit more work and other powershell experts will likely be able to help more, but for number 2, what I would do is just do a second search where you are looking for a second pattern.

For searching multiple folders, you would need multiple get-childitem calls UNLESS you mean recursively down from the root, in which case there is an argument for get-childitem to handle this.

For requirements 1 and 2, I would store your results in a variable and just append to the variable with each call to get-childItem and after you have the full result set, then export to csv.

The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!
I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system.

Bruin SSCrazy Points: 2058 More actions · Answer 2

The multiple folders are from different machines.

Mr. Brian Gale SSC-Insane Points: 24800 More actions · Answer 3

Then you will need multiple get-childitem calls and you will either need to append them all to the same CSV OR store it all in a variable (ie in memory) and dump it to disk after getting the variable full.

The above is all just my opinion on what you should do.
As with all advice you find on a random internet forum - you shouldn't blindly follow it. Always test on a test server to see if there is negative side effects before making changes to live!
I recommend you NEVER run "random code" you found online on any system you care about UNLESS you understand and can verify the code OR you don't care if the code trashes your system.

Bruin SSCrazy Points: 2058 More actions · Answer 4

That is my struggle multiple get-childitem calls and storing outputs to single variable to dump to csv.

Thanks.

Jeffrey Williams SSC Guru Points: 90303 More actions · Answer 5

For multiple servers\locations - define those in a separate file. Then - get the contents of that file and loop...

$match_array = @("failed","error");
$fileLocations = Get-Content "\\somelocation\somewhere\ListOfFolders.txt"
$fileLocations | % {
    $location = $_;
    $match_array | {
        $found_files = Get-ChildItem $location | ? {$_.Name -like "*$($_)*"};
    }
}

# if $found_files is not empty - send mail message
Send-MailMessage ...

Of course - this is completely untested code, but should give you an idea of how to approach the problem. The % symbol is short-hand for the foreach-object and the ? is short-hand for where-object. This would build a list of files in the $found_files variable - and from that you can pull all the file details you need on output.

Jeffrey Williams
“We are all faced with a series of great opportunities brilliantly disguised as impossible situations.”

― Charles R. Swindoll

How to post questions to get better answers faster
Managing Transaction Logs

Bruin SSCrazy Points: 2058 More actions · Answer 6

Thanks very much for reply, and I know you said un-tested code, but I was trying to get values for the $found_files to see

what was returned and trying to understand what this is telling me.

}

At line:5 char:20

+ $match_array | {

+ ~

Expressions are only allowed as the first element of a pipeline.

+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException

+ FullyQualifiedErrorId : ExpressionsMustBeFirstInPipeline

Jeff Moden SSC Guru Points: 1004329 More actions · Answer 7

Bruin wrote:

This script works, but I would like to add a few more things to the process, and need some help.
1) Search multiple folders 2) Search for multiple strings patterns (error,failed) 3) use a dynamic error log filename instead of hard coded in the export-csv 4) Email results.
Get-ChildItem `
        -Path "\\ftpsrv\logs\" | `
    Select-String -pattern "failed" | `
    Select-Object -Property Path,LineNumber,Line | `
    Export-CSV "C:\ftp_in\archive\errors.csv" -Append

I'm don't know enough about PowerShell to write code in it and so I can't actually help here.

If you have a moment, I AM curious, though. What is this code doing?

My guess is that you're identified a UNC to a share, are doing the equivalent of a filtered DIR looking for file names that have the word "failed" somewhere in the file name, and then writing the path and a couple of other things out a CSV on some local system that you're running all of this from.

What does "LineNumber" contain? Is that just some sequential line number created by ChildItem? And, what is in Line? Is that where the filename is actually present and, if so, is there other data that is also available in the Line?

Last but not least, you're saving whatever it is that's being built (would love to see a few lines from the CSV), you're appending it to an existing CSV. What do you intend to do with the contents of that CSV?

Like I said, I can't actually help here but I'd like to know much more about the problem that you're trying to solve so I can do a little research on how one might solve the problem in PoSh and also understand why you need to do this and store it in a CSV to begin with.

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

Bruin SSCrazy Points: 2058 More actions · Answer 8

I'm running thru folders where output from job processing happens and sometimes the product we use to run\schedule jobs

doesn't detect and error condition even though the SP or scripts output and error message. I want to save to csv to mail to me

any of these conditions so I can track down nightly processing to fix scripts or calling process.

Jeff Moden SSC Guru Points: 1004329 More actions · Answer 9

Bruin wrote:

I'm running thru folders where output from job processing happens and sometimes the product we use to run\schedule jobs doesn't detect and error condition even though the SP or scripts output and error message. I want to save to csv to mail to me any of these conditions so I can track down nightly processing to fix scripts or calling process.

Ah... Understood. I still need a couple of questions to be answered because I may have figured out a way to do all of this without having to fart around with a CSV file and a bunch of PowerShell loops, etc.

So where do the words "error" or "failed" show up? In the file name or somewhere in the file?
And you still haven't answered the question of what's in the file... What "fields" are in the file?

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

Bruin SSCrazy Points: 2058 More actions · Answer 10

Sample output from csv:

#TYPE Selected.Microsoft.PowerShell.Commands.MatchInfo

"Path","LineNumber","Line"

"\\ftpsrv\logs\EXTRACT.20741061-1.stdout","102","Authentication failed."

It shows the Path with filename, and gives me the line in the file it found a match then the MESSAGE.

Jeffrey Williams SSC Guru Points: 90303 More actions · Answer 11

So you are actually looking in each file for matching strings in the contents of the file. Okay - the same general principle applies....

Create a list of folders to check
Get the contents from the file containing the list of folders
Loop over the list of folders - use get-childitem on each folder to get matching contents into an object
output object

$match_array = "error", "failed";
$folders = Get-Content "File with list of folders to search";
$folders | % {
    $folder = $_;
    $match_array | % {
        $matched_items = Get-ChildItem $folder | Select_String -Pattern $_ | Select-Object -Property Path,LineNumber,Line;
        Export-CSV "your output file here" -Append -NoTypeInformation;
    }
}

Send-MailMessage ...

Or - you can use a traditional approach to looping. https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_foreach?view=powershell-7.2

Again - sample code only. It won't work as-is and must be modified to work in your environment according to your coding standards. There is plenty of help available online - and looping in PS is well documented.

Jeffrey Williams
“We are all faced with a series of great opportunities brilliantly disguised as impossible situations.”

― Charles R. Swindoll

How to post questions to get better answers faster
Managing Transaction Logs

Jeff Moden SSC Guru Points: 1004329 More actions · Answer 12

I don't know but it sounds like the source of the files never deletes the old files. With that, what are you doing about identical entries every time you run this?

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

Bruin SSCrazy Points: 2058 More actions · Answer 13

I would like to maybe add a date to the process to show which day the error refers too.

Jeff Moden SSC Guru Points: 1004329 More actions · Answer 14

While this is all very interesting, I have to ask the question... Have you actually called the software vendor to tell them of your problem and to see what they can do about it?

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)