Query AD - Script Out of memory

Question

Query AD - Script Out of memory

T2000

SSC Eights!

Points: 894
More actions
September 11, 2014 at 9:11 am

#310871

I have a problem with a script to query ActDir. I have found some code to search what I need, but the script stops after some time, "Exception of type 'System.OutOfMemoryException' was thrown". Our AD contains about 4K accounts.
What it should do: for every useraccount, check who is member of act.dir group "GPR1_*" and count how many "GRP1_*" for that account.
(the get-qaduser is equivalent to get-aduser)
CLS
GET-QADUSER -SearchRoot 'mydomain/OU1/OU2' -sizelimit 0 | foreach {
$profile = GET-QADUSER -SearchRoot 'mydomain/OU1/OU2' -Name $_.name –Properties name,MemberOf
$MembGRP1 = $profile.MemberOf -like "*GRP1_*"
$CountGRP1 = $MembGRP1.Count
$inp = $_.Name + ";" + $CountGRP1
Write-Host $inp
}
Anyone who can help?
thx in advance!

Viewing 10 posts - 1 through 9 (of 9 total)

You must be logged in to reply to this topic. Login to reply

Gazareth One Orange Chip Points: 27737 More actions · Answer 1

There's some info here on configuring Memory usage in Powershell: http://blogs.technet.com/b/heyscriptingguy/archive/2013/07/30/learn-how-to-configure-powershell-memory.aspx

Beyond that, as a complete guess, try writing/appending to a file instead of the console?

T2000 SSC Eights! Points: 894 More actions · Answer 2

thx for the link & tip Gazareth, I will have look. Maybe output to file solves it...

Gary Varga SSC Guru Points: 82166 More actions · Answer 3

Allegedly, the ForEach-Object cmdlet processes each item in turn as it is passed through the pipeline whereas foreach generates the whole collection first. So the following might help alleviate memory issues:

# Formatted for forum readability

GET-QADUSER -SearchRoot 'mydomain/OU1/OU2' -sizelimit 0 |

ForEach-Object -Process

{

$profile

= GET-QADUSER

-SearchRoot 'mydomain/OU1/OU2'

-Name $_.Name

–Properties name,MemberOf

$MembGRP1 = $profile.MemberOf -like "*GRP1_*"

$CountGRP1 = $MembGRP1.Count

$inp = $_.Name + ";" + $CountGRP1

Write-Host $inp

}

I hope that this helps.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!

Jeff Moden SSC Guru Points: 1004362 More actions · Answer 4

T2000 (9/11/2014)
... but the script stops after some time, "Exception of type 'System.OutOfMemoryException' was thrown".

I'm curious... what kind of system did you run this on and what happened to the system when the code ran it out of memory?

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

Gary Varga SSC Guru Points: 82166 More actions · Answer 5

Jeff Moden (9/16/2014)
T2000 (9/11/2014)
... but the script stops after some time, "Exception of type 'System.OutOfMemoryException' was thrown".
I'm curious... what kind of system did you run this on and what happened to the system when the code ran it out of memory?

This is not in place of T2000 responding but from a .NET perspective (as that is all PowerShell is at a basic level) this exception is raised when the .NET runtime cannot allocate any more memory to the current AppDomain (read as process for simplicity). The stack gets unwound and memory is garbage collected.

The rest of the system should not be affected by this except from the lack of memory available as it is allocated and the processor time required by the Garbage Collector after the stack is unwound.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!

Jeff Moden SSC Guru Points: 1004362 More actions · Answer 6

Gary Varga (9/16/2014)
Jeff Moden (9/16/2014)
T2000 (9/11/2014)
... but the script stops after some time, "Exception of type 'System.OutOfMemoryException' was thrown".
I'm curious... what kind of system did you run this on and what happened to the system when the code ran it out of memory?
This is not in place of T2000 responding but from a .NET perspective (as that is all PowerShell is at a basic level) this exception is raised when the .NET runtime cannot allocate any more memory to the current AppDomain (read as process for simplicity). The stack gets unwound and memory is garbage collected.
The rest of the system should not be affected by this except from the lack of memory available as it is allocated and the processor time required by the Garbage Collector after the stack is unwound.

Thanks, Gary.

If this happened on an SQL Server, do you have any insight as to how SQL Server would be affected? Would it substantially slow SQL Server down or have some other detrimental effect? I'm asking because I've never actually seen a server run out of memory.

--Jeff Moden

RBAR is pronounced "ree-bar" and is a "Modenism" for Row-By-Agonizing-Row.
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.

Change is inevitable... Change for the better is not.

Helpful Links:
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)

Gary Varga SSC Guru Points: 82166 More actions · Answer 7

Depending on configuration, doesn't SQL Server allocate its memory on start up? If so then it is only the processor usage that could have any affect. Isn't that also configurable albeit at a OS level?

I would be surprised if it would take it down. Also surprising if there isn't a scenario where it would have an affect either.

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!

Gazareth One Orange Chip Points: 27737 More actions · Answer 8

I'm thinking that 'System.OutOfMemoryException' means you're out of the memory you're allowed, rather than the server itself is out of memory. In Powershell the default seems to be 1GB so it shouldn't be too detrimental to the server.

Presumably even if you set this to something higher than the total system memory then Windows server starts applying it's own hard limits.

Even on a low-memory server (4GB or so) then I think the worst knock-on effect for SQL server would be a reduced buffer pool size.

Of course, this is mostly speculation 🙂

schleep SSChampion Points: 12585 More actions · Answer 9

Here's a slightly different approach: start with the the matching groups, so you don't process users that aren't members of any of those groups.

Then count the groups for each of the distinct members.

I'm sure there is an infinitely faster way of counting the groups per user than what I have below...

cls

## Get the groups

$strFilter = "(&(objectCategory=group))"

$objDomain = New-Object System.DirectoryServices.DirectoryEntry

$objSearcher = New-Object System.DirectoryServices.DirectorySearcher

$objSearcher.SearchRoot = $objDomain

$objSearcher.PageSize = 2500 # Default is 1000. If you have less than 1000 groups, you can omit this line

$objSearcher.Filter = $strFilter

$Results = @()

$colProplist = "cn", "member"

foreach ($i in $colPropList)

{

$null = $objSearcher.PropertiesToLoad.Add($i)

}

#Write-date "Search - Groups"

$colResults = $objSearcher.FindAll()

foreach ($objResult in $colResults)

{

$objItemName = $objResult.Properties.cn

$objItemPath = $objResult.Path

[string]$strMember = $objResult.Properties.member

[string]$strName = $objItemName

if (($strName -like "GRP1_*"))

{

$a = $strMember -split "CN="

foreach ($item in $a)

{

$item0 = $item -split ","

$strUser = $item0[0]

if ($strUser -ne "")

{

$Results += ("$strName,$strUser")

}

# This is the slow part:

$Users = @()

$Users += $Results | %{$_ -split ","[1]}

$Users = $Users | Sort-Object -Unique

foreach ($User in $Users)

{

$Count = ($Results | ? {$_ -match $User}).Count

if (!($Count)) {$Count = 1}

Write-Host $User"; "$Count

}