How to pass a custom function inside a ForEach-Object -Parallel

Question

I can't find a way to pass the function. Just variables.

Any ideas without putting the function inside the ForEach loop?

function CustomFunction {
    Param (
        $A
    )
    Write-Host $A
}

$List = "Apple", "Banana", "Grape" 
$List | ForEach-Object -Parallel {
    Write-Host $using:CustomFunction $_
}

Either package your function in a module, or (re-)define it _inside_ the `-Parallel` block — Mathias R. Jessen, Apr 17 '20 at 14:03
As an aside: [`Write-Host` is typically the wrong tool to use](http://www.jsnover.com/blog/2013/12/07/write-host-considered-harmful/), unless the intent is to write _to the display only_, bypassing the success output stream and with it the ability to send output to other commands, capture it in a variable, redirect it to a file. To output a value, use it _by itself_; e.g., `$value` instead of `Write-Host $value` (or use `Write-Output $value`, though that is rarely needed). See also: the bottom section of https://stackoverflow.com/a/50416448/45375 — mklement0, Apr 17 '20 at 14:21

mklement0 · Accepted Answer · 2022-04-22T18:53:20.230

The solution isn't quite as straightforward as one would hope:

# Sample custom function.
function Get-Custom {
  Param ($A)
  "[$A]"
}

# Get the function's definition *as a string*
$funcDef = ${function:Get-Custom}.ToString()

"Apple", "Banana", "Grape"  | ForEach-Object -Parallel {
  # Define the function inside this thread...
  ${function:Get-Custom} = $using:funcDef
  # ... and call it.
  Get-Custom $_
}

^{Note: This answer contains an analogous solution for using a script block from the caller's scope in a ForEach-Object -Parallel script block.}

Note: If your function were defined in a module that is placed in one of the locations known to the module-autoloading feature, your function calls would work as-is with ForEach-Object -Parallel, without extra effort - but each thread would incur the cost of (implicitly) importing the module.
The above approach is necessary, because - aside from the current location (working directory) and environment variables (which apply process-wide) - the threads that ForEach-Object -Parallel creates do not see the caller's state, notably neither with respect to variables nor functions (and also not custom PS drives and imported modules).
- Update: js2010's helpful answer shows a more straightforward solution that passes a System.Management.Automation.FunctionInfo instance, obtained via Get-Command, which can be invoked directly with &. The only caveat is that the original function should be side-effect-free, i.e. should operate solely based on parameter or pipeline inputs, without relying on the caller's state, notably its variables, as that could lead to thread-safety issues. The stringification technique above implicitly prevents any problematic references to the caller's state, because the function body is rebuilt in each thread's context.
As of PowerShell 7.1, an enhancement is being discussed in GitHub issue #12240 to support copying the caller's state to the threads on demand, which would make the caller's functions available.

Note that making do without the aux. $funcDef variable and trying to redefine the function with ${function:Get-Custom} = ${using:function:Get-Custom} is tempting, but ${function:Get-Custom} is a script block, and the use of script blocks with the $using: scope specifier is explicitly disallowed.

However, ${function:Get-Custom} = ${using:function:Get-Custom} would work with Start-Job; see this answer for an example.
It would also work with Start-ThreadJob, where you could even do & ${using:function:Get-Custom} $_, because ${using:function:Get-Custom} is preserved as a script block (unlike with Start-Job, where it is deserialized as a string, which is itself surprising behavior - see GitHub issue #11698). However, it is unclear whether this behavior is supported by design, because it is subject to the same potential cross-thread issues noted above.

${function:Get-Custom} is an instance of namespace variable notation, which allows you to both get a function (its body as a [scriptblock] instance) and to set (define) it, by assigning either a [scriptblock] or a string containing the function body.

Thank you very much. It is not the cleaner solution I was hoping for but it works. Performance-side every iteration is basically instantiating a new function. It was like inserting the function inside the foreach but more cleaner visually, right? — smark91, Apr 17 '20 at 15:02
Glad to hear it was helpful, @smark91. The technique is primarily useful if you have a preexisting function that you want to use in the `ForEach-Object -Parallel` block; directly inserting the function definition is probably faster, though I'm not sure it makes much difference in practice. — mklement0, Apr 17 '20 at 15:10
This is all great for one-offs but if you have several modules imported, more functions defined, variables up in the air, essentially a whole house of cards going, it's too much trouble and too prone to error. Here's to hoping the PowerShell Core crew decide to make runspace copying an option. — Max Cascone, Apr 21 '21 at 18:48

js2010 · Answer 2 · 2020-11-25T17:23:42.120

5

I just figured out another way using get-command, which works with the call operator. $a ends up being a FunctionInfo object. EDIT: I'm told this isn't thread safe, but I don't understand why.

function hi { 'hi' }
$a = get-command hi
1..3 | foreach -parallel { & $using:a }

hi
hi
hi

edited Nov 25 '20 at 17:23

answered Nov 25 '20 at 16:12

js2010

17,785
4
45
50

1

Nicely done; while there could be thread-safety issues in principle, it should be fine as long as the function is side-effect-free, as your example is (i.e., as long as it it operates only on _parameters_ or pipeline input and doesn't rely on the caller's state (notably with respect to its variables)). – mklement0 Jul 05 '21 at 18:12

Shenk · Answer 3 · 2022-03-14T21:05:48.353

So I figured out another little trick that may be useful for people trying to add the functions dynamically, particularly if you might not know the name of it beforehand, such as when the functions are in an array.

# Store the current function list in a variable
$initialFunctions=Get-ChildItem Function:

# Source all .ps1 files in the current folder and all subfolders
Get-ChildItem . -Recurse | Where-Object { $_.Name -like '*.ps1' } |
     ForEach-Object { . "$($_.FullName)" }

# Get only the functions that were added above, and store them in an array
$functions = @()
Compare-Object $initialFunctions (Get-ChildItem Function:) -PassThru |
    ForEach-Object { $functions = @($functions) + @($_) }

1..3 | ForEach-Object -Parallel {
    # Pull the $functions array from the outer scope and set each function
    # to its definition
    $using:functions | ForEach-Object {
        Set-Content "Function:$($_.Name)" -Value $_.Definition
    }
    # Call one of the functions in the sourced .ps1 files by name
    SourcedFunction $_
}

The main "trick" of this is using Set-Content with Function: plus the function name, since PowerShell essentially treats each entry of Function: as a path.

This makes sense when you consider the output of Get-PSDrive. Since each of those entries can be used as a "Drive" in the same way (i.e., with the colon).

score 0 · Answer 4 · answered May 05 '22 at 18:30

If you're a pro, of course you added the -Parallel flag on purpose because you really needed parallel processing (so see the accepted answer)

Newbies, like me, might consider removing the -Parallel flag because you didn't realize the code you copied from somewhere else doesn't really need it.. and then your function calls just work like normal.

How to pass a custom function inside a ForEach-Object -Parallel

4 Answers4

Linked

Related