When iinvoking PowerShell scripts winrm connection is created. MaxMemoryPerShellMB parameter controls the maximum memory that can be used while executing the script on the remote machine.
I would suggest to figure out which PS script is executed when the erro appears and execute directly from PS console. Then check how much memory does it consume.