In iTest we can execute scripts by GUI and command line - iTestRT . In both the cases we need sufficient memory allocation for execution if we don't have sufficient memory we hit OOM (Out of Memory) issue.
For GUI we need to add more memory to Xmx clause in iTest.ini file, but for iTestRT we need to increase the Xmx value in the itestrt.bat file inside the iTestRT folder.
NOTE: Both the iTest.ini file and iTestRT folder are located inside iTest installation folder.