It finally works !
A month ago I had almost given up - 2 months of running against a wall and I still was not able to introduce the "_vmware_user_" pseudo NT-acount that I needed to actually start a VM. Now I know it was worth the time.
The trick was simply to export the SAM-hive at the right time and to merge it into BartPE at the right time.
I never seen any other plugin make use of the SAM-hive - the pebuilder is not even able to process this kind of registry stuff - so I needed some time until I started looking into non-obvious directions . Another tricky aspect was the sequence in which the different ingredients have to be loaded at boot-time.
Anyway - now I got a way to build a working version based on a 2k3 build with some kind of explorer shell. It is repeatable on different builds, different methods to load network and different explorer-shells. Here is a short outline of the required actions: make a clean install of VMware into R:\ws452 - use installrite to capture classes. Hand edit the registry entries for the drivers. Create a profile-dir with settings for VMware - all users and default users. Manually copy installer dir into build dir as the pebuilder won't accept the dir name. Setup a config.ini and a preferences.ini that fit this special working conditions. Create a BartPE with a RAM-drive as large as possible. Install classes and drivers via *.inf. At boot-time register some *.ocx files, register the 3 dlls needed, copy profiles to RAM-drive, add System-acount to administrator group and finally merge the _vmware_user_ and the group settings.
Well - the working speed is slow compared with a standard install. But
I don't think that this argument counts much in those situations where
a VMware-LiveCD would be useful.
Desaster recovery - you can start your VMs when your host-system is unusable. You can extract files from Linux or BSD VMs and so on.
P2V - you can easily find out if a target-system will run in VMware - just go to the target, boot it from BartPE and run the target-system as a VM. Create raw-disks to convert them into vmdk-files.
Forensic - boot a target-system once and be able to read any filesystem natively just by starting small "translator VMs"
Not all functions are fully working at this time - the biggest problem - passing the check of the _vmware_user_ - is solved and everything else will be simple.
Don't expect a plugin tomorrow - as this will become the most complicated plugin I have ever seen. Actually I use a mixture of *.infs, editing build dir and merging *.reg-files at runtime. Now it runs on a 2k3 build with XPE. I will try XPsp1 as well but I see no point in adopting it for XPsp2 at this time.
Amazingly it doesn't matter if you put the VMware-dir in the ram-drive or onto the CD. I think the option to put it into program-dir looks more promising so I will follow that road.
Another idea that I will try is to load the complete BartPE with VMware into RAM so that I can change CDs and insert media with prebuilt VMs.
It is possible to install Virtual PC into the same BartPE build.
My current build includes VMware 4.5.2, Virtual PC, SSH-client with user-keys (client from ssh.com), an older demo-version of X-win 32, various remote control clients like ultravnc, Citrix-clients and some disk-image mounting tools like file-disk, Ken Katos vdk.exe and some P2V scripts. I hope to replace Virtual PC by Virtual Server and to add Svista-serenity as well. I don't have access to these versions now so that may take some time but I am sure that it is possible.