After several tests the problem finally located, although the reason is very strange.
At first, it must be noted that my motherboard (P4G8X) is running in dual channel DDR configuration. This means that rising the FSB speed with standard DDR333 memories is highly possible keeping 100% stability. In addition, the AGP/PCI speed can be locked to default ratio, thus rising the FSB affects only the CPU and memory.
I've been using FSB155 instead of the standard 133MHz for months, without any problem. All benchmarks, games and application were rock stable at all time.
But, when I launch PicoScope, it reports communication failure when the FSB is set any higher than 146MHz.
This is strange, I really can't imagine how the parallel port can be affected by the FSB settings. Besides, as I mentioned earlier, other applications (such as VMware) are able to use the LPT port regardless of the FSB value.
It seems, the timings in the communication protocol of Pico sw. somehow very sensitive, would be good to check is there any way to improve this.
To test the maximum throughput of parallel port handled by Pico, I recorded 5000 samples @ 1ms, on two channels.
At FSB133 the recording time is 13790ms while at 146MHz it is 13463ms.
I've been playing with other settings as well (LPT mode is BIOS, LPT irq handling in WinXP) but they didn't make any difference.
To be honest, I expected much better throughput. This is 2x 372 samples per sec, which is 118 times slower than what a basic sound card can do!!
Because of this, I can't use PicoLog to capture and analyse longer non-periodic signals, like communication protocols.
I hope there will be an USB version of ADC, which could solve all the issues described above, and even more.
Ps. I don't know whether Rod's system has the same problem due to overclock, but restoring the factory values in bios seems to be a good idea. Anyway, I hope that it will be ok soon.