Win10 PS6 and PS4444 Zombie Lockups

[bboyes]

So I am starting a new thread since the current issue does not seem to be a memory leak problem, but different. I call it a "zombie lockup". Upon arriving at the site, here is what I found today, with some notes about other similar zombie lockups in the past 48 hours:

• The PS6 application 6.13.15 can be maximized on screen, some controls partly are accessible, such as hovering over the stop button. But nothing actually works. No adjustments to range or scale work. The "running" indication is active and cannot change to "stopped". File menu does not work, nor can the application be closed with the upper-right corner controls. The only way to terminate is with Task Manager
• Sometimes the PS4444 itself does not respond when I restart the PS6 application: PS6 tells me no device is available. I have to unplug and replug USB to get the scope hardware to respond.This happens about half the time when there is a zombie lockup. Is the PS4444 hardware or firmware defective?
• in the event logs there were no application events but there was a non-error hardware entry, see below
• Task Manager showed normal memory usage (34% total, about 200 MB for PS6 if I recall)
• PS6 trace file and config files attached. The trace file shows the application unexpectedly closed the night before Apr 29 at 23:17, less than an hour after I left the site. Yet the application screen was still displaying the next day Apr 30 at 14:46 when I zipped everything up? What is really going on here?
• Windows event viewer shows an event 51, error writing to disk (I can't open that file right now on this Linux desktop PC). Event file attached.So I went to the disk menu to run a diagnostic scan and windows told me I didn't need to because there were no errors. I ran it anyway: no errors.
• I restarted the application. The pre-trigger data starts streaming for a few seconds, then stops. Is this normal? Should streaming continue without stopping (with the PS3425 and PS6 6.13.15 on another identical notebook PC, there is no pretrigger streaming, just an empty screen while waiting for a trigger).

Now I am away from the remote site and there is no new data in the cloud... so is the application running or not?

I may go out again today and swap out the PS4444 PC for another. We *really* need to get the PS4444 working reliably in the field. The PS3425 system is working for days or weeks at a time; why is the PS4444 on identical hardware so less reliable?

I could swap the two PCs and run the PS4444 on the one now running the PS3425. Recall that in our office running a 24/7 test with the PS4444 monitoring its rear panel square wave and alarming/capturing, we saw a memory leak bug. Now that the PS4444 is in the field we seem to be seeing different issues...

Should I try the interim release PS6 6.13.20 again?

[Martyn]

I see that you are writing files to a GoogleDrive on the PC which I guess syncs to the cloud on a regular basis. I will see if I can factor this in to the testing, it is the type of external action that can potentially cause issues.

[bboyes]

Yes we are using Google Backup and Synch. It has been working fine with PS6.3.15 and PS3425. We tested it in the office first with the PS4444, synching to Google Drive via a fast cable modem connection. In the field we are stuck with a slower 4GLTE modem, but we have an Ethernet switch so it should appear the same to the connected PC. We did do testing in the lab with the 4GLTE modem and all the same settings before we moved to the field. Again, the PS3425 is pretty stable. The PS4444, similar PC and same PS6 3.15, is flaky.

The PS4444 not being recognized by a freshly restarted PS6.3.15 seems to point the finger at the PS4444 firmware or its USB interface, yes?

[bboyes]

Also why is the pre-trigger data streaming (or not) different on PS3425 vs PS4444?

[bboyes]

Going through every Windows power option. I set HDD to never turn off, but as all these systems have SSDs I don't think this setting means anything. Turning off display should not matter, right? I have PCs set to never sleep when plugged in, and do nothing when lid closed. This lets me minimize the app, lock the screen/keyboard. and close the lid. This all has been working in testing in the lab (minimize app, lock keyboard, close lid, walk away, screen turns off after 30 min, PC never turns off) and in the field with the PS3425.

[Martyn]

With respect to the streaming check that Tools->Preferences->Sampling->SlowSamplingTransition is set the same as this will affect whether the trace is drawn to the trigger point or not when waiting for a trigger.

Are you running the 4444 with the power supply plugged in, or just using the power from the USB port ? and is it USB2 or USB3 ?

[bboyes]

Yes we are using the PS4444 power supply. Everything in the field is also on a largish UPS. In addition the notebook PCs have their own batteries of course.

Aha I apparently don't have those Tools->Preferences->Sampling->SlowSamplingTransition set the same. Will try to get out there to fix that today. I am not sure I understand that setting. I'd prefer to NOT have the screen streaming prior to a capture just to cut the data flow and reduce the chance of issues related to that. I'll check in the manuals... how do I optimize that setting for a given sample rate and time scale?

Every trip to the field is at least 2 hours roundtrip not including prep time and requires official customer accompaniment for site access. I have made four so far this week and it's taking a toll. But we are also into the damage season and in fact are seeing more suspicious electrical events than a month ago, so there is a big incentive to get to the root cause of our customer's problem, and for that we need these instruments. The PS4444 and PS3425 have some unique abilities, specific to this task, which is why are are using them.

The ThinkPad 13 systems have USB3. The Yoga 11e has USB2.

I swapped in the Yoga 11e last night and thought I was losing my mind (still maybe true). I started with PS 6.13.20 but it froze up immediately so uninstalled and back to 6.13.15. The PS4444 would immediately start up and trigger continuously, due to old default .pssettings files I now realize, and it was like wrestling an octopus to try to get it to stop so that I could configure it properly.

I'd like to have a phone call and maybe an email link to try to expedite getting to the bottom of this. This forum is cumbersome and there is a big time shift between us. Some things I think we could clear up in a few minutes with an interactive live phone call. And I'd like to be able to speculate on causes and share customer information which I don't feel free to do on this forum. I do appreciate your support thus far.

Sincerely
Bruce

[bboyes]

Also last night a couple of times when I had to terminate the app with Task Manager, when I restarted it, it could not connect to the PS4444. I had to unplug the USB and try again. The same thing happened with two different notebook PCs.

Also a couple of times when PS6 would freeze up the cursor changed to the crossed arrows whenever hovering over the PS6 application window. At those times nothing was actually usable in the interface and it had to be killed with TM.

[bennog]

Every trip to the field is at least 2 hours roundtrip not including prep time and requires official customer accompaniment for site access. I have made four so far this week and it's taking a toll

If you are syncing to google drive with 3g. Why are you not using something like teamviewer to acces the laptop remotely ? (then you can have a faster response on tests and the root cause of the problem will be found sooner)

Also the sync with google drive on 3g is slow, so what if the sync process starts while picoscope software is writing to the file and this causes some locking conflict. If this is the cause of your problem then you will probably see this 1 in 1000 on a fast internet connection but e.g. 1 in 5 on a slow 3g connection.

Benno

[Martyn]

Just make sure that

Tools->Preferences->Sampling->SlowSamplingTransition

is greater than the Collection Time setting you are using, and then the scope will collect to it's memory and only transfer when the scope triggers.

[Martyn]

You already have an email ticket, I will respond so that you have it in your inbox.

[bboyes]

@bennog We have 4GLTE via modem. I am not familiar with Teamviewer. I am familiar with join.me... We just want to get captured files. This is a metered data connection so how would sending screens across the data connection help? You are saying instead of pushing the data, just save it locally and then retrieve it later using a remote control app like Teamviewer? That's an interesting idea if we didn't burn up a lot of data trying to refresh screens when we don't care about that. Also an interesting idea that maybe the larger 4444 data files start to synch but then PS6 crashes when Google is trying to synch the file? Does PS6 lock the file until finished so any other app will hold off? It's something to consider... I guess we could have the 4444 save files locally and just use the cloud for smaller 3425 files, then retrieve the 4444 files once every few days... All these apps like join.me are meant to have a person on each end to OK sharing the screen and restore a connection. We don't have a person on the end at the remote site.

[bboyes]

File lock clashing could explain GUI becoming unresponsive and in a zombie state?
Also while I am there I can manually trigger 30-40 MB captures, they synch fine. Also one day I had the trigger set too sensitive. Each train passing would trigger five captures, that was 400 in one day. They all came through fine, and were still doing so when I got on site. It seems to fail only when waiting a long time (several hours) for a trigger, and only the 4444. The 3425 seems to have none of these issues. Does that give an idea of the root cause?

[bennog]

if you start teamviewer in te laptop and node the number and pincode then you can leave it alone and login remotely using the code and pin.

I suggest first try it somewhere where you can acces both machines (e.g. at home and at the office)

We use a little program called call_benno (sorry for the name) I developed this almost 20 years ago it was designed to traverse various NAT routers and firewalls (same solution as teamviewer does) but it can do a lot more than teamviewer can do .

Benno

[bboyes]

Is call_benno something that we should also be using?