65kSample limit on text file size

Forum for discussing PicoScope version 6 (non-automotive version)
Post Reply
ctrawick
Active User
Active User
Posts: 11
Joined: Thu May 24, 2007 3:12 pm
Location: Atlanta, Georgia, USA

65kSample limit on text file size

Post by ctrawick »

I just received my 5204 yesterday, and after some struggles with the included software concluded that I needed the newer code. So I am running R6.0.9 beta.

I need to be able to post-process lots of data-- the 128MSample memory of the 5204 was the main selling point. Our post-processing tools include video chip simulators and Matlab/Simulink models. But I can only get 2^16 samples when I dump txt and csv files.

I realize that you may have envisioned exporting text to Excel, which is indeed limited to 65536 lines. But is there a way to turn off this limitation?

ziko
Advanced User
Advanced User
Posts: 1705
Joined: Fri Dec 01, 2006 10:03 am
Location: St Neots

Post by ziko »

Hi and thank you for your post.

After having a word with one of the software engineers here is his input

"There is no way to turn off the limitation at the moment. Although Excel influenced our choice of 2^16 samples, the main limitation is that a csv text file with 128 million lines in it just isn't feasible (we could be talking about gigabytes of data in a single file).

We are looking at other ways of exporting the entire data set future, this will probably include a simple binary format (would require a small amount of programming knowledge to use). If you need to capture a large amount of data, you can program with the driver directly, the driver SDK is being release very soon."

Hope this helps
Ziko

Technical Specialist

ctrawick
Active User
Active User
Posts: 11
Joined: Thu May 24, 2007 3:12 pm
Location: Atlanta, Georgia, USA

A happy medium would be good

Post by ctrawick »

Yes, a 128 megasample file would be very big. But as things stand, the biggest text capture I can write will fit on a 3.5" floppy disk!

What I would actually be capturing would likely be in the 20 megasample range-- 15 video frames at 24nS sample rate. The file would weigh in at about 450MBytes.

Having 128 megasample capture without the ability to export for post-processing means that right now the ONLY WAY to actually use the data captured is to surf through the waves on the Picoscope software. Not what we bargained for!

Binary would be fine also-- we'd just change our input routine. Or maybe you could give us a clue about the format of your native format file (*.pstdata). I tried to tease it apart, but it looks like you are compressing the sample data. Have you published info on this format anywhere? Could you?

ctrawick
Active User
Active User
Posts: 11
Joined: Thu May 24, 2007 3:12 pm
Location: Atlanta, Georgia, USA

Can I use R5.x with 5204?

Post by ctrawick »

I see in the "Save as TXT problem" thread that "PicoScope 5.x can still be used for this purpose..." until R6 gets there. I further note that there is a switch in the .ini file for earlier versions of your code which allows datafile compression to be turned off. So... will version 5.16.2 work with the 5204? Or is there a compression switch I can turn off on R6?

alan
Site Admin
Site Admin
Posts: 195
Joined: Thu May 25, 2006 8:02 am

Post by alan »

Hi,

Unfortunately the .ini switches you mention are specific to PicoScope 5 so will not help here. We need to work out the best way of helping you with this.

Our current plans are to release the API / SDK in the next few days (early in June) with examples in C++ etc. Will this be of help to you? (From your posts about tearing apart our file format you obviously have software skills!).

Longer term we plan to have a file export "wizard" to help users save the potentially huge amounts of data the device can generate, we may be able to move this up in the priority list if using the API is not an option.

A middle ground may be to include an option not to compress the data, but we will be reluctant to publically release details of the file format as this may change over time - it makes more sense for customers to use the export wizard as above.

As I say we want to do our best to help you - please can you let us know which option sounds best to you.

Thanks
Alan

ctrawick
Active User
Active User
Posts: 11
Joined: Thu May 24, 2007 3:12 pm
Location: Atlanta, Georgia, USA

API/SDK versus Export Wizard

Post by ctrawick »

In terms of work at this end, the Export Wizard looks much more attractive-- the programming environment in which we do our postprocessing is on the Linux side, while the PicoScope is running on a Windows machine. So if asking for the Wizard means it would arrive in 4-6 weeks, that's what we would do. (A raw binary dump will serve.)

Having said that, I'd like to see the API/SDK stuff when it becomes available.

Thanks!

alan
Site Admin
Site Admin
Posts: 195
Joined: Thu May 25, 2006 8:02 am

Post by alan »

Hi,

You mention you use Linux as your programming enviroment. Would a Linux driver be a better solution to you than exporting the data from PicoScope 6?

If so please send me a message using the PM (private message) button below. Our Linux driver is not quite ready for the prime time yet - the documentation is a bit sparse but it the driver works OK.

If on the other hand the export from PicoScope is more important then I will see if we can move this up the priority list.

Alan

Anders Westergaard
User
User
Posts: 8
Joined: Mon Sep 04, 2006 12:30 pm

Export wizzard

Post by Anders Westergaard »

Hi - Sorry to intervean your thread, but the export wizzard is a major priority for us too I was actually considering to return the scope to the dealer as we have no benefit from it at all considering the current state of the software.
As I stated in an early thread I too need to post process a lot of data and I need Pico scope to be sure that i have triggered at the correct data package. So therefore the wizzard would be a very much appreciated form our point of perspective.

Br Anders

alan
Site Admin
Site Admin
Posts: 195
Joined: Thu May 25, 2006 8:02 am

Post by alan »

Hi,

As it stands we will have a binary export feature and / or an option to save in Matlab format (essentially the same) for the release due out at the end of June.

We are looking into whether we can delay the release due out any day now to include this feature early.

Details of what makes it into the next release of PicoScope 6 will be in our monthly email newsletter which will go out some time next week.

Regards
Alan

ctrawick
Active User
Active User
Posts: 11
Joined: Thu May 24, 2007 3:12 pm
Location: Atlanta, Georgia, USA

MATLAB matrix file output in version 6.0.10

Post by ctrawick »

After having no luck with the SDK, I was very happy to see that you'd released the promised new version of Picoscope 6. And that it supports export in the MATLAB binary matrix format. This is an excellent choice of formats, working out to 4 bytes/sample/channel, and very portable.

While I will probably be back with more questions, count me as a satisfied customer.

Regards,
Charles Trawick

Leo Jensen
User
User
Posts: 2
Joined: Thu Aug 02, 2007 12:33 pm
Location: Denmark

Need +65K export to .csv-file

Post by Leo Jensen »

Hi

We have a similar issue here: Picoscope 3423, Software 6.0.8

We capture data with 2 seconds per screen. We need to export appr. 8 seconds of data, but get only the first 65k from every screen, that is 262 msec at the moment.
We would like to do the calculations in Excel, 2007-version should be able to handle 1.000.000 rows.
To do this we need to export more than 65k, and we need to be able to reduce the samplerate. I can find an option, that saves 40 captures pr. second, next step is unlimited. We need something in between.
BR

Leo Jensen - Dino A/S - Denmark
www.dino.dk

ziko
Advanced User
Advanced User
Posts: 1705
Joined: Fri Dec 01, 2006 10:03 am
Location: St Neots

Post by ziko »

Hi , I am happy to say that there are definite plans to increase the limit to become 2^20 points (approx 1 million) for each channel in the near future.

Kind regards
Ziko

Technical Specialist

Leo Jensen
User
User
Posts: 2
Joined: Thu Aug 02, 2007 12:33 pm
Location: Denmark

Until then...

Post by Leo Jensen »

Until then, how do I export a selected part of the sample window ?
At the moment I know how to save the first 65k, but if the interesting data is later in the sample, I want to save that section.
BR

Leo Jensen - Dino A/S - Denmark
www.dino.dk

ziko
Advanced User
Advanced User
Posts: 1705
Joined: Fri Dec 01, 2006 10:03 am
Location: St Neots

Post by ziko »

Hi the next release with this addition will be added very soon, within the next week or 2.

Kind regards
Ziko

Technical Specialist

Post Reply