Save As CSV for 2 million samples

[colindavidfoster]

If I manage to sample 2 million samples the export to CSV the file only contains half of the samples.

Anybody else having the same problem?

PicoScope 6.4.64.0
Model: PicoScope 2203

[ziko]

Hi PicoScope limits the number of samples to 1million per channel on the csv file, this is because the files would be quite large.

A workaround is to save the file into a .mat format (binary format) this is not limited.

Please email support@picotech.com for more information.

Kind regards

[colindavidfoster]

Hmmm, 50Mb of text file is nothing these days.

To be able to record 2 million (or more) samples and not be able save that data in all of the suggested formats on the grounds of too much data is, IMHO, a pretty poor excuse. The whole point of the (Pico)scope is to record data and the whole point of saving is to preserve all the data. Suppose Microsoft Word only saved the first half of your novel when exported it.

Anyway, enough ranting.

The suggestion of using the .mat format seems a pretty good alternative. I'll have to rework my Perl scripts, but that's not a problem. It may even turn out to be faster.

Cheers

[alan]

I think CSV was limited to 1million samples as above this caused Excel to crash. Tech support got fed up with complaints about the CSV file not working when there was no problem with the CSV file itself.

If someone wants to use CSV to export large files into different programs / their own programs then we can look at the limit again but as mentioned above its far more efficient to save the data as a binary file.

Regards
Alan

[jokkebk]

Hello,

I just spent two hours tweaking PicoScope software settings with 5 MS captures, because no matter what, I could not get more than 1024^2 (1M) samples to the text formats.

Such an undocumented (is it documented somewhere?) feature is a terrible idea, I was just going to post a new problem topic to the forum when I by chance googled "picoscope csv maximum" and found this thread. Could you please indicate it more clearly in the save dialog, or add an option to Preferences to change this? I'm parsing the files in C so any limitations in Excel do not mind me.

I guess I'll take a look at the Matlab format then. If only I could get those two hours of my life back.

Joonas

[Hitesh]

Hi Joonas,

There is some text in the Help file under 'CSV files, exporting ' that states

The 1 million values per channel limit prevents excessively large files being created.

I've put in a request for indication in the Save As dialog or preferences.

Best wishes,

[wcressman2k]

Please please change this limitation. I use JMP as well as various scripting tools to post-process scope data from other more expensive scopes and it seems very odd to me that this "Deep Memory" scope that can only output a tiny fraction of it's memory in the most commonly used data format in the world (CSV).

Excel is no longer limited in the number of records it can contain. If you want to protect yourself from complaints posted by from users who have old Excel, how about defaulting to the 1M record limit but having a setting where a user can click OK to a warning and then get the full sample depth in a CSV output?

thanks
Bill

[Hitesh]

Hi Bill,

I have passed on your comments to members of the Development Team through our system.

We will provide an update as soon as we get an update on this.

Best wishes,