Save As CSV for 2 million samples

Post any questions you may have about our current range of oscilloscopes
Post Reply
colindavidfoster

Save As CSV for 2 million samples

Post by colindavidfoster »

If I manage to sample 2 million samples the export to CSV the file only contains half of the samples.

Anybody else having the same problem?

PicoScope 6.4.64.0
Model: PicoScope 2203

ziko
Advanced User
Advanced User
Posts: 1705
Joined: Fri Dec 01, 2006 10:03 am
Location: St Neots

Re: Save As CSV for 2 million samples

Post by ziko »

Hi PicoScope limits the number of samples to 1million per channel on the csv file, this is because the files would be quite large.

A workaround is to save the file into a .mat format (binary format) this is not limited.

Please email support@picotech.com for more information.

Kind regards
Ziko

Technical Specialist

colindavidfoster

Re: Save As CSV for 2 million samples

Post by colindavidfoster »

Hmmm, 50Mb of text file is nothing these days.

To be able to record 2 million (or more) samples and not be able save that data in all of the suggested formats on the grounds of too much data is, IMHO, a pretty poor excuse. The whole point of the (Pico)scope is to record data and the whole point of saving is to preserve all the data. Suppose Microsoft Word only saved the first half of your novel when exported it.

Anyway, enough ranting.

The suggestion of using the .mat format seems a pretty good alternative. I'll have to rework my Perl scripts, but that's not a problem. It may even turn out to be faster.

Cheers

alan
Site Admin
Site Admin
Posts: 195
Joined: Thu May 25, 2006 8:02 am

Re: Save As CSV for 2 million samples

Post by alan »

I think CSV was limited to 1million samples as above this caused Excel to crash. Tech support got fed up with complaints about the CSV file not working when there was no problem with the CSV file itself.

If someone wants to use CSV to export large files into different programs / their own programs then we can look at the limit again but as mentioned above its far more efficient to save the data as a binary file.

Regards
Alan

jokkebk
Newbie
Posts: 0
Joined: Tue May 15, 2012 4:27 pm

Re: Save As CSV for 2 million samples

Post by jokkebk »

Hello,

I just spent two hours tweaking PicoScope software settings with 5 MS captures, because no matter what, I could not get more than 1024^2 (1M) samples to the text formats.

Such an undocumented (is it documented somewhere?) feature is a terrible idea, I was just going to post a new problem topic to the forum when I by chance googled "picoscope csv maximum" and found this thread. Could you please indicate it more clearly in the save dialog, or add an option to Preferences to change this? I'm parsing the files in C so any limitations in Excel do not mind me.

I guess I'll take a look at the Matlab format then. If only I could get those two hours of my life back. :(

Joonas

Hitesh

Re: Save As CSV for 2 million samples

Post by Hitesh »

Hi Joonas,

There is some text in the Help file under 'CSV files, exporting ' that states
The 1 million values per channel limit prevents excessively large files being created.
I've put in a request for indication in the Save As dialog or preferences.

Best wishes,

wcressman2k
Newbie
Posts: 0
Joined: Mon Apr 21, 2014 6:52 pm

Save As CSV for More than 1 million samples

Post by wcressman2k »

Please please change this limitation. I use JMP as well as various scripting tools to post-process scope data from other more expensive scopes and it seems very odd to me that this "Deep Memory" scope that can only output a tiny fraction of it's memory in the most commonly used data format in the world (CSV).

Excel is no longer limited in the number of records it can contain. If you want to protect yourself from complaints posted by from users who have old Excel, how about defaulting to the 1M record limit but having a setting where a user can click OK to a warning and then get the full sample depth in a CSV output?

thanks
Bill

Hitesh

Re: Save As CSV for 2 million samples

Post by Hitesh »

Hi Bill,

I have passed on your comments to members of the Development Team through our system.

We will provide an update as soon as we get an update on this.

Best wishes,

Post Reply