Test and Measurement Forum

Save As CSV for 2 million samples

Post any questions you may have about our current range of oscilloscopes

Save As CSV for 2 million samples

Postby colindavidfoster » Mon Feb 14, 2011 3:32 pm

If I manage to sample 2 million samples the export to CSV the file only contains half of the samples.

Anybody else having the same problem?

PicoScope 6.4.64.0
Model: PicoScope 2203
colindavidfoster
 

Re: Save As CSV for 2 million samples

Postby ziko » Wed Feb 16, 2011 4:43 pm

Hi PicoScope limits the number of samples to 1million per channel on the csv file, this is because the files would be quite large.

A workaround is to save the file into a .mat format (binary format) this is not limited.

Please email support@picotech.com for more information.

Kind regards
Ziko

Technical Specialist
ziko
Zen Master
Zen Master
 
Posts: 1705
Joined: Fri Dec 01, 2006 10:03 am
Location: St Neots

Re: Save As CSV for 2 million samples

Postby colindavidfoster » Wed Feb 16, 2011 6:02 pm

Hmmm, 50Mb of text file is nothing these days.

To be able to record 2 million (or more) samples and not be able save that data in all of the suggested formats on the grounds of too much data is, IMHO, a pretty poor excuse. The whole point of the (Pico)scope is to record data and the whole point of saving is to preserve all the data. Suppose Microsoft Word only saved the first half of your novel when exported it.

Anyway, enough ranting.

The suggestion of using the .mat format seems a pretty good alternative. I'll have to rework my Perl scripts, but that's not a problem. It may even turn out to be faster.

Cheers
colindavidfoster
 

Re: Save As CSV for 2 million samples

Postby alan » Thu Feb 17, 2011 12:57 am

I think CSV was limited to 1million samples as above this caused Excel to crash. Tech support got fed up with complaints about the CSV file not working when there was no problem with the CSV file itself.

If someone wants to use CSV to export large files into different programs / their own programs then we can look at the limit again but as mentioned above its far more efficient to save the data as a binary file.

Regards
Alan
alan
Site Admin
Site Admin
 
Posts: 165
Joined: Thu May 25, 2006 8:02 am

Re: Save As CSV for 2 million samples

Postby jokkebk » Sat Jun 29, 2013 1:03 pm

Hello,

I just spent two hours tweaking PicoScope software settings with 5 MS captures, because no matter what, I could not get more than 1024^2 (1M) samples to the text formats.

Such an undocumented (is it documented somewhere?) feature is a terrible idea, I was just going to post a new problem topic to the forum when I by chance googled "picoscope csv maximum" and found this thread. Could you please indicate it more clearly in the save dialog, or add an option to Preferences to change this? I'm parsing the files in C so any limitations in Excel do not mind me.

I guess I'll take a look at the Matlab format then. If only I could get those two hours of my life back. :(

Joonas
jokkebk
Newbie
 
Posts: 0
Joined: Tue May 15, 2012 4:27 pm

Re: Save As CSV for 2 million samples

Postby Hitesh » Mon Jul 01, 2013 10:48 am

Hi Joonas,

There is some text in the Help file under 'CSV files, exporting ' that states

The 1 million values per channel limit prevents excessively large files being created.


I've put in a request for indication in the Save As dialog or preferences.

Best wishes,
Hitesh

Technical Specialist
Pico Technology
Hitesh
Site Admin
Site Admin
 
Posts: 1997
Joined: Tue May 31, 2011 3:43 pm
Location: St. Neots, Cambridgeshire

Save As CSV for More than 1 million samples

Postby wcressman2k » Mon Apr 21, 2014 6:59 pm

Please please change this limitation. I use JMP as well as various scripting tools to post-process scope data from other more expensive scopes and it seems very odd to me that this "Deep Memory" scope that can only output a tiny fraction of it's memory in the most commonly used data format in the world (CSV).

Excel is no longer limited in the number of records it can contain. If you want to protect yourself from complaints posted by from users who have old Excel, how about defaulting to the 1M record limit but having a setting where a user can click OK to a warning and then get the full sample depth in a CSV output?

thanks
Bill
wcressman2k
Newbie
 
Posts: 0
Joined: Mon Apr 21, 2014 6:52 pm

Re: Save As CSV for 2 million samples

Postby Hitesh » Tue Apr 22, 2014 8:31 am

Hi Bill,

I have passed on your comments to members of the Development Team through our system.

We will provide an update as soon as we get an update on this.

Best wishes,
Hitesh

Technical Specialist
Pico Technology
Hitesh
Site Admin
Site Admin
 
Posts: 1997
Joined: Tue May 31, 2011 3:43 pm
Location: St. Neots, Cambridgeshire


Return to USB PC Oscilloscopes

Who is online

Users browsing this forum: No registered users and 0 guests