Why am I losing samples in my application?

Post your C and C++ discussions here
Post Reply
Ben321
User
User
Posts: 6
Joined: Tue May 23, 2017 8:09 am

Why am I losing samples in my application?

Post by Ben321 »

I'm writing an application for the Picoscope 2204A. Unfortunately it's dropping samples. Maximum scope sample rate is is 1MHz. And just like the manual says, don't do anything in the callback function other than saving samples, so that's exactly what I'm doing. I'm copying them to an intermediate buffer, and then outside of the callback function, I'm using the C function fwrite to write that buffer to a file. Is fwrite actually too slow for this? Here's what my callback function looks like.

Code: Select all

void __stdcall CaptureBufferCallback
(
	int16_t** overviewBuffers,
	int16_t overflow,
	uint32_t triggeredAt,
	int16_t triggered,
	int16_t auto_stop,
	uint32_t nValues
){
	int16_t* ptrCap = *overviewBuffers;
	SizeOfCapture=nValues;
	memcpy(Buffer,ptrCap,nValues*2);
	return;
}
And here's the function outside of that, which calls ps2000_get_streaming_last_values in order to get the scope to call the callback function.

Code: Select all

void CaptureBuffer(FILE* file,int16_t hScope){
	while (ps2000_get_streaming_last_values(hScope,(GetOverviewBuffersMaxMin)CaptureBufferCallback)==0){
		//do nothing
	}
	fwrite(Buffer,2,SizeOfCapture,file);
	TotalBytes += (SizeOfCapture*2);
	return;
}
And here's the way I'm calling the the function to initialize fast streaming.

Code: Select all

ps2000_run_streaming_ns(hScope,SampleDuration,PS2000_NS,1000000,0,1,1000000)
SampleDuration is 1000 by (set elsewhere in the code), meaning 1000ns per sample, and thus the sample rate is 1MHz.

Martyn
Site Admin
Site Admin
Posts: 4491
Joined: Fri Jun 10, 2011 8:15 am
Location: St. Neots

Re: Why am I losing samples in my application?

Post by Martyn »

It is possibly the fwrite that is causing the problem particularly if you are calling it on each loop, you may need to loop a few times to collect a bigger chunk of data before writing to the file. You may also need to consider the size of the buffer you setup in the driver as this could overflow as there is very little protection with the 2204a's on board memory. If you were using one of the deeper memory scopes from our range the onboard buffer memory helps to protect against memory loss.
Martyn
Technical Support Manager

bennog
Advanced User
Advanced User
Posts: 206
Joined: Mon Nov 26, 2012 9:16 am
Location: Netherlands

Re: Why am I losing samples in my application?

Post by bennog »

If you are familiar with threads you can use multiple threads (1 for reading the data from the scope)
and a second thread for writing accumulated data to the file.

Think careful about the FIFO between the read and write thread.
because you want to do this without (b)locking and there are some points where you can cause a race condition.

Benno

Ben321
User
User
Posts: 6
Joined: Tue May 23, 2017 8:09 am

Re: Why am I losing samples in my application?

Post by Ben321 »

Martyn wrote:
Mon Dec 30, 2019 1:57 pm
It is possibly the fwrite that is causing the problem particularly if you are calling it on each loop, you may need to loop a few times to collect a bigger chunk of data before writing to the file. You may also need to consider the size of the buffer you setup in the driver as this could overflow as there is very little protection with the 2204a's on board memory. If you were using one of the deeper memory scopes from our range the onboard buffer memory helps to protect against memory loss.
I think that could be a problem, that I'm not letting it accumulate enough in the buffer, though I did specify buffer size being 1000000 samples and at 1MHz that means the buffer is one second long. It shouldn't take over 1 second to write 1000000 samples to a file. The below code shows how I'm calling the function to start fast streaming.

Code: Select all

ps2000_run_streaming_ns(hScope,SampleDuration,PS2000_NS,1000000,0,1,1000000)
That theoretically means that now every time I call ps2000_get_streaming_last_values I will get either no data (if I call it too soon) or exactly 1000000 samples placed into my buffer (and thus removing those 1000000 samples from the hardware buffer in the scope itself, preventing overflow).

It seems I'm misunderstanding how the Picoscope works though, as my assumptions seem to result in a malfunctioning program.

Martyn
Site Admin
Site Admin
Posts: 4491
Joined: Fri Jun 10, 2011 8:15 am
Location: St. Neots

Re: Why am I losing samples in my application?

Post by Martyn »

The streaming buffer is a circular buffer defined by the value of max_samples this can be quite large (PC dependent) and is used to buffer data in the driver. The overview buffer is the one used to read data back to the application and the documentation recommends using 15000 to start with and then gradually increasing this to tune to your application and PC setup. You are losing data because both buffers are the same size, and the scope has insufficient memory to buffer any additional data. Each call to get last values will return the 15000 values if there are 15000 new values to return.

I would suggest copying last values to an application buffer that is much bigger than this and only when the application buffer is full use fwrite to save to file. The driver max samples buffer can be tuned to provide sufficient buffering so that you do not lose data.

You should not need to add threads, as the driver provides this mechanism already.

If you are still having difficulties then email your full code to support@pictech.com and we will take a look for you
Martyn
Technical Support Manager

Ben321
User
User
Posts: 6
Joined: Tue May 23, 2017 8:09 am

Re: Why am I losing samples in my application?

Post by Ben321 »

Martyn wrote:
Tue Dec 31, 2019 9:57 am
The streaming buffer is a circular buffer defined by the value of max_samples this can be quite large (PC dependent) and is used to buffer data in the driver. The overview buffer is the one used to read data back to the application and the documentation recommends using 15000 to start with and then gradually increasing this to tune to your application and PC setup. You are losing data because both buffers are the same size, and the scope has insufficient memory to buffer any additional data. Each call to get last values will return the 15000 values if there are 15000 new values to return.

I would suggest copying last values to an application buffer that is much bigger than this and only when the application buffer is full use fwrite to save to file. The driver max samples buffer can be tuned to provide sufficient buffering so that you do not lose data.

You should not need to add threads, as the driver provides this mechanism already.

If you are still having difficulties then email your full code to support@pictech.com and we will take a look for you
So there are 3 buffers then? One is in the scope hardware itself (the scope's own RAM), and the other 2 as defined by the max_samples (for the driver's steaming buffer) and overview buffer size (for the driver's overview buffer) are implemented in the computer's RAM? How are these 3 buffers actually related to each other, and the picoscope api functions?

Also I thought that the max_samples parameter only is used if the autostop parameter is set to true, and is ignored otherwise. In the case of my program here, I've set the autostop to false, so that it will continue running until the stop function is called manually. So I don't think that the max_samples parameter is even used in this case (at least based on my understanding of ehat the SDK's programmer's guide says). Am I misunderstanding something here?

Martyn
Site Admin
Site Admin
Posts: 4491
Joined: Fri Jun 10, 2011 8:15 am
Location: St. Neots

Re: Why am I losing samples in my application?

Post by Martyn »

When streaming the scopes memory acts as a Fifo to help with the flow of data over the USB bus, this data then goes into the driver buffer defined by max_samples, this is a circular buffer and is over written but when you stop streaming either manually or via auto stop it will then have the last max_samples raw data values that can still be accessed. During streaming when you ask for data it is transferred into the overview buffer and passed to your application.

This mechanism has been implemented to support the operation of Picoscope 6, which streams data for a set screen time defined by max_samples and the chosen sample rate, but allows down sampled data to be read from the overview buffer to draw live chunks on the screen until the screen waveform is complete. At this point streaming is stopped and the application then reads all the raw data from the max_samples buffer.

Try setting max_samples to 10000000 and overview buffer size to 50000 and see if you collect all the data.
Martyn
Technical Support Manager

Ben321
User
User
Posts: 6
Joined: Tue May 23, 2017 8:09 am

Re: Why am I losing samples in my application?

Post by Ben321 »

Martyn wrote:
Tue Dec 31, 2019 1:36 pm
When streaming the scopes memory acts as a Fifo to help with the flow of data over the USB bus, this data then goes into the driver buffer defined by max_samples, this is a circular buffer and is over written but when you stop streaming either manually or via auto stop it will then have the last max_samples raw data values that can still be accessed. During streaming when you ask for data it is transferred into the overview buffer and passed to your application.

This mechanism has been implemented to support the operation of Picoscope 6, which streams data for a set screen time defined by max_samples and the chosen sample rate, but allows down sampled data to be read from the overview buffer to draw live chunks on the screen until the screen waveform is complete. At this point streaming is stopped and the application then reads all the raw data from the max_samples buffer.

Try setting max_samples to 10000000 and overview buffer size to 50000 and see if you collect all the data.
Thanks. I'll try that. It will require some time though, as the changes I'd need to make to implement what you've talked about in this post and previous posts, would basically require a complete rewrite of my program.

Post Reply