Subtracting Average Spectrum

Having problems ? let us know the details here
Post Reply
RichardKCollins
User
User
Posts: 4
Joined: Thu Sep 26, 2019 6:30 pm

Subtracting Average Spectrum

Post by RichardKCollins » Thu Sep 26, 2019 7:25 pm

I am just getting started with PicoScope 6 software. I ave my signal in the top panel ("view") and the average spectrum in another pane ("view"). I wanted to also see the amplitude spectrum (instantaneous view of changes buffer by buffer). But it repeats the first spectrum.

To be clear what I am trying. I click Views, Add View, Spectrum twice to get two Spectrum views.

When I try to modify one of the spectrum views to show the Average spectrum - both of the spectrum views change. There does not seem to be any sense of each view being independent. I might have reasons for wanting ten views of the spectrum.

The Math Channel does not include a basic FFT, so I cannot simply ask it to do the amplitude, do the average and subtract them and show me the changes by frequency.

There does not seem to be any way to simply add my own functions to PicoScope 6. That is simple - allow me to add a button, select the data I need to be sent to my function and when it is to be called, then let me program in one of the common languages. I suggest Javascript, since this IS the Internet age.

Gerry
PICO STAFF
PICO STAFF
Posts: 493
Joined: Mon Aug 11, 2014 11:14 am

Re: Subtracting Average Spectrum

Post by Gerry » Fri Sep 27, 2019 10:42 am

Hi Richard,

Could you just explain your reason for needing to see the actual process of the waveform being averaged over time?

Regards,

Gerry
Gerry
Technical Specialist

RichardKCollins
User
User
Posts: 4
Joined: Thu Sep 26, 2019 6:30 pm

Re: Subtracting Average Spectrum

Post by RichardKCollins » Fri Sep 27, 2019 9:47 pm

I am making long term (days, weeks, months, years) variations in the electromagnetic power spectral density globally. There are many sensors that pick up information related to this. I am trying to use them all.

The Pico "oscilloscope" from my point of view is an amplifier and ADC for collecting time series of currents or voltages from various sensors (antennas, coils, detectors) that I can find.

The PicoScope 6 software collects the time series from the ADC. It can stream it to disk. It also has FFT capabilities, and perhaps can store the FFTs.

The Pico software has "Math Channels" but they are very basic. I cannot, for instance, take the FFT as a channel (time series of FFT values at a given frame rate), treat it as a channel, and then apply mathematical operations.

Here I was describing the screens which show the instantaneous amplitude spectrum, which are being updated as new spectra are calculated. And the "average amplitude spectrum" which I hope is a simple average of the amplitudes from the individual frames of the amplitude spectrum that have been accumulated since it was last zeroed.

If you could look at the amplitude spectrum and the average amplitude spectrum as individual panels on the screen. You could try to compare them visually. But humans are not good at that. They get tired and their eyesight and attention are poor.

But it was be a trivial exercise to take each frame of the amplitude spectrum as it is completed, display that. Add the new amplitudes (by frequency bin) to the accumulating average FFT spectrum - update that.

And then subtract the average amplitudes by frequency bin from the current amplitude spectrom to get their difference.

The averages are quite complex. The amplitude spectrum is quite complex. Subtracting the average from the current amplitude spectrum leaves a tiny residual of what is changing by frequency.

Those changes at particular frequencies are clues to sources that have not been investigated yet.

I am asking if you can allow the FFT to be considered a channel. Let me give it a name. I think it has names like "Spectrum 1". Let me have access to "Average Spectrum `" and let me calculate and show "Spectrum 1" minus "Average Spectrum 1".

I cannot wait for Pico Technology to write software. But I encourage you to spend more time on FFT displays. The Pico devices can all be used as Software Defined Radios (SDRs) for their respective frequencies. I do not remember the particular sampling rate and resolution for this devices, I am reviewing many similar devices and cannot remember your particulars. But it is likely 20 Msps down to less than one sample per second. So Nyquist as as guide, this could be used for software radio (attached to an antennas and appropriate amplifiers) from DC to 10 MHz.

I am looking at the region from microHertz to kiloHertz. Your "oscilloscope" seems to be stable and reliable. I want to see if it can fit into a global network for certain groups who have common interests in electromagnetics and in traditional uses of oscilloscopes. It is only a few changes in software.

One thing I would recommend is for your company to take some time and write a device driver for the Pico line of instruments to work with SDR# ("SDR sharp"), CubicSDR, rtl_power, and the most common of the SDR software. These all have FFTs, waterfalls and experience in monitoring the electromagnetic field. They are doing it for their own purposes, but generally are moving in the direction of increasingly quantitative and scientific applications. As they build more detectors and antennas, they could use more stable and calibrated instruments like Pico scopes.

Richard Collins, The Internet Foundation

Gerry
PICO STAFF
PICO STAFF
Posts: 493
Joined: Mon Aug 11, 2014 11:14 am

Re: Subtracting Average Spectrum

Post by Gerry » Tue Oct 01, 2019 11:35 am

Hi Richard,

Thanks for your detailed feedback on a non-established potential application area for our PicoScopes. We are always interested in new and interesting ways of using our products and welcome informative constructive feedback.

Currently we only plot a 'Linear' Spectrum that has had coherent gain applied to ensure that the the voltage amplitude values for deterministic signals have minimal error. So, yes, we do enable averaging of the Spectra as a rolling average of the calculated Spectrum amplitudes collected since the first capture. This is so that the variance of non-deterministic signals, such as noise, along with their average computed amplitudes, can be reduced, while the variance of deterministic signals can also be reduced, but their accumulated mean can be held at a value for measurement.

The fact that the rolling average updates all of the values collected instead of just the last one is an error in the software that is to be looked at by our development team. Your request for Math Channel operations on Spectrum Amplitudes will be added to an existing enhancement request for our development team. However, be aware that Math channel operations on Spectrum data would be necessarily limited because some functions would result in inaccurate or meaningless data.

We don't need to rewrite our drivers for different applications, as we have the application code separated into an API that can be developed using our Software Development Kit (see here: https://www.picotech.com/library/oscilloscopes/picoscope-software-development-kit-sdk). Users can select the programming environment of their choice and download example code from Github (see here: https://github.com/picotech) that performs the basic functionality required to interact with the Scope, leaving them to add the 'bells and whistles' of their application.

Regards,

Gerry
Gerry
Technical Specialist

RichardKCollins
User
User
Posts: 4
Joined: Thu Sep 26, 2019 6:30 pm

Re: Subtracting Average Spectrum

Post by RichardKCollins » Tue Oct 01, 2019 5:40 pm

Gerry,

Thanks for the clear and complete reply. I try to give precise feedback. You have good products, and it seems a good development plan and community. So I am happy to help any way I can.

I will look at the API. I am going through all the low cost oscilloscopes and SDRs and related ADC devices, or as many as I can find and review. Yours is one of the more precise, but I need lots more data for most of the potential applications.

One thought. If you wrote a driver to let the Pico products be used with Software Defined Radio software, that would allow a large and growing number of people to do scientific and technical problems, that they are doing only qualitatively now. I am going to be using the Pico 2204A that I have as the ADC for radio frequency monitoring in the microHertz to megaHertz range. I want to make a path for people to follow, then they can buy better instruments as they get into it more.

The SDR software mainly centers around the ability to show wide band (0-2000 MHz) RF spectrum, down to subHertz views. The waterfall is a color coded intensity map. Those are useful visualizations for many related problems. The rather large and growing community is looking for better access to the data from ADCs. I am trying to add the oscilloscopes to the mix for low frequencies. Much audio work as well.

I wish I could clone myself.

Richard Collins, The Internet Foundation

Gerry
PICO STAFF
PICO STAFF
Posts: 493
Joined: Mon Aug 11, 2014 11:14 am

Re: Subtracting Average Spectrum

Post by Gerry » Wed Oct 23, 2019 2:42 pm

Hi Richard,

Sorry for the long delay in responding, (it has been a hectic month and sometimes we can't get back to everything that we would like to in a timely manner).

I can appreciate your desire for more specialized software, but the driver is what controls and communicates with our hardware, which is purely based around data capture and transfer. So, there is no scope (if you'll forgive the pun) for applying the fundamental low level software of the driver to SDR, as SDR is an application area that happens to use data acquisition rather than a means of data acquisition (so none of our driver code would be directly applicable to SDR, and no SDR code would be directly applicable to capturing data and sending it to the computer). The Application code that we provide in our SDK is also fundamental to collecting the data, but it is primarily a starting block and is there to be expanded into a fully functional application specifically targeted towards a unique application goal that our general purpose Data Acquisition Software, PicoScope 6, can't perform.

As I'm sure you can appreciate, our Software Team already have their work cut out in meeting the targets that we have, in order to produce the data acquisition products that will take us in the direction defined by our business goals. So, diluting their focus by taking up suggestions (even great suggestions) that our customers have, for developing applications that aren't directly related to data acquisition is, unfortunately, not something that we choose to do. We would prefer our customers (who, typically, have much more experience in their specific application areas) to write the non data acquisition aspects of applications, and share what they have written, if they are so inclined, on platforms such as our forum (for example, as has been done here: https://bitbucket.org/hexamer/fra4picoscope/wiki/Home) which is a much more effective use of any ones time. We can certainly help with the code that needs to either work with our hardware, or display the data on a computer using our PicoScope 6 software (which is where our expertise is), and welcome suggestions for improving either in order to better support specific applications, as alluded to previously. In that regard, a waterfall plot is something that is in our list of feature enhancement requests to be implemented.

..and, I hear ya, the limited time that we have is probably the biggest issue that many of us face.

Regards,

Gerry
Gerry
Technical Specialist

RichardKCollins
User
User
Posts: 4
Joined: Thu Sep 26, 2019 6:30 pm

Re: Subtracting Average Spectrum

Post by RichardKCollins » Wed Oct 23, 2019 3:34 pm

Gerry,

Thanks for the nice reply. It is hard to know what companies and people are doing without asking.

Most of what I do involves connecting groups across discipline and technology boundaries. So today I am trying to get seismology and gravity groups to talk to each other. The ones who are most likely to try new things are not the established ones, who are "fine thank you", but those who want to do something new or different. And it is often the ones from small universities, small countries, small companies, and many individuals, who want to try something new. And they are limited in skills, access, and money. But there are usually a LOT of them. I am sure your company knows your customers well. It would be nice if they fit into neat groups with identical skills and unlimited resources.

I have pretty good skills from many decades of learning different things. Today I am taking data from a not very good, but promising, instrument that has kind of been shuffled aside. It is a seismometer, but not very stable. It wanders all over the place for a variety of reasons. But the basic signal is there and primarily made up of the gravity signal from the sun and moon. So I am matching this "seismometer", calibrating day by day and hour by hour against the sun and moon gravity signal, which I can calculate exactly from the precise positions of the sun moon earth and station, and producing a low cost absolute gravimeter data stream. I am doing essentially the same thing with what are called "MEMS gravimeters". They are cheap, sensitive, easy to connect to networks, but they need a continuous calibration reference source.

When I have time, I will try to read the data from the Pico scope. If it is picking up data from an amplified gravimeter signal, then it becomes a "gravimeter". If it is picking up data from an amplified seismometer signal, it becomes a 'seismometer'. The "real" things cost tens or hundreds of thousands of dollars and have very very low sampling rates. To do localization and characterization of sources requires many sensors and precise timings.

The difference between the "software defined radios" and the "oscilloscopes" is minor. They have high sampling rates, controllable center frequencies and amplification, and decent software that lets users work with complex signals. The underlying hardware - chips and components - seem to be converging, because of emerging common uses - software defined, software enabled, data sharing. People who have to work with data every day want sensors and systems that make it easy to use data. They are not primary signal watchers, but enabling others to get to the data. They do not want to play with an oscilloscope knobs and buttons, but to use data - LOTS of data - in new and useful combinations.

Whether I do anything or not doesn't really matter. I have watched technologies developing for 21 years for the Internet Foundation using the Internet. And for three decades before that by more traditional methods. What I see, literally every day, is that good ideas emerge, grow and become new industries. Often dozens or hundreds of people try exactly the same things. Sometimes that runs into the thousands of groups all trying the same thing - until one group tackles it seriously, and makes it into a new industry. And all the others buy from them. It is not quite that simple, because there are so many failures.

Since I originally wrote, I have been using time series of FFTs for various things. I meet them in every technology field. I understand how the FFT algorithm is used. What is missing, mostly because of software and just doing it, is the recording, storage, playback and analysis of time series of FFTs. The SDRs can use 22 bit FFTs with no problem. That is 4,194,304 samples per FFT. With a 10 Msps ADC, that is roughly two of those per second. For stations monitoring 24 hours a days, for decades, that kind of data can be golden. It cuts downstream processing and guesswork. That is too much data to pass along. But the hourly, daily and monthly summaries are golden. I am going through nuclear magnetic resonance, magnetic resonance imaging, infrasound, and many other application areas.

For many "audio" applications where the data has an important component in the 0 to 50 kHz range, people are or can be using 16 bit FFTs. Coupling a 20 Msps 12 bit data stream to a 16 bit FFT process gives many FFTs per second. The SDRs are using that for improving communication channels. And the same thing, scaled to other sampling rates, is being tried in many places.

And it comes down to "Averages and other statistical summaries of streams of FFTs". I wish I had time to give you a cookbook, and a map of the future. All I can do, like Hercule Poirot, is to "stir the little grey cells".

The guts of oscilloscopes these days could be very useful to many data networks. But the "big, expensive, box with many dials and switches" needs to change to "an amplifier and ADC box that gives me a useful datastream, in a format that I can quickly adapt to new uses". That is easy to share on the Internet, and convert to derivative statistical streams and FFT and analytical streams that can be higher value. If the data gathering produces most of what you have to do on the receiving end anyway, why not produce those analytic streams right at the beginning and charge a little more to a much larger group.

Companies make money, not from printers, but from ink. Not from a few enduser displays where the difficulty to use them limits the number of end users, but from high quality data collection and dissemination systems that whole corporation and networks can use. And charge by the end user. Just a litle bit, like all the Internet group sites are doing, fairly successfully.

Richard Collins, The Internet Foundation

Post Reply