To answer your questions:
1) A spectrum analyzer, sweeps a narrow band filter over the frequency range of a signal of interest to look at the magnitude of the frequencies. You are effectively performing the digital equivalent when you perform a Fast Fourier Transform, i.e. applying an FFT calculation to the discrete sample values of the signal enclosed by a windowing function (the digital equivalent of the filter) replicated up and down the spectrum. The windowing function is a complex valued function (or time-variant function) that will change the relative magnitudes and phase of the non-zero values of the signal, creating components not present in the original signal (known as spectral leakage), and leading to components that are reduced/distorted due to signal and window offsets (scalloping loss). These errors interfere with the analysis, for instance, the spectral leakage can make it hard to distinguish between similar amplitude spectral components. So, to increase the visibility of certain spectral components relative to others, different windowing functions can be applied when calculating the FFT. These differences also provide other benefits (this is explained in more detail on the Wikipedia page here: https://en.wikipedia.org/wiki/Window_function
Most of the windowing functions have errors components associated with what happens at the endpoints (including overlapping) and the flatness of the window. The more leakage, and error components a windowing function has the greater will be the contribution to the measurement. A rectangular windowing function has perfect reproduction of a single frequency sinewave, but also has scalloping loss, and minimal dynamic range (which doesn’t affect a Total power calculation). Because the power is computed as the square of the spectral components, the RMS value of a power spectrum based upon a rectangular window function gives an accurate value. Other windowing functions will give greater values of RMS due to the error components created in addition to the sinewave.
dBm is an expression of a power ratio. It is the power measured relative to 1mW dissipated into a 600Ω load. As you are using the 6.12 Beta version of PicoScope 6, you will notice that when you select the dBm scale the impedance value of 600Ω is also selected, indicating that all references are now to a 600Ω load. So, the actual load that the voltage is driving is not used in the calculation and you should perform the power calculation with this in mind, as follows:
Power = 0.707^2 / 600
= 8.33 * 10^-4 = 833uW
As you can see from the screenshot of the Spectrum plot, and the psdata file when the Logarithmic unit chosen for the scale is dBm the measured ‘Total Power’ is 836uW, which confirms the calculation.
2) dBV is an expression of a voltage ratio. It is the voltage measured relative to 1 volt, regardless of impedance. You will notice that when you select the dBV scale the impedance value is not displayed as there is no fixed impedance value that is relevant to the calculation. So you now consider the dB ratio being referenced to 1V and driving a load that will have an effect on the power calculation, as follows:
Power = 0.707^2 / 1000,000
From the screenshot of the Spectrum plot, when the Logarithmic unit chosen for the scale is dBV the measured Total Power is 502nW, which confirms the calculation.
3) Although originally derived using a 600Ω load, dBu is an expression of a voltage ratio. It is the voltage measured relative to 0.775V, regardless of impedance. So, once again the Impedance is not shown when selecting it for the scale in the spectrum plot, and using impedance for power calculations relative to a 600Ω load is not applicable. You therefore, once again, need to consider the input impedance of the PicoScope in the power calculation (which will be the same as before), and the measurement of ‘Total Power’, once again, confirms this (as shown in the screenshot for dBu).