Python, improved splitMSODataFast, 70 times faster for 5Msamples

[Anders_Eriksson]

Hello,

I have been using the "save to mat4 file" option in Picoscope6 together with Octave, but because of some bugs in the saved mat file, I have started to use python and Jupyter notebook, numpy is really fast!
So converting from ctypes to numpy arrays improves speed, In my case with 5000000 (5Msamples), it is about 70 times faster and I am also impressed by "%matplotlib notebook" and it is quite fast to zoom and pan in a plot with 5Msamples, see below the pasted code + screen-dump of my Jupyter notebook

Kind regards,
Anders

%matplotlib notebook
import matplotlib.pyplot as plt


def splitMSO_np_version(buffer):
np_buffer = np.ctypeslib.as_array(buffer)
d7 = ((np_buffer & 0x80) > 1) * 1
d6 = ((np_buffer & 0x40) > 1) * 1
d5 = ((np_buffer & 0x20) > 1) * 1
d4 = ((np_buffer & 0x10) > 1) * 1
d3 = ((np_buffer & 0x08) > 1) * 1
d2 = ((np_buffer & 0x04) > 1) * 1
d1 = ((np_buffer & 0x02) > 1) * 1
d0 = (np_buffer & 0x01)
return [d0,d1,d2,d3,d4,d5,d6,d7]
%timeit d0_d7 = splitMSO_np_version(bufferDPort0Raw)
177 ms ± 1.68 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

cTotalSamples
c_long(5000000)

%timeit d0_d7 = splitMSODataFast(cTotalSamples, bufferDPort0Raw)
12.5 s ± 329 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

12.5 / 0.177
70.62146892655367

type(bufferDPort0Raw)
__main__.c_short_Array_5000000

[Anders_Eriksson]

Hi again,

An even better version, from 177ms to 107ms

def splitMSO_np_version_3(buffer):
np_buffer = np.ctypeslib.as_array(buffer)
d = lambda n: (np_buffer << (15-n)) >> 15
return [d(n) for n in range(8)]

%timeit d0_d7 = splitMSO_np_version_3(bufferDPort0Raw)
107 ms ± 873 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

=> 116 times faster!
12.5 / 0.107
116.82242990654206

[Christian B.]

Maybe a stupid question but is there a particular reason to not use numpy.unpackbits for this task?

[Anders_Eriksson]

Thank you Christian!

It is even more than 2 times faster!