Python, improved splitMSODataFast, 70 times faster for 5Msamples

Post general discussions on using our drivers to write your own software here
Post Reply
Anders_Eriksson
Newbie
Posts: 0
Joined: Fri Nov 15, 2019 10:09 am

Python, improved splitMSODataFast, 70 times faster for 5Msamples

Post by Anders_Eriksson » Tue Jan 14, 2020 8:50 am

Hello,

I have been using the "save to mat4 file" option in Picoscope6 together with Octave, but because of some bugs in the saved mat file, I have started to use python and Jupyter notebook, numpy is really fast!
So converting from ctypes to numpy arrays improves speed, In my case with 5000000 (5Msamples), it is about 70 times faster and I am also impressed by "%matplotlib notebook" and it is quite fast to zoom and pan in a plot with 5Msamples, see below the pasted code + screen-dump of my Jupyter notebook

Kind regards,
Anders

%matplotlib notebook
import matplotlib.pyplot as plt


def splitMSO_np_version(buffer):
np_buffer = np.ctypeslib.as_array(buffer)
d7 = ((np_buffer & 0x80) > 1) * 1
d6 = ((np_buffer & 0x40) > 1) * 1
d5 = ((np_buffer & 0x20) > 1) * 1
d4 = ((np_buffer & 0x10) > 1) * 1
d3 = ((np_buffer & 0x08) > 1) * 1
d2 = ((np_buffer & 0x04) > 1) * 1
d1 = ((np_buffer & 0x02) > 1) * 1
d0 = (np_buffer & 0x01)
return [d0,d1,d2,d3,d4,d5,d6,d7]
%timeit d0_d7 = splitMSO_np_version(bufferDPort0Raw)
177 ms ± 1.68 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

cTotalSamples
c_long(5000000)

%timeit d0_d7 = splitMSODataFast(cTotalSamples, bufferDPort0Raw)
12.5 s ± 329 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

12.5 / 0.177
70.62146892655367

type(bufferDPort0Raw)
__main__.c_short_Array_5000000
Attachments
improved_splitMSODataFast.PNG

Anders_Eriksson
Newbie
Posts: 0
Joined: Fri Nov 15, 2019 10:09 am

Re: Python, improved splitMSODataFast, 70 times faster for 5Msamples

Post by Anders_Eriksson » Tue Jan 14, 2020 1:58 pm

Hi again,

An even better version, from 177ms to 107ms

def splitMSO_np_version_3(buffer):
np_buffer = np.ctypeslib.as_array(buffer)
d = lambda n: (np_buffer << (15-n)) >> 15
return [d(n) for n in range(8)]

%timeit d0_d7 = splitMSO_np_version_3(bufferDPort0Raw)
107 ms ± 873 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

=> 116 times faster!
12.5 / 0.107
116.82242990654206

Christian B.
Newbie
Posts: 0
Joined: Fri Mar 06, 2020 10:05 am

Re: Python, improved splitMSODataFast, 70 times faster for 5Msamples

Post by Christian B. » Mon Mar 09, 2020 3:40 pm

Maybe a stupid question but is there a particular reason to not use numpy.unpackbits for this task?

Anders_Eriksson
Newbie
Posts: 0
Joined: Fri Nov 15, 2019 10:09 am

Re: Python, improved splitMSODataFast, 70 times faster for 5Msamples

Post by Anders_Eriksson » Wed Apr 15, 2020 2:32 pm

Thank you Christian!

It is even more than 2 times faster!
splitMSO_np_unpackbits_version.PNG

Post Reply