Disk usage when capturing with PicoLog 6

PicoLog 6 opens up new possibilities with Pico data loggers and oscilloscopes. With sample rates up to 1 kS/s per channel (device-dependent), the use of up to 20 devices connected at once, and a maximum of 16 channels per device (device-dependent), you can capture a great deal of information from your sensors in fine detail.

What's not to like?

As it happens, modern-day PCs are very capable at capturing and storing all your short, medium and long-term captures and much, much more. With hard disk space now in the terabytes, you'd think that the days of managing disk space are over. 

Yes, most PCs do have big disks, but not all. Consider that some modern PC configurations have two drives: a smaller SSD (solid state disk) with the operating system installed on it to save money, and a larger, slower mechanical disk (HDD) for data storage. Take a Raspberry Pi too: it might only have a 4 GB SD card that is nearly full with the operating system. Think about using a much larger SD card -- they're very inexpensive these days.

The PicoLog 6 database by definition is situated on the same drive as the operating system and cannot be relocated, so it is imperative that this OS disk has enough space for your capture. 

With great power and flexibility comes great responsibility (and a lot of disk space as it turns out)

You may have read that our PicoLog 6 software has almost unlimited logging capability, Well, what does this actually mean? 

What this all adds up to is a data logger application that's designed to log for days, weeks, months and years and can record your captured data at a theoretical maximum of 320 channels and 1000 samples per channel. I'll explain the math in a short while, but that's 1.28 M samples per second, or 76.8 M samples per minute, or to put it another way 460.8 M samples per hour... you can now see that we can fill a hard disk in a matter of a few hours.

Keep an eye on that capture size, or perhaps use the fixed-length capture mode (circular buffer) to start overwriting the database after a certain amount of time, if that is possible with your application. 

Hang on: 1000 x 320 is 320 k samples, not 1.28 M samples?

Yes, sorry, I forgot to tell you: PicoLog 6 stores each sample in the database as a 32-bit float value (4 x 1 byte) because we need to support all the Pico data loggers and scopes working together simultaneously. 

The different ADC resolutions of our devices produce data with a sample width in the range of 8 to 24 bits. To enable all the loggers and scopes to store samples directly into the same database at the same time, with time-correlated samples, we scale everything to 32-bit values in the database. So yes, if you're using an 8-bit scope for example, the sample stored occupies 4 times as much space as it needs to, but if you're using a TC-08 or ADC-24 which has 24-bit A/D converters, then the 24-bit sample is the smallest it can be. 

Anything else you're not telling us?

Ok, I'll come clean again. Picture this: you've just captured 10 GB of data on your PC, and you want to look closely at an event that happened last Tuesday at 10:45 pm. So you start zooming in, spinning that mouse wheel as though your life depended on it. PicoLog 6 serves up the zoomed-in data almost instantly with ruthless efficiency. Think about it, that's 10 GB of data suddenly appearing on the screen and redrawing almost instantly from a disk that can read files at 20 MB/s to 100 MB/s. Some kind of magic going on here...

No magic. While PicoLog 6 is writing your raw data to the database, it is also saving a downsampled version in parallel, which produces a smaller database for fast lookups. It makes zooming and loading waveforms considerably faster, but there is a trade-off to achieving this performance boost, and that's an additional 22% space required in the database. 

Saving and exporting

Captures can be saved as .picolog files, which contain a compressed binary form of the PicoLog 6 database. They contain both the raw data and fast lookup database for improved zooming and waveform interaction. You can expect the PicoLog file to be smaller than the PicoLog 6 database, but the amount of compression depends on waveform shape and repeating values.

Saving a .picolog file produces a copy of the current live acquisition, but remember, when you reload a saved file you cannot restart capture and append data. The live database, however, is designed to be restarted and paused at will.

Many customers need to export the data in human-readable format, either for printing or post-processing in another application (such as MATLAB). PicoLog 6 is well placed to export the data as a summarized report in PDF format or as CSV. The CSV feature is especially neat as it has the option of resampling the raw data into any new sampling rate to reduce the data to a more manageable size, or exporting only a small area of the capture according to the current graph bounds or table view bounds. 

But beware...

ASCII files can be between 10 and 20 times larger than binary source files because they contain ASCII-encoded symbols to allow the binary to be human-readable. The size also depends on the number of decimal places for each stored data point.

PicoLog will warn you about the CSV output size, and in the example to the right you can see a large 10 GB capture gets very big very quickly when you request an export as a CSV. A 137 GB CSV file is not practical and may is likely to cause your system to hang or worse.... Maybe consider exporting a portion of the data. 

Do math channels count as normal channels?

No. Math channels are calculated in real time at the point they're drawn on the screen or exported, so they take up no extra room in the database. Think of them as free channels. That said, large numbers of math channels can put the extra load on the processor and disk, so if you’re performing math, make sure you're using a beefy computer.

Raspberry Pis and math channels can work, but with slower sample rates. Think about the processor and the disk type when you configure an acquisition. The Pi 3B is not ideal for applications that require high sample rate and math channels, so get yourself a Pi 4B.