Data format

PDM data is acquired, triggered and time-stamped in the Zynq board. This data is then passed to the CPU. The CPU also acquires data from the other subsystems and packages this together with the PDM data during nominal night-time observations. Data is acquired every 5.24 s (128 x 128 x 128 x 1 GTU, 1 GTU = 2.5 us). The CPU generates 3 types of files, CPU_RUN_MAIN containing the standard data acquistion, CPU_RUN_SC for S-curve data and CPU_RUN_HV for HV data. All data files have a matryoshka structure that is summarised below.

The CPU_RUN file

Files named CPU_RUN_....dat are the output files of the Mini-EUSO software. Information is what this file contains is stored in the CpuFileHeader which is put together at runtime. This structure has the following fields:

  • spacer: a HEX ID tag 0xAA55AA55 for easy checking of the files in a hex viewer

  • header: a set of bytes defining the instrument, file type and file version (they types and versions are defined in minieuso_data_format.h and the header is put together in DataAcquisition::BuildCpuHeader())

  • run_info: a text field containing the information on the command line options and configuration at runtime (put together in DataAcquisition::BuildCpuFileInfo())

  • run_size: the number of CPU_PACKET in the standard acquisition run

The file is closed with a CpuFileTrailer. This also contains the spacer and run_size fields as well as crc which stores a 32 bit CRC, calcluated over the whole file excluding the trailer and appended using SynchronisedFile::Checksum() accessed through Access::GetChecksum() within DataAcquisition::CloseCpuRun().

  1. The CPU_RUN_MAIN file format

_images/cpu_format.png

At present the Zynq data format is fixed so that each ZYNQ_PACKET contains:

  • N1: the number of D1 packets

  • N2: the number of D2 packets

  • level1_data: N1 x 128 GTU packet of D1 data (1 byte/pixel)

  • level2_data: N2 x 128 GTU_L2 packet of D2 data (2 bytes/pixel)

  • level3_data: 1 x 128 GTU_L3 packet of D3 data (4 bytes/pixel)

The data format holds for both triggered and non-triggered readout.

  1. The CPU_RUN_SC file format

_images/sc_data_format.png

The CPU_RUN_SC has a fixed size which represents the maximum number of threshold steps (0 - 1023). For S-curves taken over a smaller threshold ranges, the file is simply padded with the value 0xFFFFFFFF. S-curve accumulation is calculated on-board the Zynq FPGA using the HLS scurve_adder (https://github.com/cescalara/zynq_ip_hls) allowing for S-curves to be taken with high statistics and stored in a small file size.

  1. The CPU_RUN_HV file format

This file also has a fixed size and is used to store information on the HV status at the end of a run. This information is additional and complementary to that stored inside the ZYNQ_PACKET.

The format is described in detail by the two header files minieuso_pdmdata.h (the Zynq data format - depends on the firmware version) and minieuso_data_format.h (the CPU data format - depends on the CPU software version). The minieuso_data_format.h file is documented below.

A 32 bit CRC is calculated for each CPU_RUN file prior to adding the CpuFileTrailer (the last 10 bytes). This CRC is appended to each CPU_RUN file as part of the CpuFileTrailer.

minieuso_data_format.h

Defines

VERSION 9.5
VERSION_DATE_STRING "13/10/2020"
INSTRUMENT "Mini-EUSO"
INSTRUMENT_ME_PDM 1
ID_TAG 0xAA55AA55
ID_TAG_SUB 0xBB66BB66
RUN_SIZE 25
RUN_INFO_SIZE 512
CPU_FILE_TYPE 'C'
SC_FILE_TYPE 'S'
HV_FILE_TYPE 'H'
SC_FILE_VER 1
HV_FILE_VER 1
CPU_FILE_VER 1
THERM_PACKET_TYPE 'T'
HK_PACKET_TYPE 'H'
HV_PACKET_TYPE 'V'
SC_PACKET_TYPE 'S'
CPU_PACKET_TYPE 'P'
TRAILER_PACKET_TYPE 'Q'
THERM_PACKET_VER 1
HK_PACKET_VER 1
HV_PACKET_VER 1
SC_PACKET_VER 2
CPU_PACKET_VER 2
N_CHANNELS_PHOTODIODE 4
N_CHANNELS_SIPM 64
N_CHANNELS_THERM 10
MAX_PACKETS_L1 4
MAX_PACKETS_L2 4
MAX_PACKETS_L3 1
struct CpuFileHeader
#include <minieuso_data_format.h>

cpu file header 12 bytes

Public Members

uint32_t spacer = ID_TAG
uint32_t header
char run_info[RUN_INFO_SIZE]
uint32_t run_size
struct CpuFileTrailer
#include <minieuso_data_format.h>

cpu file trailer 12 bytes

Public Members

uint32_t spacer = ID_TAG
uint32_t header
uint32_t run_size
uint32_t crc
struct CpuPktHeader
#include <minieuso_data_format.h>

generic packet header for all cpu packets the zynq packet has its own header defined in minieuso_pdmdata.h 16 bytes

Public Members

uint32_t spacer = ID_TAG
uint32_t header
uint32_t pkt_size
uint32_t pkt_num
struct SubPktHeader
#include <minieuso_data_format.h>

generic packet header for all hk sub packets the zynq packet has its own header defined in minieuso_pdmdata.h 16 bytes

Public Members

uint32_t spacer = ID_TAG_SUB
uint32_t header
uint32_t pkt_size
uint32_t pkt_num
struct CpuTimeStamp
#include <minieuso_data_format.h>

timestamp 4 bytes unix time in s

Public Members

uint32_t cpu_time_stamp
struct THERM_PACKET
#include <minieuso_data_format.h>

thermistor packet for temperature data 60 bytes

Public Members

CpuPktHeader therm_packet_header
CpuTimeStamp therm_time
float therm_data[N_CHANNELS_THERM]
struct HK_PACKET
#include <minieuso_data_format.h>

housekeeping packet for sipm and photodiode data 296 bytes

Public Members

SubPktHeader hk_packet_header
CpuTimeStamp hk_time
float photodiode_data[N_CHANNELS_PHOTODIODE]
float sipm_data[N_CHANNELS_SIPM]
float sipm_single
struct ZYNQ_PACKET
#include <minieuso_data_format.h>

zynq packet passed to the CPU every 5.24 s variable size, depending on configurable N1 and N2 NB: vector itself is not written to file, just contents which are contiguous in memory

Public Members

uint8_t N1
uint8_t N2
std::vector<Z_DATA_TYPE_SCI_L1_V2> level1_data
std::vector<Z_DATA_TYPE_SCI_L2_V2> level2_data
Z_DATA_TYPE_SCI_L3_V2 level3_data
struct CPU_PACKET
#include <minieuso_data_format.h>

CPU packet for incoming data every 5.24 s variable size

Public Members

CpuPktHeader cpu_packet_header
CpuTimeStamp cpu_time
HK_PACKET hk_packet
ZYNQ_PACKET zynq_packet
struct CPU_FILE
#include <minieuso_data_format.h>

CPU file to store one run shown here as demonstration only variable size

Public Members

CpuFileHeader cpu_file_header
CPU_PACKET cpu_run_payload[RUN_SIZE]
CpuFileTrailer cpu_file_trailer
struct SC_PACKET
#include <minieuso_data_format.h>

SC packet to store S-curve from Zynq 9437220 bytes

Public Members

CpuPktHeader sc_packet_header
CpuTimeStamp sc_time
uint16_t sc_start
uint16_t sc_step
uint16_t sc_stop
uint16_t sc_acc
Z_DATA_TYPE_SCURVE_V1 sc_data
struct SC_FILE
#include <minieuso_data_format.h>

SC file to store a single S-curve shown here as demonstration only 9437244 bytes (~9 MB)

Public Members

CpuFileHeader cpu_file_header
SC_PACKET scurve_packet
CpuFileTrailer cpu_file_trailer
struct HV_PACKET
#include <minieuso_data_format.h>

HV packet for HV switching data variable file size

Public Members

CpuPktHeader hv_packet_header
CpuTimeStamp hv_time
uint32_t N
ZynqBoardHeader zbh
std::vector<DATA_TYPE_HVPS_LOG_V1> hvps_log
struct HV_FILE
#include <minieuso_data_format.h>

HV file to store a single HVPS log shown here as demonstration only variable file size

Public Members

CpuFileHeader cpu_file_header
HV_PACKET hv_packet
CpuFileTrailer cpu_file_trailer