File I/O

Opening and closing a file

Let us see how we can open a file. First define some variable names that represent the file names of the files we are going to use:

>>> filename_lofar = LOFARSOFT+"/data/lofar/VHECR_example.h5"

We can create a new file object, using the open() method , which is an interface to the LOFAR CRTOOLS datareader class.

The following will open a data file and return a DataReader object:

>>> datafile = open(filename_lofar)

The associated filename can be retrieved with:

>>> datafile["FILENAME"]
'/home/username/lofarsoft/data/lofar/VHECR_example.h5'

The file will be automatically closed (and the object will be destroyed), whenever the open object is deleted or overwritten, e.g. with datafile = 0.

Retrieving and setting metadata

Now we need to access the metadata in the file. This can be done by providing a keyword to the datafile object, e.g. type:

>>> datafile["FREQUENCY_RANGE"]
[(0.0, 100000000.0),
 (0.0, 100000000.0),
 ...
 (0.0, 100000000.0),
 (0.0, 100000000.0)]

to get a list of frequency ranges for each antenna.

To get a list of valid keywords type:

>>> datafile.keys()
['OBSERVATION_START_UTC',
 'DIPOLE_CALIBRATION_DELAY',
 ...
 'FILEDATE',
 'FILENAME']

A complete list of the keywords, sorted by name, and their values can be obtained by:

>>> datafile.info()
[TBB_Timeseries] Summary of object properties
ALIGNMENT_REFERENCE_ANTENNA        : 16
ANTENNA_POSITION                   : [3826579.4925, 461005.1052, 5064892.578, '...', 3826607.8225, 461026.5902, 5064869.371]
...
TIME_DATA                          : hArray(float, [1024], fill=[0,5e-09,1e-08,1.5e-08,2e-08,...,5.095e-06,5.1e-06,5.105e-06,5.11e-06,5.115e-06], name="Time") # len=1024 slice=[0:1024])
TIME_HR                            : [ 2012-04-28 05:00:21, ...] x96 with <type 'str'>
These keywords are UNDEFINED       : [FILEDATE , NOTES , OBSERVATION_END_TAI , OBSERVATION_START_TAI , PIPELINE_NAME , PIPELINE_VERSION , PROJECT_CO_I , PROJECT_ID , PROJECT_PI , SYSTEM_VERSION , ]

To set the data attributes you can simply use the same attribute naming as mentioned above, e.g.:

>>> datafile["BLOCKSIZE"] = 2048

Reading in data

The next step is to actually read in data. This is done with the read() method.

Before this is done, one has to allocate the memory in which the data is put. Although this requires one to program carefully and understand the data structure, this improves speed and efficiency.

Let’s first create a FloatArray of the correct dimensions, naming it fxdata and with the unit set to counts:

>>> fxdata = datafile.empty("TIMESERIES_DATA")

This is now a large vector filled with zeros and the name E-Field(t).

Now we can read in the raw timeseries data, either by using datafile.read() and a keyword, or by using the read() method of arrays, or by using the method readTimeSeries, e.g.:

>>> datafile.read("TIMESERIES_DATA", fxdata)

or:

>>> fxdata.read(datafile, "TIMESERIES_DATA")

or:

>>> datafile.getTimeseriesData(fxdata)

The types of data that can be read are TIMESERIES_DATA, FFT_DATA, FREQUENCY_DATA, and TIME_DATA. The corresponding get-methods of the datafile object are getTimeseriesData(), getFFTData(), getFrequencies(), and getTimeData().

You can also use these keywords with the empty() method of the datafile object to create an empty hArray of the correct type and size with the proper name and unit set.

Below is an example that shows how to read in data to calculate an average spectrum:

>>> fftdata = datafile.empty("FFT_DATA")
>>> nblocks = datafile['MAXIMUM_READ_LENGTH'] / datafile['BLOCKSIZE']
>>> avspectrum = hArray(float, dimensions=fftdata, name="Average spectrum")
>>> for block in range(nblocks):
>>>     datafile["BLOCK"] = block
>>>     fftdata.read(datafile, "FFT_DATA")
>>>     hSpectralPower(avspectrum[...], fftdata[...])