icartt¶
icartt is an ICARTT file format reader and writer for Python
The ICARTT data format is described here: https://www-air.larc.nasa.gov/missions/etc/IcarttDataFormat.htm
Examples¶
Reading an existing dataset¶
Simple format (FFI 1001)¶
import icartt
import pathlib
# load a new dataset from an existing file
wd = pathlib.Path(__file__).parent
ict = icartt.Dataset(wd / ".." / "example_data" / "expect_ok" / "DC8-20160517.ict")
# read some metadata
ict.PIName
ict.PIAffiliation
ict.missionName
ict.dataSourceDescription
ict.dataID
ict.locationID
# list variable names
[x for x in ict.variables]
# some info on a variable
ict.variables["Alt_ft"].units
ict.variables["Alt_ft"].miss
# get data for variable 'UTC':
ict.data["UTC"]
# get all data (NumPy array):
ict.data[:]
# get the altitude in feet for those data where UTC < 86400.0:
ict.data[ict.data["UTC"] < 86400.0]["Alt_ft"]
More complex (FFI 2110)¶
Identical to FFI1001, only the data structure is more complex.
import icartt
import pathlib
# load a new dataset from an existing file
wd = pathlib.Path(__file__).parent
ict = icartt.Dataset(
wd / ".." / "example_data" / "expect_warn" / "AR_DC8_20050203_R0.ict"
)
# list variable names
[x for x in ict.variables]
# independent, independent bounded, dependent, auxiliary variables?
ict.independentVariable.shortname
ict.independentBoundedVariable.shortname
", ".join([x for x in ict.auxiliaryVariables])
", ".join([x for x in ict.dependentVariables])
# some info on a variable (units, missing value of "Latitude")
ict.variables["Latitude"].units
ict.variables["Latitude"].miss
# get steps for which data is available:
tsteps = [x for x in ict.data]
# let's look at the first time step data
ict.data[tsteps[0]]
# auxiliary data at this time step:
ict.data[tsteps[0]]["AUX"][:]
# dependent data at this time step:
tstepdata = ict.data[tsteps[0]]["DEP"][:]
# get the ozone mixing ratio for those data where Altitude < 10000.0:
tstepdata[tstepdata["Altitude[]"] < 10000.0]["O3_MR[]"]
Creating a new dataset¶
Simple format (FFI 1001)¶
import datetime
import icartt
# ------------------------------------------------------
# Phase 1: define file format, properties and variables
# ------------------------------------------------------
ict = icartt.Dataset(format=icartt.Formats.FFI1001)
ict.PIName = "Knote, Christoph"
ict.PIAffiliation = "Faculty of Medicine, University Augsburg, Germany"
ict.dataSourceDescription = "Example data"
ict.missionName = "MBEES"
# to simplify dates and avoid time zone issues, these dates are simple 3 item lists [YYYY, MM, DD]
ict.dateOfCollection = datetime.datetime.utcnow().timetuple()[:3]
ict.dateOfRevision = datetime.datetime.utcnow().timetuple()[:3]
ict.dataIntervalCode = [0]
ict.independentVariable = icartt.Variable(
"Time_Start",
"seconds_from_0_hours_on_valid_date",
"Time_Start",
"Time_Start",
vartype=icartt.VariableType.IndependentVariable,
scale=1.0,
miss=-9999999,
)
ict.dependentVariables["Time_Stop"] = icartt.Variable(
"Time_Stop",
"seconds_from_0_hours_on_valid_date",
"Time_Stop",
"Time_Stop",
scale=1.0,
miss=-9999999,
)
ict.dependentVariables["Payload"] = icartt.Variable(
"Payload", "some_units", "Payload", "Payload", scale=1.0, miss=-9999999
)
# Normal and Special Comments
# note: all comments can be multi-line per standard.
# To ensure portability the decision was made to represent them as lists,
# with each list item representing a line.
# E.g., Special Comments are simple lists, which we can append to:
ict.specialComments.append("Some comments on this dataset:")
ict.specialComments.append("They are just examples!")
ict.specialComments.append("Adapt as needed.")
# Normal Comments are separated into freeform and (required) keyword comments.
# Freeform comments are, just like special comments, a list with one line per entry
ict.normalComments.freeform.append("free comment line 1")
ict.normalComments.freeform.append("free comment line 2")
# Keywords are mandatory, all need to exist, and default to N/A
# they can be set as follows:
ict.normalComments.keywords["PI_CONTACT_INFO"].append(
"Christoph Knote, MBEES, Faculty of Medicine, University Augsburg, Augsburg, Germany"
)
ict.normalComments.keywords["PI_CONTACT_INFO"].append(
"christoph.knote@med.uni-augsburg.de"
)
ict.normalComments.keywords["PLATFORM"].append("ICARTT")
ict.normalComments.keywords["PROJECT_INFO"].append(
"A format reader/writer for ICARTT files"
)
# ... and so forth
# note for revisions: we assume you take care of the required documentation of past versions
# and setting the correct date for the current revision (ict.dateOfRevision, see above)
ict.normalComments.keywords["REVISION"].append("R1")
ict.normalComments.keywords["REVISION"].append("R1: current status")
ict.normalComments.keywords["REVISION"].append("R0: early beta")
ict.endDefineMode()
# ------------------------------------------------------
# Phase 2: after ending define mode, add data
# ------------------------------------------------------
# all data are stored as NumPy arrays, and need to be provided as such.
import numpy as np
# ivar dvar1 dvar2
data = np.array([(15.4, 15.0, 5.24524952903484e5)])
ict.data.add(data)
# Note 1: you are responsible to ensure that the order of elements in a data line
# corresponds to variable listing below:
print([x for x in ict.variables])
# Note 2: evenfor single lines, you still need to make it an array!
# Note 3: multiple data lines can be added in bulk!
# ivar dvar1 dvar2 ivar dvar1 dvar2 ...
data = np.array([(13.4, 14.0, 2.348925e5), (14.1, 14.9, 2.3425634e5)])
ict.data.add(data)
# Now, look at it in ICARTT form:
ict.write()
# And you could simply write to file:
# with open('output.ict', 'w') as f:
# ict.write(f=f)
More complex (FFI 2110)¶
Again, like for FFI 1001, but using a more complex data structure.
import datetime
import numpy as np
import icartt
# ------------------------------------------------------
# Phase 1: define file format, properties and variables
# ------------------------------------------------------
ict = icartt.Dataset(format=icartt.Formats.FFI2110)
ict.PIName = "Knote, Christoph"
ict.PIAffiliation = "Faculty of Medicine, University Augsburg, Germany"
ict.dataSourceDescription = "Example data"
ict.missionName = "MBEES"
ict.dateOfCollection = datetime.datetime.utcnow().timetuple()[:3]
ict.dateOfRevision = datetime.datetime.utcnow().timetuple()[:3]
ict.dataIntervalCode = [0]
ict.independentVariable = icartt.Variable(
"Time_Start",
"seconds_from_0_hours_on_valid_date",
"Time_Start",
"Time_Start",
vartype=icartt.VariableType.IndependentVariable,
scale=1.0,
miss=-9999999,
)
ict.independentBoundedVariable = icartt.Variable(
"Altitude",
"altitude_above_ground_in_meters",
"Altitude",
"Altitude",
vartype=icartt.VariableType.IndependentBoundedVariable,
scale=1.0,
miss=-9999999,
)
# ICARTT convention: first aux variable contains number of dependent elements
ict.auxiliaryVariables["nAltitudes"] = icartt.Variable(
"nAltitudes",
"number_of_dependent_variable_items",
"variable",
"nAltitudes",
scale=1.0,
miss=-9999999,
)
ict.auxiliaryVariables["Time_Stop"] = icartt.Variable(
"Time_Stop",
"seconds_from_0_hours_on_valid_date",
"Time_Stop",
"Time_Stop",
scale=1.0,
miss=-9999999,
)
ict.auxiliaryVariables["Longitude"] = icartt.Variable(
"Longitude",
"longitude_in_degrees",
"Longitude",
"Longitude",
scale=1.0,
miss=-9999999,
)
ict.auxiliaryVariables["Latitude"] = icartt.Variable(
"Latitude", "latitude_in_degrees", "Latitude", "Latitude", scale=1.0, miss=-9999999
)
ict.dependentVariables["Payload1"] = icartt.Variable(
"Payload1", "some_units", "Payload1", "Payload1", scale=1.0, miss=-9999999
)
ict.dependentVariables["Payload2"] = icartt.Variable(
"Payload2", "some_units", "Payload2", "Payload2", scale=1.0, miss=-9999999
)
ict.specialComments.append("Some comments on this dataset:")
ict.specialComments.append("They are just examples!")
ict.specialComments.append("Adapt as needed.")
ict.endDefineMode()
# ------------------------------------------------------
# Phase 2: after ending define mode, add data
# ------------------------------------------------------
import numpy as np
# same as for 1001, but data needs to be added in chunks for each ivar value
# note, the second variable ('4') is the number of dependent lines to follow
# ivar, ndepvar, auxvar1, auxvar2, auxvar3
auxData = np.array([(12.3, 4, 12.5, 48.21, 10.3)])
# ibvar, dvar1, dvar2 ... (repeat ndepvar times)
depData = np.array(
[(0, 123, 8.4e4), (100, 122, 9.1e4), (250, 115, 9.3e4), (500, 106, 9.8e4)]
)
ict.data.add(auxData, depData)
# ... and so forth for the next chunk:
auxData = np.array([(12.4, 2, 12.8, 48.41, 12.1)])
# ibvar, dvar1, dvar2
depData = np.array([(0, 153, 7.3e4), (270, 172, 8.9e4)])
ict.data.add(auxData, depData)
# Now, look at it in ICARTT form:
ict.write()
# And you could simply write to file:
# with open('output.ict', 'w') as f:
# ict.write(f=f)
API¶
Constants¶
Reported here for completeness, do not need to be changed by the user. Can be defined by the user upon Dataset
/ Variable
creation and when writing to output.
- icartt.dataset.DEFAULT_NUM_FORMAT = '%g'¶
Default number format for output. Provides the fmt parameter of
numpy.savetxt()
internally.
- icartt.dataset.DEFAULT_FIELD_DELIM = ','¶
Default field delimiter
- icartt.dataset.DEFAULT_SCALE_FACTOR = 1.0¶
Default variable scale factor
- icartt.dataset.DEFAULT_MISSING_VALUE = -9999.0¶
Default variable missing value
Dataset¶
- class icartt.Dataset(f=None, loadData=True, delimiter=DEFAULT_FIELD_DELIM, format=Formats.FFI1001)¶
An ICARTT dataset that can be created from scratch or read from a file, manipulated, and then written to a file.
- Parameters
f (str or file object, optional) – file path or file handle to use, defaults to None
loadData (bool, optional) – whether to load data as well (or only header if False), defaults to True
delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM
format (Formats, optional) – ICARTT file format to create, defaults to 1001
- endDefineMode()¶
Fixes the variables structure of the dataset. Sets up the data store, so data can be added. Needs to be called after variable definition and before adding data.
- isValidFileName(name)¶
test whether file name complies with ICARTT standard:
ICARTT standard v2 2.1.1 3)
Filename: Uppercase and lowercase ASCII alphanumeric characters (i.e. A-Z, a-z, 0-9), underscore, period, and hyphen. File names can be a maximum 127 characters in length.
- makeFileName(dateFormat='%Y%m%d')¶
Create ICARTT-compliant file name based on the information contained in the dataset
- read(delimiter=DEFAULT_FIELD_DELIM)¶
Read ICARTT data and header
- Parameters
delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM
- readData(delimiter=DEFAULT_FIELD_DELIM)¶
Read ICARTT data (from file)
- Parameters
delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM
- readHeader(delimiter=DEFAULT_FIELD_DELIM)¶
Read the ICARTT header (from file)
- Parameters
delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM
- property times¶
Time steps of the data
- Returns
array of time steps
- Return type
- property variables¶
Variables (independent + dependent + auxiliary)
- Returns
dictionary of all variables
- Return type
- write(f=sys.stdout, fmt=DEFAULT_NUM_FORMAT, delimiter=DEFAULT_FIELD_DELIM)¶
Write header and data
- Parameters
f (handle, optional) –
file object to write to, defaults to sys.stdout
fmt (str or sequence of str, optional) – format string for output, accepts anything
numpy.savetxt()
would, defaults to DEFAULT_NUM_FORMATdelimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM
- writeData(f=sys.stdout, fmt=DEFAULT_NUM_FORMAT, delimiter=DEFAULT_FIELD_DELIM)¶
Write data
- Parameters
f (handle, optional) –
file object to write to, defaults to sys.stdout
fmt (str or sequence of str, optional) – format string for output, accepts anything
numpy.savetxt()
would, defaults to DEFAULT_NUM_FORMATdelimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM
- writeHeader(f=sys.stdout, delimiter=DEFAULT_FIELD_DELIM)¶
Write header
- Parameters
f (handle, optional) –
file object to write to, defaults to sys.stdout
delimiter (str, optional) – field delimiter character(s) for output, defaults to DEFAULT_FIELD_DELIM
DataStore¶
The .data attribute of a Dataset is a DataStore, which can be accessed to add data as follows:
- class icartt.DataStore1001(ivar, dvars)¶
Data model for FFI1001
- add(newData)¶
(bulk) add data, providing a (structured) numpy array.
Array has to have shape [ (ivar, dvar, dvar, …), … ], missing values have to be set to
numpy.nan
.- Parameters
newData (numpy.ndarray) – data to be added
- class icartt.DataStore2110(ivar, ibvar, auxvars, dvars)¶
Data model for FFI2110
- add(newAuxData, newDepData)¶
(bulk) add data, providing (structured) numpy arrays for both the auxiliary and dependent data line(s) for a given ivar value.
Arrays have to have shape [ (ivar, auxvar, auxvar, …) ] and [ (ibvar, depvar, depvar, …), … ] for auxiliary and dependent data line(s), respectively. missing values have to be set to
numpy.nan
.- Parameters
newAuxData (numpy.ndarray) – auxiliary data line to be added
newDepData (numpy.ndarray) – auxiliary data line(s) to be added
Formats¶
Variable¶
- class icartt.Variable(shortname, units, standardname, longname, vartype=VariableType.DependentVariable, scale=DEFAULT_SCALE_FACTOR, miss=DEFAULT_MISSING_VALUE)¶
An ICARTT variable description with name, units, scale and missing value.
- Parameters
shortname (str) – Short name of the variable
units (str) – Units of the variable
standardname (str) – Standard name of the variable
longname (str) – Long name of the variable
vartype (VariableType, optional) – Variable type (unbounded/bounded independent or dependent), defaults to VariableType.dependentVariable
scale (float, optional) – Scaling factor for the variable, defaults to DEFAULT_SCALE_FACTOR
miss (float, optional) – Missing value for the variable, defaults to DEFAULT_MISSING_VALUE
Variable types¶
- enum icartt.VariableType(value)¶
ICARTT Variable Types
- Member Type
Valid values are as follows:
- IndependentVariable = <VariableType.IndependentVariable: 1>¶
- IndependentBoundedVariable = <VariableType.IndependentBoundedVariable: 2>¶
- AuxiliaryVariable = <VariableType.AuxiliaryVariable: 3>¶
- DependentVariable = <VariableType.DependentVariable: 4>¶