icartt¶

icartt is an ICARTT file format reader and writer for Python

The ICARTT data format is described here: https://www-air.larc.nasa.gov/missions/etc/IcarttDataFormat.htm

Examples¶

Reading an existing dataset¶

Simple format (FFI 1001)¶

import icartt
import pathlib

# load a new dataset from an existing file
wd = pathlib.Path(__file__).parent
ict = icartt.Dataset(wd / ".." / "example_data" / "expect_ok" / "DC8-20160517.ict")

# read some metadata
ict.PIName
ict.PIAffiliation
ict.missionName
ict.dataSourceDescription
ict.dataID
ict.locationID

# list variable names
[x for x in ict.variables]

# some info on a variable
ict.variables["Alt_ft"].units
ict.variables["Alt_ft"].miss

# get data for variable 'UTC':
ict.data["UTC"]

# get all data (NumPy array):
ict.data[:]

# get the altitude in feet for those data where UTC < 86400.0:
ict.data[ict.data["UTC"] < 86400.0]["Alt_ft"]

More complex (FFI 2110)¶

Identical to FFI1001, only the data structure is more complex.

import icartt
import pathlib

# load a new dataset from an existing file
wd = pathlib.Path(__file__).parent
ict = icartt.Dataset(
    wd / ".." / "example_data" / "expect_warn" / "AR_DC8_20050203_R0.ict"
)

# list variable names
[x for x in ict.variables]

# independent, independent bounded, dependent, auxiliary variables?
ict.independentVariable.shortname
ict.independentBoundedVariable.shortname
", ".join([x for x in ict.auxiliaryVariables])
", ".join([x for x in ict.dependentVariables])

# some info on a variable (units, missing value of "Latitude")
ict.variables["Latitude"].units
ict.variables["Latitude"].miss

# get steps for which data is available:
tsteps = [x for x in ict.data]

# let's look at the first time step data
ict.data[tsteps[0]]

# auxiliary data at this time step:
ict.data[tsteps[0]]["AUX"][:]

# dependent data at this time step:
tstepdata = ict.data[tsteps[0]]["DEP"][:]

# get the ozone mixing ratio for those data where Altitude < 10000.0:
tstepdata[tstepdata["Altitude[]"] < 10000.0]["O3_MR[]"]

Creating a new dataset¶

Simple format (FFI 1001)¶

import datetime

import icartt

# ------------------------------------------------------
# Phase 1: define file format, properties and variables
# ------------------------------------------------------

ict = icartt.Dataset(format=icartt.Formats.FFI1001)

ict.PIName = "Knote, Christoph"
ict.PIAffiliation = "Faculty of Medicine, University Augsburg, Germany"
ict.dataSourceDescription = "Example data"
ict.missionName = "MBEES"
# to simplify dates and avoid time zone issues, these dates are simple 3 item lists [YYYY, MM, DD]
ict.dateOfCollection = datetime.datetime.utcnow().timetuple()[:3]
ict.dateOfRevision = datetime.datetime.utcnow().timetuple()[:3]

ict.dataIntervalCode = [0]

ict.independentVariable = icartt.Variable(
    "Time_Start",
    "seconds_from_0_hours_on_valid_date",
    "Time_Start",
    "Time_Start",
    vartype=icartt.VariableType.IndependentVariable,
    scale=1.0,
    miss=-9999999,
)

ict.dependentVariables["Time_Stop"] = icartt.Variable(
    "Time_Stop",
    "seconds_from_0_hours_on_valid_date",
    "Time_Stop",
    "Time_Stop",
    scale=1.0,
    miss=-9999999,
)

ict.dependentVariables["Payload"] = icartt.Variable(
    "Payload", "some_units", "Payload", "Payload", scale=1.0, miss=-9999999
)

# Normal and Special Comments

# note: all comments can be multi-line per standard.
# To ensure portability the decision was made to represent them as lists,
# with each list item representing a line.

# E.g., Special Comments are simple lists, which we can append to:
ict.specialComments.append("Some comments on this dataset:")
ict.specialComments.append("They are just examples!")
ict.specialComments.append("Adapt as needed.")

# Normal Comments are separated into freeform and (required) keyword comments.

# Freeform comments are, just like special comments, a list with one line per entry
ict.normalComments.freeform.append("free comment line 1")
ict.normalComments.freeform.append("free comment line 2")

# Keywords are mandatory, all need to exist, and default to N/A
# they can be set as follows:
ict.normalComments.keywords["PI_CONTACT_INFO"].append(
    "Christoph Knote, MBEES, Faculty of Medicine, University Augsburg, Augsburg, Germany"
)
ict.normalComments.keywords["PI_CONTACT_INFO"].append(
    "christoph.knote@med.uni-augsburg.de"
)
ict.normalComments.keywords["PLATFORM"].append("ICARTT")
ict.normalComments.keywords["PROJECT_INFO"].append(
    "A format reader/writer for ICARTT files"
)
# ... and so forth

# note for revisions: we assume you take care of the required documentation of past versions
# and setting the correct date for the current revision (ict.dateOfRevision, see above)
ict.normalComments.keywords["REVISION"].append("R1")
ict.normalComments.keywords["REVISION"].append("R1: current status")
ict.normalComments.keywords["REVISION"].append("R0: early beta")

ict.endDefineMode()

# ------------------------------------------------------
# Phase 2: after ending define mode, add data
# ------------------------------------------------------

# all data are stored as NumPy arrays, and need to be provided as such.

import numpy as np

#                 ivar  dvar1 dvar2
data = np.array([(15.4, 15.0, 5.24524952903484e5)])
ict.data.add(data)

# Note 1: you are responsible to ensure that the order of elements in a data line
# corresponds to variable listing below:
print([x for x in ict.variables])

# Note 2: evenfor single lines, you still need to make it an array!

# Note 3: multiple data lines can be added in bulk!
#                 ivar  dvar1 dvar2       ivar  dvar1 dvar2      ...
data = np.array([(13.4, 14.0, 2.348925e5), (14.1, 14.9, 2.3425634e5)])

ict.data.add(data)

# Now, look at it in ICARTT form:
ict.write()

# And you could simply write to file:
# with open('output.ict', 'w') as f:
#    ict.write(f=f)

More complex (FFI 2110)¶

Again, like for FFI 1001, but using a more complex data structure.

import datetime

import numpy as np

import icartt

# ------------------------------------------------------
# Phase 1: define file format, properties and variables
# ------------------------------------------------------

ict = icartt.Dataset(format=icartt.Formats.FFI2110)

ict.PIName = "Knote, Christoph"
ict.PIAffiliation = "Faculty of Medicine, University Augsburg, Germany"
ict.dataSourceDescription = "Example data"
ict.missionName = "MBEES"
ict.dateOfCollection = datetime.datetime.utcnow().timetuple()[:3]
ict.dateOfRevision = datetime.datetime.utcnow().timetuple()[:3]

ict.dataIntervalCode = [0]

ict.independentVariable = icartt.Variable(
    "Time_Start",
    "seconds_from_0_hours_on_valid_date",
    "Time_Start",
    "Time_Start",
    vartype=icartt.VariableType.IndependentVariable,
    scale=1.0,
    miss=-9999999,
)

ict.independentBoundedVariable = icartt.Variable(
    "Altitude",
    "altitude_above_ground_in_meters",
    "Altitude",
    "Altitude",
    vartype=icartt.VariableType.IndependentBoundedVariable,
    scale=1.0,
    miss=-9999999,
)

# ICARTT convention: first aux variable contains number of dependent elements
ict.auxiliaryVariables["nAltitudes"] = icartt.Variable(
    "nAltitudes",
    "number_of_dependent_variable_items",
    "variable",
    "nAltitudes",
    scale=1.0,
    miss=-9999999,
)

ict.auxiliaryVariables["Time_Stop"] = icartt.Variable(
    "Time_Stop",
    "seconds_from_0_hours_on_valid_date",
    "Time_Stop",
    "Time_Stop",
    scale=1.0,
    miss=-9999999,
)

ict.auxiliaryVariables["Longitude"] = icartt.Variable(
    "Longitude",
    "longitude_in_degrees",
    "Longitude",
    "Longitude",
    scale=1.0,
    miss=-9999999,
)

ict.auxiliaryVariables["Latitude"] = icartt.Variable(
    "Latitude", "latitude_in_degrees", "Latitude", "Latitude", scale=1.0, miss=-9999999
)

ict.dependentVariables["Payload1"] = icartt.Variable(
    "Payload1", "some_units", "Payload1", "Payload1", scale=1.0, miss=-9999999
)

ict.dependentVariables["Payload2"] = icartt.Variable(
    "Payload2", "some_units", "Payload2", "Payload2", scale=1.0, miss=-9999999
)

ict.specialComments.append("Some comments on this dataset:")
ict.specialComments.append("They are just examples!")
ict.specialComments.append("Adapt as needed.")

ict.endDefineMode()

# ------------------------------------------------------
# Phase 2: after ending define mode, add data
# ------------------------------------------------------

import numpy as np

# same as for 1001, but data needs to be added in chunks for each ivar value

# note, the second variable ('4') is the number of dependent lines to follow
#                   ivar, ndepvar, auxvar1, auxvar2, auxvar3
auxData = np.array([(12.3, 4, 12.5, 48.21, 10.3)])
#                   ibvar,  dvar1,  dvar2  ... (repeat ndepvar times)
depData = np.array(
    [(0, 123, 8.4e4), (100, 122, 9.1e4), (250, 115, 9.3e4), (500, 106, 9.8e4)]
)

ict.data.add(auxData, depData)

# ... and so forth for the next chunk:

auxData = np.array([(12.4, 2, 12.8, 48.41, 12.1)])
#                   ibvar,  dvar1,  dvar2
depData = np.array([(0, 153, 7.3e4), (270, 172, 8.9e4)])

ict.data.add(auxData, depData)

# Now, look at it in ICARTT form:
ict.write()

# And you could simply write to file:
# with open('output.ict', 'w') as f:
#    ict.write(f=f)

API¶

Constants¶

Reported here for completeness, do not need to be changed by the user. Can be defined by the user upon Dataset / Variable creation and when writing to output.

icartt.dataset.DEFAULT_NUM_FORMAT = '%g'¶: Default number format for output. Provides the fmt parameter of numpy.savetxt() internally.

icartt.dataset.DEFAULT_FIELD_DELIM = ','¶: Default field delimiter

icartt.dataset.DEFAULT_SCALE_FACTOR = 1.0¶: Default variable scale factor

icartt.dataset.DEFAULT_MISSING_VALUE = -9999.0¶: Default variable missing value

Dataset¶

class icartt.Dataset(f=None, loadData=True, delimiter=DEFAULT_FIELD_DELIM, format=Formats.FFI1001)¶

An ICARTT dataset that can be created from scratch or read from a file, manipulated, and then written to a file.

Parameters

f (str or file object, optional) – file path or file handle to use, defaults to None
loadData (bool, optional) – whether to load data as well (or only header if False), defaults to True
delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM
format (Formats, optional) – ICARTT file format to create, defaults to 1001

endDefineMode()¶: Fixes the variables structure of the dataset. Sets up the data store, so data can be added. Needs to be called after variable definition and before adding data.

isValidFileName(name)¶

test whether file name complies with ICARTT standard:

ICARTT standard v2 2.1.1 3)

Filename: Uppercase and lowercase ASCII alphanumeric characters (i.e. A-Z, a-z, 0-9), underscore, period, and hyphen. File names can be a maximum 127 characters in length.

Parameters: name (str) – file name
Returns: is file name valid according to ICARTT standard?
Return type: bool

makeFileName(dateFormat='%Y%m%d')¶

Create ICARTT-compliant file name based on the information contained in the dataset

Parameters: dateFormat (str, optional) – date format to use when parsing, defaults to ‘%Y%m%d’
Returns: file name generated
Return type: str

property nHeader¶

Header line count

Returns: line count
Return type: int

read(delimiter=DEFAULT_FIELD_DELIM)¶

Read ICARTT data and header

Parameters: delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM

readData(delimiter=DEFAULT_FIELD_DELIM)¶

Read ICARTT data (from file)

Parameters: delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM

readHeader(delimiter=DEFAULT_FIELD_DELIM)¶

Read the ICARTT header (from file)

Parameters: delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM

property times¶

Time steps of the data

Returns: array of time steps
Return type: numpy.ndarray

property variables¶

Variables (independent + dependent + auxiliary)

Returns: dictionary of all variables
Return type: dict

write(f=sys.stdout, fmt=DEFAULT_NUM_FORMAT, delimiter=DEFAULT_FIELD_DELIM)¶

Write header and data

Parameters

f (handle, optional) –
file object to write to, defaults to sys.stdout
fmt (str or sequence of str, optional) – format string for output, accepts anything numpy.savetxt() would, defaults to DEFAULT_NUM_FORMAT
delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM

writeData(f=sys.stdout, fmt=DEFAULT_NUM_FORMAT, delimiter=DEFAULT_FIELD_DELIM)¶

Write data

Parameters

f (handle, optional) –
file object to write to, defaults to sys.stdout
fmt (str or sequence of str, optional) – format string for output, accepts anything numpy.savetxt() would, defaults to DEFAULT_NUM_FORMAT
delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM

writeHeader(f=sys.stdout, delimiter=DEFAULT_FIELD_DELIM)¶

Write header

Parameters

f (handle, optional) –
file object to write to, defaults to sys.stdout
delimiter (str, optional) – field delimiter character(s) for output, defaults to DEFAULT_FIELD_DELIM

DataStore¶

The .data attribute of a Dataset is a DataStore, which can be accessed to add data as follows:

class icartt.DataStore1001(ivar, dvars)¶

Data model for FFI1001

add(newData)¶

(bulk) add data, providing a (structured) numpy array.

Array has to have shape [ (ivar, dvar, dvar, …), … ], missing values have to be set to numpy.nan.

Parameters: newData (numpy.ndarray) – data to be added

class icartt.DataStore2110(ivar, ibvar, auxvars, dvars)¶

Data model for FFI2110

add(newAuxData, newDepData)¶

(bulk) add data, providing (structured) numpy arrays for both the auxiliary and dependent data line(s) for a given ivar value.

Arrays have to have shape [ (ivar, auxvar, auxvar, …) ] and [ (ibvar, depvar, depvar, …), … ] for auxiliary and dependent data line(s), respectively. missing values have to be set to numpy.nan.

Parameters

newAuxData (numpy.ndarray) – auxiliary data line to be added
newDepData (numpy.ndarray) – auxiliary data line(s) to be added

Formats¶

enum icartt.Formats(value)¶

ICARTT File Format Indices (FFI)

Member Type: int

Valid values are as follows:

FFI1001 = <Formats.FFI1001: 1001>¶

FFI2110 = <Formats.FFI2110: 2110>¶

Variable¶

class icartt.Variable(shortname, units, standardname, longname, vartype=VariableType.DependentVariable, scale=DEFAULT_SCALE_FACTOR, miss=DEFAULT_MISSING_VALUE)¶

An ICARTT variable description with name, units, scale and missing value.

Parameters

shortname (str) – Short name of the variable
units (str) – Units of the variable
standardname (str) – Standard name of the variable
longname (str) – Long name of the variable
vartype (VariableType, optional) – Variable type (unbounded/bounded independent or dependent), defaults to VariableType.dependentVariable
scale (float, optional) – Scaling factor for the variable, defaults to DEFAULT_SCALE_FACTOR
miss (float, optional) – Missing value for the variable, defaults to DEFAULT_MISSING_VALUE

desc(delimiter=DEFAULT_FIELD_DELIM)¶

Variable description string as it appears in an ICARTT file

Parameters: delimiter (str, optional) – field delimiter character(s), defaults to DEFAULT_FIELD_DELIM
Returns: description string
Return type: str

Variable types¶

enum icartt.VariableType(value)¶

ICARTT Variable Types

Member Type: int

Valid values are as follows:

IndependentVariable = <VariableType.IndependentVariable: 1>¶

IndependentBoundedVariable = <VariableType.IndependentBoundedVariable: 2>¶

AuxiliaryVariable = <VariableType.AuxiliaryVariable: 3>¶

DependentVariable = <VariableType.DependentVariable: 4>¶

icartt¶

Examples¶

Reading an existing dataset¶

Simple format (FFI 1001)¶

More complex (FFI 2110)¶

Creating a new dataset¶

Simple format (FFI 1001)¶

More complex (FFI 2110)¶

API¶

Constants¶

Dataset¶

DataStore¶

Formats¶

Variable¶

Variable types¶

Indices and tables¶

Table of Contents

This Page