Skip to content

add numpy array to data container (ict.data.add) - copy vs. view

I supply data as

ict.data.add(arr)

# with

type(arr)
Out[20]: numpy.ndarray

arr.shape
Out[21]: (86677, 3)

arr.dtype
Out[22]: dtype('float64')

But it fails as soon as certain variables are called by key, e.g. if I call ict.write, which then tries to convert np.nan to vmiss:

Traceback (most recent call last):

  File "C:\Users\va6504\AppData\Local\Temp\ipykernel_16448\2416392629.py", line 1, in <cell line: 1>
    ict.write()

  File "D:\PROGRAMMING\Python\icartt_dev\src\icartt\dataset.py", line 876, in write
    self.writeData(f=f, fmt=fmt, delimiter=delimiter)

  File "D:\PROGRAMMING\Python\icartt_dev\src\icartt\dataset.py", line 865, in writeData
    self.data.write(f=f, fmt=fmt, delimiter=delimiter)

  File "D:\PROGRAMMING\Python\icartt_dev\src\icartt\dataset.py", line 104, in write
    d = self.denanify(self.data)

  File "D:\PROGRAMMING\Python\icartt_dev\src\icartt\dataset.py", line 97, in denanify
    dd[k][np.isnan(dd[k])] = miss

IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

After further inspection, I found that the problem was that I transposed the array before; np.transpose returns a view, not a copy. If I feed ict.data.add(arr.copy()), everything works fine.

So I was thinking if we might add a check if the user is passing a view or a copy. Reading How can I tell if NumPy creates a view or a copy? and this linked question, only .flags['OWNDATA'] seems like a viable option to me. But not sure how reliable that is for given question. In any case, it must be clear from the docs that ict.data.add expects a copy, not a view.

Edited by Florian Obersteiner
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information