low performance: creating & reading files with many lines of data rather slow

I noted that creating and reading large files (i.e. 10k+ lines of data, FFI 1001) is rather slow, on the order of seconds to minutes.

creation

ict = icartt.Dataset(format=icartt.Formats.FFI1001)
# ...

I create the data section from a 2D np array. Internally, that calls add for each line along the vertical axis of the array. That function has two conditionals, a loop and two nested conditionals. Also, it calls np.append for each line of data, which is inefficient compared to native Python's append.

reading

ict = icartt.Dataset(f)
# ...

For the data section, this calls addBulkFromTxt, from there on, it's basically the same as for creation above.

---

At the moment, I'm not sure what would be the best way to approach this. What I'm certain about is that this needs improvement, as I think this can scare off users. Speaking for myself, that's exactly what makes using the only "sort-of-official" NASA Ames Python package nappy painful ;-)

---

ideas

• load data section with np.loadtxt or np.genfromtxt?

  • we have that dependency already and it would offer convenient features, such as "vectorized" methods to replace vmiss with np.nan etc.

• load the data section line by line, but intermediately use a list of lists

  • Python's list methods are very efficient, so I think reading data to a list of lists first, then convert to a numpy array, then use vectorized methods e.g. for vmiss-to-nan would improve performance

added Ideas/Discussion Refactor labels

changed the description

removed Ideas/Discussion label

added Bug label

changed the description

changed the description

changed the description

changed the description

I agree.

I would have gone with using np.loadtxt / np.genfromtxt from the beginning, but did not due to the increased complexity when reading FFI2110 files (data structure: ivar + nvar + auxvars line, then nvar * "ibvar + depvar" lines, then again ivar + auxvars, and so forth, with the number of ibvar + depvar lines possibly different for each portion ...).

For FFI1001 using np.loadtxt / np.genfromtxt is straightforward, and we could implement that first, then continue with FFI2110 bulk reading.

The alternative you suggested (using native lists first) seems appealing as it could more easily cover the FFI2110 format and this would potentially mean more reusable code.

No strong feelings either way at the moment, will need to look at it in detail.

changed milestone to %publish as v2.0.0 on pypi

Now that test data is called correctly, this also becomes obvious when running the tests (bulkIO)

assigned to @christophknote

Have a look at the speedup_io branch in my fork, @FObersteiner . I have implemented a Numpy genfromtxt - based input solution, which is way quicker. Works for both FFI1001 and FFI2110. Users are required to prepare NumPy arrays as input, and make sure they put the data in the right order. But that is probably ok to ask from them.

See the tests for usage, have not done documentation yet. That should do the trick, no?

@christophknote cool, I take a look later tonight. In the meantime, I just pushed revisions for handling normal comments to my dev branch, https://mbees.med.uni-augsburg.de/gitlab/FObersteiner/icartt-dev/-/tree/dev;

• nLines are now also calculated correctly if the user supplies a multi-line comment as a single string
• keywords/ingest now handles multi-line comments that include a : but no new keyword
• bulk IO tests now enforce UTF-8

@christophknote I can't seem to find your fork, is it still private maybe?

Oops - was set to internal, which external users cannot see. Is public now, @FObersteiner !

@christophknote ok working now The revision looks fine to me! Took me a while to figure out that one has to set usemask=True to make the replacement with vmiss working. numpy docs are not very verbose there ^^

Do you want to merge this into MBEES/master first? As commented above, I found more bugs / "unexpected features" within the normal comments handling - at least half of those were probably introduced by myself ;-) I'll go ahead and make a PR as soon as you integrated the IO speed-up.

Good. Will merge into main now, work on docs and tests afterwards, so you can go ahead and make a PR for the normal comments...

... merged into master, @FObersteiner. Leaving issue open until tests and docs are up to date.

created branch 17-low-performance-creating-reading-files-with-many-lines-of-data-rather-slow to address this issue

Done with merging !18 (merged) .

closed