BUG: file hash uint64's converted to float64's in darshan.report.mod_read_all_records()

[nawtrey]

As mentioned here, we are seeing some file system hashes that are floats despite the name records (report.data["name_records"]) containing integer values.

I looked into this and apparently the issue is log-dependent, where some logs are generating pandas dataframes with ids/hashes of type int64 and other logs are returning dataframes with ids/hashes of type float64. I traced this back to darshan.report.mod_read_all_records(), in the if dtype == pandas code block. Each record (pre-concatenation) has id's that are either int64's or uint64's, but upon concatenation, get uint64's get converted to float64's. This is apparently expected behavior for pandas since np.int64's cannot contain the largest values from uint64's. I was able to find some discussion about it here.

This is unwanted behavior since we want to be able to make safe comparisons between the file hashes stored in report.data["name_records"] and the counter/fcounter dataframes. As far as solutions go, I don't think we can force pandas to keep the uint64 type when it concatenates the dataframes, but we could use astype() to change the float64's back to uint64's after the the concatenation step.

[tylerjereddy]

Looks like @jakobluettgau set the hash to typedef uint64_t darshan_record_id; in the api_def_c.py source--perhaps uint64_t could be enforced consistently farther up in the control flow.

Indeed, this seems to do the trick locally so that we never cast to or from floats for the id at all (for the pandas case):

index 817f22f..587b780 100644
--- a/darshan-util/pydarshan/darshan/backend/cffi_backend.py
+++ b/darshan-util/pydarshan/darshan/backend/cffi_backend.py
@@ -356,6 +356,7 @@ def log_get_generic_record(log, mod_name, dtype='numpy'):
             })
 
     if dtype == "pandas":
+        rec['id'] = np.uint64(rec['id'])
         df_c = pd.DataFrame(cdict, index=[0])
         df_fc = pd.DataFrame(fcdict, index=[0])

If that seems to work, then the regression testing may not be too hard if we already have example logs that flip between the types based on the size their integer hashes happen to be before pandas casts them.

[tylerjereddy]

(as an aside, this entire control flow looks pretty inefficient with while rec != None: calling that generic function over and over again to generate 1-row dataframes (!) and then append them, which may be one of the inefficiencies you have noticed, but probably best to uncouple the bug fix from attempts to speed up for now).

[nawtrey]

(as an aside, this entire control flow looks pretty inefficient with while rec != None: calling that generic function over and over again to generate 1-row dataframes (!) and then append them, which may be one of the inefficiencies you have noticed, but probably best to uncouple the bug fix from attempts to speed up for now).

Yes, this was the other major bottleneck I found slowing down the data_access_by_filesystem generation, along with some of the array/dataframe handling in log_get_generic_record(). I'll open an issue about potential performance improvements.

[nawtrey]

index 817f22f..587b780 100644
--- a/darshan-util/pydarshan/darshan/backend/cffi_backend.py
+++ b/darshan-util/pydarshan/darshan/backend/cffi_backend.py
@@ -356,6 +356,7 @@ def log_get_generic_record(log, mod_name, dtype='numpy'):
             })
 
     if dtype == "pandas":
+        rec['id'] = np.uint64(rec['id'])
         df_c = pd.DataFrame(cdict, index=[0])
         df_fc = pd.DataFrame(fcdict, index=[0])

If that seems to work, then the regression testing may not be too hard if we already have example logs that flip between the types based on the size their integer hashes happen to be before pandas casts them.

Okay, this works for me as well, and simpler than what I was looking at.

For the logs in tests/input, here are the id data types from the output dataframe:

• sample.darshan -- int
• sample-goodost.darshan -- float
• sample-dxt-simple.darshan -- float

I'll try to write a regression test for it.

[tylerjereddy]

Fixed by #442