Fill data_archive with Ethica data

[RileyLazarou]

Investigate the Ethica data dump /project/6026434/interact/incoming_data/all_data.tar.gz and move all of it into the data_archive

TODO:

• Investigate data. Confirm it is complete (edit: incomplete, but this is known)
• Wave 0 saskatoon accel
• Wave 0 saskatoon gps
• Wave 1 victoria accel
• Wave 1 victoria gps
• Wave 1 vancouver accel
• Wave 1 vancouver gps
• Wave 1 saskatoon accel
• Wave 1 saskatoon gps
• Wave 1 montreal accel
• Wave 1 montreal gps
• Wave 2 victoria accel
• Wave 2 victoria gps
• Wave 2 vancouver accel
• Wave 2 vancouver gps
• Wave 2 saskatoon accel
• Wave 2 saskatoon gps
• Wave 2 montreal accel
• Wave 2 montreal gps
• Wave 3 victoria accel
• Wave 3 victoria gps
• Wave 3 vancouver accel
• Wave 3 vancouver gps
• Wave 3 saskatoon accel
• Wave 3 saskatoon gps
• Wave 3 montreal accel
• Wave 2 montreal gps

[RileyLazarou]

Ethica data seems good so far. Some Ethica data missing, see next comment

Ethica data contains:

Accelerometer:

• 395 (Montreal W1 (eng))
• 396 (Montreal W1 (fr))
• 1082 (Montreal W2 (fr))
• 1083 (Montreal W2 (eng))
• 2342 (Montreal W3 (fr and eng))
• 406 (Saskatoon W0)
• 448 (Saskatoon W1)
• 1318 (Saskatoon W2)
• 2341(Saskatoon W3)
• 212 (Victoria W1)
• 730 (Victoria W2)
• 1655 (Victoria W3)
• 376 (Vancouver W1)
• 1058 (Vancouver W2)
• 2340 (Vancouver W3)
• 2501 (COHESION 2.1)
• 2760 ((COHESION 2.2))
• 1127 (COHESION 1)
• 1128 (COHESION 1) (why does this have the same name as 1127?)
• 947 (Montreal REM (eng))
• 948 (Montreal REM (fr))
• 2160 (REM 2)

All GPS data files are accounted for, but several of them are empty, including:

• montreal/wave_01/ethica/395/395_gps.csv
• saskatoon/wave_00/ethica/406/406_gps.csv
• saskatoon/wave_01/ethica/448/448_gps.csv
• vancouver/wave_01/ethica/376/376_gps.csv
• victoria/wave_01/ethica/212/212_gps.csv

Similarly, montreal wave 1 study 396 (fr) only has 18 records in its GPS file.

[RileyLazarou]

There's accel data missing. Specifically:

• wave 0 saskatoon (406)
• wave 1 victoria (212)
• wave 1 vancouver (376)
• wave 1 saskatoon (448)
• wave 1 montreal (395, 396)

Also "missing" is the accel data for "Montreal REM" (947, 948) (idk what this is, but it was in Zoe's spreadsheet).

The tarball contained these directories:

• study_212 # empty
• study_376 # empty
• study_395 # empty
• study_396 # contains output-000001.csv and output-000002.csv
• study_406 # empty
• study_448 # empty
• study_947 # contains output-000001.csv and output-000002.csv
• study_948 # contains output-000001.csv and output-000002.csv

As you can see, these correspond to the missing accel data, but most of them are empty. The output-00000?.csv files seem to be accelerometer data, but they're in a different format from the other accel files. There is overlap in the user_ids and device_ids between the two files. Clarification regarding why the files are in the format (and how to reconcile them), and why five of the studies have no data (and how to get it) is required

[RileyLazarou]

Two new issues have been created to address this missing data

• MISSING DATA: Accelerometer #8
• MISSING DATA: GPS #9

[walkabilly]

OK. The W1 GPS and Accel data are gone from the Ethica side. We have backups. We need to file the gap that is missing with our existing backups to recreate the archive. I need to check the Montreal output-0000?.csv files. Maybe it had something to do with the fact that for W2 we used ProtoBuff for the file transfer and the extract made the data format slightly different? @bthierry-udm @zoeps any thoughts on this?

[bthierry-udm]

Sorry -- I can't really help here as the Ethica side of the data was Jeff's stuff.
Now, I think we should not spend to much time on this unless to patch known missing data in the datasets we already have on CC. This dump is more a ceinture et bretelles approach to our backup strategy, ain't it?

[RileyLazarou]

I got the missing data from the old permanent archive 👍