Check sample uniqueness in the accucor loader script

[hepcat72]

BUG DESCRIPTION

Problem

Issue #45's requirements include a requirement that the sample names must be unique. In the Issue Owner / Requirements section, it says as one of the requirements for the script (i.e. not an assumption):

sample names are unique with the file

PR #55 did not implement this requirement from the associated issue. It is unclear what will happen in the code if a sample name in the column headers of an accucor output file is not unique (whether the data beneath it is the same or different).

Steps to reproduce

1. Take an existing accucor output file. Copy two of the sample name column headers and paste them over top of other existing column headers of different columns (containing different data)
2. Run the accucor data loader script

Current behavior

Unknown

Expected behavior

The data should not load and an error should be printed indicating the duplicate sample names and the column numbers with a message that sample names must be unique.

Suggested Change

Check the uniqueness of the sample names in the data validation step.

Comment

Originally posted by @hepcat72 in #55 (comment)

B. There's no check in here about the per-file uniqueness of the sample names, which was mentioned as a requirement in the issue (although perhaps I inferred that - either way, I think it probably should be checked).

---

ISSUE OWNER SECTION

Assumptions

• List of assumptions made WRT the code
• E.g. We will assume input is correct (explaining why there is no validation)

Requirements

• List of conditions to be met for the feature
• E.g. Every column/row must display a value, i.e. cannot be empty

Limitations

• A list of things this work will not fix
• E.g. Getting edge case X to work requires too much effort and will not be
  fixed in this effort.

Affected/Changed Components

• Files
• Environment variables
• External executables
• Database tables
• Cron job settings
• Etc.

DESIGN

GUI Change description

Describe changes the user will see.

Code Change Description (Pseudocode optional)

Describe code changes planned for the fix.

Tests

A test should be planned for each requirement (above), where possible.

[jcmatese]

Just noting that the default behavior (pandas) is that the duplicate columns names (in xlsx) are auto-postfixed with .1 after reading and the loader would likely fail with an AssertionError (missing sample), unless the auto-prefixed name matched some other sample in the database, by chance. For example if the sample name BAT-xz971 were duplicated in another column, you would most likely get "Could not find sample BAT-xz971.1 in the database." AssertionError: 1 samples are missing.

[hepcat72]

I wrote a test. I need to verify it, but it seems that if you provide a duplicate sample in the input file, you get the following nonsensical exception:

ERROR: setUpClass (DataRepo.tests.test_models.DataLoadingTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rleach/PROJECT-local/TRACEBASE/REPO/TraceBase/.venv/lib/python3.9/site-packages/django/test/testcases.py", line 1123, in setUpClass
    cls.setUpTestData()
  File "/Users/rleach/PROJECT-local/TRACEBASE/REPO/TraceBase/DataRepo/tests/test_models.py", line 375, in setUpTestData
    call_command(
  File "/Users/rleach/PROJECT-local/TRACEBASE/REPO/TraceBase/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "/Users/rleach/PROJECT-local/TRACEBASE/REPO/TraceBase/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/Users/rleach/PROJECT-local/TRACEBASE/REPO/TraceBase/DataRepo/management/commands/load_samples.py", line 36, in handle
    loader.load_sample_table(
  File "/Users/rleach/PROJECT-local/TRACEBASE/REPO/TraceBase/DataRepo/utils.py", line 107, in load_sample_table
    name=row[self.headers.STUDY_NAME]
KeyError: 'Experiment Name'

I will see if I can turn this into a sensible error about duplicate sample names.

[hepcat72]

The above error is a red herring. I had a typo in the file name. It appears that 1 record results from when I supply rows with duplicate sample names. Now I need to determine whether the second of the duplicate is overwriting or whether it generates an error (my test has its own issue, so hopefully once I fix that, I'll know the answer).

[hepcat72]

Alright, I've worked out more of what happens now. In my dupe sample column test, it seems that something is appending .<#> to force sample names to be unique. I suspect pandas is doing this when it reads the file - but I will check on that. The first sample name instance is unmodified. The second one gets .1 appended (presumably to make it unique) and ultimately produces a N samples are missing error with a print above it saying Could not find sample <sample name>.1 in the database..