Retired characteristic names - corrected for WQX data but not USGS data

[ehinman]

Describe the bug
There are several characteristic names existing in WQX-submitted data that have been retired in favor of more concise or clearer names. These characteristic names are denoted by [old name]retired use [new name], and have the corrected target name (new name) in the WQX QAQC validation table. The TADA team recently created a new function that automatically substitutes the target name into the TADA.CharacteristicName field. However, I noticed that USGS data may still carry the old name, and without the retired use [new name] suffix in the CharacteristicName field, they do not join to the WQX QAQC validation table and thus do not get corrected.

To Reproduce
I found this issue with the characteristic names: Inorganic nitrogen (nitrate and nitrite) and Inorganic nitrogen (nitrate and nitrite) retireduse Nitrate + Nitrite

Code to reproduce the behavior:

library(TADA)

test = TADA_DataRetrieval(startDate = "2022-10-01", statecode = "OR")
test = subset(test, grepl("Inorganic nitrogen",test$CharacteristicName))
unique(test$CharacteristicName)
unique(test$TADA.CharacteristicName)

Expected behavior
All of the results in the example above should be converted to TADA.CharacteristicName "NITRATE + NITRITE" for data harmonization purposes.

Reminders for TADA contributors addressing this issue

New features should include all of the following work:

• Create the function/code.

• Document all code using comments to describe what is does.

• Create tests in tests folder.

• Create help file using roxygen2 above code.

• Create working examples in help file (via roxygen2).

• Add to appropriate vignette (or create new one).

[ehinman]

Proposed solution: Use WQX characteristic domain table to identify retired characteristics, grepl everything before the retired characters, and join to NWIS data to convert NWIS characteristics to current WQX characteristic structures. This would happen in autoclean.

[hillarymarler]

I'd like to work on this issue.

[hillarymarler]

This solution does work (tried on the OR example above as well as with random state and national datasets), but I am not sure if it is similar to what you had in mind originally. I used TADA_GetCharacteristicRef() to create a data frame and then filter for "retired" CharacteristicNames and then modified them to be compatible with the NWIS names. These additional CharacteristicName options can then be joined to the original ref.table of deprecated CharacteristicNames.

 # read in characteristic reference table with deprecation information, filter to deprecated terms and for "retired" in CharactersticName.
  # remove all characters after first "*" in CharacteristicName and remove any leading or trailing white space to make compatible with deprecated NWIS CharactersticName.
  nwis.table <- TADA_GetCharacteristicRef() %>%
    dplyr::filter(
      Char_Flag == "Deprecated",
      grepl("retired", CharacteristicName)
    ) %>%
    dplyr::mutate(CharacteristicName = trimws(stringr::str_split(CharacteristicName, "\\*", simplify = T)[, 1]))

  # read in characteristic reference table with deprecation information and filter to deprecated terms.
  # join with deprecated NWIS CharacteristicName data.frame.
  ref.table <- TADA_GetCharacteristicRef() %>%
    dplyr::filter(Char_Flag == "Deprecated") %>%
    full_join(nwis.table)

  rm(nwis.table)

[cristinamullin]

This solution makes sense to me. When you are ready you can create a pull request and then I can review and test the code before we merge it in. Happy to walk though this process together on our call later this week.