Improving Usability of ASCII Strings #1304
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request has been linked to Shortcut Story #20965: Unicode-to-ASCII and ASCII-to-Unicode in TileDB-Py. |
* Previously `TILEDB_STRING_ASCII` data was inconsistently displayed
as `bytes`
* There is a need to coerce to `str` everywhere because (1) previously
the resulting dataframe displayed ASCII as bytes with Pyarrow disabled
but as str with Pyarrow enabled, and (2) this fix would remove the
need to copy large amounts of data to convert back and forth in the
TileDB-SingleCell Python API
* Warning now emitted to the user to pass `dtype="ascii"` for string dim
types in lieu of `np.bytes_` or `np.str_` for clarity. All three still
work and under the hood use `np.str_` and `TILEDB_STRING_ASCII`
* `repr` of string dimensions is now always displayed as `dtype="ascii"`.
Calling `.dtype()` will return `np.dtype('U')` as the return signature
of `dtype` requires a Numpy dtype
nguyenv
force-pushed
the
viviannguyen/sc-20965/unicode-to-ascii-and-ascii-to-unicode-in
branch
from
3202aee to
dead46e
Compare
|
@nguyenv testing now -- thanks! :) |
|
@nguyenv using SOMA The Also, I see: This is (as a reminder) in reference to single-cell-data/TileDB-SOMA#99 -- please see there for the reason that we cannot store these attributes as Unicode within the TileDB storage. |
|
We have resolved the original problem that prompted this PR. However, this branch still contains several features that may be important for usability such as consistent presentation of ASCII as |
nguyenv
changed the title
TILEDB_STRING_ASCII Now Displaying As UTF-8 / str Everywhere
nguyenv
changed the title
This was referenced Aug 31, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TILEDB_STRING_ASCIIdata was inconsistently displayedas
bytesstreverywhere because (1) previouslythe resulting dataframe displayed ASCII as bytes with Pyarrow disabled
but as str with Pyarrow enabled, and (2) this fix would remove the
need to copy large amounts of data to convert back and forth in the
TileDB-SingleCell Python API
dtype="ascii"for string dimtypes in lieu of
np.bytes_ornp.str_for clarity. All three stillwork and under the hood use
np.str_andTILEDB_STRING_ASCIIreprof string dimensions is now always displayed asdtype="ascii".Calling
.dtype()will returnnp.dtype('U')as the return signatureof
dtyperequires a Numpy dtype