Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reevaluate dtypes on importing from SBDF file #75

Open
bbassett-tibco opened this issue Nov 26, 2024 · 0 comments
Open

Reevaluate dtypes on importing from SBDF file #75

bbassett-tibco opened this issue Nov 26, 2024 · 0 comments

Comments

@bbassett-tibco
Copy link
Collaborator

Given the following script:

from spotfire import sbdf

x = sbdf.import_data("spotfire/test/files/sbdf/10001.sbdf")
print("column Boolean")
print(x['Boolean'].loc[[0,1,2]])
print("column String")
print(x['String'].loc[[0,1,2]])

Output:

column Boolean
0    False
1     True
2     None
Name: Boolean, dtype: object
column String
0      The
1    quick
2     None
Name: String, dtype: object

Note how both of these columns report as being dtype object. However, it's probably better to use the nullable boolean (https://pandas.pydata.org/docs/user_guide/boolean.html) and string (https://pandas.pydata.org/docs/user_guide/text.html#text) dtypes for these columns. (We're actually using the nullable integer and float dtypes for other column types, so our current requirements on Pandas should be untouched.)

Current workarounds are to manually cast the columns using Pandas' astype method:

>>> x['Boolean'].loc[[0,1,2]].astype('boolean')
0    False
1     True
2     <NA>
Name: Boolean, dtype: boolean
>>> x['String'].loc[[0,1,2]].astype('string')
0      The
1    quick
2     <NA>
Name: String, dtype: string

Make this the native result of the import_data function, and make sure that these dtypes correctly export to SBDF in the export_data function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant