Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-think use of num_rows and proportion for detection #14

Open
Leobouloc opened this issue Jun 22, 2017 · 0 comments
Open

Re-think use of num_rows and proportion for detection #14

Leobouloc opened this issue Jun 22, 2017 · 0 comments

Comments

@Leobouloc
Copy link
Contributor

(port to the forked version)

In the code below, if the target proportion is not 1, the validation functions are called on all the values of the serie that is passed (serie.apply(test_func).sum() ). This should be replaced, at the very least, by a call on num_rows values and ideally should be replaced by some heuristic that tests a limited number of values before testing more as is done when proportion == 1.

def test_col(serie, test_func, proportion=0.9, skipna=True, num_rows=50):
    ''' Tests values of the serie using test_func.
         - skipna : if True indicates that NaNs are not counted as False
         - proportion :  indicates the proportion of values that have to pass the test
    for the serie to be detected as a certain type
    '''
    serie = serie[serie.notnull()]
    ser_len = len(serie)
    if ser_len == 0:
        return False
    if proportion == 1:  # Then try first 1 value, then 5, then all
        for _range in [
            range(0, min(1, ser_len)),
            range(min(1, ser_len), min(5, ser_len)),
            range(min(5, ser_len), min(num_rows, ser_len))
        ]:  # Pour ne pas faire d'opérations inutiles, on commence par 1,
            # puis 5 puis num_rows valeurs
            if all(serie.iloc[_range].apply(test_func)):
                pass
            else:
                return False
        return True
    else:
        return serie.apply(test_func).sum() > proportion * len(serie)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant