Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configuration for credentials, proxies, and progress #162

Closed
wants to merge 63 commits into from
Closed
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
b673ada
Use configuration dictionary
tfardet Jul 18, 2023
4e86896
Correct environment variables for credentials
tfardet Jul 18, 2023
49bd3a9
Remove pynsee_print_url from env
tgrandje Jul 25, 2023
3cee35d
Fix get_location, add token logging + alt text
tfardet Aug 8, 2023
871df17
bugfix
hadrilec Sep 2, 2023
c735cc4
Move idbank and file list to settings, improved token management
tfardet Sep 4, 2023
f7c37f9
Use configuration dictionary
tfardet Jul 18, 2023
73f6f84
Correct environment variables for credentials
tfardet Jul 18, 2023
15f0631
Remove pynsee_print_url from env
tgrandje Jul 25, 2023
9d455be
Move idbank and file list to settings, improved token management
tfardet Sep 4, 2023
f01f66c
Merged remote
tfardet Sep 4, 2023
6ed4f77
Merge branch 'master' into settings
tfardet Sep 4, 2023
22d0f23
restore old imports
tfardet Sep 5, 2023
9d44564
Merge remote-tracking branch 'refs/remotes/origin/settings' into sett…
tfardet Sep 5, 2023
a382201
Correct missing imports
tfardet Sep 5, 2023
3fe8df0
Final correction of imports
tfardet Sep 5, 2023
b1d3f5c
bug fix geofrdf import
hadrilec Sep 9, 2023
a629c9a
bug fix - ci/cd - trigger pkg tests on pr head commit
hadrilec Sep 9, 2023
d1d0fb8
Merge branch 'master' into settings
tfardet Sep 9, 2023
a26877f
Fix test imports + update them
Sep 9, 2023
f3c86e7
Fix tests except number 2
Sep 9, 2023
7a32ef2
Fix logging
Sep 9, 2023
156b830
cleanup
Sep 9, 2023
aec3a9d
Move to platformdirs and add config file
Nov 14, 2023
cb0fd7a
Merge master, fix test decorator
Nov 14, 2023
0f567dd
Fix tests
Nov 14, 2023
9fbeae2
Insee key and secret must be passed simultaneously
Nov 15, 2023
41cc4b9
Restore order for token: env, then config, then request from insee cred
Nov 15, 2023
3cf4a1b
Fix _get_credentials
Nov 15, 2023
2b9b436
Finalize credentials logic
Nov 16, 2023
2c8626b
doc requirements adjustement - tests fix
hadrilec Sep 24, 2023
32882f6
readme.md
hadrilec Oct 3, 2023
f464ee0
modif readme
hadrilec Oct 3, 2023
c5f72e3
setup include readme.md
hadrilec Oct 3, 2023
75a1dd7
readme update
hadrilec Oct 3, 2023
cb2162b
readme update
hadrilec Oct 3, 2023
d171c08
readme update
hadrilec Oct 3, 2023
2c3ce82
test new release in pypitest
hadrilec Oct 3, 2023
a9faa7b
markdown readme
hadrilec Oct 14, 2023
fca3a87
geopandas doc requirement
hadrilec Oct 14, 2023
860e27a
tests only on py 3.8+3.11, geopandas version fix
hadrilec Nov 18, 2023
68d4c0e
multiprocessing backup solution if it fails
hadrilec Nov 18, 2023
5922208
warning removed by explicitly dropping cols before concat df
hadrilec Nov 18, 2023
82ca453
wait_api_query not needed anymore as 529 error triggers some sleep
hadrilec Nov 18, 2023
7880833
notebooks update and fix
hadrilec Nov 18, 2023
34c3121
gitignore update
hadrilec Nov 18, 2023
67a8a98
conflicts solved
hadrilec Nov 18, 2023
0383d69
syntax fix
hadrilec Nov 18, 2023
5a1125c
func not used anymore rmv
hadrilec Nov 18, 2023
772e6dd
Update pynsee/geodata/_get_geodata.py
hadrilec Nov 18, 2023
6de48c4
Update pynsee/localdata/_find_latest_local_dataset.py
hadrilec Nov 18, 2023
1e1163d
Update pynsee/macrodata/_download_idbank_list.py
hadrilec Nov 18, 2023
147d025
Update pynsee/utils/_clean_insee_folder.py
hadrilec Nov 18, 2023
e52ade6
Update pynsee/utils/_create_insee_folder.py
hadrilec Nov 18, 2023
7b614d0
Update pynsee/metadata/get_legal_entity.py
hadrilec Nov 18, 2023
8117f61
cleaning tests
hadrilec Nov 18, 2023
fb413cb
user agent used to try to avoid connection error
hadrilec Nov 18, 2023
269154a
Merge branch 'settings' of https://github.com/tfardet/pynsee into set…
hadrilec Nov 18, 2023
2366eb8
use _request_insee in token config func
hadrilec Nov 18, 2023
7763812
catch error 429 and sleep
hadrilec Nov 18, 2023
72e13ac
init_conn - catch error 429
hadrilec Nov 18, 2023
0afbd0f
share connections when testing the token
hadrilec Nov 18, 2023
977b67d
Cleanup code, move to return early pattern
Nov 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pynsee/geodata/_get_geodata.py
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ def _get_geodata(
disable=pynsee.get_config("hide_progress")
))

except:
except Exception:

func_settings(args)
list_output = []
Expand Down
2 changes: 1 addition & 1 deletion pynsee/localdata/_find_latest_local_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def _find_latest_local_dataset(dataset_version, variables, nivgeo, codegeo, upda
if df['OBS_VALUE'][0] is None:
raise ValueError('No data found')

except:
except Exception:
if dv == list_dataset_version[-1]:
msg = '!!! Latest dataset version not found !!!\n'
msg += 'Please, consider having a look at api.insee.fr or get_local_metadata function'
Expand Down
2 changes: 1 addition & 1 deletion pynsee/macrodata/_download_idbank_list.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def _download_idbank_list(update=False):
file_to_dwn_default = "idbank_" + str(todays_date.year) + str(todays_date.month)

insee_folder = _create_insee_folder()
file = insee_folder + "/" + _hash(file_to_dwn_default)
file = os.path.join(insee_folder, _hash(file_to_dwn_default))

trigger_update = False

Expand Down
2 changes: 1 addition & 1 deletion pynsee/metadata/get_legal_entity.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def get_legal_entity(codes, print_err_msg=True, update=False):

filename = _hash("get_legal_entity" + "".join(codes))
insee_folder = _create_insee_folder()
file_legal_entity = insee_folder + "/" + filename
file_legal_entity = os.path.join(insee_folder, filename)

if (not os.path.exists(file_legal_entity)) or update:
list_data = []
Expand Down
2 changes: 1 addition & 1 deletion pynsee/utils/_clean_insee_folder.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ def _clean_insee_folder():

if len(list_file_insee) > 0:
for f in list_file_insee:
os.remove(insee_folder + "/" + f)
os.remove(os.path.join(insee_folder, f))
2 changes: 1 addition & 1 deletion pynsee/utils/_create_insee_folder.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ def _create_insee_folder():
# os.mkdir(insee_folder)

# test if saving a file is possible
test_file = insee_folder + "/" + _hash("test_file")
test_file = os.path.join(insee_folder, _hash("test_file"))
with open(test_file, "w") as f:
f.write("")
f.close()
Expand Down
6 changes: 6 additions & 0 deletions pynsee/utils/_request_insee.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,16 @@ def _request_insee(
if api_url is not None:
token = pynsee.get_config("insee_token")

try:
tfardet marked this conversation as resolved.
Show resolved Hide resolved
username = os.environ['USERNAME']
except:
username = "username"

if token:
headers = {
"Accept": file_format,
"Authorization": "Bearer " + token,
'User-Agent': f"python_pynsee_{username}"
}

results = requests.get(
Expand Down
8 changes: 7 additions & 1 deletion pynsee/utils/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,9 +152,15 @@ def _register_token(
"!!! Token is missing, please check that insee_key and "
"insee_secret are correct !!!")
else:
try:
tfardet marked this conversation as resolved.
Show resolved Hide resolved
username = os.environ['USERNAME']
except:
username = "username"

headers = {
"Accept": "application/xml",
"Authorization": "Bearer " + (token or "")
"Authorization": "Bearer " + (token or ""),
'User-Agent': f"python_pynsee_{username}"
}

url_test = "https://api.insee.fr/series/BDM/V1/data/CLIMAT-AFFAIRES"
Expand Down
21 changes: 0 additions & 21 deletions tests/download/test_pynsee_download.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,24 +72,3 @@ def test_download_file_all(self):
if __name__ == '__main__':
unittest.main()

"""
import unittest

class KnownGood(unittest.TestCase):
def __init__(self, input):
super(KnownGood, self).__init__()
self.input = input
def runTest(self):
self.assertEqual(type(download_file(self.input)), pd.DataFrame)

def suite():
suite = unittest.TestSuite()
list_file = ["COG_COMMUNE_2018", "AIRE_URBAINE", "FILOSOFI_COM_2015", "DECES_2020",
"PRENOM_NAT", "ESTEL_T201_ENS_T", "FILOSOFI_DISP_IRIS_2017",
"BPE_ENS", "RP_MOBSCO_2016"]
suite.addTests(KnownGood(input) for input in list_file)
return suite

if __name__ == '__main__':
unittest.TextTestRunner().run(suite())
"""
Loading