Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error exporting the csv file #4

Open
shakythesherpa opened this issue Feb 6, 2022 · 14 comments
Open

error exporting the csv file #4

shakythesherpa opened this issue Feb 6, 2022 · 14 comments

Comments

@shakythesherpa
Copy link

The code is choking up on this last step. I get a csv output but with only the column headers and empty data values. Below is my error message. Also, just had to replace the clip function from earthpy with clip form geopandas library in funcs.py

FionaCPLError Traceback (most recent call last)
Input In [81], in
----> 1 clusters = conditioning(clusters, workspace, popunit.value)

File /var/folders/3y/qvxb1s8s0nb51g2l7lzv8km00000gn/T/ipykernel_51993/481285332.py:167, in conditioning(clusters, workspace, popunit)
164 clusters["Y_deg"] = clusters.geometry.centroid.y
166 #clusters.to_file(workspace + r"\output.shp", driver='ESRI Shapefile')
--> 167 clusters.to_file(workspace + r"\output.csv", driver='CSV')
168 print(datetime.datetime.now())
169 return clusters

File ~/opt/anaconda3/envs/OnSSET_extraction/lib/python3.10/site-packages/geopandas/geodataframe.py:724, in GeoDataFrame.to_file(self, filename, driver, schema, index, **kwargs)
681 """Write the GeoDataFrame to a file.
682
683 By default, an ESRI shapefile is written, but any OGR data source
(...)
720 GeoSeries.to_file
721 """
722 from geopandas.io.file import _to_file
--> 724 _to_file(self, filename, driver, schema, index, **kwargs)

File ~/opt/anaconda3/envs/OnSSET_extraction/lib/python3.10/site-packages/geopandas/io/file.py:257, in _to_file(df, filename, driver, schema, index, mode, crs, **kwargs)
253 crs_wkt = crs.to_wkt("WKT1_GDAL")
254 with fiona.open(
255 filename, mode=mode, driver=driver, crs_wkt=crs_wkt, schema=schema, **kwargs
256 ) as colxn:
--> 257 colxn.writerecords(df.iterfeatures())

File ~/opt/anaconda3/envs/OnSSET_extraction/lib/python3.10/site-packages/fiona/collection.py:361, in Collection.writerecords(self, records)
359 if self.mode not in ('a', 'w'):
360 raise IOError("collection not open for writing")
--> 361 self.session.writerecs(records, self)
362 self._len = self.session.get_length()
363 self._bounds = None

File fiona/ogrext.pyx:1291, in fiona.ogrext.WritingSession.writerecs()

File fiona/ogrext.pyx:354, in fiona.ogrext.OGRFeatureBuilder.build()

File fiona/_err.pyx:269, in fiona._err.exc_wrap_int()

FionaCPLError: The wrapped function returned an error code, but no error message was set.
@babakkhavari
Copy link
Owner

OK, thanks for the heads-up on clipping I will update the clips to be through geopandas instead of earthpy.

For the other issue, I will have to look into it. Will try to get to that this week.

@shakythesherpa
Copy link
Author

I think this may be something to do with Fiona? I am able to export the output as a shape file but not as a csv. However, the shape file output does not seem to come with all the columns for some reason. I tried exporting a csv file directly using pandas and that did export a csv file. However, for some reason when I use this as an input in the the OnSSET gep runner code to create a calibrated file it throws an error saying: KeyError: 'NumPeoplePerHH' . Attached are my input file and specs file. Do you think you can help check this one your end?

@babakkhavari
Copy link
Owner

Hi Shaky,

Apologies, I have not had time to look into this yet. Will try to get it done ASAP.

Yes, I can have a look at it your files, however, I do not see anything attached?

@shakythesherpa
Copy link
Author

shakythesherpa commented Feb 18, 2022 via email

@babakkhavari
Copy link
Owner

babakkhavari commented Feb 18, 2022 •
edited
Loading

I do not seem to have received any invitation to one drive through any of my emails.

@shakythesherpa
Copy link
Author

shakythesherpa commented Feb 18, 2022 via email

@babakkhavari
Copy link
Owner

babakkhavari commented Feb 18, 2022 •
edited
Loading

Still nothing through my email. But, I can access the files through your link.

I opened your file called output and I see some problems that will make OnSSET crash (see image below)

image

This is because you have the geometry column when exporting. Can you try removing it by first converting your gdf to a df and then export (I think this would force you to export as a csv)?

@shakythesherpa
Copy link
Author

shakythesherpa commented Feb 19, 2022 via email

@babakkhavari
Copy link
Owner

Can you reattach the image? It seems like it did not get included in your previous message.

@shakythesherpa
Copy link
Author

shakythesherpa commented Feb 20, 2022 via email

@shakythesherpa
Copy link
Author

shakythesherpa commented Feb 20, 2022 via email

@babakkhavari
Copy link
Owner

Hi Shaky,

I received the image over email.

How did you remove the geometry column? Did you:

  1. remove it directly from your exported csv or
  2. did you remove it from your gdf through the jupyter notebook and then exported the results as a csv?

It looks like you used the first option which would explain the errors you get. The geometry column is not only an additional column, but it also risks distorting rows of data, so even if you remove the column through option 1 the faulty rows will remain. See image below of the file named "input_file" in your folder, this will make the code crash.

image

In your calibrated file the results look like this:
image

Note that the second column is your country column and every row should say "Sudan", but with "Sudan" filtered out there are still 38 rows remaining with numbers. This will crash the OnSSET.

I think there are two options:

  1. re-export your csv file from the extraction code, making sure that the geometry column does not exist in the gdf or
  2. go through your calibrated file and remove the rows that do not have "Sudan" in the country column and then run the calibration step again on the calibrated file.

Option 1 is probably the best, it will take a little bit more time though. Option 2, can work, but you might want to check the remaining rows to see if you are removing anything important (i.e. filter the country column again, but this time leave all rows with "Sudan". Now check what the total population/elecpop etc. is. If it close enough to what you would expect option 2 may be sufficient as when you go on a rerun the calibration step small distortions will be fixed with OnSSET)

Let me know if anything is unclear or if you any questions!

@shakythesherpa
Copy link
Author

Hi Bavak,
I tried option 1 and it works fine now. Thank you so much for your help. FYI, for some reason i had to manually add a GDP column in my file for the gep runner to successfully run. Is this because i did not include a custom demand layer?

@babakkhavari
Copy link
Owner

Ok, great that it worked.

The GDP column is not included in the extraction code that you run, since it is also not used in the default version of OnSSET. In other words it has nothing to do with the custom demand layer. It would perhaps make sense to add it as an optional layer in the extraction code, but this will be done later in that case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shakythesherpa @babakkhavari