Issue reading in Zip-file

[BeishuizenTimKPMG]

Not all zip-files can be used to create the sqlite file.

The Dutch public transport net cannot be processed by zip, only after unzipping. (data from www.openOV.nl)

Not a major issue, but this bug creates a minor annoyance for the users. Maybe a good case study for bug testing?

[rmkujala]

Hi @BeishuizenTimKPMG,

Can you provide a sample of your code with the error message so that we can understand what the problem is about in more detail.

[BeishuizenTimKPMG]

The code is directly taken from the example in " gtfspy/examples/example_temporal_distance_profile.py". The error can be found in the following method:

import_gtfs.import_gtfs([imported_data_path],processed_data_path)

In this line, using a direct link to the previously mentioned zip file at www.openOV.nl it can not be loaded.

The following is printed:

Beginning AgencyLoader
Importing agency.txt into agencies for
Indexing agencies
Post-import agency.txt into agencies
Beginning RouteLoader
Importing routes.txt into routes for
Indexing routes
Beginning MetadataLoader
Indexing metadata
Beginning CalendarLoader
calendar.txt missing in {'zipfile': '../data/raw/gtfs-nl.zip', 'zip_commonprefix': ''}
Indexing calendar
Beginning CalendarDatesLoader
Importing calendar_dates.txt into calendar_dates for
Beginning ShapeLoader
Importing shapes.txt into shapes for
Indexing shapes
Post-import shapes.txt into shapes
Beginning FeedInfoLoader
Importing feed_info.txt into feed_info for
Beginning StopLoader
Importing stops.txt into stops for
Indexing stops
Post-import stops.txt into stops
Beginning TransfersLoader
Not importing transfers.txt into transfers for
Beginning StopDistancesLoader
Post-import None into stop_distances
Calculating straight-line transfer distances
Copying information from transfers to stop_distances.
Beginning TripLoader
Importing trips.txt into trips for
Indexing trips
Beginning StopTimesLoader
Importing stop_times.txt into stop_times for

And the following error occurs:

---

AttributeError Traceback (most recent call last)
in
1 # Not needed to rerun, is for accessing data
----> 2 import_gtfs.import_gtfs([imported_data_path],processed_data_path)

~/Projects/GemeenteAmsterdam/gtfspy-master/gtfspy/import_gtfs.py in import_gtfs(gtfs_sources, output, preserve_connection, print_progress, location_name, **kwargs)
102
103 for loader in loaders:
--> 104 loader.import_(conn)
105
106 # Do any operations that require all tables present.

~/Projects/GemeenteAmsterdam/gtfspy-master/gtfspy/import_loaders/table_loader.py in import_(self, conn)
355 # This does insertions
356 if self.mode in ('all', 'import') and self.fname and self.exists() and self.table not in ignore_tables:
--> 357 self.insert_data(conn)
358 # This makes indexes in the DB.
359 if self.mode in ('all', 'index') and hasattr(self, 'index'):

~/Projects/GemeenteAmsterdam/gtfspy-master/gtfspy/import_loaders/table_loader.py in insert_data(self, conn)
295 from itertools import chain
296 rows = chain([row], self.gen_rows([csv_reader], [prefix]))
--> 297 cur.executemany(stmt, rows)
298 conn.commit()
299

~/Projects/GemeenteAmsterdam/gtfspy-master/gtfspy/import_loaders/stop_times_loader.py in gen_rows(self, readers, prefixes)
23 def gen_rows(self, readers, prefixes):
24 for reader, prefix in zip(readers, prefixes):
---> 25 for row in reader:
26 #print row
27 assert row['arrival_time'] != "", "Some stop_times entries is missing arrival time information."

~/Projects/GemeenteAmsterdam/gtfspy-master/gtfspy/import_loaders/table_loader.py in (.0)
217 csv_reader_stripped = (dict((k, (v.strip() if v is not None else None)) # v is not always a string
218 for k, v in row.items())
--> 219 for row in csv_reader)
220 csv_reader_generators.append(csv_reader_stripped)
221 except TypeError as e:

~/Projects/GemeenteAmsterdam/gtfspy-master/gtfspy/import_loaders/table_loader.py in (.0)
216 # The following results in a generator, the complicated
217 csv_reader_stripped = (dict((k, (v.strip() if v is not None else None)) # v is not always a string
--> 218 for k, v in row.items())
219 for row in csv_reader)
220 csv_reader_generators.append(csv_reader_stripped)

AttributeError: 'list' object has no attribute 'strip'

As you can see the error is an object mismatch. As I said before, unpacking the zip works, but using the zip directly does not.

[evelyn9191]

I have the same problem when reading files for Prague transport:

/home/miska/PycharmProjects/prague_public_transport_app/venv/bin/python /home/miska/PycharmProjects/prague_public_transport_app/search_stops/import_gtfs_data.py
Beginning AgencyLoader
Importing agency.txt into agencies for 
Indexing agencies
Post-import agency.txt into agencies
Beginning RouteLoader
Importing routes.txt into routes for 
Indexing routes
Beginning MetadataLoader
Indexing metadata
Beginning CalendarLoader
Importing calendar.txt into calendar for 
Indexing calendar
Beginning CalendarDatesLoader
Importing calendar_dates.txt into calendar_dates for 
Beginning ShapeLoader
Importing shapes.txt into shapes for 
Indexing shapes
Post-import shapes.txt into shapes
Beginning FeedInfoLoader
Importing feed_info.txt into feed_info for 
Beginning StopLoader
Importing stops.txt into stops for 
Traceback (most recent call last):
  File "/home/miska/PycharmProjects/prague_public_transport_app/search_stops/import_gtfs_data.py", line 52, in <module>
    load_or_import_example_gtfs(verbose=True)
  File "/home/miska/PycharmProjects/prague_public_transport_app/search_stops/import_gtfs_data.py", line 20, in load_or_import_example_gtfs
    location_name="Prague")
  File "/home/miska/PycharmProjects/prague_public_transport_app/venv/lib/python3.6/site-packages/gtfspy/import_gtfs.py", line 104, in import_gtfs
    loader.import_(conn)
  File "/home/miska/PycharmProjects/prague_public_transport_app/venv/lib/python3.6/site-packages/gtfspy/import_loaders/table_loader.py", line 357, in import_
    self.insert_data(conn)
  File "/home/miska/PycharmProjects/prague_public_transport_app/venv/lib/python3.6/site-packages/gtfspy/import_loaders/table_loader.py", line 297, in insert_data
    cur.executemany(stmt, rows)
  File "/home/miska/PycharmProjects/prague_public_transport_app/venv/lib/python3.6/site-packages/gtfspy/import_loaders/stop_loader.py", line 14, in gen_rows
    for row in reader:
  File "/home/miska/PycharmProjects/prague_public_transport_app/venv/lib/python3.6/site-packages/gtfspy/import_loaders/table_loader.py", line 219, in <genexpr>
    for row in csv_reader)
  File "/home/miska/PycharmProjects/prague_public_transport_app/venv/lib/python3.6/site-packages/gtfspy/import_loaders/table_loader.py", line 218, in <genexpr>
    for k, v in row.items())
AttributeError: 'list' object has no attribute 'strip'

Is there some workaround to this issue?

[rkdarst]

Looks like one of the rows's csv is being turned into a list, instead of string. I guess it's done something clever.

Can you look into the stops.txt and see if anything is weird there? and/or, use a debugger to try to figure out the bad line and value? That would help to understand what is going on...

Otherwise, can you gave an link to the exact file you are using and exact command line used?

[evelyn9191]

I cannot replicate this issue as it was solved by #24. At that time, I considered the issue to be connected with special characters that caused that a string was considered split to many due to the special chars (i.e. creating something like ["/ax instead of š).

I am using GTFS zip file that can be downloaded here and ran the script by
import_gtfs.import_gtfs(["..\\data\\traffic_source.zip"], "some.db", print_progress=verbose, location_name="Prague")