Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OOM kills the process due to lack of memory #59

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,5 @@ pip-log.txt

#Mr Developer
.mr.developer.cfg

.idea
1 change: 0 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
include LICENSE
include README.md
include tzwhere/tz_world_shortcuts.json
include tzwhere/tz_world.json.gz
35 changes: 18 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
pytzwhere [![Build Status](https://travis-ci.org/pegler/pytzwhere.svg)](https://travis-ci.org/pegler/pytzwhere) [![Coverage Status](https://coveralls.io/repos/pegler/pytzwhere/badge.svg)](https://coveralls.io/r/pegler/pytzwhere)
=========
# geodjango-tzwhere

pytzwhere is a Python library to lookup the timezone for a given lat/lng entirely offline.
geodjango-tzwhere is a Python library to lookup the timezone for a given lat/lng entirely offline using GeoDjango and PostGIS.

Version 3.0 fixes how `pytzwhere` deals with [holes](https://github.com/pegler/pytzwhere/issues/34) in timezones. It is recommended that you use version 3.0.
This package was originally forked from https://github.com/pegler/pytzwhere, and has the same API. However, instead of loading the geojson timezone data into memory each time the app is run, this package will store it in your database during migration. Lookups will be faster, memory usage negligible, and the overhead of parsing the geojson each time your app starts is avoided.

It is a port from https://github.com/mattbornski/tzwhere with a few improvements. The underlying timezone data is based on work done by [Eric Muller](http://efele.net/maps/tz/world/)
I have tested this with PostgreSQL/PostGIS. It might work with SpatiaLite as well but I have not tested it there.

## Why should I use this instead of using pytzwhere?

Pytzwhere stores all of the timezone data in memory. This uses a lot of memory and incurs a big performance hit each time you start your app, as it has to re-parse the geojson file every time. This may be compounded in a production system where you may be running multiple instances. If you are not using a spatial database then that's really your only choice, but if you are using one, then it doesn't make sense to make these sacrifices. Use this package instead to take advantage of your database.

## Usage

If used as a library, basic usage is as follows:

Expand All @@ -14,23 +19,20 @@ If used as a library, basic usage is as follows:
>>> print tz.tzNameAt(35.29, -89.66)
America/Chicago

The polygons used for building the timezones are based on VMAP0. Sometimes points are outside a VMAP0 polygon, but are clearly within a certain timezone (see also this [discussion](https://github.com/mattbornski/tzwhere/issues/8)). As a solution you can search for the closest timezone within a user defined radius.

The polygons used for building the timezones are based on VMAP0. Sometimes points are outside a VMAP0 polygon, but are clearly within a certain timezone (see also this [discussion](https://github.com/mattbornski/tzwhere/issues/8)). As a solution you can use the `forceTZ` parameter as described below.

### forceTZ

Dependencies:
If the coordinates provided are outside of the currently defined timezone boundaries, the `tzwhere` function will return `None`. If you would like to match to the closest timezone, use the `forceTZ` parameter.

* `numpy` (optional)
Unlike pytzwhere, the `forceTZ` parameter does not need to be provided at `tzwhere` init time. You can provide it with your call to `tzNameAt()`.

* `shapely`
### Dependencies:

* `django`
* `PostGIS` or possibly some other spatial database system (at your own risk)


**forceTZ**

If the coordinates provided are outside of the currently defined timezone boundaries, the `tzwhere` function will return `None`. If you would like to match to the closest timezone, use the forceTZ parameter.

Example:
### Example:

>>> from tzwhere import tzwhere
>>> tz = tzwhere.tzwhere()
Expand All @@ -41,4 +43,3 @@ Example:
>>> tz = tzwhere.tzwhere(forceTZ=True)
>>> print(tz.tzNameAt(53.68193999999999, -6.239169999999998, forceTZ=True))
Europe/Dublin

24 changes: 11 additions & 13 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,39 +1,37 @@
import os
from setuptools import setup
from setuptools import setup, find_packages

README = open(os.path.join(os.path.dirname(__file__), 'README.md')).read()

# allow setup.py to be run from any path
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))

setup(
name='tzwhere',
version='3.0.3',
packages=['tzwhere'],
name='geodjango-tzwhere',
version='0.1',
packages=find_packages(),
package_data={
'tzwhere': [
'geodjango-tzwhere': [
'tz_world.json.gz',
'tz_world_shortcuts.json'
]
},
include_package_data=True,
install_requires=[
'shapely'
'django>=1.11'
],
license='MIT License',
description='Python library to look up timezone from lat / long offline',
long_description=README,
url='https://github.com/pegler/pytzwhere',
author='Matt Pegler',
author_email='[email protected]',
maintainer='Christoph Stich',
maintainer_email='[email protected]',
url='https://github.com/cjh79/geodjango-tzwhere',
author='Chris Hawes',
author_email='[email protected]',
maintainer='Christopher Hawes',
maintainer_email='[email protected]',
classifiers=[
'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License',
'Operating System :: OS Independent',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 3',
'Topic :: Software Development :: Localization',
],
Expand Down
10 changes: 6 additions & 4 deletions tests/test_locations.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
from tzwhere import tzwhere
import datetime
import unittest

from django.test import TestCase

from tzwhere import tzwhere


class LocationTestCase(unittest.TestCase):
class LocationTestCase(TestCase):

TEST_LOCATIONS = (
( 35.295953, -89.662186, 'Arlington, TN', 'America/Chicago'),
Expand Down Expand Up @@ -51,7 +53,7 @@ class LocationTestCase(unittest.TestCase):
( 40.7271, -73.98, 'Shore Lake Michigan', 'America/New_York'),
( 50.1536, -8.051, 'Off Cornwall', 'Europe/London'),
( 49.2698, -123.1302, 'Vancouver', 'America/Vancouver'),
( 50.26, -9.051, 'Far off Cornwall', None)
( 50.26, -9.051, 'Far off Cornwall', 'Europe/Dublin'),
)

def _test_tzwhere(self, locations, forceTZ):
Expand Down
13 changes: 13 additions & 0 deletions tzwhere/admin.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
from django.contrib.gis import admin
from django.contrib.gis.admin import OSMGeoAdmin

from tzwhere.models import Timezone


class TimezoneAdmin(OSMGeoAdmin):
list_display = ['id', 'name']
search_fields = ['id', 'name']
ordering = ['name']


admin.site.register(Timezone, TimezoneAdmin)
25 changes: 25 additions & 0 deletions tzwhere/migrations/0001_initial.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# -*- coding: utf-8 -*-
# Generated by Django 1.11.7 on 2018-06-14 00:45
from __future__ import unicode_literals

import django.contrib.gis.db.models.fields
from django.db import migrations, models


class Migration(migrations.Migration):

initial = True

dependencies = [
]

operations = [
migrations.CreateModel(
name='Timezone',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=200)),
('polygon', django.contrib.gis.db.models.fields.MultiPolygonField(srid=4326)),
],
),
]
57 changes: 57 additions & 0 deletions tzwhere/migrations/0002_load_timezones.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# -*- coding: utf-8 -*-
# Generated by Django 1.11.7 on 2018-06-13 23:08
from __future__ import unicode_literals

import gzip
import json

from django.contrib.gis.geos import GEOSGeometry, MultiPolygon
from django.db import migrations

from tzwhere.tzwhere import tzwhere


def load_timezones(apps, schema_editor):
Timezone = apps.get_model('tzwhere', 'Timezone')

with gzip.open(tzwhere.DEFAULT_POLYGONS, 'rb') as f:
featureCollection = json.loads(f.read().decode('utf-8'))

for name, poly in feature_collection_polygons(featureCollection):
Timezone.objects.create(
name=name,
polygon=MultiPolygon([poly]),
)


def delete_timezones(apps, schema_editor):
Timezone = apps.get_model('tzwhere', 'Timezone')
Timezone.objects.all().delete()


def feature_collection_polygons(featureCollection):
"""Turn a feature collection
into an iterator over polygons.

Given a featureCollection of the kind loaded from the json
input, unpack it to an iterator which produces a series of
(tzname, polygon) pairs, one for every polygon in the
featureCollection. Here tzname is a string and polygon is a
list of floats.

"""
for feature in featureCollection['features']:
tzname = feature['properties']['TZID']
if feature['geometry']['type'] == 'Polygon':
poly = GEOSGeometry(json.dumps(feature['geometry']))
yield (tzname, poly)


class Migration(migrations.Migration):
dependencies = [
('tzwhere', '0001_initial'),
]

operations = [
migrations.RunPython(load_timezones, delete_timezones),
]
Empty file added tzwhere/migrations/__init__.py
Empty file.
7 changes: 7 additions & 0 deletions tzwhere/models.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
from django.contrib.gis.db import models
from django.contrib.gis.db.models import MultiPolygonField


class Timezone(models.Model):
name = models.CharField(max_length=200)
polygon = MultiPolygonField(srid=4326)
1 change: 0 additions & 1 deletion tzwhere/tz_world_shortcuts.json

This file was deleted.

Loading