Mid jan 2023 patches (#179)

* new sources invitation for images language changed.

* added Fula to multi country language groups. Requires the following db changes: A. Liveadmin 1. Change Fulani (Lower Guinea) to Fulani 2. Connect the following countries to Fula (Upper Guinea) 6011 – Guinea Bissau 6012 – Mali 6013 – Senegal 6014 – The Gambia 6021 – Guinea 6022 – Sierra Leone

* updated google analytics to v 4 (untested)

* updated maps cache

* proper past maps cache files path...

* split the language_groups tree btw. search (only show used language groups) and contribute (show all language groups)

* updated map table display according to PM & DE requests

* simpler language group switch by extending js language group class

* updated post-disembark node text, and added node filtering for origins

* languages popup display change per DE request

* *grouped* languages visual search

* ensuring valid language group nodes come through on visual search

* removed post requirement from soures search per dellamonica

* 1. links added to embark ports language table 2. cultural associations working on search bar 3. tooltip formatted properly on all nodes popups 4. require_post dropped from sources

* Fixing bug in fuzzy search sorting for Enslaved.
The non-fuzzy path applied a workaround for Django bug 28811 but the fuzzy code path did not.

* Attempt to fix the duplicate audio recordings in Origins UI.
The cached sound recordings dictionary was not being reset when reloading, which would explain duplicates in a long-lived process.

* Fixed bug: rejected enslaved contribution still shows up in editorial list.

* updated cultural association dropdown language prompt, and the po files accordingly.

* translation update

* fixed LG enumeration issue flagged by DE

* Split Enslavers Itinerary search tab into Voyage and Places. Adding Departure Port search on Enslavers page.

* Adding voyage id to OoK and full date of arrival.

* Adding departure date for OoK results table.
Changing min year date for AO to 1806 and making sure the year variable is active for the search.

* Fula grouping

Co-authored-by: John Connor (Mulligan) <[email protected]>
Co-authored-by: Domingos <[email protected]>
Co-authored-by: Leonardo Barros <[email protected]>

voyages/apps/contribute/templates/contribute/_new_source_dialog.html

@@ -57,7 +57,7 @@ <h4 class="modal-title" id="referencesModalLabel">{% trans "Add a source referen
                             <textarea class="form-control not_selectized" id="primary_info"></textarea>
                         </div>
                         <div class="form-group">
-                            <label for="primary_url">{% trans "URLs or digitized image of the source to be submitted wherever possible about source" %}</label>
+                            <label for="primary_url">{% trans "URLs of the source to be submitted wherever possible." %}</label>
                             <textarea class="form-control not_selectized" id="primary_url"></textarea>
                         </div>
                     </div>
@@ -96,7 +96,7 @@ <h4 class="modal-title" id="referencesModalLabel">{% trans "Add a source referen
                             <textarea class="form-control not_selectized" id="article_info"></textarea>
                         </div>
                         <div class="form-group">
-                            <label for="article_url">{% trans "URLs or digitized image of the source to be submitted wherever possible about source" %}</label>
+                            <label for="article_url">{% trans "URLs of the source to be submitted wherever possible." %}</label>
                             <textarea class="form-control not_selectized" id="article_url"></textarea>
                         </div>
                     </div>
@@ -147,7 +147,7 @@ <h4 class="modal-title" id="referencesModalLabel">{% trans "Add a source referen
                             <textarea class="form-control not_selectized" id="book_info"></textarea>
                         </div>
                         <div class="form-group">
-                            <label for="book_url">{% trans "URLs or digitized image of the source to be submitted wherever possible about source" %}</label>
+                            <label for="book_url">{% trans "URLs of the source to be submitted wherever possible." %}</label>
                             <textarea class="form-control not_selectized" id="book_url"></textarea>
                         </div>
                     </div>
@@ -174,7 +174,7 @@ <h4 class="modal-title" id="referencesModalLabel">{% trans "Add a source referen
                             <textarea class="form-control not_selectized" id="newspaper_info"></textarea>
                         </div>
                         <div class="form-group">
-                            <label for="newspaper_url">{% trans "URLs or digitized image of the source to be submitted wherever possible about source" %}</label>
+                            <label for="newspaper_url">{% trans "URLs of the source to be submitted wherever possible." %}</label>
                             <textarea class="form-control not_selectized" id="newspaper_url"></textarea>
                         </div>
                     </div>
@@ -201,7 +201,7 @@ <h4 class="modal-title" id="referencesModalLabel">{% trans "Add a source referen
                             <textarea class="form-control not_selectized" id="pnc_info"></textarea>
                         </div>
                         <div class="form-group">
-                            <label for="pnc_url">{% trans "URLs or digitized image of the source to be submitted wherever possible about source" %}</label>
+                            <label for="pnc_url">{% trans "URLs of the source to be submitted wherever possible." %}</label>
                             <textarea class="form-control not_selectized" id="pnc_url"></textarea>
                         </div>
                     </div>
@@ -228,7 +228,7 @@ <h4 class="modal-title" id="referencesModalLabel">{% trans "Add a source referen
                             <textarea class="form-control not_selectized" id="unpub_info"></textarea>
                         </div>
                         <div class="form-group">
-                            <label for="unpub_url">{% trans "URLs or digitized image of the source to be submitted wherever possible about source" %}</label>
+                            <label for="unpub_url">{% trans "URLs of the source to be submitted wherever possible." %}</label>
                             <textarea class="form-control not_selectized" id="unpub_url"></textarea>
                         </div>
                     </div>

voyages/apps/contribute/views.py

@@ -2253,7 +2253,7 @@ def _enslaved_contrib_base_query():
         .prefetch_related('contributed_language_groups')
 
 @login_required()
-def get_origins_contributions(request, status=0):
+def get_origins_contributions(request, status=[0]):
     q = _enslaved_contrib_base_query()
     if status:
         # The caller can explicitly set status to 
@@ -2302,16 +2302,19 @@ def create_name_match_clauses(names):
         c.enslaved.name_second,
         c.enslaved.name_third
     ])
-    propagation_candidates = Enslaved.objects \
-        .filter(voyage__voyage_itinerary__imp_principal_region_of_slave_purchase_id= \
-            c.enslaved.voyage.voyage_itinerary.imp_principal_region_of_slave_purchase_id) \
-        .filter(name_clauses) \
-        .values( \
-            'pk', 'gender', 'modern_name', 'documented_name', 'notes', \
-            'name_first', 'name_second', 'name_third', \
-            'language_group_id', 'modern_country_id', \
-            'language_group__name', 'modern_country__name', \
-            embarkation=F('voyage__voyage_itinerary__imp_principal_place_of_slave_purchase__place'))
+    if name_clauses:
+        propagation_candidates = Enslaved.objects \
+            .filter(voyage__voyage_itinerary__imp_principal_region_of_slave_purchase_id= \
+                c.enslaved.voyage.voyage_itinerary.imp_principal_region_of_slave_purchase_id) \
+            .filter(name_clauses) \
+            .values( \
+                'pk', 'gender', 'modern_name', 'documented_name', 'notes', \
+                'name_first', 'name_second', 'name_third', \
+                'language_group_id', 'modern_country_id', \
+                'language_group__name', 'modern_country__name', \
+                embarkation=F('voyage__voyage_itinerary__imp_principal_place_of_slave_purchase__place'))
+    else:
+        propagation_candidates = []
     expanded = _expand_contrib(c)
     expanded['prev_name_contributions'] = list(prev_name_contrib)
     expanded['prev_language_group_contributions'] = list(prev_lang_contrib)

voyages/apps/past/management/commands/makemaproutes.py

@@ -107,7 +107,7 @@ def get_geo_edge_distance(s_id,t_id,G):
 				id,latitude,longitude,name=languagegroup_node
 
 				##This is a complex problem. I'm going to get these past the filter here, then have hard-coded geojson polygons on the other end that I deal with some way or another....
-				if name in ["Arabic/Islamic","Islamic","Mandinka"]:
+				if name in ["Arabic/Islamic","Islamic","Mandinka","Fula (Upper Guinea)"]:
 					dist_lg=language_groups.filter(id=id)
 					dist_lg_countries=dist_lg.values_list(
 						'moderncountry__id',

voyages/apps/past/models.py

@@ -200,8 +200,9 @@ def load(cls, force_reload=False):
             cls.WORDSET_INDEX = Levenshtein_search.populate_wordset(-1, list(all_names))
             q = EnslavedName.objects.values_list('id', 'name', 'language',
                                                  'recordings_count')
+            sound_recordings = {}
             for item in q:
-                current = cls._sound_recordings.setdefault(item[1], {})
+                current = sound_recordings.setdefault(item[1], {})
                 langs = current.setdefault('langs', [])
                 lang = {}
                 lang['lang'] = item[2]
@@ -211,6 +212,7 @@ def load(cls, force_reload=False):
                     for index in range(1, 1 + item[3])
                 ]
                 langs.append(lang)
+            cls._sound_recordings = sound_recordings
             cls._loaded = datetime.now()
 
 class EnslaverNameSearchCache:
@@ -1176,7 +1178,8 @@ def add_names_sorting(sorted_name_fields, col_name, q,
             else:
                 q = q.order_by('enslaved_id')
 
-        # Save annotations to circumvent a Django bug (see below).
+        # Save annotations to circumvent Django bug 28811 (see below).
+        # https://code.djangoproject.com/ticket/28811
         aux_annotations = q.query.annotation_select.copy()
         MultiValueHelper.set_group_concat_limit()
         if is_fuzzy:
@@ -1188,7 +1191,8 @@ def add_names_sorting(sorted_name_fields, col_name, q,
         for helper in self.all_helpers:
             main_query = helper.adapt_query(main_query)
         main_query = main_query.values(*fields)
-
+            # The next line is again due to Django bug 28811.
+        main_query.query.annotation_select.update(aux_annotations)
         if is_fuzzy:
             # Convert the QuerySet to a concrete list and include the ranking
             # as a member of each object in that list.
@@ -1202,11 +1206,8 @@ def add_names_sorting(sorted_name_fields, col_name, q,
         else:
             (count_query_str, count_params) = q.order_by().distinct().query.sql_with_params()
             q = q.distinct()
-            # These two lines are due to a bug in Django (fixed in newer
-            # versions).
-            # https://code.djangoproject.com/ticket/28811
+            # The next line is again due to Django bug 28811.
             q.query.annotation_select.update(aux_annotations)
-            main_query.query.annotation_select.update(aux_annotations)
             (match_query_str, match_params) = q.query.sql_with_params()
             (main_query_str, main_params) = main_query.query.sql_with_params()
             full_query_str = f"{main_query_str} INNER JOIN ({match_query_str} LIMIT 0,0) AS matches ON matches.pk=past_enslaved.enslaved_id"
@@ -1315,6 +1316,7 @@ def __init__(self,
                  year_range=None,
                  embarkation_ports=None,
                  disembarkation_ports=None,
+                 departure_ports=None,
                  ship_name=None,
                  voyage_id=None,
                  source=None,
@@ -1334,6 +1336,8 @@ def __init__(self,
                 embarked
         @param: disembarkation_ports A list of port ids where the enslaved
                 disembarked
+        @param: departure_ports A list of port ids where the voyage
+                begin
         @param: ship_name The ship name that the enslaved embarked
         @param: voyage_id A pair (a, b) where the a <= voyage_id <= b
         @param: source A text fragment that should match Source's text_ref or
@@ -1350,6 +1354,7 @@ def __init__(self,
         self.year_range = year_range
         self.embarkation_ports = embarkation_ports
         self.disembarkation_ports = disembarkation_ports
+        self.departure_ports = departure_ports
         self.ship_name = ship_name
         self.voyage_id = voyage_id
         self.source = source
@@ -1404,6 +1409,9 @@ def add_voyage_field(q, field, op, val):
         if self.disembarkation_ports:
             # Search on MJSLPTIMP field.
             q = add_voyage_field(q, 'voyage_itinerary__imp_principal_port_slave_dis__pk', 'in', self.disembarkation_ports)
+        if self.departure_ports:
+            # Search on PORTDEP field.
+            q = add_voyage_field(q, 'voyage_itinerary__imp_port_voyage_begin__pk', 'in', self.departure_ports)
         if self.year_range:
             # Search on YEARAM field. Note that we have a 'MM,DD,YYYY' format
             # even though the only year should be present.

voyages/apps/past/templates/past-enslavers/_itinerary.html

@@ -1,12 +1,12 @@
 {% load i18n %}
 
 <li class="nav-item dropdown">
-  <v-menu-title :count="filter.itinerary.count" title="{% trans 'Itinerary' %}"></v-menu-title>
+  <v-menu-title :count="filter.itinerary.count" title="{% trans 'Voyage' %}"></v-menu-title>
 
-  <v-panel-singular @apply="apply" @reset="reset" group="itinerary" title="{% trans 'Itinerary' %}" sub-group="itinerary"
+  <v-panel-singular @apply="apply" @reset="reset" group="itinerary" title="{% trans 'Voyage' %}" sub-group="itinerary"
     :filters="filter.itinerary.itinerary" align="left">
     <div slot="v-panel-header">
-      <v-panel-header title="{% trans 'Itinerary' %}" description="{% trans '' context 'Enslaver Itinerary search panel description' %}">
+      <v-panel-header title="{% trans 'Voyage' %}" description="{% trans '' context 'Enslaver Voyage search panel description' %}">
         <div slot="v-modal-content"></div>
       </v-panel-header>
     </div>
@@ -20,16 +20,6 @@
         :filter="scope.filters.var_ship_name"
       ></v-text>
 
-      <v-treeselect
-        :filter="scope.filters.var_embarkation_ports"
-        @change="changed">
-      </v-treeselect>
-
-      <v-treeselect
-        :filter="scope.filters.var_disembarkation_ports"
-        @change="changed">
-      </v-treeselect>
-
       <v-number
         :filter="scope.filters.var_year_range"
         @change="changed">

voyages/apps/past/templates/past-enslavers/_places.html

@@ -0,0 +1,32 @@
+{% load i18n %}
+
+<li class="nav-item dropdown">
+  <v-menu-title :count="filter.places.count" title="{% trans 'Places' %}"></v-menu-title>
+
+  <v-panel-singular @apply="apply" @reset="reset" group="places" title="{% trans 'Places' %}" sub-group="places"
+    :filters="filter.places.places" align="left">
+    <div slot="v-panel-header">
+      <v-panel-header title="{% trans 'Places' %}" description="{% trans '' context 'Enslaver Places search panel description' %}">
+        <div slot="v-modal-content"></div>
+      </v-panel-header>
+    </div>
+
+    <div slot="v-panel-content" slot-scope="scope">
+      <v-treeselect
+        :filter="scope.filters.var_embarkation_ports"
+        @change="changed">
+      </v-treeselect>
+
+      <v-treeselect
+        :filter="scope.filters.var_disembarkation_ports"
+        @change="changed">
+      </v-treeselect>
+
+      <v-treeselect
+        :filter="scope.filters.var_departure_ports"
+        @change="changed">
+      </v-treeselect>
+
+    </div>
+  </v-panel-singular>
+</li>

voyages/apps/past/templates/past/database.html

@@ -105,7 +105,7 @@
 
 <script>
 	const SV_MODE = "past";
-	const minVoyageYear = 1808;
+	const minVoyageYear = 1806;
 	const enslavedDatasetOverride = {{ dataset|default:"null" }};
 </script>
 

voyages/apps/past/templates/past/enslavers.html

@@ -44,6 +44,7 @@
 				<ul class="navbar-nav">
 					{% include "past-enslavers/_enslaver-name.html" %}
 					{% include "past-enslavers/_itinerary.html" %}
+					{% include "past-enslavers/_places.html" %}
 					{% include "past-enslavers/_sources.html" %}
 					{% comment %}
 					{% include "past/_identity.html" %}
@@ -95,7 +96,7 @@
 
 <script>
 	const SV_MODE = "past";
-	const minVoyageYear = 1808;
+	const minVoyageYear = 1806;
 </script>
 
 {% compress js %}
@@ -128,6 +129,7 @@
 <!-- <script src="{{ STATIC_URL }}scripts/vue/variables/past/identity.js"></script> -->
 <!-- <script src="{{ STATIC_URL }}scripts/vue/variables/past/personal-data.js"></script> -->
 <script src="{{ STATIC_URL }}scripts/vue/variables/past-enslavers/itinerary.js"></script>
+<script src="{{ STATIC_URL }}scripts/vue/variables/past-enslavers/places.js"></script>
 <!-- <script src="{{ STATIC_URL }}scripts/vue/variables/past/cultural-association.js"></script> -->
 <!-- <script src="{{ STATIC_URL }}scripts/vue/variables/past/fate.js"></script> -->
 <script src="{{ STATIC_URL }}scripts/vue/variables/past-enslavers/sources.js"></script>

voyages/apps/past/views.py

@@ -130,8 +130,10 @@ def get_modern_countries(_):
 
 
 @csrf_exempt
-@cache_page(3600)
-def get_language_groups(_):
+@cache_page(0)
+def get_language_groups(request):
+    #we need a switch between used and unused language groups (search should only have used, contribute should have all)
+    active_only=json.loads(request.body).get('active_only')
     countries_list_key = "countries_list"
     alt_names_key = "alt_names_list"
     country_helper = MultiValueHelper(countries_list_key, ModernCountry.languages.through, 'languagegroup_id', modern_country_id='moderncountry__pk', country_name='moderncountry__name')
@@ -140,9 +142,15 @@ def get_language_groups(_):
     q = country_helper.adapt_query(q)
     q = alt_names_helper.adapt_query(q)
     items = [country_helper.patch_row(alt_names_helper.patch_row(row)) for row in q.values()]
-    return JsonResponse([{ "id": item["id"], "name": item["name"], "lat": item["latitude"], "lng": item["longitude"], "alts": item[alt_names_key], "countries": item[countries_list_key] }
-        for item in items], safe=False)
-
+    if active_only:
+        enslaved=Enslaved.objects.all().filter(enslaved_id__lte=500000)
+        used_lgids=list(set([i[0] for i in list(enslaved.values_list('language_group_id')) if i[0] is not None]))
+
+        return JsonResponse([{ "id": item["id"], "name": item["name"], "lat": item["latitude"], "lng": item["longitude"], "alts": item[alt_names_key], "countries": item[countries_list_key] }
+            for item in items if item["id"] in used_lgids], safe=False)
+    else:
+        return JsonResponse([{ "id": item["id"], "name": item["name"], "lat": item["latitude"], "lng": item["longitude"], "alts": item[alt_names_key], "countries": item[countries_list_key] }
+            for item in items], safe=False)
 
 @csrf_exempt
 @cache_page(3600)
@@ -280,6 +288,7 @@ def search_enslaved(request):
             'voyage_id',
             'voyage__voyage_ship__ship_name',
             'voyage__voyage_dates__first_dis_of_slaves',
+            'voyage__voyage_dates__date_departed_africa',
             'voyage__voyage_itinerary__int_first_port_dis__place',
             'voyage__voyage_itinerary__imp_principal_place_of_slave_purchase__place',
             'voyage__voyage_itinerary__imp_principal_port_slave_dis__place',

voyages/apps/voyage/search_views.py

@@ -823,7 +823,6 @@ def get_filtered_places(request):
     return JsonResponse(filtered)
 
 @csrf_exempt
-@require_POST
 def get_all_sources(request):
     return JsonResponse({
         'data': list(VoyageSources.objects.values('pk', 'short_ref', 'full_ref', group_name=F('source_type__group_name')))

voyages/locale/en/LC_MESSAGES/django.po

@@ -2076,8 +2076,7 @@ msgstr ""
 #: voyages/apps/contribute/templates/contribute/interim.html:1649
 #: voyages/apps/contribute/templates/contribute/interim.html:1676
 msgid ""
-"URLs or digitized image of the source to be submitted wherever possible "
-"about source"
+"URLs of the source to be submitted wherever possible."
 msgstr ""
 
 #: voyages/apps/contribute/templates/contribute/interim.html:1463
@@ -2837,6 +2836,7 @@ msgid "Names & Pronunciation"
 msgstr ""
 
 #: voyages/apps/past/templates/past/contribute.html:65
+#: voyages/apps/past/templates/past/_cultural_association.html:25
 msgid "Language Groups"
 msgstr ""
 

voyages/locale/es/LC_MESSAGES/django.po

@@ -12423,11 +12423,9 @@ msgstr "Información sobre la fuente"
 #: voyages/apps/contribute/templates/contribute/interim.html:1649
 #: voyages/apps/contribute/templates/contribute/interim.html:1676
 msgid ""
-"URLs or digitized image of the source to be submitted wherever possible "
-"about source"
+"URLs of the source to be submitted wherever possible."
 msgstr ""
-"URL o imagen digital de la fuente para ser remitida cuando sea posible sobre "
-"la fuente"
+"URLs de la fuente para ser remitida cuando sea posible."
 
 #: voyages/apps/contribute/templates/contribute/interim.html:1463
 #: voyages/apps/contribute/templates/contribute/interim.html:1514
@@ -15440,9 +15438,8 @@ msgstr ""
 
 #: voyages/apps/past/templates/past/_cultural-association.html:4
 #: voyages/apps/past/templates/past/_cultural-association.html:9
-#, fuzzy
 msgid "Cultural Association"
-msgstr "Abolición eventual"
+msgstr "Asociación Cultural"
 
 #: voyages/apps/past/templates/past/_cultural-association.html:6
 #: voyages/apps/past/templates/past/_personal-data.html:4
@@ -15541,6 +15538,7 @@ msgid "Names & Pronunciation"
 msgstr ""
 
 #: voyages/apps/past/templates/past/contribute.html:65
+#: voyages/apps/past/templates/past/_cultural_association.html:25
 #, fuzzy
 msgid "Language Groups"
 msgstr "Idioma"