Proposals to improve map load

[oscgonfer]

Let's discuss in this issue how can we improve the map load.
There is an ongoing branch by @pral2a working on a range-slider approach, that would limit the range of dates in the devices as a WIP here.

In conversations with @timcowlishaw, we also thought it would help to show only the online devices on page load, which would dramatically reduce the load time. This also would help with the everlasting question of online/offline devices when people visit the map.

What do you think? Comments welcome.

[pral2a]

Maps view

My primary concern is that the actual map no longer aligns with the platform's purpose for the last 5-6 years. The goal of a real-time worldwide sensor network changed in favour of supporting situated sensing experiments by local communities.

With it, one option is to think creatively about a better visualization that is not a map and doesn't require heavy development.

Online vs Offline

The idea of online vs. offline should be treated with care. The purpose of showing so many devices, even offline, is to show the legacy of the community and the impact the project has created. It's not for scientists or tech gurus to look for real-time online data available to use.

Loading time improvements

In my experience, the long loading time was not only connected with the API response time. The client processing time also plays a role on it. It takes a while for Angular to build all the device marker instances and then call Leaflet to load them on the map. It's relevant to note that we make Angular objects before the Leaflet instead of directly creating them on Leaflet as other apps do. That is why my slider approach did request all the devices at once but will only show them all on the map if you choose the slider.

Regarding the response time, I suggest improving RoR or NGINX level caching, a classic.

[oscgonfer]

Maps view

My primary concern is that the actual map no longer aligns with the platform's purpose for the last 5-6 years. The goal of a real-time worldwide sensor network changed in favour of supporting situated sensing experiments by local communities.

With it, one option is to think creatively about a better visualization that is not a map and doesn't require heavy development.

As we have quite a large list of changes, I would say we can leave this for after March.

Online vs Offline

The idea of online vs. offline should be treated with care. The purpose of showing so many devices, even offline, is to show the legacy of the community and the impact the project has created. It's not for scientists or tech gurus to look for real-time online data available to use.

Sure, totally agree on that. However, as the number of points is growing, a way of filtering as the first load would be beneficial, and doesn't necessarily remove the legacy, it simply doesn't make the points in the map. Similar reasoning would apply to older devices, versus those that are newer (which would be the filtering done by the range slider).

Loading time improvements

In my experience, the long loading time was not only connected with the API response time. The client processing time also plays a role on it. It takes a while for Angular to build all the device marker instances and then call Leaflet to load them on the map. It's relevant to note that we make Angular objects before the Leaflet instead of directly creating them on Leaflet as other apps do. That is why my slider approach did request all the devices at once but will only show them all on the map if you choose the slider.

I want to think it's a mix, maybe I am mistaken. Part of the problem could come from the request itselfhttps://github.com//issues/431#issue-907298001
And then part of it after.

Maybe a small assessment of the page load bottlenecks will help us.

Regarding the response time, I suggest improving RoR or NGINX level caching, a classic.

@timcowlishaw ?

[pral2a]

Follow up on proposed solutions

1. Implement a Standard Backend with Leaflet Integration

   • Establish a backend architecture that uses well-established geospatial standards to simplify frontend development and improve compatibility for future clients.
   • Recommended standards and technologies:
     • Web Feature Service (WFS)
     • GeoJSON API
     • OGC API - Features
   • Note that looks like Rails lacks robust gems to implement a server using OGC standards, unlike Python which has tools like pygeoapi.

2. Implications of Implementing a Standard Backend (Point 1)

   • Minimize the data payload for the world map by loading markers tooltip data on demand.
   • Implement filters on backend for tags and properties such as:
     • Device status (e.g., online/offline)
     • Location type (e.g., indoor/outdoor)
   • Leverage Leaflet’s built-in features such as layers, instead of using customized Angular code for filters. This will allow for a more streamlined codebase and better integration with geospatial data standards.

3. Simplified Alternative to a Standardized Backend

   • Maintain the existing endpoint structure but add a time range filter to the frontend. Correspondingly, update the backend to handle requests for the most recent devices based on their last reading update.
   • See preliminary prototype for the frontend part https://github.com/fablabbcn/smartcitizen-web/tree/feat-map-time-range-slider

Resources for Implementation

• Consider exploring the following Ruby resources for geospatial project implementation and inspiration:
  • Awesome-Geospatial Ruby Resources

[timcowlishaw]

Aha we already implement caching, but this is complicated for logged in users (we can't cache authenticated requests as we need to show private devices for logged in users, and hide for everyone else).

The query for the worldmap itself isn't too horrible, a large part of the slowness is the transfer time (the world_map json is like 4.5mb). For that reason, I think slimming down teh representation of each device to only the data needed to render the map point would be smart, then adapting the map code in angular, to do a request for /device/:id to get the remaining data when selected. If this new /devices/world_map endpoint could be made to use a standard format like GeoJSON, then so much the better!

[pral2a]

Thanks @timcowlishaw

Some random thoughts on preliminary or temporary solutions with minimal backend implications:

• Transport time: Ensure Gzip is properly enabled. Could BSON or Protobuf help reduce size over JSON?
• Front-end processing time: Does it matter? Could we move angular custom logic to Leaflet to speed up processing? Could switching from Leaflet.markercluster to https://github.com/mapbox/supercluster help reduce Leaflet processing time?

[oscgonfer]

Some low-hanging fruit from changing the world_map endpoint fields:

{
    "id":13074,
    "uuid":"fc9d12cf-b3aa-4030-9abf-2315e8f8a65d", --> Not strictly needed
    "name":"Six Mushroom Stag",
    "description":"Smart Citizen Kit 2.1 with Urban Sensor Board",
    "state":"has_published",
    "system_tags":["indoor","offline"],
    "user_tags":["Research","Inside","Ground Floor","Lab"],
    "last_reading_at":"2021-11-19T18:42:44Z", 
    "created_at":"2020-07-26T10:41:31Z", --> Not needed
    "updated_at":"2024-06-27T17:03:04Z", --> Not needed
    "notify":{"stopped_publishing":true,"low_battery":true}, --> Not needed
    "device_token":"[FILTERED]", --> Not needed
    "location":{
        "ip":null, --> Not needed
        "exposure":"indoor",
        "elevation":null, --> Not needed
        "latitude":50.82037,
        "longitude":3.29252,
        "geohash":"u1434gtd17", --> Not needed
        "city":"Kortrijk",
        "country_code":"BE", --> Not strictly needed
        "country":"Belgium"
    },
    "data_policy":{
        "is_private":false, --> Not needed (we _could_ just return non-private ones)
        "enable_forwarding":"[FILTERED]", --> Not needed
    "   precise_location":"[FILTERED]" --> Not needed
    },
    "hardware":{
        "name":"SCK 2.1",
        "type":"SCK",
        "version":"2.1",
        "slug":"sck:2,1",
        "last_status_message":"[FILTERED]" --> Not needed
    },
    "owner":{
        "id":7649,
        "uuid":"cff59239-7361-40cb-b216-99d1a359c788", --> Not needed
        "username":"BartBrowaeys",
        "url":null --> Not needed
    },
    "experiment_ids":[] --> Not needed
}