Set node_keys via a function

[BenOnTrack]

Hi,
I've set the node_keys table via a function,

function get_node_keys()
        ....
	return node_keys
end

The only way I've found to make that works is to call the function at initialization

function init_function()
	node_keys = get_node_keys()
end

and it works great - but if I understand well, the initialization is called at each object ? - I'm curious if there's a more efficient/reliable way to set the node_keys without hard-coding it.

[cldellow]

I'm curious about this, too. My init_function pulls in Wikipedia pageview data as a proxy for feature importance. Currently I'm serializing it as a text file. This is naive and slow to parse, but seemed easier than getting SQLite bindings to work. :)

Googling makes me think Lua doesn't like multiple threads sharing the same interpreter state. Thus, I think the best case is that init_function could get called 1x per thread.

I think for the non-mapsplit case, the Lua interpreter currently gets initialized:

• once on program startup to retrieve node_keys
• once per phase per block in the PBF (the generate_output parameter, passed here, used here)
  • there are 4 phases:

    tilemaker/src/read_pbf.cpp

    Line 327 in 439c6b1

    std::vector<ReadPhase> all_phases = { ReadPhase::Nodes, ReadPhase::RelationScan, ReadPhase::Ways, ReadPhase::Relations };

So not once per node/way/relation, but I think potentially still quite often.

I tried some naive things (warning: I haven't written C++ in a long time, buyer beware) to initialize OsmLuaProcessing once per thread, and to decrease the number of times the thread pool is re-created.

It mostly works. On my 16MB test PBF file, with --threads 1, I see the OsmLuaProcessing constructor run 3 times (vs 17 with the code in master). I'm a bit flummoxed that it's 3, I would have expected 2. Still, seems like strong evidence that it ought to be possible to avoid re-initing the Lua interpreter as often as it is now.

[systemed]

Yes, that's pretty much the size of it!

It shouldn't have any measurable performance impact for @BenOnTrack's case. For the Wikipedia pageview data I can see the issue. I think in those circumstances, I'd consider storing the parsed data in Redis and then pulling that out from Lua, which should be faster.

[cldellow]

Ah, I see that there exist pure-Lua clients for redis and HTTP - that's a great hint, thank you!