Link queue wrapper to record occupancy information in a file #129

constraintAutomaton · 2024-03-26T08:12:03Z

Objective

The objective of this actor is to collect information about the link queue and to serialize it into a file. The information collected are the following:

The link pushed in the queue
The link pop in the queue
The timestamp of the links pushed or popped
The reachability criteria from which the link was acquire

It is also possible to add custom link queue properties via the IOptionalLinkQueueParameters, but for their parsing the code has to be modified.

A label for each current reachability criteria also has been added. They can optionally label each link coming out of the extract-link actors. This feature doesn't only have the purpose to document the link queue occupancy but can be use in future work for engine optimization such has link queue ordering.

The @comunica/actor-rdf-resolve-hypermedia-links-queue-wrapper-info-occupance:query-identifier property can also be used to identify the query.

Example output

Config of the wrapper actor

{
  "@context": [
    "https://linkedsoftwaredependencies.org/bundles/npm/@comunica/runner/^3.0.0/components/context.jsonld",

    "https://linkedsoftwaredependencies.org/bundles/npm/@comunica/actor-rdf-resolve-hypermedia-links-queue-wrapper-info-occupancy/^0.0.0/components/context.jsonld"
  ],
  "@id": "urn:comunica:default:Runner",
  "@type": "Runner",
  "actors": [
    {
      "@id": "urn:comunica:default:rdf-resolve-hypermedia-links-queue/actors#wrapper-info-occupancy",
      "@type": "ActorRdfResolveHypermediaLinksQueueWrapperInfoOccupancy",
      "beforeActors": { "@id": "urn:comunica:default:rdf-resolve-hypermedia-links-queue/actors#fifo" },
      "mediatorRdfResolveHypermediaLinksQueue": { "@id": "urn:comunica:default:rdf-resolve-hypermedia-links-queue/mediators#main" },
      "filePath": "./link_queue_info.json"
    }
  ]
}

Config of the engine

{
    "@context": [
        "https://linkedsoftwaredependencies.org/bundles/npm/@comunica/config-query-sparql/^2.0.0/components/context.jsonld",
        "https://linkedsoftwaredependencies.org/bundles/npm/@comunica/config-query-sparql-link-traversal/^0.0.0/components/context.jsonld",
        "https://linkedsoftwaredependencies.org/bundles/npm/@comunica/actor-extract-links-predicates/^0.0.0/components/context.jsonld",
        "https://linkedsoftwaredependencies.org/bundles/npm/@comunica/actor-extract-links-solid-type-index/^0.0.0/components/context.jsonld",
        "https://linkedsoftwaredependencies.org/bundles/npm/@comunica/actor-extract-links-quad-pattern-query/^0.0.0/components/context.jsonld",
        "https://linkedsoftwaredependencies.org/bundles/npm/@comunica/runner/^3.0.0/components/context.jsonld"
    ],
    "import": [
        "ccqslt:config/config-solid-base.json",
        "ccqslt:config/rdf-resolve-hypermedia-links-queue/actors/wrapper-info-occupancy.json"
    ],
    "@graph": [
        {
            "@id": "urn:comunica:default:Runner",
            "@type": "Runner",
            "actors": [
                {
                    "@id": "urn:comunica:default:extract-links/actors#predicates-common",
                    "@type": "ActorExtractLinksPredicates",
                    "checkSubject": false,
                    "predicateRegexes": [
                        "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                        "http://www.w3.org/2002/07/owl##sameAs",
                        "http://xmlns.com/foaf/0.1/isPrimaryTopicOf"
                    ],
                    "labelLinksWithReachability":true
                },
                {
                    "@id": "urn:comunica:default:extract-links/actors#predicates-ldp",
                    "@type": "ActorExtractLinksPredicates",
                    "checkSubject": true,
                    "predicateRegexes": [
                        "http://www.w3.org/ns/ldp#contains"
                    ],
                    "labelLinksWithReachability":true
                },
                {
                    "@id": "urn:comunica:default:extract-links/actors#predicates-solid",
                    "@type": "ActorExtractLinksPredicates",
                    "checkSubject": true,
                    "predicateRegexes": [
                        "http://www.w3.org/ns/pim/space#storage"
                    ],
                    "labelLinksWithReachability":true
                },
                {
                    "@id": "urn:comunica:default:extract-links/actors#quad-pattern-query",
                    "@type": "ActorExtractLinksQuadPatternQuery",
                    "labelLinksWithReachability":true
                },
                {
                    "@id": "urn:comunica:default:extract-links/actors#solid-type-index",
                    "@type": "ActorExtractLinksSolidTypeIndex",
                    "inference": false,
                    "mediatorDereferenceRdf": {
                        "@id": "urn:comunica:default:dereference-rdf/mediators#main"
                    },
                    "labelLinksWithReachability":true
                }
            ]
        }
    ]
}

output

{
    "iris_popped": [
        {
            "url": "http://localhost:3000/pods/00000000000000000933/",
            "reachability_criteria": "cSolidStorage",
            "timestamp": 1711439007006
        },
        {
            "url": "http://localhost:3000/www.ldbc.eu/ldbc_socialnet/1.0/data/forum00000001099511627784",
            "reachability_criteria": "cCommon",
            "timestamp": 1711439008917
        }
    ],
    "iris_pushed": [
        {
            "url": "http://localhost:3000/pods/00000000000000000933/",
            "reachability_criteria": "cSolidStorage",
            "timestamp": 1711439007006
        },
        {
            "url": "http://localhost:3000/dbpedia.org/resource/Kelaniya",
            "reachability_criteria": "cMatch",
            "timestamp": 1711439007006
        },
        {
            "url": "http://localhost:3000/www.ldbc.eu/ldbc_socialnet/1.0/data/forum00000001099511627784",
            "reachability_criteria": "cCommon",
            "timestamp": 1711439008916
        }
    ],
    "started_empty": true,
    "query": {
        "type": "project",
        "input": {
            ...
         }
    }
}

If an identifier was given for the query than the query field will be this identifier.

Future works

Adding the contribution of each data source to the result bag.
Piping the result somewhere else than a file at least giving the option.

…lass

…creating it

rubensworks · 2024-03-26T09:04:07Z

packages/actor-extract-links-all/lib/ActorExtractLinksAll.ts

@@ -19,7 +27,7 @@ export class ActorExtractLinksAll extends ActorExtractLinks {
    return {
      links: await ActorExtractLinks.collectStream(action.metadata, (quad, links) => {
        for (const link of getNamedNodes(getTerms(quad))) {
-          links.push({ url: link.value });


The current approach with annotateLinkWithTheReachabilityCriteria and reachabilityLabel seems a bit complicated.

Could we just do something like links.push({ url: link.value, metadata: { actorExtractLinks: this.name } })?

I was not using the name because some actors such has predicate have multiple common variation (common, ldp, solidStorage) and I felt this was it was capture better. Another example would be the type index, maybe we would want to indicate that it is with inference or not or with other actor with notable properties.

Also I wanted to make the annotation optional hence why it is inside a method to not add to much if statement and since it will be same everywhere I placed it inside the abstract class.

some actors such has predicate have multiple common variation (common, ldp, solidStorage)

Perhaps those can be passed as additional metadata fields?

The main concern I have with this approach is that all actors have a new (optional) param now, and there is semantic redundancy with the name of the actor.

I could create a reachability criteria object documenting the specificity of the criteria (with custom equal function for when you care about the "major" reachability type or the full reachability).

For the optional new parameters, the reason it is there is to not modify the previous behavior where the metadata was empty. I defined this new parameter in the bus/abstract class to not disturbed too much the current and future implementation. I could make it so it is always in the links.

rubensworks · 2024-03-26T09:04:41Z

packages/bus-extract-links/lib/ActorExtractLinks.ts

  /**
   * @param args - @defaultNested {<default_bus> a <cc:components/Bus.jsonld#Bus>} bus
   */
  public constructor(args: IActorExtractLinksArgs) {
    super(args);
+    this.labelLinksWithReachability = args.labelLinksWithReachability ?? false;


Not sure we need this one. I guess we could just use the actor name this.name?

rubensworks · 2024-03-26T09:05:54Z

Before I review in detail, could you @RubenEschauzier first have a look at this as well?
This feels somewhat related to your prioritization work.

constraintAutomaton · 2024-03-26T10:37:22Z

I don't know why the CI doesn't pass on my end it does. Even the solidbench the one which is the one failing.

rubensworks · 2024-03-26T10:39:11Z

I don't know why the CI doesn't pass on my end it does. Even the solidbench the one which is the one failing.

The failure looks unrelated to your PR.

rubensworks · 2024-03-28T06:10:32Z

Another thought I have that would simplify this PR: Instead of writing to file, we could just write to the logger (e.g. at TRACE level). An external script could then be used to filter the log messages related to the link queue. This would make this PR work in the browser as well.

RubenEschauzier · 2024-03-28T08:27:57Z

Looks pretty good, it is a much more refined version of what I used for the link queue analysis paper. I'm unsure if it has much to do with prioritization, as (as far as I can tell) it is mainly for analyzing query execution in hindsight.

constraintAutomaton · 2024-03-28T08:47:09Z

Looks pretty good, it is a much more refined version of what I used for the link queue analysis paper. I'm unsure if it has much to do with prioritization, as (as far as I can tell) it is mainly for analyzing query execution in hindsight.

I think it is the part about the labeling of the links with the reachability criteria. At least for my work I plan to use those label to prioritize some links.

constraintAutomaton · 2024-03-28T09:11:41Z

Another thought I have that would simplify this PR: Instead of writing to file, we could just write to the logger (e.g. at TRACE level). An external script could then be used to filter the log messages related to the link queue. This would make this PR work in the browser as well.

Yes, I think it is much better! I will be also able to provide the information in a streaming manner (instead of dumping it all at every step, but I think I will provide the option to do so). I will make another repo to parse this information has a file because it make it easy for my own investigation and debugging.

constraintAutomaton added 22 commits March 25, 2024 08:44

actor to save on disk link queue information created

f516441

reachability label added to CPredicate

f52af7b

simplify the link labelling by modifiying the extract link abstract c…

97360ed

…lass

refactor the annotation of link to make it build the link instead of …

3e4a8d3

…creating it

lint fix

6e22dea

lsd module added to the context entry package

c3b8ca6

link annotation added to cMatch

2a551f4

reachability annotation added for the type index

d23961e

reachability annotation added for cAll

e0a2e4e

made the reachability metadata append instead of override

9d8404c

reachability annotation added to content policies

df04f95

reachability annotation added to TREE

76c0540

reachability annotation added to quad pattern extractor

6b04bad

readme and comments added

58c35c8

lint fix

1e0e213

documentation updated

d040fcb

config for link queue wrapper added

b3a5a01

type-link-traversal dependeency bump

63cd922

made the link queue work if there are no query string

327d197

fix random operation

d7fc6d1

support for multiple queries updated for comunica 3

228e812

labeling reachability in links added into the properties of component.js

c709ac4

constraintAutomaton marked this pull request as draft March 26, 2024 08:12

constraintAutomaton changed the title ~~Link queue wrapper to record occupancy information to a file~~ Link queue wrapper to record occupancy information in a file Mar 26, 2024

rubensworks reviewed Mar 26, 2024

View reviewed changes

static properties created for the predicate reachability

81618d9

constraintAutomaton added 2 commits March 27, 2024 07:58

random identifier implemented

2467e3a

test for priority of the defined identifier over the random one added

ffd3d00

constraintAutomaton closed this Jun 5, 2024

constraintAutomaton deleted the feature/wrapped-link-queue-info branch June 5, 2024 06:45

constraintAutomaton mentioned this pull request Jun 5, 2024

Link queue wrapper to log the link queue occupancy #136

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Link queue wrapper to record occupancy information in a file #129

Link queue wrapper to record occupancy information in a file #129

constraintAutomaton commented Mar 26, 2024 •

edited

Loading

rubensworks Mar 26, 2024

constraintAutomaton Mar 26, 2024 •

edited

Loading

rubensworks Mar 26, 2024

constraintAutomaton Mar 26, 2024 •

edited

Loading

rubensworks Mar 26, 2024

rubensworks commented Mar 26, 2024

constraintAutomaton commented Mar 26, 2024

rubensworks commented Mar 26, 2024

rubensworks commented Mar 28, 2024

RubenEschauzier commented Mar 28, 2024

constraintAutomaton commented Mar 28, 2024

constraintAutomaton commented Mar 28, 2024

Link queue wrapper to record occupancy information in a file #129

Link queue wrapper to record occupancy information in a file #129

Conversation

constraintAutomaton commented Mar 26, 2024 • edited Loading

Objective

Example output

Future works

rubensworks Mar 26, 2024

Choose a reason for hiding this comment

constraintAutomaton Mar 26, 2024 • edited Loading

Choose a reason for hiding this comment

rubensworks Mar 26, 2024

Choose a reason for hiding this comment

constraintAutomaton Mar 26, 2024 • edited Loading

Choose a reason for hiding this comment

rubensworks Mar 26, 2024

Choose a reason for hiding this comment

rubensworks commented Mar 26, 2024

constraintAutomaton commented Mar 26, 2024

rubensworks commented Mar 26, 2024

rubensworks commented Mar 28, 2024

RubenEschauzier commented Mar 28, 2024

constraintAutomaton commented Mar 28, 2024

constraintAutomaton commented Mar 28, 2024

constraintAutomaton commented Mar 26, 2024 •

edited

Loading

constraintAutomaton Mar 26, 2024 •

edited

Loading

constraintAutomaton Mar 26, 2024 •

edited

Loading