Refactor LoadInfo metrics layout schema

[sultaniman]

This issue was reported by a community member in slack and related issue #1043

When we capture load_info data in the destination database the following occurs:

• Pipeline execution slows down significantly after x number of incremental pipeline runs (it is back to the original speed when load_info is not captured in the database). My specific job (sourcing data from rest API capturing it in MotherDuck) slows down from 1.5 minutes to over 10 minutes (after a few hundred runs) and seems to keep getting slower.
• Each pipeline run creates a new table with one record (at least in my simple pipeline) with names like _load_info__metrics___1709356777_4431682.

TODO

• Adjust _ExtractInfo.metrics from Dict[str, List[ExtractMetrics]] to just List[ExtractMetrics],
• Add load_id field to StepMetrics,
• Adjust dependent code to use and extract information collection to lookup load ids etc.

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	f428b25
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/65e5e3eb97ffab00081467e7

[rudolfix]

1. please add load_info, normalize_info and extract_info.
2. please load to a separate schema (you have schema argument to run()
3. use this second schema to compare hashes (mind that it wont be a default)

[rudolfix]

this is OK and overall even better than a dictionary. But in general the shape of the data is changed in asdict() like here:

def asdict(self) -> DictStrAny:
        # to be mixed with NamedTuple
        d: DictStrAny = self._asdict()  # type: ignore
        d["pipeline"] = {"pipeline_name": self.pipeline.pipeline_name}
        d["load_packages"] = [package.asdict() for package in self.load_packages]
        if self.metrics:
            d["started_at"] = self.started_at
            d["finished_at"] = self.finished_at
        return d

and the problem was that we didn't reformat metric to convert form dict to list.

[rudolfix]

please load something more complicated than this. ie. source with a resource that have several hints

[rudolfix]

here let's load again but we should add another source with different resource and some schema hints

[rudolfix]

you may have a schema difference when loading extract_info and the new resource has a new hint type. then indeed we may add column dynamically.

[sultaniman]

closing in favor of #1051