Raise/warn on incomplete columns in normalize

[steinitzu]

Description

Turns the "unbound column" warning into an exception for not-null columns and move it to normalize

Related Issues

• Fixes Wrong Merge Key Not Throwing Error #1463

Additional Context

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	c1e2c85
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/66b3a4b0f38083000809eb94

[sh-rp]

I'm wondering if we should not do this in the extraction step already. All columns that are non-nullable (and merge and primary keys should be that) should raise if not populated. Extraction spends time on I/O mostly and not on python code as the normalizer, so the check would not make a big difference in performance in my opinion.

[steinitzu]

I'm wondering if we should not do this in the extraction step already. All columns that are non-nullable (and merge and primary keys should be that) should raise if not populated. Extraction spends time on I/O mostly and not on python code as the normalizer, so the check would not make a big difference in performance in my opinion.

I agree it would be much better to fail early if possible. Ideally we could tell right after the first data item is extracted.
But I wasn't sure if we can always tell whether the column is populated in extract. The "seen data" marker for the table is only set in normalize so I was going by that. But I'll give it a try.

[sh-rp]

Can you write a test (or check if one exists) to see what happens when we do a merge on merge keys but some rows have null in the merge key? It's not super important right now, but if it would be interesting to know what happens :)

[steinitzu]

I couldn't find a test so I added one. This was raising an exception already through schema.coerce_row in normalize

[sh-rp]

Looks good, small requests

[steinitzu]

I'm wondering if we should not do this in the extraction step already. All columns that are non-nullable (and merge and primary keys should be that) should raise if not populated. Extraction spends time on I/O mostly and not on python code as the normalizer, so the check would not make a big difference in performance in my opinion.

Was looking into if this was possible also, but I don't think so without moving a lot of normalize logic into extract. I wasn't sure how much schema inferrence is done in extract, seems there is none.

[rudolfix]

@steinitzu this needs merge from devel. we did a lot of updates. @sh-rp otherwise this PR is good to go?

[steinitzu]

@steinitzu this needs merge from devel. we did a lot of updates. @sh-rp otherwise this PR is good to go?

branch is up to date now and tests passing

[rudolfix]

LGTM!