Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix duplicate entries Missing and not Missing #106

Merged
merged 11 commits into from
Feb 27, 2024

Conversation

BobSilent
Copy link
Contributor

@BobSilent BobSilent commented Jan 19, 2024

I have the issue as well as described here:

fix: #87
fix: [Issue]: Missing Episode Fetcher Seems to Create Double Season Folders #9999
or here https://forum.jellyfin.org/t-jellyfin-displaying-duplicate-missing-files-for-episodes
https://www.reddit.com/r/jellyfin/comments/t4z9jf/show_missing_episode_showing_episodes_i_already/

improves: Unhelpful error messages #71

mainly if the items were added as missing first, they never get deleted after adding media files.

@BobSilent BobSilent force-pushed the fix-duplicate-entries branch 2 times, most recently from 6762746 to 05a69da Compare January 19, 2024 01:08
@crobibero
Copy link
Member

@BobSilent we just merged in tvdb v4 changes, if this PR is still needed please update

@BobSilent
Copy link
Contributor Author

BobSilent commented Feb 12, 2024

@crobibero I need to test and check, do I have some time (I think latest until the weekend).
Hopefully it can make it into the release then.

A first quick code check shows me that the PR is still required, I will rebase.

@BobSilent BobSilent force-pushed the fix-duplicate-entries branch from 05a69da to 148e901 Compare February 12, 2024 23:20
@BobSilent
Copy link
Contributor Author

@crobibero I am currently facing some issues with #93, I will prepare a separate PR for these bug fixes!

@BobSilent
Copy link
Contributor Author

BobSilent commented Feb 19, 2024

Please see #110

I will rebase this PR on #110

@BobSilent BobSilent force-pushed the fix-duplicate-entries branch from 148e901 to feb22ef Compare February 19, 2024 00:29
@BobSilent
Copy link
Contributor Author

BobSilent commented Feb 23, 2024

@crobibero a little bit off topic for this PR, but I have prepared another cleanup and I saw this comment here

/// <summary>
/// Create an independent ServiceProvider because registering HttpClients directly into Jellyfin
/// causes issues upstream.
/// </summary>
/// <param name="applicationHost">Instance of the <see cref="IApplicationHost"/>.</param>
/// <returns>The service provider.</returns>
private ServiceProvider ConfigureService(IApplicationHost applicationHost)

Are the mentioned "issues upstream" the fact that "suddenly" logging is no longer working as configured?
This cause i could identify:
the service collection is configured from the plugins quite early:
https://github.com/jellyfin/jellyfin/blob/e93d03d8cbff2122d7296f477604146f64758a73/Jellyfin.Server/Program.cs#L188-L189

But Serilog is added quite late.
https://github.com/jellyfin/jellyfin/blob/e93d03d8cbff2122d7296f477604146f64758a73/Jellyfin.Server/Program.cs#L349-L351

if this is changed like that:
image
in Jellyfin.Server/Program.cs#L188-L189 the AddHttpClient can be used.
Internally the AddHttpClient calls AddLogging() and this is called before adding serilog, therefore the configuration gets lost and only parts will be logged afterwards.

Shall I create a PR for this in jellyfin/jellyfin for the 10.8 branch?

@BobSilent
Copy link
Contributor Author

BobSilent commented Feb 23, 2024

@crobibero during testing I saw some issues with the Tvdb.Sdk dll.
Sometimes I get System.Text.Json.JsonException: The JSON value could not be converted... Exceptions as the schema seems to be quite strict, where it seems that TVDB send null instead of a value:

Tvdb.Sdk.SearchException: Could not deserialize the response body stream as Tvdb.Sdk.Response45.

Status: 200
Response: 

 ---> System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.data[0].movie.runtime | LineNumber: 0 | BytePositionInLine: 281.
 ---> System.InvalidOperationException: Cannot get the value of a token type 'Null' as a number.
   at System.Text.Json.Utf8JsonReader.TryGetInt32(Int32& value)
   at System.Text.Json.Utf8JsonReader.GetInt32()
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
   at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.ReadAllAsync[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
   at Tvdb.Sdk.SearchClient.ReadObjectResponseAsync[T](HttpResponseMessage response, IReadOnlyDictionary`2 headers, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Tvdb.Sdk.SearchClient.ReadObjectResponseAsync[T](HttpResponseMessage response, IReadOnlyDictionary`2 headers, CancellationToken cancellationToken)
   at Tvdb.Sdk.SearchClient.GetSearchResultsByRemoteIdAsync(String remoteId, CancellationToken cancellationToken)
   at Jellyfin.Plugin.Tvdb.TvdbClientManager.GetSeriesByRemoteIdAsync(String remoteId, String language, CancellationToken cancellationToken)
...

Post it as separate issue here: crobibero/tvdb-sdk-csharp?

@crobibero
Copy link
Member

Shall I create a PR for this in jellyfin/jellyfin for the 10.8 branch?

Thank you for investigating the figuring out the logging issue! We aren't planning on making any more 10.8 releases, so a PR targeting master would be greatly appreciated!

Post it as separate issue here: crobibero/tvdb-sdk-csharp?

That sdk is 99.97% auto-generated, any schema issues should be filed in the upstream repository (https://github.com/thetvdb/v4-api). The difficulty there lies with trying to figure out which model the issue is actually in

I think this PR can be merged without fixing the deserialization issues

@BobSilent
Copy link
Contributor Author

I think this PR can be merged without fixing the deserialization issues

Yes, this is more like a fyi...

Thank you for investigating the figuring out the logging issue! We aren't planning on making any more 10.8 releases, so a PR targeting master would be greatly appreciated!

Ok, first need to setup a system here on my side, as I saw there are also other (bigger) changes in, like jellyfin/jellyfin#9078

@BobSilent
Copy link
Contributor Author

Shall I rebase to latest master?

@crobibero
Copy link
Member

I'm not sure what you're asking but I'm fairly certain that the issue still persists in latest master of jellyfin/jellyfin

@BobSilent
Copy link
Contributor Author

sorry maybe confusing due too many topics here ;)

I just wanted to know if i shall rebase this PR to latest?

@crobibero
Copy link
Member

Don't worry about rebasing this PR, I'm going to start reviewing it shortly

Copy link
Member

@crobibero crobibero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still have some templated logging fixes to be made

@scampower3
Copy link
Member

scampower3 commented Feb 23, 2024

@crobibero during testing I saw some issues with the Tvdb.Sdk dll. Sometimes I get System.Text.Json.JsonException: The JSON value could not be converted... Exceptions as the schema seems to be quite strict, where it seems that TVDB send null instead of a value:

Tvdb.Sdk.SearchException: Could not deserialize the response body stream as Tvdb.Sdk.Response45.

Status: 200
Response: 

 ---> System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.data[0].movie.runtime | LineNumber: 0 | BytePositionInLine: 281.
 ---> System.InvalidOperationException: Cannot get the value of a token type 'Null' as a number.
   at System.Text.Json.Utf8JsonReader.TryGetInt32(Int32& value)
   at System.Text.Json.Utf8JsonReader.GetInt32()
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
   at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.ReadAllAsync[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
   at Tvdb.Sdk.SearchClient.ReadObjectResponseAsync[T](HttpResponseMessage response, IReadOnlyDictionary`2 headers, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Tvdb.Sdk.SearchClient.ReadObjectResponseAsync[T](HttpResponseMessage response, IReadOnlyDictionary`2 headers, CancellationToken cancellationToken)
   at Tvdb.Sdk.SearchClient.GetSearchResultsByRemoteIdAsync(String remoteId, CancellationToken cancellationToken)
   at Jellyfin.Plugin.Tvdb.TvdbClientManager.GetSeriesByRemoteIdAsync(String remoteId, String language, CancellationToken cancellationToken)
...

Post it as separate issue here: crobibero/tvdb-sdk-csharp?

This serialization problem is caused by the MovieBaseRecord and possibly MovieExtendedRecord schemas. Should have little effect on the plugin since its scope is limited to Series only. Also, if you can provide the remoteID that causes the problem that will be great. Can't seem to replicate it on my end.

@BobSilent
Copy link
Contributor Author

@crobibero during testing I saw some issues with the Tvdb.Sdk dll. Sometimes I get System.Text.Json.JsonException: The JSON value could not be converted... Exceptions as the schema seems to be quite strict, where it seems that TVDB send null instead of a value:

Tvdb.Sdk.SearchException: Could not deserialize the response body stream as Tvdb.Sdk.Response45.

Status: 200
Response: 

 ---> System.Text.Json.JsonException: The JSON value could not be converted to System.Int32. Path: $.data[0].movie.runtime | LineNumber: 0 | BytePositionInLine: 281.
 ---> System.InvalidOperationException: Cannot get the value of a token type 'Null' as a number.
   at System.Text.Json.Utf8JsonReader.TryGetInt32(Int32& value)
   at System.Text.Json.Utf8JsonReader.GetInt32()
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.ReadJsonAndSetMember(Object obj, ReadStack& state, Utf8JsonReader& reader)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   --- End of inner exception stack trace ---
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan`1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase)
   at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.ReadAllAsync[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo, CancellationToken cancellationToken)
   at Tvdb.Sdk.SearchClient.ReadObjectResponseAsync[T](HttpResponseMessage response, IReadOnlyDictionary`2 headers, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Tvdb.Sdk.SearchClient.ReadObjectResponseAsync[T](HttpResponseMessage response, IReadOnlyDictionary`2 headers, CancellationToken cancellationToken)
   at Tvdb.Sdk.SearchClient.GetSearchResultsByRemoteIdAsync(String remoteId, CancellationToken cancellationToken)
   at Jellyfin.Plugin.Tvdb.TvdbClientManager.GetSeriesByRemoteIdAsync(String remoteId, String language, CancellationToken cancellationToken)
...

Post it as separate issue here: crobibero/tvdb-sdk-csharp?

This serialization problem is caused by the MovieBaseRecord and possibly MovieExtendedRecord schemas. Should have little effect on the plugin since its scope is limited to Series only. Also, if you can provide the remoteID that causes the problem that will be great. Can't seem to replicate it on my end.

@scampower3
you can try it for id 253323

seriesClient.GetSeriesExtendedAsync(id: 253323, meta: null, @short: true);

@crobibero crobibero added the bug This PR or Issue describes or fixes something that isn't working label Feb 27, 2024
@crobibero
Copy link
Member

Going to merge this and create an issue to track the deserialization issue

@crobibero crobibero merged commit 60b3018 into jellyfin:master Feb 27, 2024
4 checks passed
@BobSilent
Copy link
Contributor Author

Shall I create a PR for this in jellyfin/jellyfin for the 10.8 branch?

Thank you for investigating the figuring out the logging issue! We aren't planning on making any more 10.8 releases, so a PR targeting master would be greatly appreciated!

Post it as separate issue here: crobibero/tvdb-sdk-csharp?

That sdk is 99.97% auto-generated, any schema issues should be filed in the upstream repository (https://github.com/thetvdb/v4-api). The difficulty there lies with trying to figure out which model the issue is actually in

I think this PR can be merged without fixing the deserialization issues

Just for completeness I tested this on 10.9, here with the changes from jellyfin/jellyfin#9078 this is fixed.
I have migrated the plugin to 10.9, i can share the code here.

@crobibero
Copy link
Member

crobibero commented Feb 27, 2024

Thanks for closing the loop! I also tested 10.9 and thought I was going crazy haha

Feel free to open a PR here, I can just mark it as blocked until 10.9 is released

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This PR or Issue describes or fixes something that isn't working
Projects
None yet
3 participants