Feature request: Add TMDB-ID and add individual schedule per target #134
Replies: 32 comments 45 replies
-
|
@jojo141185 The issue with fetching VOD info for each entry can result in a ban. This is the same problem as with series when using an Xtream source with M3U output. When I implemented the The difference between series and VOD is that series can change if they are not finished, which is why I need to fetch them every time. VOD, on the other hand, does not change. I could imagine building a database that is created only once when no VOD info exists, to avoid repeatedly fetching the same VOD information.
I’m not sure what exactly is taking so much time. Are you referring to the The scheduling based of types is a nice idea. I have to think about howto implement this. |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Yes, I am aware that some providers have problems with too many requests in too short a time. For this reason, your idea with the database and only updating the information in the event of an update (e.g. by checking the timestamp in the added field) is a brilliant idea! I have currently noticed that when the reading of the series info is active (xtream_resolve_series: true) the info of ALL available series is parsed before the filter is applied! Is that correct? If so, that is probably the reason for the long update times of the series list. |
Beta Was this translation helpful? Give feedback.
-
|
Yes you are right, the series info is fetched for the original list. The problem is, the captured series episodes need to be processed too. I wanted to avoid multiple times fetching the same series info. |
Beta Was this translation helpful? Give feedback.
-
|
That would be fantastic if only series information from the filtered list is processed in the future. Combined with a database where existing data is cached, and only new items are updated, added, or removed as needed, the processing time for repeated tasks could be significantly reduced. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 the challenge with series is that I have no indicator to know when a dataset has been updated. As a result, I need to repeatedly fetch the series information to ensure I have the latest data. Is there a specific value or property that I could use to determine if an update has occurred without fetching the series information? |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Action get_series should return a field called "last_modified" with a timestamp for each entry. For get_vod_streams the field is called "added". |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 I have now implemented the vod kodi style. This only works with You need to set some options on target like: targets:
- name: simple_kodi
output:
- type: strm
filename: /home/kodi/movies/
filter: "!SIMPLE_CHAN_VOD_INFO_TEST!"
options: {kodi_style: true, xtream_resolve_video: true, xtream_resolve_video_delay: 2}
|
Beta Was this translation helpful? Give feedback.
-
|
@euzu Thank you for your efforts! Seems to be the same problem as #132 Info to my system: |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 no i had no time, please use the web folder from a release. |
Beta Was this translation helpful? Give feedback.
-
|
@euzu But with series something isn't working properly. After the individual series information has been read out, the following error message appears: I use the following settings for the target: Apart from this topic, it would be useful to write some more information in the log where the user receives feedback on which goal/output is currently being worked on - especially with the long processing times that can occur on the first start. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 sorry i did not clearly tell you that you should only test vod part. The series part is currently wip. I try to implement it like vod processing to update only when an update is available. The tmdb processing for series is currently not implemented. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 it took a while but i think it is working now, at least for my source. Please can you test xtream_resolve_vod branch.
I have renamed the option to sample templates:
- name: SIMPLE_CHAN
value: 'Name ~ "...." ... <Write you filter here> '
sources:
- inputs:
- type: xtream
name: xtream
url: 'http://my-provider.tv' <---- your provider url, this is only fake
username: test
password: test.secret
options: {xtream_skip_live: true, xtream_skip_vod: false, xtream_skip_series: false} <---- enable/disable as wished
epg_url: /tmp/m3u-test/settings/xtream-local/epg.xml
targets:
- name: xtream
filter: "!SIMPLE_CHAN!"
options: {ignore_logo: false, kodi_style: true, xtream_resolve_series: true, xtream_resolve_series_delay: 2, xtream_resolve_vod: true, xtream_resolve_vod_delay: 2} <------- kodi_style and resolve options
output:
- type: xtream
- type: strm <----- kodi output
filename: /tmp/kodi/library <----- kodi output directory is mandatory
sort:
match_as_ascii: true
groups:
order: asc
mapping:
- simple-su |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Nice to hear from you and I hope you're well again! I created a new build from your feature branch xtream_resolve_vod and did a quick test to resolve some series via xtream. I watched the debug log and first all series information was processed (...&action=get_series_info&series_id=XXXXXX), then the following error messages were displayed: I think that's OK, because the series info doesn't contain any tmdb ids. Have I configured something wrong? My target settings: |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 thank you, i am now better. The error message indicated that serde was not able to parse the series info. I hope it works now. |
Beta Was this translation helpful? Give feedback.
-
|
Great how quickly you solved this :-) Example in Log: File output: In addition, the tmdb id 0 is added to the name even though it was not found, which can lead to confusion. There are also a few new (type) errors in the debug log: |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 this errors indicate that the series_info has missing fields. |
Beta Was this translation helpful? Give feedback.
-
|
@euzu The last commit was successful - no more errors and the strm files of the series are now fully exported.
|
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 i have pushed to the branch. Can you test it please. We need to discuss about one point. The Folder structure of Kodi needs a folder for each series. For example when you have a TV-Show named The structure is I am taking the first folder name from the series info which is not modified during the processing (rename or mapping is not applied). For example you apply a renaming from Hello World to Goodby World then the result would be As you can see the first folder name is not processed and keeps the same, because it is comming from the series info attribute which is not processed. |
Beta Was this translation helpful? Give feedback.
-
|
i mean the commit today, |
Beta Was this translation helpful? Give feedback.
-
|
This is the last commit i can see: I sent you some files via email with API output and the result of the test with the last commit (cea636c) |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 sorry i thought i pushed the changes but they were rejected due to a merge problem. Now i have pushed the changes. |
Beta Was this translation helpful? Give feedback.
-
|
Tested with your newest commit (452e7e5). Unfortunately there are still multiple years specified in the folder name and thus episodes of the same series are distributed across different folders (see mail). |
Beta Was this translation helpful? Give feedback.
-
|
Ok i think i need to test with your data. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185, there is another issue with Kodi. I can't apply reverse proxy URLs properly. The reverse proxy URL rewrite only works when accessing the playlist through the API with credentials. However, with One potential solution could be to assign a username to the Kodi output, like this: output:
- type: strm
filename: /tmp/kodi/media
username: kodiThe |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I've noticed that too. Neither in the m3u playlist nor in the strm files is the URL adjusted when using the reverse proxy option. That's why I still use cascaded iptv-proxy. |
Beta Was this translation helpful? Give feedback.
-
|
But the m3u playlist should rewrite the urls when the acccessing user has reverse proxy setting. This is implemented. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you're absolutely right. The streaming links in the M3U playlist retrieved via the API are proxified. |
Beta Was this translation helpful? Give feedback.
-
|
But data is the working directory to store the local processed data. The problem with strm could be solved with a specific user. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, with the newest commit and the correct username defined in output it works and also the stream url in strm file is proxified. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 Note: I made some changes to the stored data. Before testing, please delete the I’ve implemented the reverse proxy mode for STRM output. To enable reverse URLs, you’ll need to set a username for the STRM output and define an Xtream output as well. Here’s why this is necessary:
Let me know if you encounter any issues! |
Beta Was this translation helpful? Give feedback.
-
|
I did a test with commit (a027e80). I think we're almost there. However, there are still a few series in different folders (with and without year of publication). I've sent you another email with the results. |
Beta Was this translation helpful? Give feedback.
-
|
Did another test of the reverse proxy functionality in conjunction with the STRM files based on commit 406ae84. The problem is that the original URL is resolved incorrectly (missing extension). |
Beta Was this translation helpful? Give feedback.
-
|
i really dont understand why this happens, need to think about a solution. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 i have pushed some changes to fix the reverse proxy url extension problem. If you are able to debug, can you try to figure out the problem with the Series names without year? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, with the last commit the stream URL from the STRM file is now a mix of both: If I correct the URL to Unfortunately, my test instance is currently running in a Docker container, which makes debugging quite difficult. Can’t the issue with the folder names be reproduced using the JSON files I sent you by email? |
Beta Was this translation helpful? Give feedback.
-
|
sorry i could not test it. i need the output action=get_series without category_id
|
Beta Was this translation helpful? Give feedback.
-
|
Of course... I sent you the output via email. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 i have send you an email, and i have pushed the changes to the feature branch. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, euzu! With your last commit the strm output is now working like a charm.
|
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 i have pushed some changes to the branch. 1.) do you use 2.) should be fixed the the latest commit. |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Series is working now on direct connect. Great! |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 can you send me get_vod_categories ? share_stream is wip, i will ask you to test when it is ready to test. Normally reverse proxy does handle jumping back and forward. But it would be helpfull to send me called api endpoints with header and param information if there are other api endpoints which i did not implement. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 i have tested vod processing and it is working, i need get_vod_categories and at least one get_vod_info output to check the kodi output. But as i can see it is working, Maybe your filter is excluding all vods ? |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Happy New Year! |
Beta Was this translation helpful? Give feedback.
-
|
HI @jojo141185 happy new year, i sent you an email, with the latest branch it is working. I have used yur |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 when does this happen ? Did you configure m3u output ? If not the access over m3u api will not work. You need to define a m3u output. The rule is ->
|
Beta Was this translation helpful? Give feedback.
-
|
@euzu I have deleted the data folder for testing. Since neither the LiveTV nor the VOD playlists can be generated when I connect directly (without iptv-proxy), the error occurred because a client tried to start a LiveTV stream, but the m3u.db was not generated and available. I am now accessing cascaded via iptv-proxy again and everything seems to be working fine! |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Did a complete run tonight connected over iptv-proxy. Process panicked with error message: |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 i have pushed changes to the master which handle oversized nodes for bplustree. |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Yes, that solved the problem at first. The process has continued to run, but after renaming further errors occur. Some may be related to the docker problem mentioned here. |
Beta Was this translation helpful? Give feedback.
-
|
What I also just noticed is that I wanted to start a manual update only from the target pl1_SERIES. Nevertheless, both targets pl1_VOD and pl1_SERIES are updated.
I think this should not be the case. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 this should be fixed on master (you need to rebuild the frontend) |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Great, seems to work as intended! |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 i have send you an email. i have pushed changes to the frontend to fix the manual update problem. I have merged the branch to master. If you want to test, please use the master branch. |
Beta Was this translation helpful? Give feedback.
-
|
Did an update from master but the problem persits (data from the last run was not deleted). PS: If I find some time today, I will try to run a debug process to find out why there are problems with direct connect. |
Beta Was this translation helpful? Give feedback.
-
|
I run the test again with your provided Direct ... files, and there are no issues. There must be something which i dont have in my content files. |
Beta Was this translation helpful? Give feedback.
-
|
I think I have found the problem with the direct connection. As you already suspected, I had set a url filter on my local domain. I had overlooked it and forgot to adjust it :-( |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 indiviual schedule with target pushed to master Breaking change: multi scheduler config with optional targets. # sec min hour day of month month day of week year
schedules:
- schedule: "0 0 8 * * * *"
targets:
- vod_channels <---- executes only vod_channels target
- schedule: "0 0 10 * * * *"
targets:
- series_channels <---- executes only series_channels target
- schedule: "0 0 20 * * * *" <---- executes all targets |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Thank you, that feature looks promising! The problem occurs when I run it on a docker container with shared volumes. I think instead of rename, the file should be copied and then deleted or temp files placed in the same shared volume. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 I have pushed a fix for the rename functionality. I suggest closing this discussion and opening a new one if further discussion is needed. |
Beta Was this translation helpful? Give feedback.
-
|
@euzu Thanks, that solved the problem and the export works again. Thank you very much for the quick fix! Otherwise, the features discussed here seem to work perfectly and I will close the discussion here as you suggested. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 i have pushed changes to the master. unfortunately id dont have any id at that point, only a series name, i hope this helps. |
Beta Was this translation helpful? Give feedback.
-
|
Based on the current master, the errors are now completely gone. Explicitly checking whether series_info.episodes exists ensures that the code handles null or missing fields reasonably and avoids trying to access non-existent data. Brilliant, you had the right idea! |
Beta Was this translation helpful? Give feedback.
-
|
But this is somehow strange because series info without episodes ? It is junk. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it makes no sense, but it cannot be ruled out that the database is sometimes poorly maintained. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 WebUI has now a ScheduleEditor. Pushed to master. |
Beta Was this translation helpful? Give feedback.
-
|
@jojo141185 stats have now target information {
"stats": [
{
"inputs": [
{
"errors": 0,
"name": "xtream",
"processed": {
"channels": 38713,
"groups": 639
},
"raw": {
"channels": 235230,
"groups": 819
},
"took": "8 secs",
"type": "xtream"
}
],
"targets": [
{
"success": true,
"target": "m3u"
},
{
"success": true,
"target": "xtream"
}
]
}
]
} |
Beta Was this translation helpful? Give feedback.
-
Hello Euzu,
You have really created a fantastic M3U filter tool here. Well done!
But I have two suggestions for improvement:
The tmdb-id should generally be accessible via the info page using the xtream-api (field tmdb_id):
/player_api.php?username=&password=c&action=get_vod_info&vod_id=
If available, the file name should look like this according to Kodi's naming convention:
Blade Runner (2017) {tmdb=335984}.strm
I'm afraid the disadvantage is that you would have to parse the individual info pages (similar to series).
Depending on the size of the database, this can take a lot of time!
One question on this: Does the reading of series info actually only take place for the filtered content? Despite a very restricted filter, I notice very long processing times.
What do you think? Or is there a better way to integrate XTream codes directly into Kodi e.g. via an add-on to only access the data on demand.
Beta Was this translation helpful? Give feedback.
All reactions