Adding new quality metrics

[DonaldTsang]

• Status Report aizvorski/video-quality#8
  • VIF (sometimes called VIF-P or VIFP), Visual Information Fidelity
  • RECO, Relative Polar Edge Coherence
  • NIQE, Natural Image Quality Evaluator
  • MS-SSIM, MultiScale Structural Similarity Metric
  • 3SSIM, 3-Component Structural Similarity Metric
  • VQUAD-HD
  • VQM
  • UIQ, Universal Image Quality
  • MSSIM, Motion SSIM
• https://github.com/Rolinh/VQMT
  • MS-SSIM
  • VIFp
  • PSNR-HVS
  • PSNR-HVS-M

[DonaldTsang]

rolinh/VQMT#12

[slhck]

Good suggestions, and a good overview of what's out there. But this script is really just built for using ffmpeg to calculate the metrics, not calculate them itself.

If you would like ways to compute these metrics, you need to either:

• have ffmpeg implement them natively (preferred solution IMO)
• provide a library for ffmpeg (like libvmaf)
• add Python code to @aizvorski's repository or C code to @rolinh's repository

[DonaldTsang]

@slhck here is one of the problems of @aizvorski 's repos: The guy has MIA for 3 years.
In that case contacting @rolinh or ffmpeg would be much more doable.
(I would much prefer seeing these repos using Python, or wrapping libraries in Python).
What about OpenCV, do they handle video quality as well?

[slhck]

I agree that having a general Python interface would be useful for the community. Getting in touch with Alex and co-maintaining the existing scripts would be useful, since there's a lot already out there. Or you just fork the repo and continue developing them – I'd be happy to help.

OpenCV can be used for such tasks (see e.g. https://github.com/Telecommunication-Telemedia-Assessment/SITI/).

For this project though I am generally against implementing anything in addition.

[slhck]

PS: When/if forking, consider creating a new organization for it. Or I can create one in https://github.com/vqeg. This way it becomes easier having multiple people involved.

[DonaldTsang]

@slhck I would like to see changes happen for @aizvorski's repo, how should I go about and help the cause? Are social media and bountysosurce a good place to start?

[slhck]

Ultimately, if he's not responding to Pull Requests, there's no use. Your code will just be waiting for approval forever. I'd wait for a response, and if nothing happens, create a fork.

I have no experience with attracting contributions via social media and bountysource—these are niche topics, so we can be happy that there is some code out there at all! :) Of course, if you have money to spend … but for these metrics you need domain-specific knowledge, and I wouldn't trust a programmer without that knowledge to implement them properly. We don't have proper test cases either.

[DonaldTsang]

@slhck most academic papers on the matter does not have test cases or code for the public either. I dunno how to approach this, but a first move needs to be made.

[slhck]

That's true, and I agree.

I can't promise to find time for coding, but in terms of organization, I can help.

[DonaldTsang]

@slhck thanks

[aizvorski]

@slhck here is one of the problems of @aizvorski 's repos: The guy has MIA for 3 years.

@DonaldTsang I understand your frustration to some extent but I think you're simply wrong on the facts, as well as somewhat unfair. It is true that I haven't been actively involved in the video software field for a while. It is not true that I'm not supporting code that I wrote and maintain (at some level - remember, this is volunteer work). If there is anything in particular you need for work, you are of course welcome to write me a note and offer a bounty.

See:
aizvorski/video-quality#3
aizvorski/h264bitstream@34f3c58
etc

I don't see your pull requests to my repos. I also don't see you trying to get in touch with me and consistently not being able to.

In general, if I see a pull request which I can understand, and has a reasonable test case, I will review it and try to run it and then merge it within a fairly short time frame (<1 week). If it is not ready for prime time - as some of the stalled pull requests on h264bitstream - it may take me a long time to provide good feedback and/or to fix it up. YMMV.

[aizvorski]

@slhck "Ultimately, if he's not responding to Pull Requests, there's no use" - Quick score card:
Number of pull requests to video-metrics (ever): 1
Number of pull requests to video-metrics which had a test case: 0

Please do bring on the pull requests! Seriously, I wish there was more of an active community around this, but it seems to be mostly an academic field with not much industry involvement. I've ported some (now dated) academic code here, but there hasn't exactly been a flood of contributions. I think the community is - well, pretty much just us here. I'm open to working together in whatever format, but I don't have much time for it.

[slhck]

@slhck "Ultimately, if he's not responding to Pull Requests, there's no use" - Quick score card:
Number of pull requests to video-metrics (ever): 1
Number of pull requests to video-metrics which had a test case: 0

My apology if this sounded like I was implying you were indeed inactive. I was just outlining the scenario in which (hypothetical) contributions would not be accepted. My suggestion was to get in touch with you and see where we can go from there. Of course, if there are no actual contributions planned, there's no point in discussing maintainership at length.

I think the community is - well, pretty much just us here. I'm open to working together in whatever format, but I don't have much time for it.

Same here. Some of the metrics are quite “obscure”. These days most people are using PSNR, (MS-)SSIM, VIF, maybe VMAF.