Compression project >> Video Area Home

Project, ideas: Dr. Dmitriy Vatolin
Implementation: Oleg Petrov

List of the Frequently Asked Questions

• [0] What is this page about?
• [1] What is this tool for?
• [2] What does "subjective video quality" mean?
• [3] How should I measure "subjective video quality"?
• [4] Which subjective testing methodology is the most popular?
• [5] What "task file" is?
• [6] What video formats are supported in this tool?
• [7] How expert compares video?
• [8] What is stored in file with results of an expert?
• [9] "Task file" contains different versions of a single video sequence. But usually a number of different sequences are used for the comparison; how can I handle it?
• [10] Ok, experts went through task; how can I count average results?
• [11] What is the range of marks?

Answers to Frequently Asked Questions

Q: What is this page about?

A: This page is a list of Frequently Asked Questions about MSU Perceptual Video Quality tool.

Q: What is this tool for?

A: This tool aids to make subjective video quality evaluation.

Q: What does "subjective video quality" mean?

A: You can measure video quality using a mathematical formula, like PSNR or more complex VQM and SSIM. These methods are implemented in our MSU Video Measurement tool. It is "objective video quality". But results of objective measurements do not always correlate with subjective impressions of a person, so the only way to predict users' opinion is to ask them! Video quality marks that are given by experts are named "subjective video quality".

Q: How should I measure "subjective video quality"?

A: If you want to get reliable results you should follow recommendations on subjective assessments (ITU-T BT.500), but simplified actions are:

1. Choose video sequences for testing (they are often named SRC).
2. Choose settings of systems that you want to compare (often named HRC).
3. Choose test methodology (how sequences are presented to experts and how their opinion is collected).
4. Invite sufficient number of experts (not less than 15 are recommended).
5. Calculate average marks for each HRC based on their opinion.

Q: Which subjective testing methodology is the most popular?

A: DSCQS type 2 is used quite often (for instance, "Subjective Quality Assessment of The Emerging AVC/H.264 Coding Standard"), SAMVIQ ("Subjective Quality of Internet Video Codecs", our "Subjective Comparison of Video Codecs") and DSIS.

Q: What "task file" is?

A: "Task file" is created by "MSU PVQ - task manager". Task is a collection of video files + information about how they will be shown to experts. All video files in task are supposed to be different versions of one video sequence. In some test methods you must choose "task reference"; it is supposed to be unimpaired (uncompressed) version of the sequence.

Q: What video formats are supported in this tool?

A: This tool supports .avi format, including all sorts of compressed video files, and .avs (AviSynth scripts). Warning: to play video files, you should have all appropriate codecs installed!
Through AviSynth you can use lots of other video formats. For more information go to AviSynth page.

Q: How expert compares video?

A: Expert runs "MSU perceptual video quality player", inputs his or her name and opens task file (you can simplify it; see following question). When expert finishes task, subfolder with name of task is created in folder that contains task file. Results from all experts are stored there. Results for particular expert are stored in file "expert_name".csv.

Q: What is stored in file with results of an expert?

A: Results for expert are stored in a following format:

task type,	[string with type of task]
average framerate,	[0|1]	//is "average framerate" option enabled
pause allowed,	[0|1]
rewind allowed,	[0|1]
one to each,	[0|1]
number of tests,	[number of tests]
number of videos,	[number of videos]
reference video,	[reference video name]	//only if task is "one to each"
video,	mark
[video name],	[mark]	//meaning of mark depends on test methodology
...,	...
screen resolution,	[resolution values]
video,	decompressor
[video name],	[decompressor name]
...,	...
time of assessment,	[time]

Q: "Task file" contains different versions of a single video sequence. But usually a number of different sequences are used for the comparison; how can I handle it?

A: You can create .tsk file for each sequence that you use for comparison. Then you can create .bat file with following text:

"MSU perceptual video quality player.exe" "c:\tasks\task1.tsk" "c:\tasks\task2.tsk" ...

When expert runs this file, he ought to go through all this tasks (he doesn't have to open them).

Q: Ok, experts went through task; how can I count average results?

A: Results for all experts are stored in subfolder with name of task in folder that contains task file. To count average results, open task file and press button "count results". Results will be saved in file "average mark.csv" in folder with experts' results.

Q: What is the range of marks?

A: In file with results of the particular expert, range of marks depends on the test methodology. In average results, all marks are from 0 to 10, the higher the better.

Download

See also MSU Video Quality Metric

MSU Video Quality Metrics
• MSU Video Quality Metric - program for objective comparison
  • Features
  • Setting & screenshots
  • FAQ
  • Metrics info
  • PRO-version
  • Download & purchase
  • Plugins & Plugin SDK
    • MSU Brightness Flicking Metric
    • MSU Brightness Independent PSNR
    • MSU Drop Frame Metric
    • MSU Noise Estimation Metric
    • MSU Scene Change Detector
• MSU Human Perceptual Quality Metric - several metrics for exact visual tests
  • Implemented metrics short info:
    ITU-R BT.500-11: DSIS, DSCQS I & II, SCACJ; EBU: SAMVIQ; MSUCQE
  • MSU PVQ Metric Tool FAQ
  • MSU Perceptual Video Quality Tool PDF

Other resources

Video resources:

Public MSU video filters Here are available VirtualDub and AviSynth filters. Commonly we develop a whole family of some kind of a filter. Generally there are also versions optimized for PC and hardware implementations (ASIC/FPGA/DSP). These optimized versions can be licensed to companies. Please contact us for details via video(at)graphics.cs.msu_ru. MSU Cartoon Restore (New!) MSU Cartoon Restore (New!) MSU Noise Estimation (New!) MSU Frame Rate Conversion (New!) MSU Image Restoration (New!) MSU Denoising (Top!) MSU Old Cinema MSU Deblocking (Top!) MSU Smart Brightness and Contrast (Top!) MSU Smart Sharpen (Top!) MSU Noise generation MSU Noise estimation MSU Motion Estimation Information MSU Subtitles removal MSU Logo removal (Top!) MSU Deflicker (Top!) MSU Field Shift Fixer AviSynth plug-in MSU StegoVideo MSU Cartoonizer (Top!) MSU SmartDeblocking (Top!) MSU Color Enhancement MSU Old Color Restoration MSU filters FAQ (Read me!) MSU filters statistics	MSU filters for companies We are working with Intel, Samsung, RealNetworks and other companies on adapting our filters other video processing algorithms for specific video streams, applications and hardware like TV-sets, graphics cards, etc. Some of such projects are non-exclusive. Also we have internal researches. Please let us know via video(at)graphics.cs.msu_ru if you are interested in acquiring a license for such filters or making a custom R&D project on video processing, compression, computer vision. Automatic Objects Segmentation (New!) Semiautomatic Objects Segmentation New Frame Rate Conversion (New!) New Deinterlacer (New!) MSU-Samsung Deinterlacing Project Digital TV Signal Enhancement Old Film Recovery Tuner TV Restore Panorama Video2Photo SuperResolution SuperPrecision MSU-Samsung image and video resampling MSU-Samsung Frame Rate Conversion Motion Phase filter Deshaker (video stabilization) Film Grain/Degrain filter Deblurring filter Video Content Search
Codecs comparisons Objective and subjective quality evaluation tests for video and image codecs 4-th MPEG4-AVC/H.264 Comparison (New!) Lossless Video Codecs Comparison 2007 (New!) 3-rd MPEG4-AVC/H.264 Comparison HD Photo and JPEG 2000 Comparison MPEG-2 Decoders Crash Test (Top!) Subjective Compar. of 4 Modern Codecs (Top!) 2-nd MPEG-4 AVC/H.264 Comparison (Top!) FAQ of 2-nd H.264 Comparison 1-st MPEG-4 AVC/H.264 Codecs Comparison FAQ of 1-st H.264 Comparison JPEG 2000 Image Codecs Comparison (Top!) Lossless Video Codecs Comparison 2004 MPEG-4 SP/ASP Codecs Comparison (Top!) Video codecs comparison (old) Ext. link: x264 parameters efficiency comparison	Video quality metrics Programs with different objective and subjective video quality metrics implementation MSU Video Quality Metric - objective metrics for codecs and filters comparison (Top!) Implemented metrics short info: PSNR, Delta, MSAD, MSE, SSIM, VQM, MSU Blurring, MSU Blocking MSU Brightness Flicking Metric MSU Brightness Independent PSNR MSU Drop Frame Metric MSU Noise Estimation Metric MSU VQM Metric Tool FAQ MSU Human Perceptual Quality Metric - several metrics for exact visual tests (Top!) Implemented metrics short info: ITU-R BT.500-11: DSIS, DSCQS I & II, SCACJ; EBU: SAMVIQ; MSUCQE MSU PVQ Metric Tool FAQ
Video codecs projects Different research and development projects on video codecs MSU Lossless Video Codec (Top!) MSU Screen Capture Lossless Codec (Top!) MSU MPEG-2 Video Codec x264 Codec Improvement	Other Other information Crazy gallery (filters screams :) License for commercial usage of MSU VideoGroup Public Software (please be careful: some soft like metrics has another license!)

Bookmark this page:   Add to Del.icio.us     Digg It     reddit
 

Last updated: 07-December-2007

---

Search (Russian): Server size: 7489 files, 921Mb (Server statistics)

Project updated by
Server Team and MSU Video Group

Project sponsored by YUVsoft Corp.

Project supported by MSU Graphics & Media Lab