现在增加了一个filter属性，所以可以很好和opencv结合。转一篇文章(http://blog.qt.io/blog/2015/03/20/introducing-video-filters-in-qt-multimedia/)：
Introducing video filters in Qt Multimedia
Published Friday March 20th, 2015
3 Comments
Posted in Graphics, Multimedia, OpenGL, Qt Quick
Qt Multimedia makes it very easy to get a video stream from the camera or a video file rendered as part of your application’s Qt Quick scene. What is more, its modularized backend and video node plugin system allows to provide hardware accelerated, zero copy solutions on platforms where such an option is available. All this is hidden from the applications when using Qt Quick elements like Camera, MediaPlayer and VideoOutput, which is great. But what if you want to do some additional filtering or computations on the video frames before they are presented? For example because you want to transform the frame, or compute something from it, on the GPU using OpenCL or CUDA. Or because you want to run some algorithms provided by OpenCV. Before Qt 5.5 there was no easy way to do this in combination with the familiar Qt Quick elements. With Qt 5.5 this is going to change: say hello to QAbstractVideoFilter.
QAbstractVideoFilter serves as a base class for classes that are exposed to the QML world and are instantiated from there. They are then associated with a VideoOutput element. From that point on, every video frame the VideoOutput receives is run through the filter first. The filter can provide a new video frame, which is used in place of the original, calculate some results or both. The results of the computation are exposed to QML as arbitrary data structures and can be utilized from Javascript. For example, an OpenCV-based object detection algorithm can generate a list of rectangles that is exposed to QML. The corresponding Javascript code can then position some Rectangle elements at the indicated locations.
Let’s see some code
import QtQuick 2.3
import QtMultimedia 5.5
import my.cool.stuff 1.0
Item {
Camera {
id: camera
}
VideoOutput {
source: camera
anchors.fill: parent
filters: [ faceRecognitionFilter ]
}
FaceRecognizer {
id: faceRecognitionFilter
property real scaleFactor: 1.1 // Define properties either in QML or in C++. Can be animated too.
onFinished: {
console.log("Found " + result.rects.length + " faces");
... // do something with the rectangle list
}
}
}
The new filters property of VideoOutput allows to associate one or more QAbstractVideoFilter instances with it. These are then invoked in order for every incoming video frame.
The outline of the C++ implementation is like this:
QVideoFilterRunnable *FaceRecogFilter::createFilterRunnable()
{
return new FaceRecogFilterRunnable(this);
}
...
QVideoFrame FaceRecogFilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags)
{
// Convert the input into a suitable OpenCV image format, then run e.g. cv::CascadeClassifier,
// and finally store the list of rectangles into a QObject exposing a 'rects' property.
...
emit m_filter->finished(result);
return *input;
}
...
int main(..)
{
...
qmlRegisterType
...
}
Here our filter implementation simply passes the input video frame through, while generating a list of rectangles. This can then be examined from QML, in the finished signal handler. Simple and flexible.
While the registration of our custom filter happens from the main() function in the example, the filter can also be provided from QML extension plugins, independently from the application.
The QAbstractVideoFilter – QVideoFilterRunnable split mirrors the approach with QQuickItem – QSGNode. This is essential in order to support threaded rendering: when the Qt Quick scenegraph is using its threaded render loop, all rendering (the OpenGL operations) happen on a dedicated thread. This includes the filtering operations too. Therefore we have to ensure that the graphics and compute resources live and are only accessed on the render thread. A QVideoFilterRunnable always lives on the render thread and all its functions are guaranteed to be invoked on that thread, with the Qt Quick scenegraph’s OpenGL context bound. This makes creating filters relying on GPU compute APIs easy and painless, even when OpenGL interop is involved.
GPU compute and OpenGL interop
All this is very powerful when it comes to avoiding copies of the pixel data and utilizing the GPU as much as possible. The output video frame can be in any supported format and can differ from the input frame, for instance a GPU-accelerated filter can upload the image data received from the camera into an OpenGL texture, perform operations on that (using OpenCL – OpenGL interop for example) and provide the resulting OpenGL texture as its output. This means that after the initial texture upload, which is naturally in place even when not using any QAbstractVideoFilter at all, everything happens on the GPU. When doing video playback, the situation is even better on some platforms: in case the input is already an OpenGL texture, D3D texture, EGLImage or similar, we can potentially perform everything on the GPU without any readbacks or copies.
The OpenCL-based example that comes with Qt Multimedia demonstrates this well. Shown below running on OS X, the input frames from the video already contain OpenGL textures. All we need to do is to use OpenCL’s GL interop to get a CL image object. The output image object is also based on a GL texture, allowing us to pass it to VideoOutput and the Qt Quick scenegraph as-is.
