Dominik Goldschmidt develops multimedia application since 2001, with a focus on kiosk apps for museum and public spaces. He has been living in Toulouse, France for 5 years now, where he currently works as Senior Developer and Technical Director at Anagram Audiovisuel. His preferred languages are AS3 and C#.View all articles by Dominik GOLDSCHMIDT
This tutorial describes how to create a chroma key effect - removing a fix color (blue screen or green screen) from a video source in real time - using Adobes Pixel Bender technology, and how to use this effect in a simple demonstration program.
Adobe Pixel Bender Tool Kit
Flash Develop / Flex SDK or any other Action Script 3 IDE.
compatible graphic card
Programming the filter
Adobe Pixel Bender Technology is the gateway to using hardware acceleration in Flash. It lets you program small pieces of code (shaders) which are directly executed by the graphic card's hardware. To learn more about Pixel Bender Technology go to http://www.adobe.com/devnet/pixelbender.html
Our first step consist in creating the pixel bender shader. We open the Pixel Bender Toolkit program and create a new kernel filter (file -> new kernel filter). To be able to see what the filter is doing we also need to load an image (file-> load image).
The basic new filter already has some preconfigured variables and function content. There is an input image "src", which corresponds in Flash to the bitmap data of the object the shader is being applied on. This input is of type image4 which means that the pixels in the image have 4 channels (red, green, blue and alpha). There is an output pixel called dst also in 4 channel format pixel4. And there is the "main" function evaluate pixel being called for every pixel of the output image.
We now need a few parameters to be able to configure the shader from within Flash. We declare three float parameters for the key color called r, g and b. In order to set the default key color to green we define r and b defaultValue to float(0) and g defaultValue to 1. Further we declare a float parameter sens which will control the sensibility/wideness of the color and a soften float parameter which defines the falloff of the key.
Then we need to extend the evaluatePixel function. The first step consists in calculating the float distance between the source color and the key color vectors in RGB space. We then set the alpha channel to zero if the distance is smaller than sens.If this is not the case we find out if the distance is lies in the falloff range (sens + soften). We calculate a new softened alpha value. But we only apply this alpha value if it is smaller than the original alpha value. This is important when using multiple instances of this shader on the same object, because we don't want to modify areas which have already been keyed by an upstream shader instance.
That's it for the code part in Pixel Bender.
The last step is exporting the shader into byte code which can be loaded in AS3. Go to "File->Export Kernel Filter for Flash Player..." and save the file.
Using the filter
Now we need to create a simple flash movie to test the new shader. My
preferred tool is Flash Develop but you can accomplish this also in
Flash CS4 or whatever tool you prefer.
We create a base class derived from the Sprite class. We add a "Video" member, a "Camera" member. We have to get a camera object by calling Camera.getCamera(). We attach this camera to the video object and finally add the video object to the stage. In order to see something more than a poor thumbnail image we set the resolution to something higer than the default 160 * 120 pixels.
There are two methods for using pixel bender shaders in Flash Player: embedding or loading at runtime. In this tutorial we will be using the embed method as it requires less code. We create an "embed"-tag with a source attribute pointing to the byte code shader file. The class member following the embed-tag needs to be of type "Class". To load the shader we simply create a new Shader object with a new instance of our embedded class as constructor parameter. The new shader has to be pushed into an array instance. This array has to be 'applied' to the filter property of our video object. If we don't want to use the default parameters of the shader object we need to access these parameters prior to applying the shader to the video object. The shader object has a member called 'data' (of type ShaderData) which contains a ShaderParameter object for each parameter of the filter. Each parameter has a value property containing an array of values. In the case of a float parameter this array has a length of 1. For example to change the red value we have to access shader.data.r.value. Just note that only array parameters have more than one value, the problem being that the Flash Player Pixel Bender API does not support those array types at the moment.
Running the filter
That's it. The example code adds some functionality which lets you choose multiple key colors by clicking on the webcam image. Every time you click a new chroma key shader is added to the filter array.