These Days Labs

Guest blog post by Stefan Van Camp from Ventigrate.

While I was working on at project here at These Days I ran into an opportunity to use one of the new Silverlight 3 features, custom built pixel shaders! The specific reason why we needed pixel shaders was because These Days wanted to use Alpha Channel encoded video with Silverlight. Of course, since Windows Media Video (.wmv) does not support alpha channels, the media stream used in the project would need to use pixel shader in order to filter out a predefined color.

It soon became clear to me that there was no simple answer to the question “Which pixel shader will give the best result?”. Some people claimed the Synergy effect provided the best results, other people referred to the shader from the WPF effects library, some even wrote their own shader… After having tried a couple of different methods I wasn’t completely pleased with any of the results they gave me. And therefore, since I had nowhere else to turn, I solicited Microsoft for help via Katrien De Graeve – Microsoft developer evangelist.

Microsoft’s explanation for the poor performance of the pixel shaders was that the shader’s calculations were based on the RGB values of the color that needed to be filtered out. They suggested using the HSV values of the color. This technique proved to be a significant improvement over the other shaders, but it still was not perfect. Green lines on the sides were still showing and wherever the video had a motion blur effect, the background color wasn’t filtered out.

Obviously, the first part of an acceptable solution was removing the motion blur effect from the movies (which were rendered 3D animations with effects from After Effects). Once we had a new render of the movie, we retested with the new HSV shader Microsoft had delivered and immediately noticed the decrease in green matte. Now we had only the green around the subject edges to worry about.

Because pixel shaders perform their ’shading’ by calculating a given color value to another value pixel by pixel, I got the idea of increasing the number of pixels that the movie would feed to the shader as a logical consequence. Since Expression Encoder didn’t seem to provide any means of increasing the amount of pixels in the form of changing a “dpi” property or something, I decided to encode the source movie at twice the dimensions of it’s original size. In my test case my movie was 856×806 big, I encoded it at 1600×1432 to make at least the width of the movie a nice round number. When I used this movie in my Silverlight project using the original dimensions of the movie (otherwise encoding to increase the scale had no point), the results were more than satisfying. The green contours where almost all gone, only people who actually knew to look for green edges would occasionally be able to spot one. Taken into account that most people don’t hug their screens and look at the movie over and over again, this technique proved to deliver a satisfying result that can actively be used in Silverlight applications using the correct shader.

I am, however, aware that at a certain point in the future, some bright mind will think up a shader that’s absolutely perfect. But since mathematics and creating shaders aren’t really my cup of tea, I will leave that to someone else. In the meantime, I believe we’ve come up with an acceptable temporary solution which will satisfy most of your clients.

One remark though: exaggerating the dimensions will have a negative impact on the playback performance in Silverlight. Depending on your system’s specifications you will run into performance issues sooner or later. The best method to find the correct dimensions for your target audience is trial and error I’m afraid. Make sure to test your application on a less competent machine.

On a side note, I would like to point out one bug I encountered while applying these shader effects to the media elements. Most of the time there will be a thin line surrounding the element. I don’t know whether this is caused by a bug in the shader or if the effect has an adverse reaction on the media element. In any case, you can work around this problem by simply setting the clip property of the media element to remove those ugly lines.

Resources:

• Demo page
• Sources of demo page