OT: smart deinterlace

[mattias]

hey there all fcp users. i just managed to tanslate my "smart deinterlacer" (no interpolation in areas without motion) to fxscript and it now runs natively in fcp. anybody interested in beta testing it, just let me know...

/matt

[MovieStuff]

Hi, Matt!

Isn't this the same as the "flicker removal" found in Premier? It only affects areas where there is motion artifacts between fields; areas of no motion are unaffected. Maybe I'm not understanding what you're talking about. Can you fill me in?

Roger

[mattias]

afaik "flicker removal" in premiere completetly interpolates one of the fields from the other, reducing interlace flicker over the entire image, with the drawback that your vertical resolution is reduced by a factor two. in any case, this is how the built in deinterlacer in fcp works.

my filter only reduces flicker (and comb artifacts) in motion areas, where flicker isn't visible anyway. so, do not use my filter to reduce flicker, but do use it for web and vcd creation, as well as part of your "film look" process. you will notice the higher resolution and lesser artifacts over the built in deinterlace filters in most nle's.

/matt

[This message has been edited by mattias (edited September 20, 2001).]

[mattias]

looked it up. flicker removal in premiere performs a blending of the two fields, which potentially doesn't remove as much detail, but can cause severe artifacts near moving edges, and does in fact reduce the resolution quite a lot.

/matt

[MovieStuff]

Yes, I did some experimenting on my own, since I use the flicker removal all the time and I didn't remember it removing one field. The "always deinterlace" function will reduce the vertical resolution by half, for sure, because it DOES remove one field. However, I use the flicker removal quite often when generating titles or other footage for film restoration on video and I don't seem to notice any resolution loss at all since it keeps both fields and only affects areas of motion. I also don't notice any artifacts around the motion areas other than the typical blurring of the areas of motion which looks pretty natural to me.

Still I would like to see some examples of your process to better understand what you're doing that the flicker removal on Premier won't do.

Roger

[mattias]

ah, what i do (and all the other adaptive/smart/motion compensating deinterlacers out there do) that the flicker removal doesn't:

i don't do anything at all with static areas, which is the real point of the filter. the usual deinterlacing method of blending fields causes a slight vertical blur and interpolation reduces detail.

i do whatever you like with the moving areas: currently you can decide to throw away the odd field, or the even field, or blend them. since the resolution in motion areas is lower due to motion blur, and because it's hard to judge resolution of moving areas, it's generally safe to throw one field away.

and it was the blurring i meant when i said artefacts. you might like them, but they are indeed artefacts and they really aren't supposed to be there from a technical point of view. personally, i think they look more like a double exposure than like motion blur. in any case, you can still use the blend function if you want to. the point of my filter is to avoid touching image data that doesn't have to be touched, i.e. areas without motion. what to do about the motion areas is up to you.

if you don't have fcp, i'll post a few images here soon...

/matt

[mattias]

tech description:

i apply a horizontal line detecting kernel to the image, which gives me a map of all motion areas plus all horizontal edges. i then apply an edge detecing kernel to the image and subtract this result from the first. the result is a map of all moving areas in the frame, which i then use as my alpha channel when blending the original image with a deinterlaced version of it (or a field blended one). the next step will be to use a kernel to detect diagonal edges as well, so i can improve the interpolation and avoid introducing aliasing when reducing the resolution. working on that right now.

/matt

[MovieStuff]

Hi Mattias!

Let me see if I understand how this works:

In essence, your method delinterlaces only the areas of movement, then creates an alpha channel to allow just that deinterlaced area to appear in front of the remaining non-deinterlaced frame?

Assuming this is the case, is the area soft edged or hard edged? You said you were still working on some of the aliasing issues. Just curious how you would handle, say, someone's arm moving rapidly upward. If the system works like I think it does, which is pretty clever, the alpha channel can't just start and stop at the arm's edge, since there would still be the "other" arm from the previous field peeking out from underneath the deinterlaced overlay of the "new arm". Or does the system you came up with deinterlace an area surrounding the moving object to effectively cover any "trailing" of the moving object from the previous field? If so, then wouldn't there be a "halo" effect of lower resolution on the background image surrounding the moving object?

Roger

[mattias]

> In essence, your method delinterlaces only the areas of movement, then creates an alpha channel to allow just that deinterlaced area to appear in front of the remaining non-deinterlaced frame?

correct.

> does the system you came up with deinterlace an area surrounding the moving object to effectively cover any "trailing" of the moving object from the previous field?

exactly. there's a slider called "motion sensitivity" that you can tweak until the motion is entirely covered by the deinterlaced area.

> If so, then wouldn't there be a "halo" effect of lower resolution on the background image surrounding the moving object?

correct again, even though the matte isn't completely hard edged, this can happen. it's hardly noticeable though, and tweaking the sliders for every clip where you can see it helps. this is really the part where i need beta testers. i want to make sure the plugin can handle as many situations as possible. perhaps i should add an option blurring the alpha channel? i'll look into it.

another option might be to do the motion estimation on a slightly blurred version of the source frame, to avoid interpreting noise and dropouts as motion and to create a softer matte. i'll look into that as well.

/matt

[MovieStuff]

Clever, Mattias, very clever.

Here's an idea: Perhaps you need three layers instead of two. The top layer is the object in motion that is deinterlaced and isolated on its own alpha channel with NO surrounding "halo" to cover anything else. It would be form fitting to the object. The next layer down would be another alpha channel in the shape of the same object from the PREVIOUS field position (the one you are tying to cover), only this would be an inverted selection that revealed background material from the next or previous field.

In other words, instead of having a "halo" of deinterlaced material big enough to cover the previous object position of the previous field, you would use interlaced background material from the NEXT field that was "cut out" to fit the shape of the object you are trying to cover. That way, only the object in motion is deinterlaced and ends at the object edge. The previous field position would be covered, not by a "halo" surrounding the top layer object, but by a form fitting matte containing the appropriate BG material still interlaced.

I would think you could use the same tracking software you've developed and make two passes. On the first pass, it would isolate the object in the even fields and cut an alpha "hole" that would be filled with material from the next field (or previous field; same difference). Then, on the second pass, the software would do its job as usual, isolating the remaining moving object and essentially "pasting" it, via an alpha channel, onto the previously created "clean" layer where the object has been "removed" from its previous field position by covering it up with form fitting BG material from an adjacent field.

Does this make sense?

Roger