Questions around "deshake"

[mgritsch]

Just started to woork with videopad and really love ist. It is fascinating how slender and performant but yet fully functional this piece of software is. One of the few cases where I paid the full pro licence price gladly without hesitation

One of the many reasons why I like it so much is the very well working deshake filter and the seamless integration of this plugin.

But thats also where my questions start:

 

To the developers of videopad:

• In the current download version of videopad, the deshake-plugin version is 2.4. The current version published by the author of deshake (http://www.guthspot.se/video/deshaker.htm) is 3.0 which has been improved a lot plus it is available as a 32 and 64 bit version. I tried to "update" by replacing the .vdf file that came along my videopad-installation, but that just didnt work. I assume that the integration of the vdf has changed in a way. When will there be an update on this filter? Any choice of 32 vs. 64 bit version?
  

 

To the users of deshake:

• One of my intended uses of deshake is for underwater footage. It works fine as long as there is enough light and some solid ground in the picture, but apparently has no whatsoever clue when the only thing it gets to 'see' is a big fish (slowly) swimming through the blue water. It also has big problems when the footage has been taken under low light conditions (although the video might still look pretty smooth) or in turbid water so that there are no crisp structures. Anybody experienced similar problems? Any clue as to which set of parameters could still work under those circumstances?
  

Any help greatly appreciated!

Martin

[Nationalsolo]

Hi

 

This is a copy of some guide data regarding deshaker that I collected quite some time ago. You may have already seen it but I'm posting for anybody that might be interested..

 

The Long Guide

 

This section describes all the settings in detail.

The default parameter settings are chosen to work pretty good under most circumstances. However, you can always get better results by adjusting some settings. For example, some complicated clips might require that you change some pass 1 settings, or you will get jerky video. Also, you'll need to change some pass 2 settings if you want to take care of the black borders.

Remember that all settings have a purpose, or they wouldn't be there. Most people I hear from seem to use the default settings or change very few of them. And many clips I've seen from other users are far from as good as they could have been. I know the settings may seem a little hard to understand and maybe a little too many, but if you want perfect looking video you should at least try.

To understand these settings it is necessary for you to understand how the filter works, so here comes a brief description.

How the Filter Works

During Pass 1 the filter tries to find the panning, rotation and zoom that, when applied to the current image, makes it look like the previous image (almost). It does this by taking blocks of pixels in the two images and finding the shift that makes them match as good as possible. First, to find the "big" motion, the images are reduced a lot in scale and the initial shift is calculated from the whole (scale-reduced) images. This shift is then fintuned in a scale twice as big and with four times as many blocks, and so on. Finally we have the block shifts (or motion vectors) in the full image. The filter then finds the best values for panning, rotation and zoom based on these motion vectors and those values are written to the log file.

At the start of Pass 2 the values in the log file are read and optimal camera motion is calculated based on the smoothness settings. The filter then transforms each image of the video clip according to this motion.

 

The Settings

Parameter

Description

 

Source pixel aspect

Tells the filter what pixel aspect ratio the source clip has. It's very important that you set this correctly, or the rotation (for example) won't be made correctly. You may enter any value if you can't find the preset you're looking for.

 

Video type

Selects between progressive video and interlaced video (for input and output). If the video is interlaced you must also determine if the video has the first or last field on the first line. In general, DV uses lower field first and all other formats (including HDV) use upper field first. You can easily check that your setting is correct by looking at the motion vectors for the two fields when there's motion in the clip. If the vectors point in opposite directions most of the time the setting is incorrect.

 

Camcorder has a rolling shutter Most modern camcorders (those with a CMOS sensor instead of a CCD sensor) don't capture the whole frames at once, but instead use a "rolling shutter". Here the lines of a frame are captured one after another at slightly different (although usually overlapping) times, which can cause some ugly effects. For example, vertical lines will get slanted when doing a fast horizontal pan, and when panning up/down everything will get squeezed and stretched.

For Deshaker, a rolling shutter means that the camcorder shake parameters (panning, rotation and zoom) aren't necessarily constant over the entire frame, but can change slightly from one line to the next. If you enable this setting, Deshaker will therefore allow these parameters to change linearly in the vertical direction during pass 1 (and you will see a range of values under the output video). This makes extracting these parameters more reliable and it also makes it possible for pass 2 to remove the rolling shutter distortions if you keep the setting enabled during that pass too. Although the actual camcorder shake isn't linear during the capture of a frame, it's usually a good enough approximation to make the distortions disappear almost completely. But I still strongly recommend using the internal camcorder stabilizer too.

If you have one of those compact CMOS cameras with bad (or no) internal stabilization, and the clip has very fast shakes (for example if the camera is mounted on a bike or similar), you probably won't be able to get a very good result from Deshaker, since the distortions will be far from linear in this case. Get a CCD camera capable of high shutter speeds for these kind of videos.

Different camcorder models may have different rolling shutter speeds (how fast the shutter rolls downwards). And a certain model could possibly even have different speeds under different conditions. To be able to handle all speeds properly, you can enter an amount from 0 to 100%. You can try which amount setting looks best, or you can measure it more precisely yourself using the following procedure:

Put the camcorder on a tripod (or something flat) and point it towards some vertical line (or bar) that covers the whole height of the frame. Now, using a fast shutter speed (to make the line sharp), record some video where you pan left and right (direction doesn't matter) at a pretty fast constant speed, as well as when it's completely still. Then, pick a still frame and a panning frame (where the vertical line is slanted). If you recorded interlaced video, the panning frame will contain two fields (and the line will appear twice), which is what you need. If you recorded progressive video, you'll need two consecutive frames instead. Then, find the horizontal position (x-pos) of the vertical line at 5 places, namely at St = top of still frame, Sb = bottom of still frame, Ft = top of first panning field/frame, Fb = bottom of first panning field/frame, Nt = top of next panning field/frame. Now, the rolling shutter amount should be calculated as: 100 * (Fb - Ft + St - Sb) / (Nt - Ft).

The default rolling shutter amount is 88%, which was measured from the Sony HDR-HC1(E) camcorder. (Deshaker versions before 2.4 had 88% fixed.)

If you measure the amount for another camcorder, please let me know so I can publish the values for other users. Here are the ones I've got so far:

 

Canon EOS 500D: 71%

Canon EOS 550D: 66%

Canon EOS 5D Mark II: Probably somewhere between 66% - 72%

Canon HV20, at 1440x1080 25p: 38%

Casio Exilim EX-F1 Pro, at 1280x720 30fps: 22%

Casio Exilim EX-F1 Pro, at 1920x1080 60fps: 72%

HD Hero, at 1280x720 60fps: 82%

iPhone 4: 97% (+/- 2)

(Note: Deshaker assumes that the rolling shutter rolls downwards over the frames. For this to happen, the iPhone must record video in landscape mode with the lens in the upper part of the phone. Otherwise you'll probably need to rotate the video before Deshaker, and rotate it back afterwards.)

Kodak Zi8: Somewhere between 80% - 100%

Oregon Scientific ATC5K: 100%

Panasonic GH1, at 1080p 24fps: 66%

Panasonic HDC-SD300 EG-K: 73%

Pentax K7, at 1536x1024 30fps: 75%

Sanyo Xacti HD1010, at 1280x720 60fps: 48%

Sanyo Xacti HD1010, at 1920x1080 30fps: 28%

Sony HDR-FX7: 86.4% (+/- 2.0)

Sony HDR-HC1: 88%

Sony HDR-HC1E: 88%

Sony HDR-V1U: 86.4% (+/- 2.0)

Sony HVR-A1E: 82%

Note that only rolling shutter distortions that are caused by moving/shaking the camcorder can be removed. Fast moving objects within the frame will sadly still look as strange as before.

You cannot use the deep analysis algorithm with rolling shutter video.

 

Log file

The name of the log file to write in pass 1 and read in pass 2.

 

Append to file

If checked, it appends to the log file instead of overwriting it. This is useful if you want to process certain frames again.

 

Video output

What kind of output to generate during pass 1. If you select Motion vectors you will see the previous image with arrows showing how to move the blocks to match the current source image. Only white arrows are used to calculate panning, rotation and zoom factor. For interlaced video both of the fields are shown, the first field above the last.

 

Block size

The width and height of the blocks used during the image matching.

 

Scale

The maximum scale to do the image matching on. Full is best but very slow.

 

Use pixels

How many pixels in the blocks to actually use when matching. All is best but very slow.

 

Color mode

The image matching can be made in color or grayscale. Grayscale is a little faster but the matching is worse. This setting has nothing to do with the color of the final output of pass 2.

 

Initial search range

In the most reduced scale this percentage gives the maximum shift while matching. Lower = slightly faster.

 

Differential search range

When going from a reduced scale to twice the size, the filter already has approximate shifts. The differential search range is how many pixels up, down, left and right from these approximate shifts to look when matching. Lower values make processing a lot faster but when objects move quickly over the frames, or if you have very fast/big rotations and/or zooms, matching can fail for some blocks.

 

Detect rotation/zoom

If Deshaker has problems finding a stable background to match on, and you know there is no notable rotation/zoom in the problematic frames, you can turn off detection of rotation and/or zoom (individually) for these frames (or the entire clip). This will make Deshaker assume there is no rotation/zoom and therefore not accept groups of motion vectors that indicate rotation/zoom. And this should make it easier for Deshaker to find the background. But as I said, there mustn't be any rotation/zoom in the analyzed frames for this to work. In fact, there might be problems even when the camera is moving forward quickly, because it looks rather similar to zooming.

 

Discard motion of blocks that have match value less than X

If the best match value found while matching a certain block is below X this block is discarded. The blocks in higher scales that depend on this block are also discarded. These discarded blocks show no motion vector in the output video of pass 1.

 

Discard motion of blocks that have 2nd best match larger than best minus X

If the second best match value found while matching a certain block is close to the best match value, this means that the best shift found isn't very reliable. It could just as well be this second shift that's the correct one. For example a blue sky has good match values for all shifts. Setting this parameter correctly discards those unreliable blocks. The blocks in higher scales that depend on these blocks are also discarded. These discarded blocks show no motion vector in the output video of pass 1.

 

Discard motion of blocks that move more than X pixels in wrong direction.

When calculating the best values for panning, rotation and zoom based on all motion vectors, the filter soon discovers that some blocks have motion that don't quite fit in with the "main motion". If a block has a motion vector that is more than X pixels off this main motion it is discarded in order to get more precise values. The motion vectors for these blocks are drawn in red in the output video of pass 1 and are not used in any way. Since the panning, rotation and zoom are all done in only 2 dimensions, it's possible to get a lot of these blocks that don't seem to fit in with the main motion when the effect of perspective gets big. This happens when the camera is set to wideangle and not just rotating (along any axis), but actually moving. Here you should use high X values (or maybe limit the matching area to the most distant parts). When zoomed in a lot or when the camera is stationary, the perspective effect is low and you should use low X values (even 1) to discard motion of objects moving even very slightly.

 

Discard motion of blocks that move more than X pixels (absolute motion)

Simply discards all blocks that have a motion vector that is longer than X pixels. Might be useful in some cases.

 

Remember discarded areas to next frame

When enabled, this feature makes Deshaker try to ignore approximately the same areas from one frame to the next. Deshaker will then become a lot more successful in ignoring moving objects. As long as they enter the scene rather slowly (by not covering too much of the background), Deshaker will usually be able to ignore those objects even if they eventually grow to cover most of the frame.

 

Deep analysis if less than X % of vectors are ok When set to 0% (which is the default value) this does nothing. To enable it, set it to 25% or something. Then, if less than 25% of the found vectors get white using the standard algorithm, this deep analysis algorithm kicks in. It does a much better job finding an object to lock on where the standard algorithm can be confused and lock on two separate objects at the same time, usually resulting very few white vectors and weird values. Use this in combination with the discard setting above to control how much the vectors in the object you want to lock on (usually the background) may vary. For example, the blocks on a moving person usually don't move "coherently" as the background usually does (if the camcorder is stationary). So by using a low value for the discard setting above (maybe < 1) the person would probably get very few white vectors and this algorithm would therefore hopefully lock on the background instead, even if it's a rather small part of the frame. This algorithm is very good, but also VERY slow (depending on other settings and how complicated the video is).

You cannot use the deep analysis algorithm with rolling shutter video.

 

Skip frame if less than X percent of all blocks are ok.

If less than X percent of all blocks are ok (i.e. the rest of the blocks either have no acceptable match or have been discarded), skip this frame (set panning and rotation to 0 and zoom scale to 1)

 

Ignore pixels outside area

Only area is used during matching. If you check Let area follow motion, the area will move along with the motion vectors between different frames. You can use it if you want to stabilize on a moving object instead of the background. Just don't expect this to work perfectly

 

Ignore pixels inside area

area is not used during matching.

 

Ignore pixels darker than X % brightness Most camcorders seem to use some kind of temporal smoothing internally to reduce noise. This means that the noise that is left will look very similar between two successive frames, and may cause Deshaker to incorrectly match on it. This problem gets worse when the camcorder is forced to used high gain (when filming in dark places) and it's also a lot worse in dark areas of the frame. By using this option you can ignore dark pixels during matching. What threshold to use depends on your camcorder and the gain used in the clip. The default 15% seems fine for my camcorder in daylight but may not be ideal for your camcorder/clip.

 

Same destination properties as source

If checked, the destination video will get the same size and pixel aspect ratio as the source and the two settings below will be ignored.

 

Destination pixel aspect

Tells the filter to generate video with a certain pixel aspect ratio. If you plan to watch the clip on a computer monitor you'll probably want to use square pixels. You may enter any value if you can't find the preset you're looking for.

 

Destination video size

The size of the output video you want from pass 2.

 

Generate "interlaced progressive" video Usually, Deshaker will generate video with the same video type (interlaced/progressive) as the source video. But if you enable this setting, it can turn interlaced video into progressive video with twice the framerate. Earlier versions of VirtualDub didn't allow for filters to change the frame rate, so the two progressive frames are interlaced into the same output frame. In order to get the true progressive video, you must separate the fields of this video afterwards. In later versions of VirtualDub you can do that by adding a Bob doubler filter after Deshaker, using the deinterlacing method None - alternate fields. You could also do it with AVISynth and its SeparateFields function (and a preceeding ComplementParity for "upper field first" video). You should never deinterlace this video in any other way than to simply separate the fields.

The difference between true interlaced output and this interlaced progressive output is that the video becomes twice as high (in order to fit two frames into one) and more importantly that the lower field (or rather "frame" in this case) isn't calculated with a one line offset as in interlaced video.

 

Resampling

The algorithm to use when calculating pixel colors from the source image. Note that none of these are very good when the destination size is a lot lower than the source. Instead, add a separate 2:1 reduction filter (or something) after the Deshaker to reduce scale a lot.

 

Edge compensation

A side-effect from stabilizing video is black or strange looking borders caused by moving the image around to compensate the shaking. To reduce or eliminate this effect you can zoom into the image. The following options do this in different ways:

None: No zoom is added.

Adaptive zoom: Determines, for each frame, how much zoom is needed to avoid borders. This additional zoom is then smoothed together with the stabilizing zoom changes using the zoom smoothness parameter, so the borders still become visible sometimes.

Adaptive zoom only: Same as adaptive zoom, but the zoom smoothness parameter is only applied to this adaptive zoom and not for stabilization. So, with this setting no smoothing will be made to the zooming made in the clip.

Fixed zoom: Deshaker will add a constant zoom to completely eliminate any borders. This zoom may be different for each scene of the clip.

Adaptive+fixed: First, adaptive zoom will be used. Then, a smaller fixed zoom will be added on top of this to completely eliminate any borders. This will usually result in less zoom than fixed zoom.

Adaptive+fixed only: First, adaptive zoom only will be used. Then, a smaller fixed zoom will be added on top of this to completely eliminate any borders.

If you're using any edge compensation (i.e. anything but None), your can limit the amount of zoom needed to avoid the borders by using lower values for Motion smoothness and especially Max. correction limits.

 

Previous and future frames to fill in borders

This is a kind of additional edge compensation. When the current frame doesn't contain any image data in the "real world area" we want in our output, it can search in past and future frames and use that image data instead. As long as that area is part of the object (or background) that was stabilized on in pass 1, this usually works perfectly. But when other objects move in these areas, it can look a bit strange. It's usually better than black anyway, so I strongly recommend using this feature. Just make sure you crop any borders from the source clip before Deshaker. Otherwise this won't work very well at all.

If the current frame is number 50, it searches frames in the following order to find the image data it needs: 50, 49, 51, 48, 52, 47, 53, 46, 54 etc. So for each pixel the closest frame (chronologically) is always used. You can set how many frames to store internally for this feature. More is always better but keep in mind that each frame needs up to 8 MB of memory (for full HD) and it can take a long time to search through all frames for image data.

Try using this feature on a clip with lots of panning, and zoom out by setting Extra zoom factor to something like 0.6. Then you'll get a panorama like effect. It's slow, but fun.

Soft borders If enabled, all relevant previous and future frames that Deshaker has stored in memory will be used to calculate the border (as opposed to just using the closest frame in time). For each pixel, it takes an average of the corresponding pixels in all these frames, weighted so that the frames closest to the current frame (in time) get higher importance.

This option will also allow you to make the transitions between a frame and the frame(s) beneath it smoother. The transition width value you supply is how far in from the edge of a frame the transition will start. The transition will be applied also to the current frame, so you'll actually lose some information in the edges of the main frame by using this option. But I'd say it's worth it. If you don't like it, just set the transition width to 0.

Enabling 'Soft borders' improves the appearance of the border quite a lot, especially when there are objects moving there, but it is also very slow when there are lots of border pixels.

 

Extrapolate colors into border If enabled, any black border pixel that couldn't be fixed in another way, is replaced by a color extrapolated from the edge of the known video pixels. This option is very slow when there are lots of border pixels, and I strongly recommend using 'Previous and future frames to fill in borders' in combination with this option.

Extra zoom factor

An additional zoom factor to apply to the video. This can for example be used together with fixed zoom edge compensation to zoom out just enough for the borders to become slightly visible but still remain hidden in the overscan area of the TV. Then you will see more of the video while still not seeing any borders on the TV. Or you can use it to zoom in slightly to get less borders. This setting won't cause another resampling, so using a little extra zoom to finetune your results is perfectly safe, quality wise. (Btw, all output pixels are always resampled directly from the original source pixels exactly once no matter what settings you use. So, no double resampling ever. This includes areas taken from previous or future frames.)

 

Motion smoothness

These values determine how smooth the motion will be. You can set a parameter to 0 to turn off the smoothing completely. This can be useful if you don't want to stabilize zoom for example. But you must use a rather high zoom smoothness if you use adaptive zoom as edge compensation, or the video will zoom in and out very fast to avoid the borders. The motion smoothness calculation is not time-based but rather based on frame count, so you'll probably want to use larger values for video with high number of frames per second. And in this case, the fps for interlaced video will count as twice as high, since each of its frames contains two "sub frames" (fields). If you enter -1 as smoothness you will get "infinite smoothness", meaning the camera will appear to be stationary at the position of the first frame that was processed in pass 1. This feature usually doesn't work perfectly, though.

 

Max. correction limits

To keep the corrections (that cause the black borders) small during fast pans (for example), you can limit the maximum corrections. Especially when using fixed zoom as edge compensation these settings keep the zoom from being too large. What these settings really do is automatically lower the motion smoothness values in certain parts of the video when it becomes necessary.

 

 

Things to Think About

Deshaker works best for video shot using high shutter speeds (at least 1/200 sec or so). Otherwise you might get motion blur without motion in the deshaked video. Also, the matching will work better.

Always try to make sure the white vectors end up on the background in the frames for best results, i.e. try to make Deshaker completely ignore objects moving in front of the background. If the white vectors jump around between different objects from frame to frame, you need to do something to get a good result. Either use the "ignore area" settings or something else. There are many things you can do. Experiment with the settings!

Don't forget to set the source video pixel aspect ratio. For example if your source video has resolution 720x480, 704x480 or 352x240, it's probably NTSC, and if it has 720x576, 704x576 or 352x288, it's probably PAL.

If there is no zooming in the video clip, use Adaptive zoom only as edge compensation or if you don't want any edge compensation, set zoom smoothness to 0. This is because the zoom stabilizing can't distinguish between zooming due to the camera moving towards something or if it's actually zooming. You may think it's the same thing, but it isn't. At least not for wideangle shots.

Often there's conflicting motion of large objects in the video clip. In this case the filter tries to "follow" the motion of the biggest object (usually the background). The setting that discards blocks that move more than X pixels in wrong direction can be used to some extent to control this. A low value makes it follow only the biggest object, while a larger value makes it follow the average motion of all objects (that aren't moving too much in the "wrong" direction). You can also use the image matching area to control which object to follow or try enabling the "deep analysis", which is designed specifically for this problem.

Video with a high degree of barrel or pincussion distortion will not look good when deshaked. If you get a lot of distortion in your video, try avoiding the extreme zoom positions of your camcorder. Extreme wide angles will unfortunately always result in a less than perfect looking stabilized video due to some geometrical effects that Deshaker doesn't deal with.

If you're a perfectionist (like me), always use the slowest settings. This makes it *really* slow!

 

Human Guidance

Sometimes it can get nearly impossible to automatically match two successive images correctly (or rather 'the way you want to'). This usually happens when the images have very little in common or when there's conflicting motion of large objects (see 'Things to Think About' above). In order to make a complete video clip look perfect, you might want to use different settings during pass 1 for different parts of the clip (or even single frames). You can do this in two ways:

You can use the Append to file option to continue writing to the same log file. You can then change the settings, step a few frames, change again, step some more etc. The frame numbers are written to the log file so you can even go back and redo some frames. If a frame exists more than once in the log file, the last one is used in pass 2.

You can edit the log file manually. The values on each line in the file are (from left to right) frame number, x- and y-panning (in pixels), rotation (in degrees) and zoom factor. When "rolling shutter" is enabled the four values for panning, rotation and zoom are given twice; first for the middle line and then for the first line of the frame. If a frame is the first frame of a new scene, the word 'new_scene' should be at the end of that line. You can delete lines that got completely wrong (and that you don't care to try to fix in a better way). Gaps in the file are treated as zero-panning, zero rotation and no scaling.

 

Stabilizing Interlaced Video

Deshaker can deshake interlaced video, but the resulting video will look slightly softer than for progressive video since the fields of interlaced video are resampled individually and they only have half the vertical resolution of a progressive frame. The resampled fields are then reinterlaced in the output video. There is really no sense in trying to detect non-moving pixels and keep them "un-deinterlaced" since you wouldn't be deshaking video shot on a tripod anyway, and without a tripod the pixels will almost always move slightly. (Deinterlacing with motion estimation in some way might give a better result but I'll probably not try implementing that anytime soon. And for certain reasons I don't like the idea of it.)

If you want, here's another way to stabilize interlaced video that might look better when some parts of the video are completely non-moving.

Use my smooth deinterlacer (or Xesdeeni's AVISynth port of it, or other advanced deinterlacers) to generate 50 or 60 fps full-sized video.

Feed it into Deshaker as progressive video (in two passes).

Run the resulting video through this little AVISynth script (that I got from Xesdeeni):

AVISource("deshaked.avi")

SeparateFields()

SelectEvery(4, 1, 2) (or SelectEvery(4, 0, 3) depending on the output field polarity)

Weave()

 

If you stop after the second step you'll have smooth 50 or 60 fps full sized video that will look great when played on a computer. That video can't be played on a TV however.

 

 

Nat

[mgritsch]

yes, I've seen this guide, however it is rather a technical description of the paramters and not really helpful in terms of "so what should I do now?"

I've played around a while but at the end gave up and resorted to a commercial software (mercalli) because it did a way better job on these files and is easier on the settings. thanks anyway.