GSoC Python Tutorials

GSoC Python Tutorials

removed white spaces

removed blank lines at EOF

removed duplicate labels

doc/py_tutorials/py_feature2d/py_brief/py_brief.rst

@@ -0,0 +1,63 @@
+.. _BRIEF:
+
+
+BRIEF (Binary Robust Independent Elementary Features)
+***********************************************************
+
+Goal
+=======
+In this chapter
+    * We will see the basics of BRIEF algorithm
+
+
+Theory
+=============
+
+We know SIFT uses 128-dim vector for descriptors. Since it is using floating point numbers, it takes basically 512 bytes. Similarly SURF also takes minimum of 256 bytes (for 64-dim). Creating such a vector for thousands of features takes a lot of memory which are not feasible for resouce-constraint applications especially for embedded systems. Larger the memory, longer the time it takes for matching.
+
+But all these dimensions may not be needed for actual matching. We can compress it using several methods like PCA, LDA etc. Even other methods like hashing using LSH (Locality Sensitive Hashing) is used to convert these SIFT descriptors in floating point numbers to binary strings. These binary strings are used to match features using Hamming distance. This provides better speed-up because finding hamming distance is just applying XOR and bit count, which are very fast in modern CPUs with SSE instructions. But here, we need to find the descriptors first, then only we can apply hashing, which doesn't solve our initial problem on memory.
+
+BRIEF comes into picture at this moment. It provides a shortcut to find the binary strings directly without finding descriptors. It takes smoothened image patch and selects a set of :math:`n_d` (x,y) location pairs in an unique way (explained in paper). Then some pixel intensity comparisons are done on these location pairs. For eg, let first location pairs be :math:`p` and :math:`q`. If :math:`I(p) < I(q)`, then its result is 1, else it is 0. This is applied for all the :math:`n_d` location pairs to get a :math:`n_d`-dimensional bitstring.
+
+This :math:`n_d` can be 128, 256 or 512. OpenCV supports all of these, but by default, it would be 256 (OpenCV represents it in bytes. So the values will be 16, 32 and 64). So once you get this, you can use Hamming Distance to match these descriptors.
+
+One important point is that BRIEF is a feature descriptor, it doesn't provide any method to find the features. So you will have to use any other feature detectors like SIFT, SURF etc. The paper recommends to use CenSurE which is a fast detector and BRIEF works even slightly better for CenSurE points than for SURF points.
+
+In short, BRIEF is a faster method feature descriptor calculation and matching. It also provides high recognition rate unless there is large in-plane rotation.
+
+BRIEF in OpenCV
+=====================
+
+Below code shows the computation of BRIEF descriptors with the help of CenSurE detector. (CenSurE detector is called STAR detector in OpenCV)
+::
+
+    import numpy as np
+    import cv2
+    from matplotlib import pyplot as plt
+
+    img = cv2.imread('simple.jpg',0)
+
+    # Initiate STAR detector
+    star = cv2.FeatureDetector_create("STAR")
+
+    # Initiate BRIEF extractor
+    brief = cv2.DescriptorExtractor_create("BRIEF")
+
+    # find the keypoints with STAR
+    kp = star.detect(img,None)
+
+    # compute the descriptors with BRIEF
+    kp, des = brief.compute(img, kp)
+
+    print brief.getInt('bytes')
+    print des.shape
+
+The function ``brief.getInt('bytes')`` gives the :math:`n_d` size used in bytes. By default it is 32. Next one is matching, which will be done in another chapter.
+
+
+Additional Resources
+==========================
+
+#. Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua, "BRIEF: Binary Robust Independent Elementary Features", 11th European Conference on Computer Vision (ECCV), Heraklion, Crete. LNCS Springer, September 2010.
+
+#. LSH (Locality Sensitive Hasing) at wikipedia.

doc/py_tutorials/py_feature2d/py_fast/py_fast.rst

@@ -0,0 +1,136 @@
+.. _FAST:
+
+FAST Algorithm for Corner Detection
+*************************************
+
+Goal
+=======
+
+In this chapter,
+    * We will understand the basics of FAST algorithm
+    * We will find corners using OpenCV functionalities for FAST algorithm.
+
+
+Theory
+=========
+
+We saw several feature detectors and many of them are really good. But when looking from a real-time application point of view, they are not fast enough. One best example would be SLAM (Simultaneous Localization and Mapping) mobile robot which have limited computational resources.
+
+As a solution to this, FAST (Features from Accelerated Segment Test) algorithm was proposed by Edward Rosten and Tom Drummond in their paper "Machine learning for high-speed corner detection" in 2006 (Later revised it in 2010). A basic summary of the algorithm is presented below. Refer original paper for more details (All the images are taken from original paper).
+
+
+Feature Detection using FAST
+------------------------------
+
+1. Select a pixel :math:`p` in the image which is to be identified as an interest point or not. Let its intensity be :math:`I_p`.
+2. Select appropriate threshold value :math:`t`.
+3. Consider a circle of 16 pixels around the pixel under test. (See the image below)
+
+    .. image:: images/fast_speedtest.jpg
+        :alt: A corner in the image
+        :align: center
+
+4. Now the pixel :math:`p` is a corner if there exists a set of :math:`n` contiguous pixels in the circle (of 16 pixels) which are all brighter than :math:`I_p + t`, or all darker than :math:`I_p − t`. (Shown as white dash lines in the above image). :math:`n` was chosen to be 12.
+5. A **high-speed test** was proposed to exclude a large number of non-corners. This test examines only the four pixels at 1, 9, 5 and 13 (First 1 and 9 are tested if they are too brighter or darker. If so, then checks 5 and 13). If :math:`p` is a corner, then at least three of these must all be brighter than :math:`I_p + t` or darker than :math:`I_p − t`. If neither of these is the case, then :math:`p` cannot be a corner. The full segment test criterion can then be applied to the passed candidates by examining all pixels in the circle. This detector in itself exhibits high performance, but there are several weaknesses:
+
+    * It does not reject as many candidates for n < 12.
+    * The choice of pixels is not optimal because its efficiency depends on ordering of the questions and distribution of corner appearances.
+    * Results of high-speed tests are thrown away.
+    * Multiple features are detected adjacent to one another.
+
+First 3 points are addressed with a machine learning approach. Last one is addressed using non-maximal suppression.
+
+
+Machine Learning a Corner Detector
+------------------------------------
+
+1. Select a set of images for training (preferably from the target application domain)
+2. Run FAST algorithm in every images to find feature points.
+3. For every feature point, store the 16 pixels around it as a vector. Do it for all the images to get feature vector :math:`P`.
+4. Each pixel (say :math:`x`) in these 16 pixels can have one of the following three states:
+
+    .. image:: images/fast_eqns.jpg
+        :alt: FAST equation
+        :align: center
+
+5. Depending on these states, the feature vector :math:`P` is subdivided into 3 subsets, :math:`P_d`, :math:`P_s`, :math:`P_b`.
+6. Define a new boolean variable, :math:`K_p`, which is true if :math:`p` is a corner and false otherwise.
+7. Use the ID3 algorithm (decision tree classifier) to query each subset using the variable :math:`K_p` for the knowledge about the true class. It selects the :math:`x` which yields the most information about whether the candidate pixel is a corner, measured by the entropy of :math:`K_p`.
+8. This is recursively applied to all the subsets until its entropy is zero.
+9. The decision tree so created is used for fast detection in other images.
+
+
+Non-maximal Suppression
+---------------------------
+
+Detecting multiple interest points in adjacent locations is another problem. It is solved by using Non-maximum Suppression.
+
+1. Compute a score function, :math:`V` for all the detected feature points. :math:`V` is the sum of absolute difference between :math:`p` and 16 surrounding pixels values.
+2. Consider two adjacent keypoints and compute their :math:`V` values.
+3. Discard the one with lower :math:`V` value.
+
+
+Summary
+-----------
+
+It is several times faster than other existing corner detectors.
+
+But it is not robust to high levels of noise. It is dependant on a threshold.
+
+
+FAST Feature Detector in OpenCV
+==================================
+
+It is called as any other feature detector in OpenCV. If you want, you can specify the threshold, whether non-maximum suppression to be applied or not, the neighborhood to be used etc.
+
+For the neighborhood, three flags are defined, ``cv2.FAST_FEATURE_DETECTOR_TYPE_5_8``, ``cv2.FAST_FEATURE_DETECTOR_TYPE_7_12`` and  ``cv2.FAST_FEATURE_DETECTOR_TYPE_9_16``. Below is a simple code on how to detect and draw the FAST feature points.
+::
+
+    import numpy as np
+    import cv2
+    from matplotlib import pyplot as plt
+
+    img = cv2.imread('simple.jpg',0)
+
+    # Initiate FAST object with default values
+    fast = cv2.FastFeatureDetector()
+
+    # find and draw the keypoints
+    kp = fast.detect(img,None)
+    img2 = cv2.drawKeypoints(img, kp, color=(255,0,0))
+
+    # Print all default params
+    print "Threshold: ", fast.getInt('threshold')
+    print "nonmaxSuppression: ", fast.getBool('nonmaxSuppression')
+    print "neighborhood: ", fast.getInt('type')
+    print "Total Keypoints with nonmaxSuppression: ", len(kp)
+
+    cv2.imwrite('fast_true.png',img2)
+
+    # Disable nonmaxSuppression
+    fast.setBool('nonmaxSuppression',0)
+    kp = fast.detect(img,None)
+
+    print "Total Keypoints without nonmaxSuppression: ", len(kp)
+
+    img3 = cv2.drawKeypoints(img, kp, color=(255,0,0))
+
+    cv2.imwrite('fast_false.png',img3)
+
+See the results. First image shows FAST with nonmaxSuppression and second one without nonmaxSuppression:
+
+    .. image:: images/fast_kp.jpg
+        :alt: FAST Keypoints
+        :align: center
+
+
+Additional Resources
+=========================
+
+#. Edward Rosten and Tom Drummond, “Machine learning for high speed corner detection” in 9th European Conference on Computer Vision, vol. 1, 2006, pp. 430–443.
+
+#. Edward Rosten, Reid Porter, and Tom Drummond, "Faster and better: a machine learning approach to corner detection" in IEEE Trans. Pattern Analysis and Machine Intelligence, 2010, vol 32, pp. 105-119.
+
+
+Exercises
+============

doc/py_tutorials/py_feature2d/py_feature_homography/py_feature_homography.rst

@@ -0,0 +1,110 @@
+.. _PY_feature_homography:
+
+
+Feature Matching + Homography to find Objects
+***********************************************
+
+Goal
+======
+In this chapter,
+    * We will mix up the feature matching and findHomography from calib3d module to find known objects in a complex image.
+
+
+Basics
+=========
+
+So what we did in last session? We used a queryImage, found some feature points in it, we took another trainImage, found the features in that image too and we found the best matches among them. In short, we found locations of some parts of an object in another cluttered image. This information is sufficient to find the object exactly on the trainImage.
+
+For that, we can use a function from calib3d module, ie **cv2.findHomography()**. If we pass the set of points from both the images, it will find the perpective transformation of that object. Then we can use **cv2.perspectiveTransform()** to find the object. It needs atleast four correct points to find the transformation.
+
+We have seen that there can be some possible errors while matching which may affect the result. To solve this problem, algorithm uses RANSAC or LEAST_MEDIAN (which can be decided by the flags). So good matches which provide correct estimation are called inliers and remaining are called outliers. **cv2.findHomography()** returns a mask which specifies the inlier and outlier points.
+
+So let's do it !!!
+
+
+Code
+=========
+
+First, as usual, let's find SIFT features in images and apply the ratio test to find the best matches.
+::
+
+    import numpy as np
+    import cv2
+    from matplotlib import pyplot as plt
+
+    MIN_MATCH_COUNT = 10
+
+    img1 = cv2.imread('box.png',0)          # queryImage
+    img2 = cv2.imread('box_in_scene.png',0) # trainImage
+
+    # Initiate SIFT detector
+    sift = cv2.SIFT()
+
+    # find the keypoints and descriptors with SIFT
+    kp1, des1 = sift.detectAndCompute(img1,None)
+    kp2, des2 = sift.detectAndCompute(img2,None)
+
+    FLANN_INDEX_KDTREE = 0
+    index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
+    search_params = dict(checks = 50)
+
+    flann = cv2.FlannBasedMatcher(index_params, search_params)
+
+    matches = flann.knnMatch(des1,des2,k=2)
+
+    # store all the good matches as per Lowe's ratio test.
+    good = []
+    for m,n in matches:
+        if m.distance < 0.7*n.distance:
+            good.append(m)
+
+
+Now we set a condition that atleast 10 matches (defined by MIN_MATCH_COUNT) are to be there to find the object. Otherwise simply show a message saying not enough matches are present.
+
+If enough matches are found, we extract the locations of matched keypoints in both the images. They are passed to find the perpective transformation. Once we get this 3x3 transformation matrix, we use it to transform the corners of queryImage to corresponding points in trainImage. Then we draw it.
+::
+
+    if len(good)>MIN_MATCH_COUNT:
+        src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
+        dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)
+
+        M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)
+        matchesMask = mask.ravel().tolist()
+
+        h,w = img1.shape
+        pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
+        dst = cv2.perspectiveTransform(pts,M)
+
+        img2 = cv2.polylines(img2,[np.int32(dst)],True,255,3, cv2.LINE_AA)
+
+    else:
+        print "Not enough matches are found - %d/%d" % (len(good),MIN_MATCH_COUNT)
+        matchesMask = None
+
+
+Finally we draw our inliers (if successfully found the object) or matching keypoints (if failed).
+::
+
+    draw_params = dict(matchColor = (0,255,0), # draw matches in green color
+                       singlePointColor = None,
+                       matchesMask = matchesMask, # draw only inliers
+                       flags = 2)
+
+    img3 = cv2.drawMatches(img1,kp1,img2,kp2,good,None,**draw_params)
+
+    plt.imshow(img3, 'gray'),plt.show()
+
+
+See the result below. Object is marked in white color in cluttered image:
+
+    .. image:: images/homography_findobj.jpg
+        :alt: Finding object with feature homography
+        :align: center
+
+
+Additional Resources
+============================
+
+
+Exercises
+==================

doc/py_tutorials/py_feature2d/py_features_harris/py_features_harris.rst

@@ -0,0 +1,154 @@
+.. _Harris_Corners:
+
+Harris Corner Detection
+****************************
+
+Goal
+=======
+
+In this chapter,
+
+    * We will understand the concepts behind Harris Corner Detection.
+    * We will see the functions: **cv2.cornerHarris()**, **cv2.cornerSubPix()**
+
+Theory
+==========
+
+In last chapter, we saw that corners are regions in the image with large variation in intensity in all the directions. One early attempt to find these corners was done by **Chris Harris & Mike Stephens** in their paper **A Combined Corner and Edge Detector** in 1988, so now it is called Harris Corner Detector. He took this simple idea to a mathematical form. It basically finds the difference in intensity for a displacement of :math:`(u,v)` in all directions. This is expressed as below:
+
+.. math::
+
+    E(u,v) = \sum_{x,y} \underbrace{w(x,y)}_\text{window function} \, [\underbrace{I(x+u,y+v)}_\text{shifted intensity}-\underbrace{I(x,y)}_\text{intensity}]^2
+
+Window function is either a rectangular window or gaussian window which gives weights to pixels underneath.
+
+We have to maximize this function :math:`E(u,v)` for corner detection. That means, we have to maximize the second term. Applying Taylor Expansion to above equation and using some mathematical steps (please refer any standard text books you like for full derivation), we get the final equation as:
+
+.. math::
+
+    E(u,v) \approx \begin{bmatrix} u & v \end{bmatrix} M \begin{bmatrix} u \\ v \end{bmatrix}
+
+where
+
+.. math::
+
+    M = \sum_{x,y} w(x,y) \begin{bmatrix}I_x I_x & I_x I_y \\
+                                         I_x I_y & I_y I_y \end{bmatrix}
+
+Here, :math:`I_x` and :math:`I_y` are image derivatives in x and y directions respectively. (Can be easily found out using **cv2.Sobel()**).
+
+Then comes the main part. After this, they created a score, basically an equation, which will determine if a window can contain a corner or not.
+
+.. math::
+
+    R = det(M) - k(trace(M))^2
+
+where
+    * :math:`det(M) = \lambda_1 \lambda_2`
+    * :math:`trace(M) = \lambda_1 + \lambda_2`
+    * :math:`\lambda_1` and :math:`\lambda_2` are the eigen values of M
+
+So the values of these eigen values decide whether a region is corner, edge or flat.
+
+    * When :math:`|R|` is small, which happens when :math:`\lambda_1` and :math:`\lambda_2` are small, the region is flat.
+    * When :math:`R<0`, which happens when :math:`\lambda_1 >> \lambda_2` or vice versa, the region is edge.
+    * When :math:`R` is large, which happens when :math:`\lambda_1` and :math:`\lambda_2` are large and :math:`\lambda_1 \sim \lambda_2`, the region is a corner.
+
+It can be represented in a nice picture as follows:
+
+    .. image:: images/harris_region.jpg
+        :alt: Classification of Image Points
+        :align: center
+
+So the result of Harris Corner Detection is a grayscale image with these scores. Thresholding for a suitable give you the corners in the image. We will do it with a simple image.
+
+
+Harris Corner Detector in OpenCV
+====================================
+
+OpenCV has the function **cv2.cornerHarris()** for this purpose. Its arguments are :
+
+    * **img** - Input image, it should be grayscale and float32 type.
+    * **blockSize** - It is the size of neighbourhood considered for corner detection
+    * **ksize** - Aperture parameter of Sobel derivative used.
+    * **k** - Harris detector free parameter in the equation.
+
+See the example below:
+::
+
+    import cv2
+    import numpy as np
+
+    filename = 'chessboard.jpg'
+    img = cv2.imread(filename)
+    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
+
+    gray = np.float32(gray)
+    dst = cv2.cornerHarris(gray,2,3,0.04)
+
+    #result is dilated for marking the corners, not important
+    dst = cv2.dilate(dst,None)
+
+    # Threshold for an optimal value, it may vary depending on the image.
+    img[dst>0.01*dst.max()]=[0,0,255]
+
+    cv2.imshow('dst',img)
+    if cv2.waitKey(0) & 0xff == 27:
+        cv2.destroyAllWindows()
+
+
+Below are the three results:
+
+    .. image:: images/harris_result.jpg
+        :alt: Harris Corner Detection
+        :align: center
+
+
+Corner with SubPixel Accuracy
+===============================
+
+Sometimes, you may need to find the corners with maximum accuracy. OpenCV comes with a function **cv2.cornerSubPix()** which further refines the corners detected with sub-pixel accuracy. Below is an example. As usual, we need to find the harris corners first. Then we pass the centroids of these corners (There may be a bunch of pixels at a corner, we take their centroid) to refine them. Harris corners are marked in red pixels and refined corners are marked in green pixels. For this function, we have to define the criteria when to stop the iteration. We stop it after a specified number of iteration or a certain accuracy is achieved, whichever occurs first. We also need to define the size of neighbourhood it would search for corners.
+::
+
+    import cv2
+    import numpy as np
+
+    filename = 'chessboard2.jpg'
+    img = cv2.imread(filename)
+    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
+
+    # find Harris corners
+    gray = np.float32(gray)
+    dst = cv2.cornerHarris(gray,2,3,0.04)
+    dst = cv2.dilate(dst,None)
+    ret, dst = cv2.threshold(dst,0.01*dst.max(),255,0)
+    dst = np.uint8(dst)
+
+    # find centroids
+    ret, labels, stats, centroids = cv2.connectedComponentsWithStats(dst)
+
+    # define the criteria to stop and refine the corners
+    criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.001)
+    corners = cv2.cornerSubPix(gray,np.float32(centroids),(5,5),(-1,-1),criteria)
+
+    # Now draw them
+    res = np.hstack((centroids,corners))
+    res = np.int0(res)
+    img[res[:,1],res[:,0]]=[0,0,255]
+    img[res[:,3],res[:,2]] = [0,255,0]
+
+    cv2.imwrite('subpixel5.png',img)
+
+Below is the result, where some important locations are shown in zoomed window to visualize:
+
+    .. image:: images/subpixel3.png
+        :alt: Corner Detection with SubPixel Accuracy
+        :align: center
+
+
+Additional Resources
+======================
+
+
+Exercises
+============

doc/py_tutorials/py_feature2d/py_features_meaning/py_features_meaning.rst

@@ -0,0 +1,52 @@
+.. _Features_Meaning:
+
+Understanding Features
+************************
+
+Goal
+=====
+
+In this chapter, we will just try to understand what are features, why are they important, why corners are important etc.
+
+Explanation
+==============
+
+Most of you will have played the jigsaw puzzle games. You get a lot of small pieces of a images, where you need to assemble them correctly to form a big real image. **The question is, how you do it?** What about the projecting the same theory to a computer program so that computer can play jigsaw puzzles? If the computer can play jigsaw puzzles, why can't we give a lot of real-life images of a good natural scenery to computer and tell it to stitch all those images to a big single image? If the computer can stitch several natural images to one, what about giving a lot of pictures of a building or any structure and tell computer to create a 3D model out of it?
+
+Well, the questions and imaginations continue. But it all depends on the most basic question? How do you play jigsaw puzzles? How do you arrange lots of scrambled image pieces into a big single image? How can you stitch a lot of natural images to a single image?
+
+The answer is, we are looking for specific patterns or specific features which are unique, which can be easily tracked, which can be easily compared. If we go for a definition of such a feature, we may find it difficult to express it in words, but we know what are they. If some one asks you to point out one good feature which can be compared across several images, you can point out one. That is why, even small children can simply play these games. We search for these features in an image, we find them, we find the same features in other images, we align them. That's it. (In jigsaw puzzle, we look more into continuity of different images). All these abilities are present in us inherently.
+
+So our one basic question expands to more in number, but becomes more specific. **What are these features?**. *(The answer should be understandable to a computer also.)*
+
+Well, it is difficult to say how humans find these features. It is already programmed in our brain. But if we look deep into some pictures and search for different patterns, we will find something interesting. For example, take below image:
+
+    .. image:: images/feature_building.jpg
+        :alt: Understanding features
+        :align: center
+
+Image is very simple. At the top of image, six small image patches are given. Question for you is to find the exact location of these patches in the original image. How many correct results you can find ?
+
+A and B are flat surfaces, and they are spread in a lot of area. It is difficult to find the exact location of these patches.
+
+C and D are much more simpler. They are edges of the building. You can find an approximate location, but exact location is still difficult. It is because, along the edge, it is same everywhere. Normal to the edge, it is different. So edge is much more better feature compared to flat area, but not good enough (It is good in jigsaw puzzle for comparing continuity of edges).
+
+Finally, E and F are some corners of the building. And they can be easily found out. Because at corners, wherever you move this patch, it will look different. So they can be considered as a good feature. So now we move into more simpler (and widely used image) for better understanding.
+
+    .. image:: images/feature_simple.png
+        :alt: Features
+        :align: center
+
+Just like above, blue patch is flat area and difficult to find and track. Wherever you move the blue patch, it looks the same. For black patch, it is an edge. If you move it in vertical direction (i.e. along the gradient) it changes. Put along the edge (parallel to edge), it looks the same. And for red patch, it is a corner. Wherever you move the patch, it looks different, means it is unique. So basically, corners are considered to be good features in an image. (Not just corners, in some cases blobs are considered good features).
+
+So now we answered our question, "what are these features?". But next question arises. How do we find them? Or how do we find the corners?. That also we answered in an intuitive way, i.e., look for the regions in images which have maximum variation when moved (by a small amount) in all regions around it. This would be projected into computer language in coming chapters. So finding these image features is called **Feature Detection**.
+
+So we found the features in image (Assume you did it). Once you found it, you should find the same in the other images. What we do? We take a region around the feature, we explain it in our own words, like "upper part is blue sky, lower part is building region, on that building there are some glasses etc" and you search for the same area in other images. Basically, you are describing the feature. Similar way, computer also should describe the region around the feature so that it can find it in other images. So called description is called **Feature Description**. Once you have the features and its description, you can find same features in all images and align them, stitch them or do whatever you want.
+
+So in this module, we are looking to different algorithms in OpenCV to find features, describe them, match them etc.
+
+Additional Resources
+=======================
+
+Exercises
+===========

doc/py_tutorials/py_feature2d/py_matcher/py_matcher.rst

@@ -0,0 +1,207 @@
+.. _Matcher:
+
+
+Feature Matching
+*********************************************
+
+Goal
+=====
+In this chapter
+    * We will see how to match features in one image with others.
+    * We will use the Brute-Force matcher and FLANN Matcher in OpenCV
+
+
+Basics of Brute-Force Matcher
+===================================
+
+Brute-Force matcher is simple. It takes the descriptor of one feature in first set and is matched with all other features in second set using some distance calculation. And the closest one is returned.
+
+For BF matcher, first we have to create the BFMatcher object using **cv2.BFMatcher()**. It takes two optional params. First one is ``normType``. It specifies the distance measurement to be used. By default, it is ``cv2.NORM_L2``. It is good for SIFT, SURF etc (``cv2.NORM_L1`` is also there). For binary string based descriptors like ORB, BRIEF, BRISK etc, ``cv2.NORM_HAMMING`` should be used, which used Hamming distance as measurement. If ORB is using ``VTA_K == 3 or 4``, ``cv2.NORM_HAMMING2`` should be used.
+
+Second param is boolean variable, ``crossCheck`` which is false by default. If it is true, Matcher returns only those matches with value (i,j) such that i-th descriptor in set A has j-th descriptor in set B as the best match and vice-versa. That is, the two features in both sets should match each other. It provides consistant result, and is a good alternative to ratio test proposed by D.Lowe in SIFT paper.
+
+Once it is created, two important methods are *BFMatcher.match()* and *BFMatcher.knnMatch()*. First one returns the best match. Second method returns `k` best matches where k is specified by the user. It may be useful when we need to do additional work on that.
+
+Like we used cv2.drawKeypoints() to draw keypoints, **cv2.drawMatches()** helps us to draw the matches. It stacks two images horizontally and draw lines from first image to second image showing best matches. There is also **cv2.drawMatchesKnn** which draws all the k best matches. If k=2, it will draw two match-lines for each keypoint. So we have to pass a mask if we want to selectively draw it.
+
+Let's see one example for each of SURF and ORB (Both use different distance measurements).
+
+
+
+Brute-Force Matching with ORB Descriptors
+--------------------------------------------
+
+Here, we will see a simple example on how to match features between two images. In this case, I have a queryImage and a trainImage. We will try to find the queryImage in trainImage using feature matching. ( The images are ``/samples/c/box.png`` and ``/samples/c/box_in_scene.png``)
+
+We are using SIFT descriptors to match features. So let's start with loading images, finding descriptors etc.
+::
+
+    import numpy as np
+    import cv2
+    from matplotlib import pyplot as plt
+
+    img1 = cv2.imread('box.png',0)          # queryImage
+    img2 = cv2.imread('box_in_scene.png',0) # trainImage
+
+    # Initiate SIFT detector
+    orb = cv2.ORB()
+
+    # find the keypoints and descriptors with SIFT
+    kp1, des1 = orb.detectAndCompute(img1,None)
+    kp2, des2 = orb.detectAndCompute(img2,None)
+
+
+Next we create a BFMatcher object with distance measurement ``cv2.NORM_HAMMING`` (since we are using ORB) and ``crossCheck`` is switched on for better results. Then we use Matcher.match() method to get the best matches in two images. We sort them in ascending order of their distances so that best matches (with low distance) come to front. Then we draw only first 10 matches (Just for sake of visibility. You can increase it as you like)
+::
+
+    # create BFMatcher object
+    bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
+
+    # Match descriptors.
+    matches = bf.match(des1,des2)
+
+    # Sort them in the order of their distance.
+    matches = sorted(matches, key = lambda x:x.distance)
+
+    # Draw first 10 matches.
+    img3 = cv2.drawMatches(img1,kp1,img2,kp2,matches[:10], flags=2)
+
+    plt.imshow(img3),plt.show()
+
+Below is the result I got:
+
+    .. image:: images/matcher_result1.jpg
+        :alt: ORB Feature Matching with Brute-Force
+        :align: center
+
+
+What is this Matcher Object?
+-----------------------------------
+
+The result of ``matches = bf.match(des1,des2)`` line is a list of DMatch objects. This DMatch object has following attributes:
+
+    * ``DMatch.distance`` - Distance between descriptors. The lower, the better it is.
+    * ``DMatch.trainIdx`` - Index of the descriptor in train descriptors
+    * ``DMatch.queryIdx`` - Index of the descriptor in query descriptors
+    * ``DMatch.imgIdx``   - Index of the train image.
+
+
+Brute-Force Matching with SIFT Descriptors and Ratio Test
+-------------------------------------------------------------
+
+This time, we will use ``BFMatcher.knnMatch()`` to get k best matches. In this example, we will take k=2 so that we can apply ratio test explained by D.Lowe in his paper.
+::
+
+    import numpy as np
+    import cv2
+    from matplotlib import pyplot as plt
+
+    img1 = cv2.imread('box.png',0)          # queryImage
+    img2 = cv2.imread('box_in_scene.png',0) # trainImage
+
+    # Initiate SIFT detector
+    sift = cv2.SIFT()
+
+    # find the keypoints and descriptors with SIFT
+    kp1, des1 = sift.detectAndCompute(img1,None)
+    kp2, des2 = sift.detectAndCompute(img2,None)
+
+    # BFMatcher with default params
+    bf = cv2.BFMatcher()
+    matches = bf.knnMatch(des1,des2, k=2)
+
+    # Apply ratio test
+    good = []
+    for m,n in matches:
+        if m.distance < 0.75*n.distance:
+            good.append([m])
+
+    # cv2.drawMatchesKnn expects list of lists as matches.
+    img3 = cv2.drawMatchesKnn(img1,kp1,img2,kp2,good,flags=2)
+
+    plt.imshow(img3),plt.show()
+
+See the result below:
+
+    .. image:: images/matcher_result2.jpg
+        :alt: SIFT Descriptor with ratio test
+        :align: center
+
+
+FLANN based Matcher
+==========================
+
+FLANN stands for Fast Library for Approximate Nearest Neighbors. It contains a collection of algorithms optimized for fast nearest neighbor search in large datasets and for high dimensional features. It works more faster than BFMatcher for large datasets. We will see the second example with FLANN based matcher.
+
+For FLANN based matcher, we need to pass two dictionaries which specifies the algorithm to be used, its related parameters etc. First one is IndexParams. For various algorithms, the information to be passed is explained in FLANN docs. As a summary, for algorithms like SIFT, SURF etc. you can pass following:
+::
+
+    index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
+
+While using ORB, you can pass the following. The commented values are recommended as per the docs, but it didn't provide required results in some cases. Other values worked fine.:
+::
+
+    index_params= dict(algorithm = FLANN_INDEX_LSH,
+                       table_number = 6, # 12
+                       key_size = 12,     # 20
+                       multi_probe_level = 1) #2
+
+Second dictionary is the SearchParams. It specifies the number of times the trees in the index should be recursively traversed. Higher values gives better precision, but also takes more time. If you want to change the value, pass ``search_params = dict(checks=100)``.
+
+With these informations, we are good to go.
+::
+
+    import numpy as np
+    import cv2
+    from matplotlib import pyplot as plt
+
+    img1 = cv2.imread('box.png',0)          # queryImage
+    img2 = cv2.imread('box_in_scene.png',0) # trainImage
+
+    # Initiate SIFT detector
+    sift = cv2.SIFT()
+
+    # find the keypoints and descriptors with SIFT
+    kp1, des1 = sift.detectAndCompute(img1,None)
+    kp2, des2 = sift.detectAndCompute(img2,None)
+
+    # FLANN parameters
+    FLANN_INDEX_KDTREE = 0
+    index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
+    search_params = dict(checks=50)   # or pass empty dictionary
+
+    flann = cv2.FlannBasedMatcher(index_params,search_params)
+
+    matches = flann.knnMatch(des1,des2,k=2)
+
+    # Need to draw only good matches, so create a mask
+    matchesMask = [[0,0] for i in xrange(len(matches))]
+
+    # ratio test as per Lowe's paper
+    for i,(m,n) in enumerate(matches):
+        if m.distance < 0.7*n.distance:
+            matchesMask[i]=[1,0]
+
+    draw_params = dict(matchColor = (0,255,0),
+                       singlePointColor = (255,0,0),
+                       matchesMask = matchesMask,
+                       flags = 0)
+
+    img3 = cv2.drawMatchesKnn(img1,kp1,img2,kp2,matches,None,**draw_params)
+
+    plt.imshow(img3,),plt.show()
+
+
+See the result below:
+
+    .. image:: images/matcher_flann.jpg
+        :alt: FLANN based matching
+        :align: center
+
+
+Additional Resources
+========================
+
+
+Exercises
+=================

doc/py_tutorials/py_feature2d/py_orb/py_orb.rst

@@ -0,0 +1,75 @@
+.. _ORB:
+
+ORB (Oriented FAST and Rotated BRIEF)
+***************************************
+
+Goal
+======
+
+In this chapter,
+    * We will see the basics of ORB
+
+
+Theory
+==========
+
+As an OpenCV enthusiast, the most important thing about the ORB is that it came from "OpenCV Labs". This algorithm was brought up by Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary R. Bradski in their paper **ORB: An efficient alternative to SIFT or SURF** in 2011. As the title says, it is a good alternative to SIFT and SURF in computation cost, matching performance and mainly the patents. Yes, SIFT and SURF are patented and you are supposed to pay them for its use. But ORB is not !!!
+
+ORB is basically a fusion of FAST keypoint detector and BRIEF descriptor with many modifications to enhance the performance. First it use FAST to find keypoints, then apply Harris corner measure to find top N points among them. It also use pyramid to produce multiscale-features. But one problem is that, FAST doesn't compute the orientation. So what about rotation invariance? Authors came up with following modification.
+
+It computes the intensity weighted centroid of the patch with located corner at center. The direction of the vector from this corner point to centroid gives the orientation. To improve the rotation invariance, moments are computed with x and y which should be in a circular region of radius :math:`r`, where :math:`r` is the size of the patch.
+
+Now for descriptors, ORB use BRIEF descriptors. But we have already seen that BRIEF performs poorly with rotation. So what ORB does is to "steer" BRIEF according to the orientation of keypoints. For any feature set of :math:`n` binary tests at location
+:math:`(x_i, y_i)`, define a :math:`2 \times n` matrix, :math:`S` which contains the coordinates of these pixels. Then using the orientation of patch, :math:`\theta`, its rotation matrix is found and rotates the :math:`S` to get steered(rotated) version :math:`S_\theta`.
+
+ORB discretize the angle to increments of :math:`2 \pi /30` (12 degrees), and construct a lookup table of precomputed BRIEF patterns. As long as the keypoint orientation :math:`\theta` is consistent across views, the correct set of points :math:`S_\theta` will be used to compute its descriptor.
+
+BRIEF has an important property that each bit feature has a large variance and a mean near 0.5. But once it is oriented along keypoint direction, it loses this property and become more distributed. High variance makes a feature more discriminative, since it responds differentially to inputs. Another desirable property is to have the tests uncorrelated, since then each test will contribute to the result. To resolve all these, ORB runs a greedy search among all possible binary tests to find the ones that have both high variance and means close to 0.5, as well as being uncorrelated. The result is called **rBRIEF**.
+
+For descriptor matching, multi-probe LSH which improves on the traditional LSH, is used. The paper says ORB is much faster than SURF and SIFT and ORB descriptor works better than SURF. ORB is a good choice in low-power devices for panorama stitching etc.
+
+
+ORB in OpenCV
+================
+
+As usual, we have to create an ORB object with the function, **cv2.ORB()** or using feature2d common interface. It has a number of optional parameters. Most useful ones are ``nFeatures`` which denotes maximum number of features to be retained (by default 500), ``scoreType`` which denotes whether Harris score or FAST score to rank the features (by default, Harris score) etc. Another parameter, ``WTA_K`` decides number of points that produce each element of the oriented BRIEF descriptor. By default it is two, ie selects two points at a time. In that case, for matching, ``NORM_HAMMING`` distance is used. If WTA_K is 3 or 4, which takes 3 or 4 points to produce BRIEF descriptor, then matching distance is defined by ``NORM_HAMMING2``.
+
+Below is a simple code which shows the use of ORB.
+::
+
+    import numpy as np
+    import cv2
+    from matplotlib import pyplot as plt
+
+    img = cv2.imread('simple.jpg',0)
+
+    # Initiate STAR detector
+    orb = cv2.ORB()
+
+    # find the keypoints with ORB
+    kp = orb.detect(img,None)
+
+    # compute the descriptors with ORB
+    kp, des = orb.compute(img, kp)
+
+    # draw only keypoints location,not size and orientation
+    img2 = cv2.drawKeypoints(img,kp,color=(0,255,0), flags=0)
+    plt.imshow(img2),plt.show()
+
+See the result below:
+
+    .. image:: images/orb_kp.jpg
+        :alt: ORB Keypoints
+        :align: center
+
+
+ORB feature matching, we will do in another chapter.
+
+Additional Resources
+==========================
+
+#. Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski: ORB: An efficient alternative to SIFT or SURF. ICCV 2011: 2564-2571.
+
+
+Exercises
+==============

doc/py_tutorials/py_feature2d/py_shi_tomasi/py_shi_tomasi.rst

@@ -0,0 +1,77 @@
+.. _shi_tomasi:
+
+Shi-Tomasi Corner Detector & Good Features to Track
+*******************************************************
+
+Goal
+=======
+
+In this chapter,
+
+    * We will learn about the another corner detector: Shi-Tomasi Corner Detector
+    * We will see the function: **cv2.goodFeaturesToTrack()**
+
+Theory
+=========
+
+In last chapter, we saw Harris Corner Detector. Later in 1994, J. Shi and C. Tomasi made a small modification to it in their paper **Good Features to Track** which shows better results compared to Harris Corner Detector. The scoring function in Harris Corner Detector was given by:
+
+.. math::
+
+    R = \lambda_1 \lambda_2 - k(\lambda_1+\lambda_2)^2
+
+Instead of this, Shi-Tomasi proposed:
+
+.. math::
+
+    R = min(\lambda_1, \lambda_2)
+
+If it is a greater than a threshold value, it is considered as a corner. If we plot it in :math:`\lambda_1 - \lambda_2` space as we did in Harris Corner Detector, we get an image as below:
+
+    .. image:: images/shitomasi_space.png
+        :alt: Shi-Tomasi Corner Space
+        :align: center
+
+From the figure, you can see that only when :math:`\lambda_1` and :math:`\lambda_2` are above a minimum value, :math:`\lambda_{min}`, it is conidered as a corner(green region).
+
+Code
+=======
+
+OpenCV has a function, **cv2.goodFeaturesToTrack()**. It finds N strongest corners in the image by Shi-Tomasi method (or Harris Corner Detection, if you specify it). As usual, image should be a grayscale image. Then you specify number of corners you want to find. Then you specify the quality level, which is a value between 0-1, which denotes the minimum quality of corner below which everyone is rejected. Then we provide the minimum euclidean distance between corners detected.
+
+With all these informations, the function finds corners in the image. All corners below quality level are rejected. Then it sorts the remaining corners based on quality in the descending order. Then function takes first strongest corner, throws away all the nearby corners in the range of minimum distance and returns N strongest corners.
+
+In below example, we will try to find 25 best corners:
+::
+
+    import numpy as np
+    import cv2
+    from matplotlib import pyplot as plt
+
+    img = cv2.imread('simple.jpg')
+    gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
+
+    corners = cv2.goodFeaturesToTrack(gray,25,0.01,10)
+    corners = np.int0(corners)
+
+    for i in corners:
+        x,y = i.ravel()
+        cv2.circle(img,(x,y),3,255,-1)
+
+    plt.imshow(img),plt.show()
+
+See the result below:
+
+    .. image:: images/shitomasi_block1.jpg
+        :alt: Shi-Tomasi Corners
+        :align: center
+
+
+This function is more appropriate for tracking. We will see that when its time comes.
+
+Additional Resources
+======================
+
+
+Exercises
+============

doc/py_tutorials/py_feature2d/py_sift_intro/py_sift_intro.rst

@@ -0,0 +1,139 @@
+.. _sift_intro:
+
+
+Introduction to SIFT (Scale-Invariant Feature Transform)
+*************************************************************
+
+Goal
+======
+
+In this chapter,
+    * We will learn about the concepts of SIFT algorithm
+    * We will learn to find SIFT Keypoints and Descriptors.
+
+
+Theory
+========
+
+In last couple of chapters, we saw some corner detectors like Harris etc. They are rotation-invariant, which means, even if the image is rotated, we can find the same corners. It is obvious because corners remain corners in rotated image also. But what about scaling? A corner may not be a corner if the image is scaled. For example, check a simple image below. A corner in a small image within a small window is flat when it is zoomed in the same window. So Harris corner is not scale invariant.
+
+    .. image:: images/sift_scale_invariant.jpg
+        :alt: Scale-Invariance
+        :align: center
+
+So, in 2004, **D.Lowe**, University of British Columbia, came up with a new algorithm, Scale Invariant Feature Transform (SIFT) in his paper, **Distinctive Image Features from Scale-Invariant Keypoints**, which extract keypoints and compute its descriptors. *(This paper is easy to understand and considered to be best material available on SIFT. So this explanation is just a short summary of this paper)*.
+
+There are mainly four steps involved in SIFT algorithm. We will see them one-by-one.
+
+1. Scale-space Extrema Detection
+--------------------------------------
+
+From the image above, it is obvious that we can't use the same window to detect keypoints with different scale. It is OK with small corner. But to detect larger corners we need larger windows. For this, scale-space filtering is used. In it, Laplacian of Gaussian is found for the image with various :math:`\sigma` values. LoG acts as a blob detector which detects blobs in various sizes due to change in :math:`\sigma`. In short, :math:`\sigma` acts as a scaling parameter. For eg, in the above image, gaussian kernel with low :math:`\sigma` gives high value for small corner while guassian kernel with high :math:`\sigma` fits well for larger corner. So, we can find the local maxima across the scale and space which gives us a list of :math:`(x,y,\sigma)` values which means there is a potential keypoint at (x,y) at :math:`\sigma` scale.
+
+But this LoG is a little costly, so SIFT algorithm uses Difference of Gaussians which is an approximation of LoG. Difference of Gaussian is obtained as the difference of Gaussian blurring of an image with two different :math:`\sigma`, let it be :math:`\sigma` and :math:`k\sigma`. This process is done for different octaves of the image in Gaussian Pyramid. It is represented in below image:
+
+
+    .. image:: images/sift_dog.jpg
+        :alt: Difference of Gaussian
+        :align: center
+
+Once this DoG are found, images are searched for local extrema over scale and space. For eg, one pixel in an image is compared with its 8 neighbours as well as 9 pixels in next scale and 9 pixels in previous scales. If it is a local extrema, it is a potential keypoint. It basically means that keypoint is best represented in that scale. It is shown in below image:
+
+    .. image:: images/sift_local_extrema.jpg
+        :alt: Difference of Gaussian
+        :align: center
+
+Regarding different parameters, the paper gives some empirical data which can be summarized as, number of octaves = 4, number of scale levels = 5, initial :math:`\sigma=1.6`, :math:`k=\sqrt{2}` etc as optimal values.
+
+
+2. Keypoint Localization
+------------------------------------
+
+Once potential keypoints locations are found, they have to be refined to get more accurate results. They used Taylor series expansion of scale space to get more accurate location of extrema, and if the intensity at this extrema is less than a threshold value (0.03 as per the paper), it is rejected. This threshold is called **contrastThreshold** in OpenCV
+
+DoG has higher response for edges, so edges also need to be removed. For this, a concept similar to Harris corner detector is used. They used a 2x2 Hessian matrix (H) to compute the pricipal curvature. We know from Harris corner detector that for edges, one eigen value is larger than the other. So here they used a simple function,
+
+.. math:
+
+    \frac{Tr(H)^2}{Det(H)} < \frac{(r+1)^2}{r} \; \text{where} \; r = \frac{\lambda_1}{\lambda_2}; \; \lambda_1 > \lambda_2
+
+If this ratio is greater than a threshold, called **edgeThreshold** in OpenCV, that keypoint is discarded. It is given as 10 in paper.
+
+So it eliminates any low-contrast keypoints and edge keypoints and what remains is strong interest points.
+
+3. Orientation Assignment
+-----------------------------------
+
+Now an orientation is assigned to each keypoint to achieve invariance to image rotation. A neigbourhood is taken around the keypoint location depending on the scale, and the gradient magnitude and direction is calculated in that region. An orientation histogram with 36 bins covering 360 degrees is created. (It is weighted by gradient magnitude and gaussian-weighted circular window with :math:`\sigma` equal to 1.5 times the scale of keypoint. The highest peak in the histogram is taken and any peak above 80% of it is also considered to calculate the orientation. It creates keypoints with same location and scale, but different directions. It contribute to stability of matching.
+
+
+4. Keypoint Descriptor
+-----------------------------------------
+
+Now keypoint descriptor is created. A 16x16 neighbourhood around the keypoint is taken. It is devided into 16 sub-blocks of 4x4 size. For each sub-block, 8 bin orientation histogram is created. So a total of 128 bin values are available. It is represented as a vector to form keypoint descriptor. In addition to this, several measures are taken to achieve robustness against illumination changes, rotation etc.
+
+5. Keypoint Matching
+----------------------------------------
+
+Keypoints between two images are matched by identifying their nearest neighbours. But in some cases, the second closest-match may be very near to the first. It may happen due to noise or some other reasons. In that case, ratio of closest-distance to second-closest distance is taken. If it is greater than 0.8, they are rejected. It eliminaters around 90% of false matches while discards only 5% correct matches, as per the paper.
+
+So this is a summary of SIFT algorithm. For more details and understanding, reading the original paper is highly recommended. Remember one thing, this algorithm is patented. So this algorithm is included in Non-free module in OpenCV.
+
+
+SIFT in OpenCV
+=================
+
+So now let's see SIFT functionalities available in OpenCV. Let's start with keypoint detection and draw them. First we have to construct a SIFT object. We can pass different parameters to it which are optional and they are well explained in docs.
+::
+
+    import cv2
+    import numpy as np
+
+    img = cv2.imread('home.jpg')
+    gray= cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
+
+    sift = cv2.SIFT()
+    kp = sift.detect(gray,None)
+
+    img=cv2.drawKeypoints(gray,kp)
+
+    cv2.imwrite('sift_keypoints.jpg',img)
+
+**sift.detect()** function finds the keypoint in the images. You can pass a mask if you want to search only a part of image. Each keypoint is a special structure which has many attributes like its (x,y) coordinates, size of the meaningful neighbourhood, angle which specifies its orientation, response that specifies strength of keypoints etc.
+
+OpenCV also provides **cv2.drawKeyPoints()** function which draws the small circles on the locations of keypoints. If you pass a flag, **cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS** to it, it will draw a circle with size of keypoint and it will even show its orientation. See below example.
+::
+
+    img=cv2.drawKeypoints(gray,kp,flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
+    cv2.imwrite('sift_keypoints.jpg',img)
+
+See the two results below:
+
+    .. image:: images/sift_keypoints.jpg
+        :alt: SIFT Keypoints
+        :align: center
+
+Now to calculate the descriptor, OpenCV provides two methods.
+
+1. Since you already found keypoints, you can call **sift.compute()** which computes the descriptors from the keypoints we have found. Eg: ``kp,des = sift.compute(gray,kp)``
+
+2. If you didn't find keypoints, directly find keypoints and descriptors in a single step with the function, **sift.detectAndCompute()**.
+
+We will see the second method:
+::
+
+    sift = cv2.SIFT()
+    kp, des = sift.detectAndCompute(gray,None)
+
+Here kp will be a list of keypoints and des is a numpy array of shape :math:`Number\_of\_Keypoints \times 128`.
+
+So we got keypoints, descriptors etc. Now we want to see how to match keypoints in different images. That we will learn in coming chapters.
+
+
+Additional Resources
+=====================
+
+
+
+Exercises
+=============

doc/py_tutorials/py_feature2d/py_surf_intro/py_surf_intro.rst

@@ -0,0 +1,145 @@
+.. _SURF:
+
+
+Introduction to SURF (Speeded-Up Robust Features)
+*****************************************************
+
+Goal
+======
+
+In this chapter,
+    * We will see the basics of SURF
+    * We will see SURF functionalities in OpenCV
+
+
+Theory
+==========
+
+In last chapter, we saw SIFT for keypoint detection and description. But it was comparatively slow and people needed more speeded-up version. In 2006, three people, Bay, H., Tuytelaars, T. and Van Gool, L, published another paper, "SURF: Speeded Up Robust Features" which introduced a new algorithm called SURF. As name suggests, it is a speeded-up version of SIFT.
+
+In SIFT, Lowe approximated Laplacian of Gaussian with Difference of Gaussian for finding scale-space. SURF goes a little further and approximates LoG with Box Filter. Below image shows a demonstration of such an approximation. One big advantage of this approximation is that, convolution with box filter can be easily calculated with the help of integral images. And it can be done in parallel for different scales. Also the SURF rely on determinant of Hessian matrix for both scale and location.
+
+    .. image:: images/surf_boxfilter.jpg
+        :alt: Box Filter approximation of Laplacian
+        :align: center
+
+
+For orientation assignment, SURF uses wavelet responses in horizontal and vertical direction for a neighbourhood of size 6s. Adequate guassian weights are also applied to it. Then they are plotted in a space as given in below image.  The dominant orientation is estimated by calculating the sum of all responses within a sliding orientation window of angle 60 degrees. Interesting thing is that, wavelet response can be found out using integral images very easily at any scale. For many applications, rotation invariance is not required, so no need of finding this orientation, which speeds up the process. SURF provides such a functionality called Upright-SURF or U-SURF. It improves speed and is robust upto :math:`\pm 15^{\circ}`. OpenCV supports both, depending upon the flag, **upright**. If it is 0, orientation is calculated. If it is 1, orientation is not calculated and it is more faster.
+
+    .. image:: images/surf_orientation.jpg
+        :alt: Orientation Assignment in SURF
+        :align: center
+
+For feature description, SURF uses Wavelet responses in horizontal and vertical direction (again, use of integral images makes things easier). A neighbourhood of size 20sX20s is taken around the keypoint where s is the size. It is divided into 4x4 subregions. For each subregion, horizontal and vertical wavelet responses are taken and a vector is formed like this, :math:`v=( \sum{d_x}, \sum{d_y}, \sum{|d_x|}, \sum{|d_y|})`. This when represented as a vector gives SURF feature descriptor with total 64 dimensions. Lower the dimension, higher the speed of computation and matching, but provide better distinctiveness of features.
+
+For more distinctiveness, SURF feature descriptor has an extended 128 dimension version. The sums of :math:`d_x` and :math:`|d_x|` are computed  separately for :math:`d_y < 0` and :math:`d_y \geq 0`. Similarly, the sums of :math:`d_y` and :math:`|d_y|` are split
+up according to the sign of :math:`d_x` , thereby doubling the number of features. It doesn't add much computation complexity. OpenCV supports both by setting the value of flag **extended** with 0 and 1 for 64-dim and 128-dim respectively (default is 128-dim)
+
+Another important improvement is the use of sign of Laplacian (trace of Hessian Matrix) for underlying interest point. It adds no computation cost since it is already computed during detection. The sign of the Laplacian distinguishes bright blobs on dark backgrounds from the reverse situation. In the matching stage, we only compare features if they have the same type of contrast (as shown in image below). This minimal information allows for faster matching, without reducing the descriptor's performance.
+
+    .. image:: images/surf_matching.jpg
+        :alt: Fast Indexing for Matching
+        :align: center
+
+In short, SURF adds a lot of features to improve the speed in every step. Analysis shows it is 3 times faster than SIFT while performance is comparable to SIFT. SURF is good at handling images with blurring and rotation, but not good at handling viewpoint change and illumination change.
+
+
+SURF in OpenCV
+====================
+
+OpenCV provides SURF functionalities just like SIFT. You initiate a SURF object with some optional conditions like 64/128-dim descriptors, Upright/Normal SURF etc. All the details are well explained in docs. Then as we did in SIFT, we can use SURF.detect(), SURF.compute() etc for finding keypoints and descriptors.
+
+First we will see a simple demo on how to find SURF keypoints and descriptors and draw it. All examples are shown in Python terminal since it is just same as SIFT only.
+::
+
+    >>> img = cv2.imread('fly.png',0)
+
+    # Create SURF object. You can specify params here or later.
+    # Here I set Hessian Threshold to 400
+    >>> surf = cv2.SURF(400)
+
+    # Find keypoints and descriptors directly
+    >>> kp, des = surf.detectAndCompute(img,None)
+
+    >>> len(kp)
+     699
+
+
+1199 keypoints is too much to show in a picture. We reduce it to some 50 to draw it on an image. While matching, we may need all those features, but not now. So we increase the Hessian Threshold.
+::
+
+    # Check present Hessian threshold
+    >>> print surf.hessianThreshold
+    400.0
+
+    # We set it to some 50000. Remember, it is just for representing in picture.
+    # In actual cases, it is better to have a value 300-500
+    >>> surf.hessianThreshold = 50000
+
+    # Again compute keypoints and check its number.
+    >>> kp, des = surf.detectAndCompute(img,None)
+
+    >>> print len(kp)
+    47
+
+It is less than 50. Let's draw it on the image.
+::
+
+    >>> img2 = cv2.drawKeypoints(img,kp,None,(255,0,0),4)
+
+    >>> plt.imshow(img2),plt.show()
+
+See the result below. You can see that SURF is more like a blob detector. It detects the white blobs on wings of butterfly. You can test it with other images.
+
+    .. image:: images/surf_kp1.jpg
+        :alt: SURF Keypoints with Orientation
+        :align: center
+
+Now I want to apply U-SURF, so that it won't find the orientation.
+::
+
+    # Check upright flag, if it False, set it to True
+    >>> print surf.upright
+    False
+
+    >>> surf.upright = True
+
+    # Recompute the feature points and draw it
+    >>> kp = surf.detect(img,None)
+    >>> img2 = cv2.drawKeypoints(img,kp,None,(255,0,0),4)
+
+    >>> plt.imshow(img2),plt.show()
+
+See the results below. All the orientations are shown in same direction. It is more faster than previous. If you are working on cases where orientation is not a problem (like panorama stitching) etc, this is more better.
+
+    .. image:: images/surf_kp2.jpg
+        :alt: Upright-SURF
+        :align: center
+
+Finally we check the descriptor size and change it to 128 if it is only 64-dim.
+::
+
+    # Find size of descriptor
+    >>> print surf.descriptorSize()
+    64
+
+    # That means flag, "extended" is False.
+    >>> surf.extended
+     False
+
+    # So we make it to True to get 128-dim descriptors.
+    >>> surf.extended = True
+    >>> kp, des = surf.detectAndCompute(img,None)
+    >>> print surf.descriptorSize()
+    128
+    >>> print des.shape
+    (47, 128)
+
+Remaining part is matching which we will do in another chapter.
+
+Additional Resources
+=======================
+
+
+Exercises
+==============

doc/py_tutorials/py_feature2d/py_table_of_contents_feature2d/py_table_of_contents_feature2d.rst

@@ -0,0 +1,170 @@
+.. _PY_Table-Of-Content-Feature2D:
+
+Feature Detection and Description
+------------------------------------------
+
+*  :ref:`Features_Meaning`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_1|     What are the main features in an image? How can finding those features be useful to us?
+
+  =========== ======================================================
+
+  .. |f2d_1|  image:: images/features_icon.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`Harris_Corners`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_2|     Okay, Corners are good features? But how do we find them?
+
+  =========== ======================================================
+
+  .. |f2d_2|  image:: images/harris_icon.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`shi_tomasi`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_3|     We will look into Shi-Tomasi corner detection
+
+  =========== ======================================================
+
+  .. |f2d_3|  image:: images/shi_icon.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`sift_intro`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_4|     Harris corner detector is not good enough when scale of image changes. Lowe developed a breakthrough method to find scale-invariant features and it is called SIFT
+
+  =========== ======================================================
+
+  .. |f2d_4|  image:: images/sift_icon.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`SURF`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_5|     SIFT is really good, but not fast enough, so people came up with a speeded-up version called SURF.
+
+  =========== ======================================================
+
+  .. |f2d_5|  image:: images/surf_icon.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`FAST`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_06|    All the above feature detection methods are good in some way. But they are not fast enough to work in real-time applications like SLAM. There comes the FAST algorithm, which is really "FAST".
+
+  =========== ======================================================
+
+  .. |f2d_06|  image:: images/fast_icon.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`BRIEF`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_07|    SIFT uses a feature descriptor with 128 floating point numbers. Consider thousands of such features. It takes lots of memory and more time for matching. We can compress it to make it faster. But still we have to calculate it first. There comes BRIEF which gives the shortcut to find binary descriptors with less memory, faster matching, still higher recognition rate.
+
+  =========== ======================================================
+
+  .. |f2d_07|  image:: images/brief.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`ORB`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_08|    SIFT and SURF are good in what they do, but what if you have to pay a few dollars every year to use them in your applications? Yeah, they are patented!!! To solve that problem, OpenCV devs came up with a new "FREE" alternative to SIFT & SURF, and that is ORB.
+  =========== ======================================================
+
+  .. |f2d_08|  image:: images/orb.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`Matcher`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_09|    We know a great deal about feature detectors and descriptors. It is time to learn how to match different descriptors. OpenCV provides two techniques, Brute-Force matcher and FLANN based matcher.
+  =========== ======================================================
+
+  .. |f2d_09|  image:: images/matching.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+*  :ref:`PY_feature_homography`
+
+  .. tabularcolumns:: m{100pt} m{300pt}
+  .. cssclass:: toctableopencv
+
+  =========== ======================================================
+  |f2d_10|    Now we know about feature matching. Let's mix it up with `calib3d` module to find objects in a complex image.
+  =========== ======================================================
+
+  .. |f2d_10|  image:: images/homography_icon.jpg
+                 :height: 90pt
+                 :width:  90pt
+
+
+.. raw:: latex
+
+   \pagebreak
+
+.. We use a custom table of content format and as the table of content only informs Sphinx about the hierarchy of the files, no need to show it.
+.. toctree::
+   :hidden:
+
+   ../py_features_meaning/py_features_meaning
+   ../py_features_harris/py_features_harris
+   ../py_shi_tomasi/py_shi_tomasi
+   ../py_sift_intro/py_sift_intro
+   ../py_surf_intro/py_surf_intro
+   ../py_fast/py_fast
+   ../py_brief/py_brief
+   ../py_orb/py_orb
+   ../py_matcher/py_matcher
+   ../py_feature_homography/py_feature_homography