-
Notifications
You must be signed in to change notification settings - Fork 222
Ch 24 - Add edge detection section #177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 82 commits
Commits
Show all changes
84 commits
Select commit
Hold shift + click to select a range
1bbfb8a
Start converting prototype to react
jasmaa 1e49a41
Finish convolution demo
jasmaa b75bd59
Improve UI, refactor code
jasmaa cbe0ee4
Add sobel image processing demo
jasmaa b048f1a
Improve sobel ui
jasmaa 87778c9
Add gradient initial explanations
jasmaa 3d99a57
Swap in image processing funcs in convolution demo, improve doc
jasmaa 2c6de14
Fix grid input bug, generalize grid input ui
jasmaa cf7e77e
Start gradient demo
jasmaa 2d963a9
Fix and improve gradient demo
jasmaa 02d8d4c
Add topology demo
jasmaa 967190a
Try to improve mobile support
jasmaa 383756a
Make convolution demo more clear, replace max spread with reduce
jasmaa 2a9a53c
Style changes, fix gradient drawing
jasmaa 32901c3
Start suppression demo
jasmaa f37d358
Start pipeline demo
jasmaa 5dda712
Add image uplaod and indicator to pipeline demo
jasmaa 6c06ab1
Add movement control, refactor code
jasmaa 21ce7db
UI and documentation improvements
jasmaa eaaec72
Start moving over explanations
jasmaa 5f910f3
Replace third party media, fix grid input update bug
jasmaa b95f853
Implement remaining edge detection processes
jasmaa de86cc2
Tweaking explanations and reworking on convolution demo
jasmaa 0898a19
Fix image processing bugs
jasmaa fbc3292
Start reworking gradient demo
jasmaa 7e1c619
Add control UI to gradient demo
jasmaa c8771d9
Start replacing 3dcss with threejs in pipeline demo
jasmaa 33f6eb0
Fix material update bug
jasmaa 359aa9b
Re-implement pipeline demo with three
jasmaa 3fc7e5d
Improve pipeline demo code and docs
jasmaa e0bd01f
Rework suppression demo
jasmaa c6d5b9b
Minor bug fixes, improve suppression demo
jasmaa fabb2e3
Add threshold section
jasmaa dc719d5
Add hysteresis and canny sections
jasmaa 4512f83
Split into multiple pages, modify header injection
jasmaa ed46d47
Reworking convolution demo
jasmaa 323575c
Reworking convolution topology demos
jasmaa 4ee7950
Testing sticky footer, add breadcrumbs in injection
jasmaa fb296a1
Rewrite explanations
jasmaa a087da8
Rebuild vis with gray colorscale for surface
jasmaa d0f0d5b
Add opacity and boldness changes for gradient vectors
jasmaa 72f2afd
Combine pillar and ramp diagrams in convolution demo
jasmaa f6637ca
Add out of bounds for convolution demo
jasmaa 92fdba4
Extend 2d pipeline demo
jasmaa 61fa32c
Add RGB demo
jasmaa 05f512b
Add grayscale demo, improve readability
jasmaa 043e9c6
Minor UI changes
jasmaa bcf1485
Start adding webcam
jasmaa cd02bb9
Clean up pipeline code and add webcam
jasmaa dbc1d57
Add pattern selection for sobel pipeline
jasmaa 12dd2a1
Improving webcam performance
jasmaa 7e5c286
Add pixel magnifier, fix image upload bug
jasmaa c570993
Change sobel colorscale to red-green
jasmaa 5bc5fe1
Rehaul text for robust edge detection
jasmaa 2aa9265
Improving mobile support
jasmaa dd020cf
Minor changes to suppression demo
jasmaa 4170a2f
Separate magnifier component
jasmaa 9a00aa4
Disable nav to section 3, pipeline improvements
jasmaa be66ec4
Modularize and lazy load demos
jasmaa ca07880
Minor changes to webcam
jasmaa 217de34
Start condensing convolution demo
jasmaa 88b8bc7
Minor improvements to convolution magnifier, add loading fallback
jasmaa bf1b0e8
Reorganize text and add noise demo
jasmaa 899d920
Add default image reset, minor ui changes
jasmaa e2033f2
Try remove modules for mobile
jasmaa b9205aa
Fixing touch and sizing for mobile
jasmaa 7cacfb1
Convert grids to canvas
jasmaa 16bf6f7
Fixing mobile bugs
jasmaa 4002101
Fixing touch scrolling bug
jasmaa 337c19b
Fixing mobile rendering bug
jasmaa 05cf27f
Minor ui changes
jasmaa 0d1afdc
Improve convolution demo
jasmaa c7c63ee
Fix css on topology demo
jasmaa f9049ea
Replace RGB demo
jasmaa 6197016
Add blurbs, rewrite grayscale func and rgb section
jasmaa e5ca541
Minor style changes
jasmaa b47dfd1
Remove unused code and assets
jasmaa 4afd532
Merge edge detection to one page
jasmaa 795be97
Merge filters in convolution demo
jasmaa 9a9bec9
Add collapsable section, clean up text
jasmaa 20673d4
Fix for mobile
jasmaa 2343490
Make requested changes
jasmaa 57c5136
Revert global changes
jasmaa 8fcc95f
Fix diff problems
jasmaa File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,256 @@ | ||
| <!DOCTYPE html> | ||
| <html lang="en"> | ||
|
|
||
| <head> | ||
| <meta charset="UTF-8"> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" /> | ||
| <title>Edge Detection - Introduction</title> | ||
|
|
||
| <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.8.2/css/all.css" | ||
| integrity="sha384-oS3vJWv+0UjzBfQzYUhtDYW+Pj2yciDJxpsK1OYPAYjqT085Qq/1cq5FLXAZQ7Ay" crossorigin="anonymous"> | ||
| <link rel="stylesheet" href="https://code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css"> | ||
| <link rel="stylesheet" href="/styles.css"> | ||
| <link rel="stylesheet" href="./style.css"> | ||
|
|
||
| <script src="https://code.jquery.com/jquery-1.12.4.js"></script> | ||
| <script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script> | ||
|
|
||
| <script type="text/javascript" src="/main.js"></script> | ||
| </head> | ||
|
|
||
| <body class="flex-content-container"> | ||
| <div class="flex-content"> | ||
|
|
||
| <div class="content-container" id="content"> | ||
| <div style="display: flex; flex-direction: column; row-gap: 1em;"> | ||
|
|
||
| <h1>Edge Detection</h1> | ||
|
|
||
| <p> | ||
| For many of us, it is easy to find both the objects in a picture and their boundaries. Humans are | ||
| very good at identifying what something is and where it is given a clear line of sight, but what if | ||
| we need to find the boundaries of many objects in an enormously large number photos? This is too | ||
| much work for one person. Why don't we just make a computer find the edges for us instead? | ||
| In computer vision, this is a general problem known as <strong>edge detection</strong>. | ||
| </p> | ||
|
|
||
| <p> | ||
| Formally, edge detection is the process of detecting the boundaries in an image that separate great | ||
| changes in brightness or <strong>intensity</strong>. One of the reasons for doing edge detection is | ||
| that it is able to reduce a complex image down to a simpler one where only edges are marked. | ||
| Because of this, edge detection often acts as a preliminary step to other computer vision processes. | ||
| </p> | ||
|
|
||
| <div id="pipeline2d-direct-root"></div> | ||
|
|
||
| <h2>Image Representation</h2> | ||
|
|
||
| <p> | ||
| Before jumping into edge detection, let's go over how computers represent images. Computers | ||
| generally store images as grids of values or <strong>pixel arrays</strong>. A color image, for | ||
| instance, is decomposable into three pixel arrays storing the RGB color channels. While RGB is | ||
| convenient for representing color, edge detection is only interested in looking at intensity values. | ||
| Fortunately for us, it is possible to directly compute intensity from RGB values, giving us a pixel | ||
| array of grayscale intensities to start working with. | ||
| </p> | ||
|
|
||
| <div id="rgb-root"></div> | ||
|
|
||
| <div class="blurb-container"> | ||
| <h4><strong>From RGB to Intensity</strong></h4> | ||
| <p> | ||
| The conversion from RGB to intensity is a somewhat involved process that delves into color | ||
| science and human perception. While RGB is a helpful model, in reality, humans do not perceive | ||
| color linearly or equally among the three channels. To convert RGB to intensity, each RGB value | ||
| must first be mapped into a linear color space. These values are then put through a weighted sum | ||
| and converted back to our non-linear RGB color space for display. | ||
| </p> | ||
| </div> | ||
|
|
||
| <h2>Finding Edges</h2> | ||
|
|
||
| <p> | ||
| How do we begin searching for edges? We need some natural intuition for what an edge is. | ||
| To aid us in this question, let us take a step back and re-imagine our problem. | ||
| Suppose our pixel array is put into 3D space as a topological map where the intensity of | ||
| each pixel becomes some height. Looking at the example below, it becomes clear that edges are | ||
| analogous to sharp drops and inclines on this map. | ||
| </p> | ||
|
|
||
| <div id="topology-root"></div> | ||
|
|
||
| <p> | ||
| How do we locate areas with these sharp changes in height? Let's pick some point on this map. | ||
| Knowing our point’s height alone is relatively unhelpful since although it may tell us how | ||
| high up it is, we get no information about how the height has changed around it. What if, in | ||
| addition, we also considered the heights of points a small radius away from our chosen point? | ||
| Ah-hah! Now by comparing these nearby heights, we can get some measurement of the height changes | ||
| around our point! | ||
| </p> | ||
|
|
||
| <img src="./images/elevation.png" alt="Elevation comparison example" class="center display-image"> | ||
|
|
||
| <p> | ||
| This intuition is the driving principle behind a well-known edge detector called the <strong>Sobel | ||
| operator</strong>. The operator works by dragging two <strong>kernels</strong> or | ||
| <strong>filters</strong> called Sobel X and Sobel Y across an image to calculate horizontal and | ||
| vertical intensity changes respectively. For each target pixel in the image, the kernel selects and | ||
| performs an operation on a neighborhood of surrounding pixels. This compares the intensity of the | ||
| neighboring pixels with each other and uses them to compute the change in intensity at the target | ||
| pixel. To demonstrate this, below is an interactive example using the Sobel X | ||
| and Sobel Y filters. Drag the filters around and see how different areas reflect different | ||
| intensity changes. | ||
| </p> | ||
|
|
||
| <div id="convolution-root"></div> | ||
|
|
||
| <div class="blurb-container"> | ||
| <h4><strong>The Math Behind the Filter</strong></h4> | ||
| <p> | ||
| Under the hood, filters are grids of numerical weights that are matched up to each neighborhood | ||
| of pixels. The filter weights and the pixel intensities are used to calculate a weighted sum | ||
| that represents the change at each target pixel. This process of dragging a grid across | ||
| a larger grid is called <strong>cross correlation</strong>. Doing this with a flipped filter is | ||
| a slightly different process called <strong>convolution</strong>. In general, we prefer doing | ||
| convolutions because of their nicer properties, such as being able to | ||
| compose a series of convolutions into a single convolution. | ||
| </p> | ||
| </div> | ||
|
|
||
| <h2>Change as a Vector</h2> | ||
|
|
||
| <p> | ||
| The end result of applying both the Sobel X and Sobel Y filters to our image is two new pixel arrays | ||
| containing values that represent horizontal and vertical changes respectively. These values can also | ||
| be thought of as the X and Y components for a grid of 2D arrows or <strong>vectors</strong>. | ||
| The vectors gotten using the Sobel operator are known as <strong>gradients</strong>. For a given | ||
| pixel, the magnitude of the gradient tells us how much overall intensity change occurs at that pixel | ||
| and the angle tells us the general direction of that intensity change. | ||
| </p> | ||
|
|
||
| <p> | ||
| Below is a demonstration that illustrates these gradients. Draw edges or select one of the preset | ||
| images and observe how the direction and magnitude of the gradient vectors change in response. | ||
| </p> | ||
|
|
||
| <div id="gradient-root"></div> | ||
|
|
||
| <p> | ||
| So far, our algorithm consists of taking a color image, converting it to grayscale intensities, | ||
| and applying the Sobel operator to it to find the image's gradients. Observing the magnitudes | ||
| of these gradients at the end of the pipeline below, we can already begin to see edges manifest | ||
| as brighter pixels where intensity change is strong. | ||
| </p> | ||
|
|
||
| <div id="pipeline2d-short-root"></div> | ||
|
|
||
| <h2>Problems and Shortcomings</h2> | ||
|
|
||
| <p> | ||
| Plugging images through this algorithm, you may have already noticed that Sobel by itself | ||
| does not perform very well. Our edge detection is not very good at producing clean and precise | ||
| edges, nor is it very good at dealing with grainy images. | ||
| </p> | ||
|
|
||
| <div id="pipeline2d-grainy-root"></div> | ||
|
|
||
| <p> | ||
| To make edge detection robust, several pre and post-processing steps are required along with Sobel. | ||
| In fact, our current algorithm with these additional steps was formulated in 1986 by John Canny and | ||
| is what we know as <strong>Canny edge detection</strong>. The full Canny edge detection | ||
| pipeline can be seen below. | ||
| </p> | ||
|
|
||
| <div id="pipeline2d-long-root"></div> | ||
|
|
||
| <p> | ||
| Since the advent of Canny edge detection, many more methods of edge detection have been | ||
| proposed and experimented with such as second-differential approaches to looking at intensity | ||
| or physics-based approaches to name a few. | ||
| </p> | ||
|
|
||
| <h2>Summary</h2> | ||
|
|
||
| <p> | ||
| Edge detection is the problem of locating the boundaries among objects in a scene. The core | ||
| principle behind locating edges is to look for areas of sharp intensity change which can be achieved | ||
| with the Sobel operator. While this is a start to finding edges, there are many more steps involved | ||
| in producing clean and reliable edges. The culmination of these steps is an early edge detection | ||
| algorithm called Canny edge detection which still remains in popular use today. | ||
| </p> | ||
| </div> | ||
| </div> | ||
|
|
||
| </div> | ||
|
|
||
| <script crossorigin src="https://unpkg.com/react@16/umd/react.development.js"></script> | ||
| <script crossorigin src="https://unpkg.com/react-dom@16/umd/react-dom.development.js"></script> | ||
| <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/106/three.js"></script> | ||
| <script src="/third-party/vis.min.js"></script> | ||
|
|
||
| <script type="text/javascript" src="./js/setup.js"></script> | ||
|
|
||
| <script type="text/javascript" src="./js/util.js"></script> | ||
| <script type="text/javascript" src="./js/imageProcessing.js"></script> | ||
| <script type="text/javascript" src="./js/ui.js"></script> | ||
| <script type="text/javascript" src="./js/demos/pipeline2d/pipelineHelpers.js"></script> | ||
|
|
||
| <script type="text/javascript" src="./js/demos/pipeline2d/Pipeline2dDirectDemo.js"></script> | ||
| <script type="text/javascript" src="./js/demos/pipeline2d/Pipeline2dRGB.js"></script> | ||
| <script type="text/javascript" src="./js/demos/pipeline2d/Pipeline2dGrayscaleDemo.js"></script> | ||
| <script type="text/javascript" src="./js/demos/TopologyDemo.js"></script> | ||
| <script type="text/javascript" src="./js/demos/ConvolutionDemo.js"></script> | ||
| <script type="text/javascript" src="./js/demos/GradientDemo.js"></script> | ||
| <script type="text/javascript" src="./js/demos/pipeline2d/Pipeline2dShortDemo.js"></script> | ||
| <script type="text/javascript" src="./js/demos/pipeline2d/Pipeline2dLongDemo.js"></script> | ||
| <script type="text/javascript" src="./js/demos/pipeline2d/Pipeline2dGrainyDemo.js"></script> | ||
|
|
||
| <script> | ||
| ReactDOM.render( | ||
| e(Pipeline2dDirectDemo, null, null), | ||
| document.getElementById('pipeline2d-direct-root') | ||
| ); | ||
|
|
||
| ReactDOM.render( | ||
| e(Revealer, { | ||
| baseId: 'pipeline2d-rgb-collapse', | ||
| promptText: 'See a Demo' | ||
| }, | ||
| e(Pipeline2dRGB, null, null), | ||
| ), | ||
| document.getElementById('rgb-root') | ||
| ); | ||
|
|
||
| ReactDOM.render( | ||
| e(TopologyDemo, null, null), | ||
| document.getElementById('topology-root') | ||
| ); | ||
|
|
||
| ReactDOM.render( | ||
| e(ConvolutionDemo, null, null), | ||
| document.getElementById('convolution-root') | ||
| ); | ||
|
|
||
| ReactDOM.render( | ||
| e(GradientDemo, null, null), | ||
| document.getElementById('gradient-root') | ||
| ); | ||
|
|
||
| ReactDOM.render( | ||
| e(Pipeline2dShortDemo, null, null), | ||
| document.getElementById('pipeline2d-short-root') | ||
| ); | ||
|
|
||
| ReactDOM.render( | ||
| e(Pipeline2dLongDemo, null, null), | ||
| document.getElementById('pipeline2d-long-root') | ||
| ); | ||
|
|
||
| ReactDOM.render( | ||
| e(Pipeline2dGrainyDemo, null, null), | ||
| document.getElementById('pipeline2d-grainy-root') | ||
| ); | ||
| </script> | ||
| </body> | ||
|
|
||
| </html> |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
revert?