So I was working on a project (KnoFolio.net) with a super cool web designer dude I know (Andrew Coyle) and there was something in his user studio design that I had a bit of trouble with. His design had this image as the avatar:

and he wanted any user uploaded image to be filtered out in a similar way. At first I was all like WTF no way is that happening. I know I've blogged about image processing before but this seemed out of scope for the project. Then after a couple of weeks working on the project I got to the page that needed this and I had already decided to use Paperclip for image uploading to Amazon S3 and I knew that I could do some filtering in there with ImageMagick but had no idea how to go about making something like this.

So I took to reading up on some ImageMagick and paperclip docs; I quickly learned that I can (without writing my own image processor cause that sounds shitty) indeed use any of ImageMagick's commands to edit the image. Paperclip just calls the convert command so I can add anything to that I want. So I took to testing in terminal and quickly found that the -page command was what I wanted (Paging Dr. Faggot). It allows me to put any additional images on top of the original image; next I took to photoshop to get what I needed.

Photoshop

Andrew had kindly given me the pattern he was using for this in a psd so I took this and made it into an image that I could layer on top of a user uploaded picture to get our effect.

Layer 1 (gradient -to fill transparency- and additional spacing added for demonstration) was simply a white covering over the negative space of the image, this essentially removes all the extra stuff in the background of the image that would otherwise be outside of our splat pattern.

I then put a simple gray at about 40% opacity behind this layer so that when placed on top of the user image it would show the image but add some gray coloring. What I ended up with was (pattern -to fill transparency- and additional spacing added for demonstration):

I played around with this on the command line awhile and finally came up with the perfect command:

The holy command

convert original_image.png -resize 120x135^ -gravity center -extent 120x135 -colorspace gray -page -10-10 profile_cover.png -layers merge +repage  -flatten -transparent -resize 180x182 output.png

Which when run on an image like this one:

Outputed this:

But lets break down the command and figure out what all is happening

This first part:

convert original_image.png -resize 120x135^ -gravity center -extent 120x135 

takes original_image.png; resizes it to 120x135. The ^ tells it to fill the space even if it has to exceed one of the dimensions givien. The -gravity center tells image magick to resize from the middle of the image instead of one of the corners (I think top left is default) and the -extent 120x135 crops the image off at that size since we may have resized it to larger in the resize command before

Next I convert the user image to gray scale

 -colorspace gray 

Then I layer the top image I created in photoshop; the -10-10 places this image -10 pixels up and -10 pixles left of the original image

-page -10-10 profile_cover.png

Now I take these two images that we've placed on top of each other and merge them; the +repage is required to reset the croping frame; otherwise we'd loose those -10-10 pixels since they're technically off frame of the original image.

-layers merge +repage 

This gives us this image (Gradient border added for demonstration):

We're so close now; just need to convert whitespace to transparent space and resize down to our desired size:

+repage -resize 180x182 output.png

Put it in paperclip already!

Ok, now that this works great in terminal lets get it into paperclip. We just need to use the convert_options in our Avatar model right? You couldn't be stupider! Yes it starts there but thats not all. This is what my attachment definition looks like for a single size:

has_attached_file :attachment,

:styles => { 

      :normal => :png,

    },

:convert_options => { 

:normal => ' -resize 120x135^ -gravity center -extent 120x135 -colorspace gray -page -10-10 profile_cover.png -layers merge +repage -flatten -transparent white -resize 180x182'

}

Notice the file type png definition; this screwed me a bit. I had only tested with png24 files before so everything worked great but then after I deployed this out somebody thought it'd be fun to upload a jpg. Well it was pretty ugly since by default paperclip maintains filetype and just dumbly converted my nice transparent clean image into an ugly minotaur eating beast.

Some Caveats

As you'll notice the images I was testing are actually headshots and have no background -thats because I striped it out. The whole thing works best that way. It does still function with a background; its just not as clean. Browse the gallery and user profiles on KnoFolio.net for examples!

Due to lack of time funds and motivation I have shut down the servers that were hosting StreetsBehind.com. This post is in memoriam:

The goal of Streets Behind was to provide a statistical framework to support meaningful solutions to improving road quality by collecting, analyzing and employing route data to automatically report potholes and poor road conditions.

This was great in concept and even great in execution but it was such a huge project and after putting all of the time in to develop it we simply didn't have the energy to market it or work on it at all anymore. The city of Milwaukee met with us but didn't seem all that interested even though other places have similar systems (Boston, Canada) that frankly don't work as well and are much more expensive and invasive.




Credit to Andrew Coyle for the images

Just a quick update. I'm still working at Siide.com and its awesome! We should have some and interesting stuff coming out pretty soon.

Also, I've registerred Martigot as a software dev contracting LLC and have taken on Tanis Brush as a client. Its a very busy and interesting time for sure!

I have accepted a position at siide and will be starting there as of March. Though a lot of risk comes with this move, so does a lot of opportunity.

I am also going to continue development on some of my side projects (streets behind, REEL, this site) and hopefully pick up a couple extra projects for development to keep my head above water and make a name for myself in this field.

I recently completed a project with @mitchellhenke for a digital image processing course at Milwaukee School of Engineering. This project involves taking an image of a face and manipulating the pixels to add emotion to an otherwise emotionless image. Report follows.

Introduction

The basis of this project is that of image morphing. It has the ability to take an image, and without reduction in quality, manipulate it and make it look like something else. The entire concept can be broken down into pixel manipulation. The program takes a pixel or a group of pixels in a particular area and moves them along a specified pattern. In doing this, the new pixel mapping produces a slightly different image, though still related to the original. The application also has the ability to apply preset morph maps. Given an image of a face it can increase or decrease the amount of smile essentially changing the mood of the photographed person.
Implementation

In order to morph an image the application first needs an image to morph and a map to morph the image along. A map is more or less a series of points within the image. These points have both an original and a current position. The relation between these and the mapped points nearby are used to calculate the shift in pixels. A map is built for each pixel in the image that dictates the source pixel it should use in the new image. No correction or modification is done after morphing to smooth any artifacts from manipulating the image in this manner. The shift is calculated by each square in the grid. Within these squares, the differences in the two dimensions of the current morph point and morph points one row and column over contribute to the pixel movement. The pixel difference in the current column and row are added to the pixel difference of the row and column over, and weighted heavier as they get further from the current morph point. This process is done for each pixel in the image, which creates the aforementioned map. Once the offset map is created, the new image is built from the original image’s shifted pixels.

The next step in the application was to be able to automatically adjust the mood of a person. In order to do this, it has to be able to find the features to manipulate. Given a selection of the face, the application has a pretty good idea of what regions to find the mouth and eyes in by using the basic proportions of the human head. According to artyfactory.com, it can be assumed that the mouth is in the bottom quarter of the image as well as in the middle half. Similarly, the eyes can be found on either side of the image, within the second quarter from the top. These regions are displayed in Figure 2 below.

When the application knows where the mouth is, it can do some simple pixel manipulation in order to find the exact location. Given the fact that the space between the lips is generally straight and darker than the surrounding lips, the location of the middle of the mouth can be found be determining the longest dark line of pixels in the area that the mouth is expected to be in. It also works significantly better to boost the contrast of the image first, making the dark colors stick out even more. An example of this can be found below in Figure 3.

This method of finding the mouth does limit the types of images that will be able to be handled by these algorithms. For instance, it will not work if the person is already smiling and showing teeth, but that’s not much of an issue because it doesn’t make much sense to make that smile even larger, and it would be impossible to hide those teeth and keep the image at a high quality. The application had originally been intended to also find the eyes using edge detection but with the limited time frame this was not completed.

Once the application knows where the mouth is, it then needs to calculate a map of where to move the pixels. In the example of making the mouth smile the map is generated to stretch the mouth while simultaneously bending it downward, this gives a morph mesh like the one in Figure 4.

The application then applies this mesh using the same method as when working with a mesh on the entire image.

A similar effect is done to make a person appear to be frowning:

Conclusion

Setting out to implement a mesh-warp algorithm required a lot of research and testing to ensure a reliable process. Much was learned, especially the importance of simplification and minimization of assumptions in the area of image processing. The current implementation allows for basic manual morphing and some automated processes using facial detection and performs with very few and small limitations.
Recommendations

The project was quite daunting initially, but by breaking it down into its components allowed for a clear separation of tasks. Each of the subsystems can be modified slightly to improve the operation of the system. For future work, a lot could be done to expand the current set of feature detection. By adding detection and manipulation of more facial features, the project could produce even more impressive output.

Bibliography

Crane, R. (1997). A Simplified Approach to Image Processing: Classical and Modern Techniques in C. Upper Saddle River, NJ: Prentice Hall PTR.
MacTaggart, J. (n.d.). Proportions of a Head. Retrieved January 2012, from Arty Factory: http://www.artyfactory.com/portraits/drawing_techniques/proportions_of_a_head_1.htm