Canvas pixels #2: convolution matrix

In the previous post I talked about manipulating and changing pixels in an image (using JavaScript and canvas) one at a time. We took a single pixel and messed around with its R, G, B or A values.

This time let's look into taking account not only the single pixel but the pixels around it. This allows you to do all kinds of effects, the most popular being emboss, edge detection, blur and sharpen.

The demo page is here

Theory

The type of manipulation we'll consider is called image convolution using a 3x3 matrix. You take 9 pixels from the image: the current pixel you're changing and the 8 immediately around it.

In other words you want to change the RGB values for the pixel in the middle based on its own value and those around it.

Let's say we have some sample values (given in red for R, blue for B and green for G in this figure):

Remember this manipulation was called convolution matrix. So you need a matrix. Below is an example of one such matrix (used in the blur effect)

Now you take one of the channels, say R for example. You take each of the 9 R values you have and multiply it by the corresponding number in the matrix. Then sum the nine numbers.

1*1 + 2*2 + 5*1 + 11*2 + 10*4 + 20*2 + 1*1 + 10*2 + 1*1 =
 1  +  4  + 5   +   22 +  40  +  40  +  1  +  20  +  1  =
                      134 

In addition to the matrix we also have a divisor and an offset, both optional. If there's no divisor (meaning it's 1, not 0), the result for Red we're looking for is 134. As you can see 134 is pretty far off from the original value of 10. But the blur effect has a divisor of 16. So the new value for red is 8.375

If the convolution asked for an offset, you add it to the end result.

Then you repeat the same for Green and Blue. You can do alpha if you want but for regular images it has constant 255 value so you'll do a lot of math and end up with 255.

You may have noticed that the divisor 16 is also the sum of the numbers in the matrix;

1 + 2 + 1 + 2 + 4 + 2 + 1 + 2 + 1 = 16

This way the result image is as bright as the original. If you have an unbalanced matrix you'll get a darker or a lighter image.

The offset is 0 most of the time, but not always. The emboss effect has offset 127 for example.

Demo matrices

My demo uses the most popular matrices out there. You can search the web for other matrices and play with them. None of them define a divisor because it's the sum of their elements, but the API I'll show you lets you use your custom divisor.

Without further ado, here are the matrices I used defined as an array of JavaScript objects:

var matrices = [
  {
    name: 'mean removal (sharpen)',
    data:
     [[-1, -1, -1],
      [-1,  9, -1],
      [-1, -1, -1]]
  },
  {
    name: 'sharpen',
    data:
     [[ 0, -2,  0],
      [-2, 11, -2],
      [ 0, -2,  0]]
  },
  {
    name: 'blur',
    data:
     [[ 1,  2,  1],
      [ 2,  4,  2],
      [ 1,  2,  1]]
  },
  {
    name: 'emboss',
    data:
     [[ 2,  0,  0],
      [ 0, -1,  0],
      [ 0,  0, -1]],
    offset: 127,
  },
  {
    name: 'emboss subtle',
    data:
     [[ 1,  1, -1],
      [ 1,  3, -1],
      [ 1, -1, -1]],
  },
  {
    name: 'edge detect',
    data:
     [[ 1,  1,  1],
      [ 1, -7,  1],
      [ 1,  1,  1]],
  },
  {
    name: 'edge detect 2',
    data:
     [[-5,  0,  0],
      [ 0,  0,  0],
      [ 0,  0,  5]],
  }
];

Results

Original

Blur

Sharpen

Edge detect

Edge 2

Emboss

Emboss (subtle)

Mean removal (sharpen a lot)

The API

The API is the same as in the previous post, same constructor and all, just adding a new method called convolve(). This is where the magic happens.

You use this method like so:

transformador.convolve([
  [1,2,1],
  [2,4,2],
  [1,2,1]
], 16, 0);

Again, 16 is optional as the method will figure it out if you omit and offset is optional too. Actually you can go to the demo and play in the console to see what happens with a different divisor, e.g.

transformador.convolve([[1,2,1],[2,4,2],[1,2,1]], 10);

or

transformador.convolve([[1,2,1],[2,4,2],[1,2,1]], 20);

convolve()

Some comments on how convolve() was implemented in this demo.

The big picture:

CanvasImage.prototype.convolve = function(matrix, divisor, offset) {
  // ...
};

Handle arguments: flat matrix is easier to work with and figure out the divisor if missing. How 'bout that array reduce, eh? ES5 ftw.

  var m = [].concat(matrix[0], matrix[1], matrix[2]); // flatten
  if (!divisor) {
    divisor = m.reduce(function(a, b) {return a + b;}) || 1; // sum
  }

Some vars more or less the same as the last time in the transform() method:

  var olddata = this.original;
  var oldpx = olddata.data;
  var newdata = this.context.createImageData(olddata);
  var newpx = newdata.data
  var len = newpx.length;
  var res = 0;
  var w = this.image.width;

Then a loop through all the image data, filter out every 4th element (because we ignore Alpha channel) and write the new image data to the canvas.

  for (var i = 0; i < len; i++) {
    if ((i + 1) % 4 === 0) {
      newpx[i] = oldpx[i];
      continue;
    }
 
    // 
    // magic...
    //
  }
  this.setData(newdata);

Remember that canvas image data is one long array where 0 is R for pixel #1, 1 is B, 2 is G, 3 is Alpha, 4 is R for pixel #2 and so on. This is different than more other code examples you'll in different languages where there are two loops in order to touch every pixel: one from 0 to width and an inner one from 0 to height.

And finally, the "magic" part:

    res = 0;
    var these = [
      oldpx[i - w * 4 - 4] || oldpx[i],
      oldpx[i - w * 4]     || oldpx[i],
      oldpx[i - w * 4 + 4] || oldpx[i],
      oldpx[i - 4]         || oldpx[i],
      oldpx[i],
      oldpx[i + 4]         || oldpx[i],
      oldpx[i + w * 4 - 4] || oldpx[i],
      oldpx[i + w * 4]     || oldpx[i],
      oldpx[i + w * 4 + 4] || oldpx[i]
    ];
    for (var j = 0; j < 9; j++) {
      res += these[j] * m[j];
    }
    res /= divisor;
    if (offset) {
      res += offset;
    }
    newpx[i] = res;

these are the pixels we want to inspect. oldpx[i] is the one in the middle which we're changing to newpx[i]. Also note how we default all pixels to oldpx[i]. This is to deal with the boundary pixels: tho top and bottom rows of pixels and the left and right columns. Because the pixel in position 0x0 has no pixels above it or to the left. Then we loop through these and multiply by the corresponding value in the matrix. Finally divide and offset, if required.

Thanks!

Thanks for reading, and now go play with the demo in the console. An easy template to start is:

transformador.convolve([[1,0,0],[0,0,0],[0,0,-1]], 1, 127); 

If you want to apply convolutions on top of each other, you can reset the original image data to the current.

transformador.original = transformador.getData();

Tell your friends about this post on Facebook and Twitter