kenfrederick: processing. triangulation + meshes

processing. triangulation + meshes

i've been slowly experimenting with meshes (mostly with triangulation algorithms), or better said trying to understand how to better control them. it started out innocent enough and has slowly progressed. after deconstructing jonathan puckey's delaunay raster scriptographer work, i created my own version and began modifying it. i expanded upon my own version by creating a version which maps brightness values of an area to line density (instead of colors).

karl pilkington, triangulated

although proud of my achievement, the entire thing is something that jonathan is exploring, but honestly without a lot of conceptual thought i kept trudging aimlessly on. i thought, well i'll take this idea to live and moving images (recently the internet has proved to me that everything has been done, with this and this) my explorations with my own puckey-delaunay-raster script proved the best illustrations come from getting in the details of the face, so when i switched to processing for moving images i created a rudimentary edge detection class which pulls out the edges as PVector arraylist.

the first attempt was terribly slow, but worked more or less. i contemplated and subsequently failed to port it over to openFrameworks (i couldn't get the edge detection to work). i realized even if the concept is nothing new i could at least learn more about optimization and harnessing opengl VBOs (vertex buffer objects). and eventually i developed a version which takes web cam input, edge detects, and triangulates all the while maintaining a frame rate of 30fps+. a z depth (height map) is generated from the brightness of the edge-detected-pixel, and by adding a multiplier an interesting "face landscape" occurs (especially when using the smooth gradient mode).

pgl.beginGL();

if(triangles.size() != 0) {
if(mode == 0 || mode == 1) {
gl.glBegin(GL.GL_TRIANGLES);

float x1,y1,z1, x2,y2,z2, x3,y3,z3;
float px1,py1,pz1, px2,py2,pz2, px3,py3,pz3;;
float r,g,b,a, r2,g2,b2;

if(mode == 0) {
/**
* smooth gradients
*/
for(int i=0; i<TriangleVBOPoints.length; i+=3) {
x1 = TriangleVBOPoints[i];
y1 = TriangleVBOPoints[i+1];
z1 = TriangleVBOPoints[i+2];

int c1 = img.pixels[ ((int)y1*capture.width) + ((int)x1) ];
r = norm((c1 >> 16) & 0xff, 0,255);
g = norm((c1 >> 8) & 0xff, 0,255);
b = norm(c1 & 0xff, 0,255);
if(bOpaque) a = 1.0;
else a = norm(FTools.luminance(c1), 0,255);

gl.glColor4f(r,g,b, a);
gl.glVertex3f(x1,y1,z1*zHeight);
}

} else if(mode == 1) {
/**
* flat colors
*/
for(int i=0; i<TriangleVBOPoints.length; i+=9) {
x1 = TriangleVBOPoints[i];
y1 = TriangleVBOPoints[i+1];
z1 = TriangleVBOPoints[i+2];

int c1 = img.pixels[ ((int)y1*capture.width) + ((int)x1) ];
r = norm((c1 >> 16) & 0xff, 0,255);
g = norm((c1 >> 8) & 0xff, 0,255);
b = norm(c1 & 0xff, 0,255);
if(bOpaque) a = 1.0;
else a = norm(FTools.luminance(c1), 0,255);

gl.glColor4f(r,g,b, a);
gl.glVertex3f(x1, y1, z1*zHeight);

x2 = TriangleVBOPoints[i+3];
y2 = TriangleVBOPoints[i+4];
z2 = TriangleVBOPoints[i+5];
gl.glVertex3f(x2, y2, z2*zHeight);

x3 = TriangleVBOPoints[i+6];
y3 = TriangleVBOPoints[i+7];
z3 = TriangleVBOPoints[i+8];
gl.glVertex3f(x3, y3, z3*zHeight);

}

}

gl.glEnd();

}

pgl.endGL();

i really liked this valley effect when increasing the z depth, and i thought it would be cool if i could start to distort the image by "folding" and "crumpling" it. similar to the pop faces work by joshua scott.

then i began thinking it would be even better to do this directly in illustrator using scriptographer after porting my edge detection code over to javascript, and creating the necessary 3d infrastructure i ran into challenges distorting the image although i did manage to create a neat kaleidoscope effect, which wasn't exactly what i was going for (although, that could lead to something)

i've pretty much abandoned the idea of achieving what i want in illustrator and have gone back to processing. using particles and physics i can distort a blank mesh of lines, however, i cannot deform an image. i can get the image mapped as a texture to a mesh, but as i distort the mesh, the image gets clipped (as if in a mask). this probably stems from me calculating the texture coordinates incorrectly.

all in all, i'm happy with what i've learned especially better understanding direct opengl calls and optimization, but i'm frustrated that i've hit this stumbling block. i think i'll probably put this idea on ice and focus on some other things (that require less programming) and maybe loop back around in the future.

Labels: processing, scriptographer