## Archive for September, 2008

### Automatic lip synch.

I wasn’t originally going to talk about this until I had something to show, but my next AIR project was going to be something like my old e2animate application that I talked about in my previous post.

I was going to aim it at non-technical users. Make it fun. Easy to use. Allow them to access an online library of clip-art. Plus I was going to incorporate a very powerful and special feature. Automatic lip synchronisation. Just drop a mouth shape into an animation, and it would automatically synchronise to a sound track.

However, someone has beat me to it with an AS3 lip-synch implementation. Samir has created a Flash component that he’s talking about licensing for free.

I am pleased that someone has possibly saved me all the hard work. But I was quite looking forward to the technical challenge of doing this myself.

I don’t know the details of Samir’s algorithm, but this is how I was going to do it…

The first thing is to determine the shape of the mouth from each segment of speech. I was going to look at two possible ways of doing this. Prediction gain, and frequency bin matching.

Predication gain is the power of the signal coming out a filter, divided by the power of the signal going in. If the filter matches the characteristics of the signal, then we get a high value (because we don’t lose much power), but if it doesn’t match, we get a low value, because the filter blocks out more of the signal.

So, imagine a bank of fixed filters, each one detects a particular sound. (A,I/E/U/L/W,Q/M, fricatives, etc. – possibly also taking different kinds of voices into account). Depending which filter gives us the best match, we select a particular mouth shape.

Frequency bin matching is like matching the power in parts of the frequency spectrum to known sets of values, again corresponding to different kinds of sound.

MP3 is a sub-band coder. An MP3 file actually contains these frequency bin values. Or they can be derived from an FFT (SoundMixer.computeSpectrum()).

I also heard that pixelbender could be used to process sound, and I was going to investigate this, and anything else Cosmo had.

Anyway, it was probably going to take a little fiddling around, possibly some time-based signal processing too, but I’m sure something based on these methods could automatically generate a mouth shape. Then, the size of the mouth is just proportional to the amplitude/power of the speech signal.

And that’s how I was going to do dynamic lip synch in ActionScript.

### Flash CS4 tweening. Nothing new.

It’s funny how technology tends to recycle old ideas. Like Macromedia Director style animation paths and tweening, now recycled into Flash CS4. Don’t get me wrong, Flash CS4 looks pretty cool, and introduces a whole bunch of stuff that Director never had.

In fact, animation paths are such a cool idea, that I used them a few years ago in a Central application. I wrote a prototype animation application called e2animate.

I’d like to recycle it to work on AIR sometime.

The pictures below shows how similar the animation paths in e2animate were to Flash CS4. See how I used bezier curve paths that could be edited on the stage.

Some people have speculated that Adobe will eventually port versions of its software (Flash, etc.) to AIR, like it has already done with PhotoShop.

Well, back in the day, before AIR, before Flex – developers like myself were already beginning to move in that direction.

### Overcoming rounding errors

Floating point variables are prone to rounding errors. (That is, the Number type in ActionScript) Often these don’t matter. But in some applications, especially those that process financial information, it is important to avoid errors. Especially errors that propagate. I am interested in eliminating rounding errors because of my spreadsheet AIR application. e2spreadsheet. The current beta version doesn’t do anything about these errors, but I want to remedy that in the next version.

For example, if you open up e2spreadsheet, and type in the following formula: =74.725*100 what answer do you expect? 7472.5 right? Well, the answer that you actually get is 74.499999999999

Because the binary representation has a finite length, it cannot accurately represent some decimal values.

So, how do we get around this? Well, my first thought was a number representation called Binary Coded Decimal (BCD). I think Cobol used to use this representation. But that would have involved many program operations to simulate each BCD calculation. Then this blog by Josh Tynjala gave me another idea. (Ok, probably not a new idea – but it was a revelation to me).

What if I represent my numbers using a radix-10 (decimal) mantissa and exponent.

Actually floating point numbers are represented internally in binary mantissa-exponent form. So that…

value=mantissa*Math.pow(2,exponent);

In other words, the mantissa is shifted left or right depending on the value of the exponent.

Rounding errors occur because finite fractional binary numbers don’t translate well into decimal numbers. So the solution is to simulate this mantissa-exponent form in decimal (radix-10). We would write a class so that value=mantissa*Math.pow(10,exponent); And this class would contain all the methods necessary to do arithmetic operations on this representation. Add, Subtract, Multiply, etc..

So, with that idea in my head, I wrote the following code. (previous blog entry which appears below). Don’t rely on this yet. I wrote and tested it quickly, and I wanted to see what other developers think about the idea. Is this a good way to eliminate rounding errors? Or is there a better way I haven’t considered?

### Overcoming rounding errors (code)

```
package { // Decimal.as by Daniel Freeman http://www.e2easy.co.cc
public class Decimal {
private var mantissa:int;
private var exponent:int;
private var fixed:Boolean=false;
public function Decimal(mantissa:int=0,exponent:int=0) {
this.mantissa=mantissa;
this.exponent=exponent;
}
public function convert(float:Number):void {
var strng:String=float.toString();
var pnt:int=strng.indexOf('.');
if (pnt<0) {
mantissa=float;
exponent=0;
normalise();
} else {
exponent=pnt-strng.length+1;
mantissa=Math.round(float*Math.pow(10,-exponent));
}
fixed=false;
}
public function converttoprecision(float:Number,exponent:int):void {
this.exponent=exponent
mantissa=Math.round(float*Math.pow(10,-exponent));
fixed=true;
}
private function normalise():void {
if (!fixed && mantissa!=0) {
while (mantissa % 10 == 0) {
mantissa=mantissa/10;
exponent++;
}
}
}
public function add(num:Decimal,minus:Boolean=false):Decimal {
var ret:Decimal=new Decimal();
var m0:int=mantissa;
var m1:int=num.mantissa;
var sum:int;
if (exponent<num.exponent) {
ret.exponent=exponent;
m1=m1*Math.pow(10,num.exponent-exponent);
} else if (exponent>num.exponent) {
ret.exponent=num.exponent;
m0=m0*Math.pow(10,exponent-num.exponent);
} else ret.exponent=exponent;
ret.mantissa=minus ? m0-m1 : m0+m1;
ret.fixed=fixed && num.fixed;
ret.normalise();
return ret;
}
public function subtract(num:Decimal):Decimal {
return add(num,true);
}
public function multiply(num:Decimal):Decimal {
var ret:Decimal=new Decimal(mantissa*num.mantissa,exponent+num.exponent);
ret.fixed=fixed && num.fixed;
ret.normalise();
return ret;
}
public function toString():String {
var ret:String;
var sign=mantissa<0;
var pos:int,i:int;
if (sign) mantissa=-mantissa;
ret=mantissa.toString();
if (exponent<0) {
pos=ret.length+exponent;
if (pos<0) {
for (i=0;i<-pos;i++) ret='0'+ret;
ret='0.'+ret;
} else {
ret=ret.substring(0,pos)+'.'+ret.substring(pos,ret.length);
if (pos==0) ret='0'+ret;
}
} else if (exponent>0) {
for (i=0;i<exponent;i++) ret+='0';
}
if (sign) {ret='-'+ret;mantissa=-mantissa}
return ret;
}
}
}
```

### Undo and redo

I’ve seen quite a few people on the internet ask about how to implement undo in an AIR application.

Most desktop applications enable the user to undo and redo their changes to a document. So users expect this functionality in an AIR application. The ability to undo changes is an invaluable feature to any application where the user makes changes to a document.

I’ve submitted a document to the AIR cookbook, that describes exactly how to achieve this. Plus code examples.

Click here to view my article.

Don’t forget to leave a high rating.

As well as solving some problems out there, I expect this will raise some questions. You can ask your questions here by leaving a comment, or at the AIR cookbook.

### Kap Lab launch

After months of preparation, the Kap IT team announced the launch of its lab yesterday.

The lab showcases their expertise in Flex, ActionScript, Flash, Air, LifeCycle and their strong partnership with Adobe since 2005.

They’ve developed free AS3 components for data visualisation, reporting, diagramming, development and test tools. (Just register and download).

There are some really impressive demos on their site, that use these components. I was particularly impressed by the “Diagrammer” demo. I think they should release this as an AIR app in its own right.

### The AIR Office

Years ago, when I used to talk about a rich client office suite, nobody thought it was good idea. When I demonstrated office type applications for Macromedia Central in 2004, nobody batted an eye-lid. (e2textedit, e2spreadsheet, e2draw, e2paint, and e2data).

But since the release of Adobe AIR, the idea has gained a little momentum.

High-profile applications such as BuzzWord and PhotoShop Express indicate a growing trend in this direction, as well as developing new ways for people to share and collaborate on documents.

What do you think of the idea of an AIR Office suite? How would it differ from the desktop software that you use? How would it support collaboration on documents?