Flex Tutorial - An Asynchronous JPEG Encoder

This weekend I needed to encode a rather large image as a jpeg using flex. And you know what I found? It was slow. And cpu hungry. While this was not entirely unexpected, it was disappointing. I could have dealt with the program being slow and unresponsive during the time it took to encode the image, except for one problem - after about 15 seconds, the flex app would run up against the script timeout error. You know, the error that goes along the lines of "A script has executed for longer than the default timeout period of 15 seconds." And because of this, the jpeg encode never actually finished.

Well, that was no good. I poked a bit at extending the default timeout period, but really that was only a partial solution. Who knows how long it might take to encode the image on a slow computer? So I started poking at what it would take to make the jpeg encoding asynchronous. Sadly, actionscript and flex do not have the concept of just saying "hey go do this and get back to me when your done." This is probably because actionscript is single threaded - everything just runs on the browser's main thread.

Above we have a small sample app showing off the asynchronous jpeg encoder that we are going to build in this tutorial. You see the spinning lag meter? Right now, it is probably spinning nice and smooth. This is because flex has plenty of cpu cycles to update the spinning circle. When there isn't enough cpu time available, that animation will start to look choppy, as Flex will only be able to update it once every few hundred milliseconds, possibly longer. If there are no spare cycles, the circle will freeze and stop spinning altogether. If you hit "Normal Encode", that is probably exactly what will happen, and your entire browser will freeze along with it. That is because by clicking "normal encode", you told flex to encode the image displayed in the sample app (which is a 2800x2100 pixels) as a jpeg. If your computer is slow like mine, you will eventually get the "script timeout" error, and the encoding will never finish.

On the right hand side, however, is the button to trigger the asynchronous jpeg encoding. When you click that, the app won't freeze and the circle will keep spinning (although not quite as smoothly). You will also get a progress bar that shows the progress of the encoding. By moving the "pixels per iteration" slider bar to the right, you will increase the speed of the encoding, but decrease the responsiveness of the interface, and if you move the slider to the left, you get the opposite effect. A little farther down I will explain how that is accomplished.

So how do you even go about making something like this asynchronous? As I said above, we only have one thread to work with, so it is all hopeless, right? Not quite - there are ways to act like a multi-threaded system, such as using something like setTimeout. You can use setTimeout to emulate threading - essentially do a little work, but then you queue more work for the future (not immediately). The app then has a chance to breathe and deal with things like UI input during the pauses between work items.

It was here that I ran up against another issue. How was I going to break the act of encoding a large jpeg into manageable little pieces? Well, there is no way to do that from outside the jpeg encoding object, so I went into the Flex source code and found the jpeg encoder. And this is where the real meat of this article starts.

Below we have the problem loop in the original jpeg encoder:

for (var ypos:int = 0; ypos < height; ypos += 8)
{
  for (var xpos:int = 0; xpos < width; xpos += 8)
  {
    RGB2YUV(sourceBitmapData, sourceByteArray, xpos, ypos, width, height);
    DCY = processDU(YDU, fdtbl_Y, DCY, YDC_HT, YAC_HT);
    DCU = processDU(UDU, fdtbl_UV, DCU, UVDC_HT, UVAC_HT);
    DCV = processDU(VDU, fdtbl_UV, DCV, UVDC_HT, UVAC_HT);
  }
}

This loop can take a long time on large images - or at least on my computer it does. So the essential idea here is that we don't want to do this whole loop at once. We want to process the image a chunk at a time, giving the rest of the app time to work between chunks. So how do we do this? Well, we write a function that can process the loop a chunk at a time:

private function AsyncLoop(xpos:int, ypos:int):void
{
  for(var i:int=0; i < PixelsPerIter; i++)
  { 
    RGB2YUV(Source, xpos, ypos, SrcWidth, SrcHeight);
    DCY = processDU(YDU, fdtbl_Y, DCY, YDC_HT, YAC_HT);
    DCU = processDU(UDU, fdtbl_UV, DCU, UVDC_HT, UVAC_HT);
    DCV = processDU(VDU, fdtbl_UV, DCV, UVDC_HT, UVAC_HT);

    xpos += 8;       
    if(xpos >= SrcWidth)
    {
      xpos = 0;
      ypos += 8;
    }

    if(ypos >= SrcHeight)
    {
      setTimeout(FinishEncode, 10);
      return;
    }

  }
  setTimeout(AsyncLoop, 10, xpos, ypos);
}

This AsyncLoop function takes a xpos and ypos into the image, and starts processing from that point. But it only loops PixelsPerIter times. After it has processed that many chunks, it leaves the loop, and calls setTimeout on AsyncLoop. In this setTimeout call, it hands the current x and y position in the image - so when the function is called again, it starts up in the right place.

By setting a 10 millisecond timeout, the application will have time to do other things before it gets back to processing this image data. The PixelsPerIter value is very important - it determines how responsive the application is during the image processing. If PixelsPerIter is 1, the application will seem perfectly responsive. But it will also take a long time for a large image to encode, because the encoding is pausing for 10 milliseconds between event chunk. If you set the value to, say, 10000, the app will become unresponsive, because it is taking multiple seconds to process data before it responds to any UI input. Playing around, I've found that 128 is a pretty good value - the app slows down a little bit, and it takes under 2 minutes to encode a 2880x2880 jpeg.

So what do we do here when we are done processing the pixels? Well we do a setTimeout to FinishEncode, and return out of AsyncLoop. FinishEncode holds all the code that was later than the main loop in the original encode function:

private function FinishEncode():void
{
  //EOI
  if (bytepos >= 0)
  {
    var fillbits:BitString = new BitString();
    fillbits.len = bytepos + 1;
    fillbits.val = (1 << (bytepos + 1)) - 1;
    writeBits(fillbits);
  }
  writeWord(0xFFD9);
}

Ok, well, that makes sense, but how to we signal the rest of the application that the encoding has completed? Its not like we can just wait for a function call to return. Because the encoding is asynchronous, the original encode call will return immediately - long before the actual encoding is finished. So instead, we use an event. In this case, I created my own event, because I wanted the event object to hold the encoded image:

public class JPEGAsyncCompleteEvent extends Event
{
  public static const JPEGASYNC_COMPLETE:String = "JPEGAsyncComplete";

  public var ImageData:ByteArray;

  public function JPEGAsyncCompleteEvent(data:ByteArray)
  {
    ImageData = data;
    super(JPEGASYNC_COMPLETE);
 }
}

To use this event on the new jpeg encoding class, we add an attribute at the top of the class, and make the class extend EventDispatcher:

[Event(name=JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE, type="JPEGAsyncCompleteEvent")]
public class JPEGAsyncEncoder extends EventDispatcher
{
...

And now to fire the event, we add a dispatchEvent call to the FinishEncode function:

private function FinishEncode():void
{
  //EOI
  if (bytepos >= 0)
  {
    var fillbits:BitString = new BitString();
    fillbits.len = bytepos + 1;
    fillbits.val = (1 << (bytepos + 1)) - 1;
    writeBits(fillbits);
  }
  writeWord(0xFFD9);
  this.dispatchEvent(new JPEGAsyncCompleteEvent(byteout));
}

I also wanted to add a progress event, so that I could show a progress bar to the user as the jpeg was encoding. To do this, I added another event to the class:

[Event(name=JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE, type="JPEGAsyncCompleteEvent")]
[Event(name=ProgressEvent.PROGRESS, type="flash.events.ProgressEvent")]
public class JPEGAsyncEncoder extends EventDispatcher
{
...

And with this progress event, I added a bunch of logic to the AsyncLoop code to fire the progress event at appropriate points:

private function AsyncLoop(xpos:int, ypos:int):void
{
  for(var i:int=0; i < ChunksPerIter; i++)
  { 
    RGB2YUV(Source, xpos, ypos, SrcWidth, SrcHeight);
    DCY = processDU(YDU, fdtbl_Y, DCY, YDC_HT, YAC_HT);
    DCU = processDU(UDU, fdtbl_UV, DCU, UVDC_HT, UVAC_HT);
    DCV = processDU(VDU, fdtbl_UV, DCV, UVDC_HT, UVAC_HT);

    xpos += 8;       
    if(xpos >= SrcWidth)
    {
      xpos = 0;
      ypos += 8;
    }

    if(ypos >= SrcHeight)
    {
      setTimeout(FinishEncode, 10);
      return;
    }

    CurrentTotalPos += 64;
    if(CurrentTotalPos >= NextProgressAt)
    {
      this.dispatchEvent(new ProgressEvent(ProgressEvent.PROGRESS, 
          false, false, CurrentTotalPos, TotalSize));
      NextProgressAt += PercentageInc;
    }
  }

  setTimeout(AsyncLoop, 10, xpos, ypos);
}

That piece of code actually uses a bunch of fields that I added to the class to keep track of total progress, so that the progress event gets fired only about once a percent or so.

Those are pretty much all the changes that I needed to make (barring the couple added fields). Below you can see the entirety of what I did to the code. I put "...." where code from the original jpeg encoder class would be - there is a lot of it, so I didn't want to paste it all here:

[Event(name=JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE, type="JPEGAsyncCompleteEvent")]
[Event(name=ProgressEvent.PROGRESS, type="flash.events.ProgressEvent")]
public class JPEGAsyncEncoder extends EventDispatcher
{

......

    private var DCY:Number = 0;
    private var DCU:Number = 0;
    private var DCV:Number = 0;
    private var SrcWidth:int = 0;
    private var SrcHeight:int = 0;
    private var Source:Object = null;
    private var TotalSize:int = 0; 
    private var PixelsPerIter:int = 128;
    private var PercentageInc:int = 0;
    private var NextProgressAt:int = 0;
    private var CurrentTotalPos:int = 0;
    private var Working:Boolean = false;

.....

    public function set PixelsPerIteration(val:int):void
    { PixelsPerIter = val; }

    public function get ImageData():ByteArray
    { return byteout; }

    public function encodeByteArray(raw:ByteArray, width:int, height:int):Boolean
    { return internalEncode(raw, width, height); }

    public function encode(image:BitmapData):Boolean
    { return internalEncode(image, image.width, image.height); }

    private function internalEncode(newSource:Object, width:int, height:int):Boolean
    {
      if(Working)
        return false;

      Working = true;
      Source = newSource;
      SrcWidth = width;
      SrcHeight = height;
      TotalSize = width*height;
      PercentageInc = TotalSize/100;
      NextProgressAt = PercentageInc;
      CurrentTotalPos = 0;

      setTimeout(StartEncode, 10);
      return true;
    }

    private function StartEncode():void
    {
      // Initialize bit writer
      byteout = new ByteArray();
      bytenew = 0;
      bytepos = 7;

      // Add JPEG headers
      writeWord(0xFFD8); // SOI
      writeAPP0();
      writeDQT();
      writeSOF0(SrcWidth, SrcHeight);
      writeDHT();
      writeSOS();

      DCY = 0;
      DCV = 0;
      DCU = 0;

      bytenew = 0;
      bytepos = 7;

      this.dispatchEvent(new ProgressEvent(ProgressEvent.PROGRESS, 
          false, false, 0, TotalSize));

      setTimeout(AsyncLoop, 10, 0, 0);
    }   

    private function AsyncLoop(xpos:int, ypos:int):void
    {
      for(var i:int=0; i < PixelsPerIter; i++)
      { 
        RGB2YUV(Source, xpos, ypos, SrcWidth, SrcHeight);
        DCY = processDU(YDU, fdtbl_Y, DCY, YDC_HT, YAC_HT);
        DCU = processDU(UDU, fdtbl_UV, DCU, UVDC_HT, UVAC_HT);
        DCV = processDU(VDU, fdtbl_UV, DCV, UVDC_HT, UVAC_HT);

        xpos += 8;       
        if(xpos >= SrcWidth)
        {
          xpos = 0;
          ypos += 8;
        }

        if(ypos >= SrcHeight)
        {
          setTimeout(FinishEncode, 10);
          return;
        }

        CurrentTotalPos += 64;
        if(CurrentTotalPos >= NextProgressAt)
        {
          this.dispatchEvent(new ProgressEvent(ProgressEvent.PROGRESS, 
              false, false, CurrentTotalPos, TotalSize));
          NextProgressAt += PercentageInc;
        }
      }

      setTimeout(AsyncLoop, 10, xpos, ypos);
    }

    private function FinishEncode():void
    {
      //EOI
      if (bytepos >= 0)
      {
        var fillbits:BitString = new BitString();
        fillbits.len = bytepos + 1;
        fillbits.val = (1 << (bytepos + 1)) - 1;
        writeBits(fillbits);
      }
      writeWord(0xFFD9);
      this.dispatchEvent(new ProgressEvent(ProgressEvent.PROGRESS, 
          false, false, TotalSize, TotalSize));
      this.dispatchEvent(new JPEGAsyncCompleteEvent(byteout));
      Working = false;    
    }

.......

}

The methods internalEncode, encode, and encodeByteArray replace methods in the original jpeg encoder class. Everything else shown above is an addition. I added all of those fields show at the top because unlike the original encoder (where almost everything was done in a single function call), I needed to remember things across function calls.

One thing to note, this encoder does not implement IImageEncoder like the original jpeg encoder does. This is because that interface expects the encode and encodeByteArray to return a byte array containing the encoded image. Obviously, since this class does all the encoding asynchronously, it can't return the byte array straight out of the original call, because the encoding hasn't actually been done yet. Instead, the functions return a boolean. If the call to encode returns false, it is because this instance of the class is already encoding an image - so it can't start encoding another one yet.

So we have this awesome asynchronous jpeg encoder class. How do we use it? Well, lets take a look at some sample code:

private function startEncode(imageBitmapData:BitmapData):void 
{
  var encoder:JPEGAsyncEncoder = new JPEGAsyncEncoder(80);
  encoder.PixelsPerIteration = 128;
  encoder.addEventListener(JPEGAsyncCompleteEvent.JPEGASYNC_COMPLETE, encodeDone);
  encoder.addEventListener(ProgressEvent.PROGRESS, encodeProg);
  encoder.encode(imageBitmapData);
}

private function encodeProg(event:ProgressEvent):void
{
  var percentage:String = ((event.bytesLoaded / event.bytesTotal)*100) + "%";
  //Display the percentage somewhere
}

private function encodeDone(event:JPEGAsyncCompleteEvent):void
{
  var data:ByteArray = event.ImageData;
  //Do something with the encoded image
}

And there you go. Thats all you need to use this asynchronous jpeg encoder class. You can grab the source code for the whole example above here, and feel free to use it for whatever you want.

Source Files:

• SOTC-JpegEncodingTutorial.zip