Hmm, this is something I started myself a while ago.
One of the problems with extending the AVR softcore is that it just didn't seem to be built for a peripheral rich setup (IO space, etc). But I didn't want to leave an open source GCC based solution. Which is why I've watched the ZPU work with interest.
I recently tried to use a quadrature interface from opencores, but it was far too featureful and resource heavy.
So instead, I created my own "simple" quadrature interface for the ZPU, and using the standard HD44780 code in the ZPUino IDE install, created a sketch to read the quad counter and write to the LCD.
It is so very nice to read a quadrature encoder as simply as:
unsigned int y=REGISTER ( IO_SLOT (8) ,0);
And to know that it is being clocked at 96Mhz.
(Of course, I most probably have created a piece of junk, full of bugs - but I think it's cute)
The code's a complete mess, but if you're interested... it's attached.
Download Source Code
Source Code Link