First of all, I sympathize! I get so involved with what I've done that anything less than the DVD quality bugs me.
I really think you'll find better answers at these two forums:
Both of these sites have forums, downloads and links you may find useful. In your quest, you may find that only "expert" tools are available which will require some technical tweaking and you'll get your education over on these.
But, every compression loses quality, lol. I guess the question becomes, "is there a further compression that most people won't notice the loss of quality".
There are several things that go together in this, D. Probably the first thing you need to target is a bit rate. Alternatively, you can target a size for the file.
Once you have that, you can try various codecs and control programs for the codecs to see what you can get. Of course DivX is a biggie these days, but it's not for free as an encoder (the decoder is). There's one derived from the same techniques that is free (forget the name) -- but it's not as good generally.
Most encoders use transform matrices of some kind, and the coefficients which go in the cells can make a big difference in quality and compression -- but they revolve around particular types of material.
Generally, images where fewer pixels change/move from frame to frame can be compressed the most. Motion is the next biggest challenge, and flashing to a completely unrelated frame is the biggest challenge. Both of those require higher bit rates to encode. Motion takes less because often similar groups of pixels are simply moving from one location to another and the encoder can detect that using motion sensing algorithms and simply supply information on the change in the pixels location and quality rather than re-encoding them all.
Those matrices I mentioned can be tweaked to give better results with different color encoding vs. black and white vs. anime/cartoons etc. Each type of material has a "depth" that comes from the chromanance/luminance in each pixel and the number of bits required to express that depends on the overall "palette" of colors in the source material. There are different color encoding schemes used like YV12, YUY2, RGB15/16, RGB24 and so on (there are quite a few -- some not used in regular video) that have an effect on what you can do and sometimes a certain scheme may be a requirement to even use an encoder -- though some encoders do "conversion" for you if you supply the wrong scheme.
A note on conversions: the more you scale ANYTHING -- colorspace, dimensions, etc. -- the more decay in "quality" you can suffer. Just a word of warning. Sometimes it pays to have lots of harddrive space and use as little compression as you can until the final cut. However, that also requires a lot of overhead in terms of data transfer for previews and what not during editing so you need very fast equipment.
Another thing which can help is just a small amount of noise filtering. Though primarily used for analog video sources, some CGI effects create noise as a side effect. Using a mild noise filtering algorithm on the final result sometimes gives you a smaller file or lower bit rate.
I guess what I'm saying is: this isn't a "get the right appliance and all your problems are solved". Video encoding by it's very nature requires attention to a lot of detail and parameters and if you want to REALLY compress something, it can be quite an experimental process. It pays to learn what's available and what's possible and then decide what direction to try.
You will find that over there, like here, experts disagree so you must sort it out for yourself!
DOH! Memory is weird when you get older. I couldn't remember the name of the DivX alternative -- it's XviD, lol -- DivX backwards!