AVI's are choppy if they contain B-VOP's, which don't seem to be supported on DSM-320. XviD, by default, has this value set to 2, for optimum performance. If you're coding using XviD, either uncheck b-vop's altogether (would also disable packed bitstream), or set the level to 0. Complex scenes are handled better using avi than with mp4 (like the dam bursting in Ice Age - The Meltdown), so if you're creating a new encoding, this is the way to go. This also creates a file that is widely supported, so you don't end up creating one copy for DSM-320, and another copy for your other players.

Gordian Knot and XviD are still by far the best encoding tools you'll ever use

Of course, if you have an existing avi file that needs converting, Avidemux conversion to mp4 would still be the way to go.