Here is the full list of formats the PS3 can play:
* Memory Stick Video Format
* - MPEG-4 SP (AAC LC)
* - H.264/MPEG-4 AVC Main Profile (AAC LC)
* MP4 file format
* - H.264/MPEG-4 AVC High Profile (AAC LC)
* MPEG-1 (MPEG Audio Layer 2)
* MPEG-2 PS (MPEG2 Audio Layer 2, AAC LC, AC3(Dolby Digital), LPCM)
* MPEG-2 TS (MPEG2 Audio Layer 2)
* AVI
* - Motion JPEG (Linear PCM)
* - Motion JPEG (μ-Law)
* AVCHD （.m2ts / .mts）
* DivX
* WMV
* - VC-1（WMA Standard V2）
As for what is best? Probably DivX or MPEG-4 AVC.