My handbrake settings for converting mkv to mp4 which works on the ps3.

Video Tab:
Video Codec: H.264 (x264)
Framerate: same as source
2 Pass: unchecked
Quality -selected- Target Size (MB): 2300 (this was for a mkv that was about 2200 megs)

H.264 Tab
Reference Frames – Number: 3
B-Frames – Number: 3
Pyramidal B-Frames: checked
Weighted B-Frames: checked
Motion Estimation – Method: Uneven Multi-Hexagon
Motion Estimation – Range: 16
Motion Estimation – Subpixel Method: 6
Motion Estimation – Analysis: Some
8x8dct: checked
Miscellaneous – Deblocking: -2 -1
Trellis: Final Macro Block
No Fast P-Skip: unchecked
No DCT Decimate: unchecked
CABAC Entropy Encoding: checked