What's the best rope for a traveller, thick or thin, presume rough is good, damn thing keeps falling out of the cleat....

Key thing is to have plaited rope rather than stuff with a cover and a core as the core frays quickly.  Stretch isn't a problem here either so cheap polyester prestretched will do, think mine is 4mm.
If its falling out is it the cleat needs replacing rather than the rope?  It can be a problem to get it to cleat in light winds

Just done a full job today on my winder mk1 traveller set up. It wasn't easy, however I have used 5mm line and replaced the cleats with Holt "small alloy cam cleats from P and B reference number 85-HA77 at £25.62 each (ouch!).
My traveller initially refused to jam, so I replaced the lines, however I didn't route the lines correctly when I did this. A fellow more experienced Merlin sailor at my club helped out here and then advised that I should change the cleats as the jaws had become worn.
It was quite tricky to remove and refit the new cleats as space is limited for two hands, a head, various tools and a visiting wasp, however I got there in the end!
Also, the length of the lines plays an important part in the traveller process for it to work properly. Leave them too short (which I did originally) and the system will not allow the strop to come across far enough to cleat on the windward side. I found this bit out when I posted a note on the forum like you have. If you need any more info, you can call me 07798631089 cheers Kev
Oh, I forgot to mention that mine works perfectly now!