This true/false option gives you the ability to turn on/off the visual aid for border less tables. If a the border of a table is set to 0 TinyMCE adds a dotted line around the table by default.
Example of usage of the visual option:
tinyMCE.init({ ... visual : false });