This option should contain the number of undo levels to keep in memory. This is set to -1 by default and such a value tells TinyMCE to use a unlimited number of undo levels. But this steals lots of memory so for low end systems a value of 10 may be better.
Example of usage of the custom_undo_redo_levels option:
tinyMCE.init({ ... custom_undo_redo_levels : 10 });