To clarify, the continuation-based writer is used internally as part of the corecursive queue transformer, but it cannot be used as an argument to the transformer.

It is believed to be impossible to define MonadFix on the standard continuation monad, but as far as I am aware, this is not definitively known. Levent Erkök wrote about this conjecture in his Ph.D. thesis Value Recursion in Monadic Computations, where he proves a theorem that seems to support the conjecture.

However, we now know that the statement of Erkök’s conjecture needs to be made more precise. For example, the codensity monad is a continuation monad that uses higher-ranked types to hide the type of the result. It appears in Edward Kmett’s category-extras and Mauro Jaskelioff’s Monatron library, and is the key to Janis Voigtländer’s paper. Interestingly, the codensity monad admits an instance for MonadFix, as demonstrated by Monatron but not category-extras. However, the codensity’s MonadFix does not appear to be well-behaved, as I don’t know of any interesting examples where it converges.

Also, I’ve heard Erkök’s conjecture informally stated as “there are no useful fixpoint operators on continuations”, which my paper “Lloyd Allison’s Corecursive Queues” throughly refutes. Not only did I demonstrate a fixpoint operator on the continuation monad, I also demonstrate a practical application where it’s use seems to be required!

If you are really interested in the studying the relationship of MonadFix, the continuation monad, and other continuation-like monads, you should probably take a look at “Recursion is a Computational Effect” by Dan Friedman and Amr Sabry. I don’t understand all the details, but they implement mfix on the continuation monad using unsafePerformIO, and use it to implement “LETREC + CALL/CC = SET!” in Haskell.

It seems likely that there is something here yet to be discovered. At the very least, there are explanations to be made and things to be definitively confirmed or refuted.