CasssiniDev, when used as a library, does execute in the same AppDomain and errors will be propagated to your tests. I am not privy to the transactional implications in your scenario.
Whether you run Cassinidev in-proc or out-of-proc, the SimpleWorkerProcess that is created for each request has it's own AppDomain. There is no way around that.
Using CasssiniDev-lib gets you as closer to the metal than you will get with any other server/technique but again, the request will be in a new appdomain.
But if I might make an observation that may provide a different perspective:
A unit test should be simple and atomic. One test should not depend on the final state of another.
Employing transactions to manage test context state seems to be adding unnecessary complexity.
What is the difference between simply setting up the context for each test and the technique you describe?
If I have misunderstood, please let me know.