Model Based Testing is the general class of testing techniques that relies on modeling the system of smart contracts to be tested using a mathematical abstraction. One example of that is State Transition Testing which models underlying contracts as a state machine.

A list of more complex and representative models for systems are listed in the Peterson's Lattice of Behavioral Models seen here. Sufficiently expressive models like Petri Nets can be used to model different types of network-level threats such as transaction front-running.

‍