Mathematical solvers can be parameterized today with a multitude of different parameters. While default parameter settings of- ten provide good results, in terms of low runtime, often parameter set- tings can be found, which speed-up the solving process for a particular model. Before considering the construction of strategies for optimizing parameter settings for particular models, it is necessary to understand the underlying search space. We do so by investigating systematically the effects of different parameter settings, taking into account the pa- rameters considered to me most important in the literature. Based on three pre-existing mathematical models, we explore runtime for solving them, systematically varying the parameters of the solver. As a result of our study, we can provide a better understanding of the underlying search space, that needs to be investigated for effectively perform pa- rameter tuning of mathematical solvers. Also we highlight that choosing bad parameters can have significant disadvantages, e.g. compared to the default parameters.