Hyper-parameters are the parameters that define the model itself and how it is learnt from data.

These are the parameters that:

cannot be learnt during the training process
define the model complexity
define the learning process For example: learning algorithm, learning-rate, number of layers in a neural net, etc.

Thus, one needs to perform a search in the hyper-parameter space in order to discover the best performing model.