What does INSTBY stand for?

Instability

A condition reached when a structure or structural member is loaded in which continued deformation results in a decrease in its load-resisting capacity. A wavefunction is expressed as a long list of parameters (basis-set expansion) that are adjusted to minimize the total energy. Sometimes the global minimum is not obtained; the local minimum that is obtained may be unstable with respect to various perturbations or liberalization of constraints.