The modulus of elasticity is an important parameter for an accurate prediction of the springback in sheet metal forming processes. With increasing plastic deformation, this modulus behaves nonlinearly and declines, which leads to an unpredictable springback behavior. The most cited reason for this nonlinearity is the dislocation movement during plastic deformation that especially occurs with multiphase steels. The present contribution investigates the nonlinear unloading behavior and the resulting decrease of the elastic modulus from a differently heat treated DP980 steel. The heat treatments set five different microstructures with martensite volume fractions in the range of 42 to 95 %. By means of the tensile test, a decline of the elastic modulus according to pre-strain was examined by evaluating the chord-modulus during unloading at different strain levels. In addition, a nano-hardness test was performed. It turned out that in all heat treatment conditions, a pronounced decrease in the modulus of elasticity up to 25% from the initial value occurred. With decreasing annealing temperature and lower martensite volume fraction, respectively, the martensite hardness increased, leading to higher hardness differences between the ferrite and the martensite phase in the microstructure. This led to an increase of strain hardening, i.e. to an increased formation of fresh mobile dislocations in the vicinity of the harder martensite phase during plastic deformation. As a result, the modulus of elasticity decreased more sharply. Thus, in the present contribution, an interplay between the martensite volume fraction and its hardness on the decrease of elastic modulus could be clearly manifested.