1.3 The Problem of the Zero-Point Vacuum Energy
One of the most important unsolved problems of physics is the problem of the divergent zero-point energy of the vacuum, shared by all fundamental fields. If a field is decomposed into harmonic oscillators, then for each oscillator quantum mechanics predicts a zero-point energy equal to (1/2)ħω. With 4πω2dω oscillators in between ω and ω + dω, the spectrum of the zero-point energy has ω3 frequency dependence, the only one which is Lorentz invariant. Because it is divergent, it should lead to an infinite vacuum energy with an infinite mass, resulting in very large gravitational fields, obviously not observed. There can be no doubt in the reality of the zero point vacuum energy, because it not only makes itself felt in the Lamb shift, but also shows up in the macroscopically observed Casimir effect. The zero-point energy of the vacuum more than anything else shows that there must be an effect by which its large mass density is compensated. Cutting off the zero-point energy at some frequency not only would destroy Lorentz invariance, but because the cutting off presumably would have to be done at some high frequency, it would left uncompensated the zero-point energy up to the cut-off.
The problem of the zero-point energy is closely connected with the problem of the vanishing cosmological constant . To be consistent with the astronomical evidence, the cosmological constant must be very small, if not zero. The smallness of the cosmological constant is, for this reason, further strong evidence for a mass-compensating effect.
One way a divergent zero point vacuum energy could be made to vanish, at least in principle, is by supersymmetry. It assumes that a bosonic elementary particle has a fermionic partner and vice verse. Because of the negative energy states occurring in the Dirac equation, the positive zero-point energy coming from the bosonic part can be made to cancel in combination with a corresponding negative zero-point energy contribution from the fermionic part. In spite of the nice symmetry it expresses, the simple fact, that no supersymmetric partners for the known elementary particles have ever been found, speaks against supersymmetry. The hope that the supersymmetric particles may be found above a not yet reached energy would really not help because it would mean that the zero-point energy remains uncompensated below this energy. One can, for this reason, say the breaking of supersymmetry generates a cosmological constant. If, for example, the supersymmetric particles would show up above ~1 GeV, the zero-point energy which would be uncompensated up to this energy, would lead to a vacuum mass density comparable to the density of nuclear matter.
In spite of its failure, the idea of supersymmetry provides a hint about the nature of the mass-compensating effect. In supersymmetry, the compensation is caused by the negative energy states and hence mass states of the Dirac equation. This suggests that negative masses are responsible for the compensation of the divergent zero-point energy, but that they are hidden from direct observation. Hypothesizing that both positive and negative masses can coexist as part of the physical vacuum, the zero point fluctuations would not be absent, but rather be compensated in the average.