Nonexistence of invariant semigroups in affine symmetric spaces