Instability and dynamic self-organization of the cancer cell system