With the development of the Internet of Things (IoT), the number of devices will also increase tremendously. However, we need more wireless communication resources. It has been shown in the literature that non-orthogonal multiple access (NOMA) offers high multiplexing gains due to the simultaneous transfer of signals, and massive multiple-input–multiple-outputs (mMIMOs) offer high spectrum efficiency due to the high antenna gain and high multiplexing gains. Therefore, a downlink mMIMO NOMA cooperative system is considered in this paper. The users at the cell edge in 5G cellular system generally suffer from poor signal quality as they are far away from the BS and expend high battery power to decode the signals superimposed through NOMA. Thus, this paper uses a cooperative relay system and proposes the mMIMO NOMA double-mode model to reduce battery expenditure and increase the cell edge user’s energy efficiency and sum rate. In the mMIMO NOMA double-mode model, two modes of operation are defined. Depending on the relay’s battery level, these modes are chosen to utilize the system’s energy efficiency. Comprehensive numerical results show the improvement in the proposed system’s average sum rate and average energy efficiency compared with a conventional system. In a cooperative NOMA system, the base station (BS) transmits a signal to a relay, and the relay forwards the signal to a cluster of users. This cluster formation depends on the user positions and geographical restrictions concerning the relay equipment. Therefore, it is vital to form user clusters for efficient and simultaneous transmission. This paper also presents a novel method for efficient cluster formation.