For 5G broadband wireless access systems, ultra-high data rates and efficient spectrum usage have become critical issues. To take the full advantage of the licensed spectrum, it is desirable to have frequency reuse-1 deployment. However, with this approach, the communication systems needs to have an effective way of handling the co-channel interference from neighboring cells on the Downlink (DL). Joint detectors have the ability to mitigate the co-channel interference, provided they are correctly initialized in terms of the knowledge of the modulation order and channel state information of all the signal streams involved in the measurements. In our work, we consider a DL interference channel model and provide a theoretical framework to analyze the effect of mis-specifying the modulation order of this interference on the error performance of a joint detection scheme.

An analytical proof is provided by deriving a new Pair-wise Error Probability (PEP) expression for the existence of an error floor in the Average Symbol Error Probability (ASEP) of the desired symbol when the joint detection metric is employed with the mis-specified modulation order for the interference. The effect of over specifying modulation order on the joint detector's ASEP is analyzed. A Unit Multiplicity Union Bound (UMUB) is also described which is tighter than the classical union bound based ASEP.

To avoid error floor due to the mis-specified modulation order at the joint detector, the classical Maximum likelihood Modulation Classification (MC) along with the joint maximum likelihood detection has been proposed in literature of Multi-user MIMO (MU-MIMO) detection. Overloaded MU-MIMO links are prone to interference due to several factors. To mitigate this interference in 1X2 overloaded MU-MIMO systems, many authors have proposed Max-Log MAP (MLM) interference modulation classification which however is computationally complex. A more efficient Layered ORthogonal lattice Detector (LORD) scheme has also been described in literature but this scheme fails if the receiver has only one antenna. In our second part of work, we propose a Quasi-MLM interference modulation classification followed by joint data detection for 1X2 overloaded MU-MIMO systems which is not only (near) optimal in bit-error rate performance, but also computationally efficient.