Electroencephalography based Brain Computer Interface (EEG -BCI) is used to provide communication, rehabilitation and control for the people suffering from severe spinal cord injury and stroke. Among various EEG signals, Motor Imagery (MI) based BCI is widely used for patients to communicate with the external world. These MI- EEG signals are recorded when the subject performs imagery motor task. Each subject has specific patterns because of the spatial origin, change in amplitude and brain signals and therefore calibration process is necessary. In present BCI systems, calibration is often tedious process which cause mental fatigue for the subject. It is obvious that long calibration time has become one the considerable snag of the BCI system. To use BCI systems for practical application it is important to reduce the long calibration period. To overcome this drawback, we use Rule adaptation (RA) based transfer learning approach for transferring knowledge from previous subjects to the new subject. The aim of this research is to design an end-to-end subject independent motor imagery-based brain computer interface system and to enhance the performance by fine tuning the subject independent model using a genetic algorithm. Experimental result shows that fine-tuning a subset of the CNN parameters with data from the target subject can significantly increase the average classification accuracy and address the inter-subject variability.