[Objective]In this paper,the objective of the research is to improve the performance of Orthogonal Frequency Divi-sion Multiplexing(OFDM)-Wavelength Division Multiplexing(WDM)optical fiber communication system.a Quadrature Grouped Subcarrier Amplitude Shaping(QGSAS)dual-Phase Conjugate Twin Waves(dual-PCTW)scheme is proposed,which can suppress the nonlinear effect of the systems.In addition,the simulation systems under different conditions are built to ex-plore the performance of the scheme.The specific experimental methods are as follows.[Methods]With the mixed program-ming of Optisystem and Matlab,a QGSAS dual-PCTW OFDM system is simulated based on 16-ary and 64-ary Quadrature Amplitude Modulation(QAM)formats.Two kinds of QGSAS(sinusoidal amplitude shaping,rectangular amplitude shaping)and three dimensions dual-PCTW(time domain,polarization domain,sub-carrier domain)are adopted for the system to form 12 different combinations.[Results]The simulation results show that the QGSAS dual-PCTW scheme can effectively suppress the nonlinear effects in the system.In several combination schemes discussed,the 16QAM dual-PCTW scheme in time domain has the best performance.When the QGSAS technology is used,the scheme of rectangular amplitude shaping is better than the sinusoidal amplitude shaping.[Conclusion]Therefore,it can be concluded that in the scheme studied in this paper,the rectan-gular amplitude shaping scheme combined with the time domain scheme dual-PCTW technology under 16QAM can effectively suppress the nonlinear effect of OFDM-WDM system.