Because of the subjective detection of depression, the use of user voice diagnosis of depression has become a more potential auxiliary way. However, the speech signals of different users are different. In this study, a CUADAN (Cross User Audio Domain Adaptation Network) is proposed to detect depression. Visual Mel spectrograms are extracted from the audio, and the feature extractor of the CUADAN model is used to extract deeper depression features from the Mel spectrograms. Since the source domain and target domain contain the voice features of different healthy users and depressed users, the domain classifier of CUADAN model is used to perform domain adaptation be- tween different user data, so that unknown users can be detected by existing classifiers. The experimental results show that the CUADAN model has a higher depression detection accuracy, with an average accuracy of 81.0 ± 2.4%. Therefore, the CUADAN model can effectively weaken the differences between different users voices and improve the accuracy of cross-user depression detection.