Periodontitis is one of the most common complications of diabetes, which seriously affects patients' life quality. It is important to find the key factors and mechanisms to improve the treatment of periodontitis. In our study, high glucose (HG) and lipopolysaccharide (LPS) treated human periodontal ligament cells (hPDLCs) and LPS treated diabetic mice was used to establish the diabetic periodontitis model in vitro and in vivo. O-linked beta-N-acetylglucosamine glycosylation (O-GlcNAcylation) and O-linked N-acetylglucosamine transferase (OGT) protein levels were detected by western blot assay. Cell counting kit-8, alkaline phosphatase (ALP), and alizarin red staining (ARS) assays were used to observe the O-GlcNAcylation and OGT effects on cell viability and osteoblast differentiation. Co-immunoprecipitation (Co-IP) assay was used to detect the relationship between OGT and ALP. The results showed that the levels of OGT and O-GlcNAcylation were significantly increased in both cell and mouse models. ALP and ARS staining results showed that silencing of OGT or inhibition of O-glycosylation notably improved osteogenic differentiation, increased the osteoprotegerin (OPG) protein levels and decreased the receptor activator for nuclear factor-κB Ligand (RANKL) protein levels of the HG and LPS treated hPDLCs. In diabetic periodontitis mice, knockdown of OGT relieved the injury of gingival tissue, increased the ALP and OPG levels and decreased the RANKL levels. Besides, ALP interacted with OGT protein, and OGT protein was found to act on ALP serine 513 glycosylation. In conclusion, our study demonstrated that excessive O-GlcNAcylation could restrain osteoblast differentiation by O-glycosylation in ALP.