Abstract A modified boundary element method for analysing hybrid modes of waveguide is presented. The coupled boundary integral equations and their reduced forms are deduced for hybrid mode problems, and these equations are discreted into a system of linear algebraic equations by moment method. Finally the dispersion characteristics of rectangular waveguide filled with two kinds of dielectrics are calculated and the numerical results are in good agreement with that obtained by transverse resonating method.