A fast direct solution of the electric current volume integral equation (JVIE) with the Sherman-Morrison-Woodbury (SMW) formula-based algorithm is presented to analyze electromagnetic scattering from inhomogeneous dielectric objects. The JVIE is discretized with the nonconformal facebased Schaubert-Wilton-Glisson (SWG) basis functions. Compared with conformal discretization that is advantageous to discrete homogeneous regions, the nonconformal discretization provides a more flexible and efficient scheme to separately handle the inhomogeneous subdomains depending on local parameters. Moreover, to take full use of both discretization methods, the mixture discretization is adopted. With the increase of object size, the impedance matrix equation arising from the JVIE becomes too large to solve and store for direct solution. In this paper, the SMW formula-based algorithm is adopted, leading to remarkable reduction on the computational complexity and memory requirement in contrast with conventional direct solution. This algorithm compresses the impedance matrix into a product of block diagonal submatrices, which can be inversed rapidly in direct way. Numerical results are given to demonstrate the efficiency and accuracy of the proposed method.