I have most of two layers of old wallpaper off the wall. Can I just prime and then texture over what is left?

Getting the rest off the wall is extremely labor intensive and I don't have the time or the strength to do it. What I have gotten off is down to the drywall and gouging the drywall mud off as well as digging into the drywall itself. The only coat of paint is coming off with the undercoat of glue.