Don't use HardieBacker. That's for ceramic/stone tile. For vinyl or linoleum you typically use luaun or some other plywood underlayment that is engineered for dimensional stability (minimal expansion/shrinkage) and does not off-gas or interact with the flooring adhesive.
I ran into the same issue in my old house. It had stick-on vinyl over approx 3/8" thick fiberboard, over a rough pine tongue-and-groove subfloor. We used a roofer's shovel to scrape up some of the tiles, until we saw how crappy the underlying fiberboard was (glue from multiple floor installs, gouges from multiple floor removals, swollen around the joints, etc.), so we just peeled off whole chunks of the fiberboard until we were down to the subfloor. Shop-vac to get rid of the crap in the grooves.
We ended up going with SurePly underlayment to replace the fiberboard. It costs more than luaun but we didn't like the environmental issues associated with luaun (google it.) We only needed about half a pint of floor leveler for the entire 11x14 space--just a skim coat over the seams and a dab on about half a dozen spot where the ring-shank nails didn't go in exactly right and left impressions in the surface. We used a single sheet of vinyl flooring and after it settled in for a day it looked fantastic.