Is the belief in nothing still a belief in something?

No. A lack of belief in something is still nothing. It's sort of like you cannot divide zero. That has no meaning. A belief in nothing is nothing.