No one will take your house for that reason by itself. But they might take it when you can no longer afford to pay the mortgage because you were convicted of a drug felony where you were given a harsher sentence than someone of a different race, after being arrested under suspicion that others wouldn’t be. As a result you can’t find a job after release from prison.