A mother and daughter – both of whom have developmental disabilities – are set to receive a five-figure payout amid claims that they were repeatedly harassed and pressured to move from their apartment.

Laura Doty, who has cerebral palsy and visual impairment, and her daughter Brenda, who has Down syndrome, along with an advocate who helped them, will get $40,000 in damages under a proposed settlement brokered by the U.S. Department of Justice.

According to a lawsuit filed by the federal agency on behalf of the Dotys, the mother and daughter moved to the Applewood Apartments in Cross Plains, Wis. in July 2013. From day one, other residents allegedly called Brenda Doty “mentally retarded” and made other offensive comments like “You don’t belong here … you belong in an institution.”

The Dotys said they were followed around and complaints to the apartment complex’s management went unheeded, according to the suit which alleges numerous violations of the federal Fair Housing Act.

Instead, the apartment’s owners – Applewood of Cross Plains LLC and William Ranguette – insisted that the Dotys develop a “plan” to deal with Brenda Doty’s behavior stemming from her disability and pressured the mother and daughter to move, the lawsuit states. Ultimately, the Dotys were told that their lease would not be renewed and no reason was given.

The Dotys ended up moving to a more expensive, less accessible and less convenient apartment where they must rely to a greater extent on others to grocery shop and do other errands, the complaint indicates.

The Justice Department intervened after the Dotys filed a complaint with the U.S. Department of Housing and Urban Development.

“The Fair Housing Act prohibits discrimination on many bases, including disability,” said U.S. Attorney John W. Vaudreuil of the Western District of Wisconsin. “Persons living with disabilities have an equal right to protection under the act and we will enforce the act when such offensive conduct interferes with their rights to use and enjoyment of their home.”

Under the terms of the settlement, the apartment owners have agreed to pay damages, attend fair housing training and maintain non-discrimination housing policies. The agreement must still be approved by a judge.