A federal agency is warning the White House that more protections are needed to ensure the parental rights of those with disabilities.

Even as an increasing number of Americans with special needs choose to become parents, laws across the country routinely undermine their rights, according to a National Council on Disability report which was sent to President Barack Obama on Thursday.

In two-thirds of states, courts are allowed to deem a parent unfit solely based on their disability. And, disability can legally be taken into account in every state when assessing what’s in the best interest of a child, the council found.

“Even today, 22 years after the passage of the Americans with Disabilities Act, parents with disabilities are the only distinct community of Americans who must struggle to retain custody of their children,” wrote Jonathan Young, chair of the National Council on Disability in a letter to the president that accompanied the report.

Currently, some 6.1 million children in the United States have parents with disabilities. They are significantly more likely than other kids to be forcibly separated from their parents, the federal agency found.

Estimates suggest that among parents with intellectual disabilities, removal rates are as high as 80 percent. Similarly high rates are seen among parents with psychiatric disabilities.

Meanwhile, the council found that people with special needs are more likely to lose custody of their children after divorce and have more difficulty adopting kids.

“A nexus should always be shown between the disability and harm to the child, so that a child is taken from a custodial parent only when the parent’s disability is creating a detriment that cannot be alleviated. However, this is not the reality,” Young said.

The council is urging the administration and Congress to collect data and fund research to better understand the experiences of people with disabilities as parents.

What’s more, the agency is recommending that new laws be implemented at the state or federal level to protect the rights of parents with disabilities. The report also indicates that social services agencies need to work to better understand and accommodate this population.