You read correct, they CAN enter your apartment and they're backed by the Federal Government! From AJC.com: 

Thousands of census workers, including many temporary employees, are fanning out across America to gather information on the citizenry. 

What many Americans don’t realize, is that census workers — from the head of the Bureau and the Secretary of Commerce (its parent agency) down to the lowliest and newest Census employee — are empowered under federal law to actually demand access to any apartment or any other type of home or room that is rented out, in order to count persons in the abode and for “the collection of statistics.”  If the landlord of such apartment or other  leased premises refuses to grant the government worker access to your living quarters, whether you are present or not, the landlord can be fined $500.00.

That’s right — not only can citizens be fined if they fail to answer the increasingly intrusive questions asked of them by the federal government under the guise of simply counting the number of people in the country; but a landlord must give them access to your apartment whether you’re there or not, in order to gather whatever “statistics” the law permits.

And your thoughts are....?