Isn’t it a conflict of interest to allow people dependant on government charity to vote?

You're right. We should also ban people who use the public roads and highways, the US Postal Service, air traffic controllers, police or fire departments, or who have had their country protected by the United States armed forces.