Why are people so afraid of population growth?

Australia does not have enough resources to support a larger population. The country is in severe drought with many many many farms shutting down and there is a desperate shortage of water. Where do you suggest all this extra food and water is going to come from? Are you willing to give up your standard of living to allow others in?