Holding the global increase in temperature caused by climate change well below 2 degrees C above pre-industrial levels, the goal affirmed by the Paris Agreement, is a major societal challenge. Meanwhile, food security is a high-priority area in the UN Sustainable Development Goals, which could potentially be adversely affected by stringent climate mitigation. Here we show the potential negative trade-offs between food security and climate mitigation using a multi-model comparison exercise. We find that carelessly designed climate mitigation policies could increase the number of people at risk of hunger by 160 million in 2050. Avoiding these adverse side effects would entail a cost of about 0.18% of global gross domestic product in 2050. It should be noted that direct impacts of climate change on yields were not assessed and that the direct benefits from mitigation in terms of avoided yield losses could be substantial, further reducing the above cost. Although results vary across models and model implementations, the qualitative implications are robust and call for careful design of climate mitigation policies taking into account agriculture and land prices.