Background: This study aimed to quantify the mortality burden attributable to non-optimal cold and hot conditions from moderate and extreme temperatures in Hong Kong during 2006-2016. Methods: The association between mortality and daily mean temperature was assessed using a distributed lag non-linear model (DLNM) integrated with a quasi-Poisson model. The cumulated effects of cold and hot temperature were firstly quantified. The attributable risks of non-optimal ambient temperature on deaths by cause-specific mortality and age groups were then estimated. Results: A reversed J-shaped relationship was found between temperature and total mortality. The highest increase in risk was at extreme cold, with the highest relative risk (RR) for injuries of 2.18 (95%CI: 1.03-4.62), followed by the respiratory and circulatory system diseases for lag 0-21 days. Cold temperature was associated with a greater burden of death than hot temperature, with attributable fractions (AF) of 4.72% and 0.16%, respectively. Moderate temperatures played a major role in all-cause mortality with AF of 4.25%, and 0.63% for extreme temperature. Conclusions: Most of the temperature-related mortality burden was attributed to moderate weather, suggesting relevant temperature-related preventive strategies and measurements should be implemented to minimize the negative impact of temperatures on population health, particularly for vulnerable sub-populations.