Rescue teams have been trying to reach people trapped in rubble since the magnitude 6.4 quake struck just before 04:00 (20:00 GMT Friday).
A baby was among the dead when a high-rise residential block collapsed. At least 155 people were injured.
President Ma Ying-jeou promised an "all-out effort" to rescue people when he arrived in the city.
Shelters would be set up for those who had lost their homes in the city of two million people, he said.
The quake was shallow, meaning its effects would have been amplified, the US Geological Survey (USGS) said.
There have also been at least five aftershocks.