Some 50 people were reported killed in the town of Roche-a-Bateau alone.
The nearby city of Jeremie saw 80% of its buildings levelled. In Sud province 30,000 homes were destroyed.
The hurricane, now a Category Three storm with sustained winds of 120mph (193km/h), is heading towards the US state of Florida.
At 03:00 local time (07:00 GMT) Matthew was still off the Florida coast, centred about 37 miles (60km) east of Vero Beach and was moving north-west at about 14mph (22km/h), the National Hurricane Center said.