Following a tour of Saudia Arabia, he has told BBC Sandeshaya that he has stressed the need for a new method to resolve issues of Sri Lankan workers in KSA.
Saudi authorities have told him that agents in both countries were responsible for most of these issues.
Helpless workers are left in the lurch when fraudulent foreign employment agencies commit heinous crimes against them, he charged.
De Silva said measures would be taken to bring down the Sri Lankans who have sought shelter in camps after fleeing their Saudi employers.
Noting that the main camp alone has 200 women, he said the main issue hampering their return was obtaining a permit to leave KSA.
When a problem arises, the Saudi employer refuses to return the permit as he has paid 25,000 Saudi Riyals to the agent, out of which the worker gets only 1,000 Saudi Riyal, he said.
However, not a single agent has been taken before courts and punished, he added.