His novel tells the story of a young black man who tries to reinstate slavery and racial segregation in a suburb of Los Angeles.
Amanda Foreman, chair of the judges, said the book managed "to eviscerate every social taboo".
Beatty's win was announced at a ceremony at London's Guildhall on Tuesday.
Picking up the £50,000 prize from the Duchess of Cornwall, Beatty, 54, was clearly overwhelmed with emotion and struggled for words as he began his acceptance speech.
"I hate writing," he admitted later in the speech.
"This is a hard book," he went on. "It was a hard for me to write, I know it's hard to read. Everyone's coming at it from different angles."
This is the third year that the £50,000 prize has been open to writers of any nationality. The shortlist included two British, two US, one Canadian and one British-Canadian writer.
The Sellout beat five other novels, including Madeleine Thien's Do Not Say We Have Nothing, the bookies' favourite, and Graeme Macrae Burnet's Scottish crime thriller His Bloody Project.