“I write for black people” is entirely different from “I don’t want white people to read my books.” A book written for black people can be just as universal as a book written with no particular audience in mind (as you point out, the default of “no color” is white people), and in a way it is more universal.
W. E. B. Du Bois observed that black people have two identities: as black people and as members of a society defined primarily by white people.
White people rarely have this double awareness. Being a white person and a member of a society defined primarily by white people are for most whites the same thing. That’s because they are unaware of the role of non-whites in the creation and maintenance of our nation and world.
The most important books for American white people to read if they want to understand the reality of their country are books written for black people.