RSG is another way to write SceneGraph (it stands for Roku SceneGraph).
Direct Publisher is basically a plug–and–play utility/method to make channels quickly with little–to–no actual programming required, but as a result it has very limited customization.
SceneGraph still uses Brightscript functions to handle logic, the main event processing loop, etc, but SceneGraph basically integrates XML on top of it to make it more flexible (and human-readable).
In other words, if you just want to throw something together quickly, without even learning a new programming language, and don't care about how customized or polished your channel looks, Direct Publisher is an easy way to do that. If you want more power and customization, you should use SceneGraph/RSG—which is more or less Brightscript with XML welded to it (although some features of Brightscript are deprecated in RSG, this is mentioned in their man files or whatever).