Recently I was trying to create a custom toolbar menu group for some bulk actions I developed. At first I thought it’d be as easy as creating a <CustomActionGroup> feature. However, as I tried and tried I realized that it was way more complicated than that.
Basically you need to implement abstract ToolBarMenuButton class. You can find implementation details in this thread from SharePoint msdn forum: