Within the current release of the OSS Activiti version, it does not look to be possible to store any relation between groups without additional modification to the source code. In the Beta version, with the updated UI to look and feel more like Enterprise - it seems as though the group relation feature was not included, either.
So you're options are either to:
Utilize the Enterprise release, which in the UI and REST API allows you to store relations between groups - essentially creating subgroups, as many layers deep as you'd like.
Or custom create some additional database column, and the infrastructure to support it, in order to allow the creation of group relations - allowing for 'subgroups'.
With either of these, you will still be able to assign that hierarchy at any point, to a task.