Splitting this apart
$block = $this->getLayout()->createBlock('core/template')->setTemplate('PATH/RELATIVE/TO/TEMPLATE/*.phtml')->toHtml();
This
$block = $this->getLayout()->createBlock('core/template');
instantiates a Mage_Core_Block_Template object. The class name comes from looking in the merged config.xml for a node inside <blocks/> named <core/>, and then appending the word Template.
then this
$block->setTemplate('PATH/RELATIVE/TO/TEMPLATE/*.phtml');
Will set a variable named template on the Mage_Core_Block_Template object.
then this
$block->toHtml();
will render the block as HTML. It just so happens a Mage_Core_Block_Template block's toHtml method renders the block by loading a phtml template set in the template variable.
Then, splitting apart this
$block = $this->getLayout()->createBlock('CLASS_GROUP_NAME/PATH_TO_BLOCK_FILE')->toHtml();
This
$this->getLayout()->createBlock('CLASS_GROUP_NAME/PATH_TO_BLOCK_FILE');
instantiates a block object. It gets the base class name from looking in the configuration for a node named <class_group_name/> under the <blocks/> configuration node. (for this example, let's say that's Packagename_Modulename_Block) — and then appending PATH_TO_BLOCK_FILE to the end of this. This is the same process as with core/template above. This gives us a block named Packagename_Modulename_Block_Path_To_Block_File.
Then this
$block->toHtml();
calls the Packagename_Modulename_Block_Path_To_Block_File's object's toHtml method, which renders the block. It maybe Packagename_Modulename_Block_Path_To_Block_File extends from Mage_Core_Block_Template, and has had it's template set inside itself, or it may be a block that contains different logic for rendering in toHtml.