In the controller, you can get the referer url by using \Magento\Framework\App\Response\RedirectInterface.
For example: vendor/magento/module-customer/Controller/Account/Edit.php
/** @var \Magento\Framework\View\Result\Page $resultPage */
$resultPage = $this->resultPageFactory->create();
$block = $resultPage->getLayout()->getBlock('customer_edit');
if ($block) {
/** @var \Magento\Framework\App\Response\RedirectInterface $_redirect */
$block->setRefererUrl($this->_redirect->getRefererUrl()); // Set referer url in your block.
}
And then, the block we can get $block->getRefererUrl()