You can do it by code:
Create a file called taxclass.php on the same level as index.php with the following content.
error_reporting(E_ALL | E_STRICT);
define('MAGENTO_ROOT', getcwd());
$mageFilename = MAGENTO_ROOT . '/app/Mage.php';
require_once $mageFilename;
Mage::setIsDeveloperMode(true);
ini_set('display_errors', 1);
umask(0);
Mage::app();
$productIds = Mage::getResourceModel('catalog/product_collection')->getAllIds();
Mage::getSingleton('catalog/product_action')->updateAttributes(
$productIds,
array('tax_class_id' => 2),
0
);
If you still want to do it in a query you can identify the tax class attribute like this:
SELECT
*
FROM
eav_attribute
WHERE
attribute_code = 'tax_class_id' AND
entity_type_id = (SELECT entity_type_id FROM eav_entity_type where entity_type_code = 'catalog_product')
then run your update with the attribute id your find. But this won't work if you have products that don't have a class id set.