The portable hole is not flammable1
Portable hole as a magic item is resistant to fire, so it is not easily set on flame.
Flammable is not a defined game term, so we default to the dictionary for what it means. The dictionary definition of flammable (from Oxford Languages) is:
easily set on fire
Not everything that can burn is flammable, only things that can be inflamed easily.
The DMG states under Magic Item Resilience (p. 141) states that "Most magic items, other than potions and scrolls, have resistance to all damage". Damage Resistance is defined in the PHB (p. 197):
Some creatures and objects are exceedingly difficult or unusually easy to hurt with certain types of damage.
If a creature or an object has resistance to a damage type, damage of that type is halved against it.
So the hole as a magic item benefits from magic item resilience and will be exceedingly difficult to damage with fire, the very opposite of easy. Therefore, like most magic items, is not flammable, no matter what shape it is in, open or folded up.
1What if the DM rules the hole is not resilient?
The DM can rule that the portable hole does not benefit from magic item resilience's resistance to all damage, without overruling any rule as merely "most" items other than potions and scolls are resistant to all damage.
They DM also could decide that such a fine, thin cloth is indeed easily set aflame. In this case, the hole would be flammable in either form, and could be set on fire.
This clearly can happen when the hole is folded up, but even when the hole is open, the cloth must still be exposed, at least at the rim of the hole, because the description says that
You can use an action to close a portable hole by taking hold of the edges of the cloth and folding it up
If the edge of the cloth was not exposed, you could not take hold of it. So it could be ignited in either form.
The DM would have to rule about the size of the hole (from tiny for cloth mass to medium for diameter of unfolded cloth) and if it is fragile or resilient, to assign hit points following the rules in DMG p. 247, and would have to decide how much damage it takes per round from being ignited.