Collocations of the form division-name + number—Volume II, Book 2, Chapter Four, Section 3, Illustration C—are capitalized because they are taken to be names of the entities they refer to, as may be seen from the fact that (just like personal names) they take determiners only when modified by preposed adnominals.
okWe find in Chapter Four that ...
okWe find in the very badly written Chapter Four that ... but not
∗ We find in the Chapter Four that ...
Note, however, that these names for larger divisions—volume, book, chapter—should be used only if they actually appear in the source; if the entire heading on the final chapter is "Happy Endings" you shouldn't call it "Chapter Sixty".
With smaller divisions—section, subsection, paragraph, page and the like—use varies. As a rule of thumb, you shouldn't use 'name' form unless the division you refer to is marked with at least a distinguishing number or letter. If this isn't further marked with an explicit division-name, you may introduce a division-name of your own, such as "section 3"; this will be treated as a name syntactically, but usually without capitalization:
We find in section 3 that . . .
It is remarked on page 267 that . . .
Sometimes, however, you will find that an explicit division-name is given in another context. For instance, I am now working with a document which has thirty numbered and titled sections: "4. Quantity", "5. Weight", "6. Commodity" There are also internal references to these sections, and these take use the explicit division-name "Clause":
... in bulk, including dockage, 5% more or less at buyer's option, and at market price (per Clause 10) ...
So in my discussion I refer to these as "Clause 1", "Clause 2" and so forth.
In other sorts of collocation these designations are common nouns; they are not capitalized and take a determiner
okWe find in the fourth chapter that ...
okWe find in the very badly written chapter on framistats that ... but not
∗ We find in fourth chapter that ...