I'm trying to work out what the correct solution is for a DCT of a 4x4 checkerboard matrix (let's call it A) and for a matrix of ones (let's call it B). So A is:
0 1 0 1
1 0 1 0
0 1 0 1
1 0 1 0
And B is:
1 1 1 1
1 1 1 1
1 1 1 1
1 1 1 1
With Octave for dct(A) I get:
1.00000 1.00000 1.00000 1.00000
-0.38268 0.38268 -0.38268 0.38268
0.00000 0.00000 0.00000 0.00000
-0.92388 0.92388 -0.92388 0.92388
And for dctmtx(4) * A * transpose(dctmtx(4)) (which should be equivalent) I get:
2.00000 0.00000 -0.00000 -0.00000
0.00000 -0.29289 -0.00000 -0.70711
-0.00000 -0.00000 0.00000 -0.00000
-0.00000 -0.70711 -0.00000 -1.70711
And then dct(B):
2 2 2 2
0 0 0 0
0 0 0 0
0 0 0 0
And dctmtx(4) * B * transpose(dctmtx(4)):
4 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
What's going on? Which is correct?