When you look at the Café Wall, you should see the 'mortar lines' (the horizontal lines between the tiles) slope alternately upward and downward to the right. This gives the impression that the tiles are wedge-shaped. The lines are actually parallel, and the tiles are all perfectly square and of the same size, as can be verified when the mortar lines disappear.
The Café Wall illusion demonstrates the effect of some simple image processing occurring at the retina combined with some complex processing in the cortical cells of the striate cortex. The incoming image is filtered by the center-surround operator of the retina (similar to a Laplacian or Difference-Of-Gaussians operator). If the receptive field of the operator is larger than the mortar width, a set of diagonal bands, similar to Fraser's twisted cords, will appear in place of the mortar line.
The apparent tilt of the mortar lines is caused by orientation-sensitive simple cells in the striate cortex. The cells interact with one another to interpret the diagonal bands produced by the retina as a single continuous line, tilted in the direction of the diagonal bands.
References:
Lulich, D.P. & Stevens, K.A. (1989). Differential Contributions of Circular and Elongated Spatial Filters to the Café Wall Illusion. Biological Cybernetics 61, 427-435.