Brueghel's Fall of IcarusDaniel Rockmore writes about the mathematical analysis of art in the June 2006 Chronicle (The Style of Numbers Behind a Number of Styles). In the essay Rockmore describes Richard Taylor's work in analyzing Jackson Pollock pieces, which may be forgeries. ( See my post on this topic)
The Pollock intro is a lead in to a description of stylometry - the mathematical/scientific analysis of literary texts that attempts to address issues of authorship. (See Bookish Math, an excellent intro to stylometry by Erica Klarreich for Science News Online.) Rockmore than describes a method he developed with co-workers Siwei Lyu and Hany Farid that uses wavelet analysis to determine unique "signatures" of different artists - in effect a stylometry for visual images.
The actual mathematics of the wavelet approach can be found in A Digital Technique for Art Authentication. Here the authors use examples of Pieter Bruegel and Perugino to test their model. They claim that their "techniques, in collaboration with existing physical authentication, to play an important role in the field of art forensics."
The wavelet technique is different from Taylor's fractal analysis of Pollock's works, but both are examples of stylometry applied to visual information. Both Taylor and Rockmore are attempting to quantify art, an activity that Rockmore admits is unsettling/impossible to some. According to Taylor, this quantification should be expected: "Both mathematics and art are all about pattern...it would be unusual that you would not apply mathematical analysis to the question."
Rockmore is more explicit about what mathematical categorization of art analysis does not do: "Fractal analysis doesn't diminish Pollock's athleticism and movement, nature and turbulence, chaos and beauty; it reveals and amplifies it."
For more on this topic, see Can Mathematical Tools Illuminate Artistic Style?, by Sara Robinson for SIAM.