Optical resolution describes the ability of an imaging system to resolve detail in the object that is being imaged. An imaging system may have many individual components including a lens and recording and display components. Each of these contributes to the optical resolution of the system, as will the environment in which the imaging is done.

Lens resolution The ability of a lens to resolve detail is usually determined by the quality of the lens but is ultimately limited by diffraction. Light coming from a point in the object diffracts through the lens aperture such that it forms a diffraction pattern in the image which has a central spot and surrounding bright rings, separated by dark nulls; this pattern is known as an Airy pattern, and the central bright lobe as an Airy disk. The angular radius of the Airy disk (measured from the center to the first null) is given by

 $\sin \theta = 1.22 \frac{\lambda}{D}$ where θ is the angular resolution, λ is the wavelength of light, and D is the diameter of the lens aperture.

Two adjacent points in the object give rise to two diffraction patterns. If the angular separation of the two points is significantly less than the Airy disk angular radius, then the two points cannot be resolved in the image, but if their angular separation is much greater than this, distinct images of the two points are formed and they can therefore be resolved. Rayleigh defined the somewhat arbitrary "Rayleigh criterion" that two points whose angular separation is equal to the Airy disk radius to first null can be considered to be resolved. It can be seen that the greater the diameter of the lens or its aperture, the greater the resolution. Astronomical telescopes have increasingly large lenses so they can 'see' ever finer detail in the stars.

Only the very highest quality lenses have diffraction limited resolution, however, and normally the quality of the lens limits its ability to resolve detail. This ability is expressed by the Optical Transfer Function which describes the spatial (angular) variation of the light signal as a function of spatial (angular) frequency. When the image is projected onto a flat plane, such as photographic film or a solid state detector, spatial frequency is the preferred domain, but when the image is referred to the lens alone, angular frequency is preferred. OTF may be broken down into the magnitude and phase components as follows:

$\mathbf{OTF(\xi,\eta)}=\mathbf{MTF(\xi,\eta)}\cdot\mathbf{PTF(\xi,\eta)}$

where

$\mathbf{MTF(\xi,\eta)} = | \mathbf{OTF(\xi,\eta)} |$
$\mathbf{PTF(\xi,\eta)} = e^{-i 2\cdot\pi\cdot\lambda (\xi,\eta)}$
and (ξ,η) are spatial frequency in the x- and y-plane, respectively.

The OTF accounts for aberration, which the limiting frequency expression above does not. The magnitude is known as the Modulation Transfer Function (MTF) and the phase portion is known as the Phase Transfer Function (PTF).

In imaging systems, the phase component is typically not captured by the sensor. Thus, the important measure with respect to imaging systems is the MTF.

Phase is critically important to adaptive optics and holographic systems.

Sensor resolution (spatial)

Some optical sensors are designed to detect spatial differences in EM (electro-magnetic) energy. These include photographic film, solid-state devices (CCD, CMOS detectors, and infrared detectors like PtSi and InSb), tube detectors (vidicon, plumbicon, and photomultiplier tubes used in night-vision devices), scanning detectors (mainly used for IR), pyroelectric detectors, and microbolometer detectors. The ability of such a detector to resolve those differences depends mostly on the size of the detecting elements.

Spatial resolution is typically expressed in line pairs per millimeter (lppmm), lines (of resolution, mostly for analog video), contrast vs. cycles/mm, or MTF (the modulus of OTF)). The MTF may be found by taking the two-dimensional Fourier transform of the spatial sampling function. Smaller pixels result in wider MTF curves and thus better detection of higher frequency energy.

This is analogous to taking the Fourier transform of a signal sampling function; as in that case, the dominant factor is the sampling period, which is analogous to the size of the picture element (pixel).

Other factors include pixel noise, pixel cross-talk, substrate penetration, and fill factor.

A common problem among non-technicians is the use of the number of pixels on the detector to describe the resolution. If all sensors were the same size, this would be acceptable. Since they are not, the use of the number of pixels can be misleading. For example, a 2 megapixel camera of 20 micrometre square pixels will have worse resolution than a 1 megapixel camera with 8 micrometre pixels, all else being equal.

For resolution measurement, film manufacturers typically publish a plot of Response (%) vs. Spatial Frequency (cycles per millimeter). The plot is derived experimentally. Solid state sensor and camera manufacturers normally publish specifications from which the user may derive a theoretical MTF according to the procedure outlined below. A few may also publish MTF curves, while others (especially intensifier manufacturers) will publish the response (%) at the Nyquist limiting frequency, or, alternatively, publish the frequency at which the response is 50%.

To find a theoretical MTF curve for a sensor, it is necessary to know three characteristics of the sensor: the active sensing area, the area comprising the sensing area and the interconnection and support structures ("real estate"), and the total number of those areas (the pixel count). The total pixel count is almost always given. Sometimes the overall sensor dimensions are given, from which the real estate area can be calculated. Whether the real estate area is given or derived, if the active pixel area is not given, it may be derived from the real estate area and the fill factor, where fill factor is the ratio of the active area to the dedicated real estate area.