The Smartphone Camera Story

The smartphone camera, in less than twenty years, has become the most-used camera in human history. More photographs are now taken every two days than were taken in the entire nineteenth century, and the vast majority of them are taken with a phone. The story of how a sensor the size of a fingernail, paired with a lens no thicker than a coin, came to outperform a professional DSLR of the 2000s is, in retrospect, one of the most surprising and consequential stories in the history of photography.

The optical camera: 200 years of physics

The first photograph, taken by Joseph Nicéphore Niépce in 1826 or 1827, required an exposure of several days and produced a barely recognisable image of a courtyard. The first practical process, the daguerreotype introduced by Louis Daguerre in 1839, required an exposure of several minutes. The first instant-photography process, the Polaroid, was introduced by Edwin Land in 1948. The first digital camera, the Dycam Model 1, was sold in 1990. The first camera phone, the Sharp J-SH04, was sold in Japan in 2000. The first iPhone, with a 2-megapixel camera, was sold in 2007. The progression, from 8-hour exposures to multi-frame computational photography, took about 200 years.[1, 2, 9]

The physics of a camera has not changed in 200 years. A camera is a light-tight box with a lens at one end and a light-sensitive surface at the other. The lens forms an image of the scene on the surface; the surface records the image. The amount of light that reaches the surface is set by the area of the lens (the entrance pupil) and the time the shutter is open. The amount of light recorded is set by the sensitivity of the surface. The resolution of the image is set by the number of grains (in film) or pixels (in a digital sensor). The depth of field is set by the aperture of the lens and the distance to the subject. The trade-offs are fundamental: a larger lens collects more light; a faster shutter freezes motion; a smaller aperture increases depth of field; a more sensitive surface allows shorter exposures. Every camera, from the daguerreotype to the iPhone, has to make these trade-offs, and the optimal design depends on the application.[1, 2, 9]

The digital revolution, which began in the 1970s and accelerated in the 1990s, replaced the film with a silicon sensor. The charge-coupled device (CCD), invented at Bell Labs in 1969, was the first practical solid-state image sensor, and the early digital cameras of the 1980s and 1990s used CCDs. The CMOS sensor, which is fabricated using the same process as a digital logic chip, became dominant in the 2000s because it could be integrated with on-chip analogue-to-digital converters, reducing the cost and the power. The modern smartphone image sensor is a back-side-illuminated stacked CMOS, with the photodiodes on the back of the chip (to avoid the metal wiring on the front) and the analog and digital circuitry stacked underneath. The result is a sensor with 50-megapixel resolution, less than 1-micrometre pixel pitch, and 14-bit dynamic range, all in a package less than 1 cm across.[1, 2, 10]

The challenge of miniaturisation

The smartphone form factor imposes severe constraints on the camera. The phone is, typically, 7–8 mm thick, and the camera module must fit within this thickness, including the lens, the sensor, and the autofocus mechanism. The result is that the lens is at most 5–6 mm thick, and the sensor is at most 4–5 mm deep. The area of the lens is constrained by the thickness and the desired focal length: a 26-mm-equivalent lens (the standard for a smartphone) at f/1.6 has an entrance pupil of about 16 mm, which is achievable in a flat package, but the design requires careful optical engineering to correct for aberrations across a wide field of view.[2, 3, 11]

The first smartphone cameras, in the 2000s, had small sensors (perhaps 3 megapixels) and slow lenses (f/2.8 or slower), and the image quality was, by modern standards, poor. The improvements over the past 15 years have been driven by a combination of better sensor technology (larger pixels, back-side illumination, stacked CMOS), better lens technology (more elements, better coatings, wider apertures), and better image processing. The sensor pixel size has, paradoxically, fallen: the original iPhone had 2-micrometre pixels; the iPhone 15 Pro has 2.44-micrometre pixels, despite having a much higher resolution. The improvement has come from the improved quantum efficiency (more of the incident photons are converted to electrons), from the lower readout noise, and from the use of multi-frame processing to combine many noisy frames into a single clean image.[2, 3, 11]

The lens design has also improved. The first smartphone lenses were 4-element designs, sufficient for a small sensor and a slow lens. The modern smartphone lens is a 7- or 8-element design, with several aspheric surfaces and a wide aperture, optimised for the small sensor and the wide field of view. The lens elements are made of plastic, not glass, because plastic can be moulded into aspheric shapes more easily and cheaply. The trade-off is that plastic lenses are more susceptible to temperature and humidity changes, and to scratching. The best smartphone lenses now use glass elements for the most demanding surfaces, particularly in telephoto and ultra-wide lenses.[2, 3, 11]

Computational photography: when software replaces glass

The fundamental insight of computational photography is that the camera is a software device, not a hardware device. The hardware (sensor, lens) captures a raw image, but the image that the user sees is the result of a complex pipeline of processing: demosaicing, denoising, sharpening, tone mapping, white balance, exposure fusion, and (in recent years) semantic segmentation and neural network enhancement. The raw image from a modern smartphone sensor is, in a real sense, an unrecognisable mess; the beautiful image that the user sees is largely the work of the image signal processor (ISP) and the neural network.[3, 4, 12]

The most important computational technique is multi-frame fusion. The camera captures a burst of 10–30 frames in quick succession (typically at 30–60 fps), aligns them at the pixel level (correcting for hand shake and subject motion), and combines them into a single image. The combination reduces noise (each frame has independent noise, which averages out), increases dynamic range (different frames are exposed for different parts of the scene), and improves detail (the best frame for each region is used). The technique, called HDR+ by Google and Deep Fusion by Apple, is now standard in every high-end smartphone, and is the reason that a phone camera can take a usable photo in conditions that would defeat a 2005 DSLR.[3, 4, 12]

The second most important technique is semantic segmentation, in which a neural network identifies the different regions of the image (sky, face, hair, skin, clothing, background) and processes each region differently. The sky can be enhanced for colour and contrast, the face can be retouched for skin tone and smoothness, the hair can be sharpened to bring out the strands, and the background can be blurred to simulate a shallow depth of field. The technique, called Semantic Image by Apple and Scene Recognition by Samsung, is the reason that phone photos often look more pleasing than DSLR photos of the same scene, even though the DSLR captured more light. The phone is, in a sense, cheating — but the cheating is what the user wants.[3, 4, 12]

Multi-frame, multi-lens: the modern smartphone

The modern high-end smartphone has three or four lenses, each optimised for a different focal length. The standard wide lens (about 26 mm equivalent) has a fast aperture (f/1.6 or wider) and a large sensor, and is used for most photos. The ultra-wide lens (about 13 mm equivalent) has a smaller aperture (f/2.2) and a smaller sensor, and is used for landscapes and for indoor shots. The telephoto lens (about 77 mm or 120 mm equivalent) has a moderate aperture (f/2.8) and a small sensor, and is used for portraits and for distant subjects. The combination of the three lenses, with the option of digital zoom between them, gives the user a focal-length range of about 13–200 mm, comparable to a DSLR with a 24–200 mm zoom lens.[4, 5, 13]

The telephoto lens is, in many ways, the most interesting. The physical constraints of the smartphone make a long focal length difficult: a 77 mm lens would require a depth of at least 15 mm, which is more than the thickness of the phone. The solution, in modern smartphones, is a periscope lens, in which the light is bent by a prism to travel horizontally through the phone, and then back to a sensor near the top. The periscope design, first used in a smartphone by Huawei in 2019, allows focal lengths of 100–200 mm in a thin phone. The trade-off is a smaller sensor and a slower aperture, but the combination of the periscope with computational zoom (a neural network upscales the image, adding detail that the sensor could not resolve) produces surprisingly good results.[4, 5, 13]

The other notable feature of the modern smartphone camera is the dedicated time-of-flight (ToF) or lidar sensor. The ToF sensor emits a brief pulse of infrared light and measures the time it takes to return, producing a depth map of the scene with centimetre accuracy. The depth map is used to enable portrait mode (the synthetic bokeh effect), to improve autofocus in low light, and to enable augmented-reality applications. The iPhone Pro models, the Samsung Galaxy S Ultra, and many other high-end phones now include a lidar or ToF sensor, and the technology is becoming standard in the mid-range.[4, 5, 13]

AI photography and the future

AI has transformed image processing in the 2020s. The introduction of dedicated neural processing units (NPUs) in smartphones, starting with the Apple A11 in 2017 and the Qualcomm Hexagon in 2018, allowed complex neural networks to run in real time on every frame. The result is a new generation of “AI photography” features that would have been impossible on a smartphone just a few years earlier. The most important are: night mode, in which the camera captures 30 or more frames over several seconds, aligns them, and uses a neural network to denoise and demosaic them, producing a usable image in near-darkness; computational zoom, in which a neural network upscales a 3x optical zoom to a 10x or 20x hybrid zoom with surprisingly good detail; semantic portrait, in which the network segments the subject from the background and applies a synthetic bokeh that mimics the look of a fast lens on a full-frame sensor; and AI scene recognition, in which the network identifies the type of scene (food, sunset, pet, plant) and adjusts the processing accordingly.[5, 6, 14]

The next step is generative AI, in which the camera does not just process the image but generates content that was not in the original scene. Adobe’s Generative Fill and similar tools can remove unwanted objects, replace the sky, or add elements that were not in the original shot. The Samsung Galaxy S24 introduced Generative Edit, which allows the user to move, resize, or erase objects in a photo, with the missing pixels filled in by an on-device generative model. The implications are profound: a photograph is no longer a record of a moment, but a creative artefact shaped by the user and the algorithm. The line between photography and image generation is, in this sense, blurring.[5, 6, 14]

The other frontier is computational video. The techniques that have transformed still photography — multi-frame fusion, semantic segmentation, neural network enhancement — are now being applied to video. The iPhone 15 Pro, in 2024, can record 4K video at 60 fps with real-time HDR and semantic processing, and the Google Pixel 8 Pro can record 4K video with continuous AI enhancement. The next step is computational video that uses a learned prior to fill in missing information, allowing extreme low-light video, real-time slow motion at very high frame rates, and even video that is “impossible” in the conventional sense (e.g. video of a person in complete darkness, with the face lit only by the noise of the sensor). The next decade of smartphone photography will be, in this sense, a story of the gradual replacement of the optical image by the computational image.[5, 6, 14]

Limits, ethics, and the meaning of a photograph

The remaining limits of the smartphone camera are physical, not algorithmic. The smartphone sensor is still much smaller than a DSLR sensor, and the lens is still much smaller. The result is that the smartphone still cannot match the DSLR in low light (a DSLR with a 50 mm f/1.4 lens collects perhaps 20 times more light than a phone with a 26 mm f/1.6 lens), in dynamic range (a modern full-frame DSLR can capture 14–15 stops of dynamic range, vs 10–12 stops for a phone), and in shallow depth of field (a phone cannot produce the same bokeh as a full-frame sensor with a fast lens). The progress in computational photography is closing the gap, but the gap remains. For professional work — sports, wildlife, fashion, product photography — the DSLR or mirrorless camera is still the tool of choice, and will be for the foreseeable future.[6, 7, 15]

The deeper story is about the meaning of a photograph. The smartphone camera, by making photography universal, has changed what a photograph is. A photograph is no longer a careful composition, taken with a deliberate choice of lens, exposure, and moment; it is a snapshot, taken in a fraction of a second, and post-processed by an algorithm before the user even sees it. The “raw” output of a modern smartphone is, in a real sense, an AI-generated image that uses the sensor data as one input among many. The implications for journalism, for evidence, for memory, and for the authenticity of photographs are profound, and are still being worked out.[6, 7, 16]

The arrival of generative AI has accelerated the crisis. The same techniques that allow a phone to denoise a low-light photo can be used to generate a photo that never existed. The same tools that allow a user to remove a power line from a landscape shot can be used to put a person in a place they never were. The provenance of a photograph — the chain of evidence that it depicts what it claims to depict — is now a technical problem, and the technical solution is the Coalition for Content Provenance and Authenticity (C2PA), an industry standard for cryptographically signing the provenance of digital content. The smartphone camera, in other words, is not just a piece of technology but a cultural shift, and its consequences for what we consider a “real” photograph are still unfolding.[6, 7, 16]

Comparison: smartphone camera specs (2024)

Phone	Wide	Ultra-wide	Telephoto	Sensor size (wide)	AI features
iPhone 15 Pro Max	48 MP, 24 mm, f/1.78	12 MP, 13 mm, f/2.2	12 MP, 120 mm periscope, f/2.8	1/1.28″	Deep Fusion, Photonic Engine, Generative Edit
Samsung Galaxy S24 Ultra	200 MP, 24 mm, f/1.7	12 MP, 13 mm, f/2.2	50 MP, 111 mm periscope, f/3.4	1/1.3″	ProVisual Engine, Generative Edit, AI Zoom
Google Pixel 8 Pro	50 MP, 25 mm, f/1.68	48 MP, 13 mm, f/1.95	48 MP, 113 mm periscope, f/2.8	1/1.31″	Magic Editor, Best Take, Audio Magic Eraser
Huawei P60 Pro	48 MP, 25 mm, f/1.4–4.0	13 MP, 13 mm, f/2.2	48 MP, 90 mm, f/2.1	1/1.4″	XD Fusion Pro, AI scene recognition
Xiaomi 14 Pro	50 MP, 23 mm, f/1.4–4.0	50 MP, 14 mm, f/2.2	50 MP, 75 mm, f/2.0	1/1.31″ (Light Fusion 800)	Leica optics, AI portrait

Key numbers

• ~1.5 trillion: photographs taken worldwide in 2024 (estimated).
• ~95 percent: share of photographs taken on a smartphone (2024).
• ~200 MP: maximum resolution of a smartphone camera (2024).
• ~f/1.4: widest aperture of a smartphone lens (Huawei, 2024).
• ~1/1.3″: largest smartphone image sensor (2024).
• ~120 mm: longest periscope telephoto on a smartphone (iPhone 15 Pro Max, 2024).
• 1826: year of the first photograph (Niépce).
• 2000: year of the first camera phone (Sharp J-SH04).
• 2007: year of the first iPhone.

References

1. Wikipedia, “History of the camera.” (Standard reference on the 200-year history of the camera.) https://en.wikipedia.org/wiki/History_of_the_camera
2. Wikipedia, “Camera phone.” (Standard reference on the camera phone, with a section on the history.) https://en.wikipedia.org/wiki/Camera_phone
3. Wikipedia, “Computational photography.” (Standard reference on computational photography.) https://en.wikipedia.org/wiki/Computational_photography
4. Wikipedia, “High dynamic range imaging.” (Standard reference on HDR, the multi-frame exposure-fusion technique used in smartphones.) https://en.wikipedia.org/wiki/High_dynamic_range
5. Wikipedia, “Apple A11.” (Standard reference on the Apple A11, the first smartphone SoC with a dedicated neural engine.) https://en.wikipedia.org/wiki/Apple_A11
6. Wikipedia, “Neural processing unit.” (Standard reference on the NPU, the dedicated hardware that runs AI models on smartphones.) https://en.wikipedia.org/wiki/Neural_processing_unit
7. Wikipedia, “Full-frame digital SLR.” (Standard reference on the full-frame DSLR, the standard against which smartphone cameras are often compared.) https://en.wikipedia.org/wiki/Full-frame_digital_SLR
8. Wikipedia, “Joseph Nicéphore Niépce.” (Standard reference on Joseph Nicéphore Niépce, who took the first photograph in 1826.) https://en.wikipedia.org/wiki/Joseph_Nic%C3%A9phore_Ni%C3%A9pce
9. Wikipedia, “Daguerreotype.” (Standard reference on the daguerreotype, the first practical photographic process.) https://en.wikipedia.org/wiki/Daguerreotype
10. Wikipedia, “Charge-coupled device.” (Standard reference on the CCD, the first solid-state image sensor.) https://en.wikipedia.org/wiki/Charge-coupled_device
11. Wikipedia, “Back-illuminated sensor.” (Standard reference on back-side-illuminated sensors, now standard in smartphones.) https://en.wikipedia.org/wiki/Back-illuminated_sensor
12. Wikipedia, “Image signal processor.” (Standard reference on the ISP, the dedicated chip that processes raw sensor data.) https://en.wikipedia.org/wiki/Image_signal_processor
13. Wikipedia, “Periscope lens.” (Standard reference on the periscope lens used in modern smartphones for telephoto.) https://en.wikipedia.org/wiki/Periscope_lens
14. Wikipedia, “Deep Fusion.” (Standard reference on Apple’s Deep Fusion, the multi-frame processing pipeline.) https://en.wikipedia.org/wiki/Deep_Fusion_(Apple)
15. Wikipedia, “Mirrorless camera.” (Standard reference on the mirrorless camera, the modern replacement for the DSLR.) https://en.wikipedia.org/wiki/Mirrorless_camera
16. Wikipedia, “Coalition for Content Provenance and Authenticity.” (Standard reference on C2PA, the industry standard for digital content provenance.) https://en.wikipedia.org/wiki/Coalition_for_Content_Provenance_and_Authenticity