![]() Our use of Spanner has proven Spanner’s ability to scale rapidly without compromise - something rare in traditional, vertically scalable SQL databases.Įqually as important, Spanner has significantly increased our operational efficiency. It’s Spanner’s high write throughput, consistency guarantees, and resource management tools that have allowed Photos to build and scale these features and pipelines by 10x with minimal re-architecture. That’s not surprising when you consider the variety of first- and third-party clients that upload media, processing pipelines performing updates, and various feature needs – many of which involve cross-user sharing. Photos also has strict consistency and concurrency needs. We need to serve our active online users quickly because we know they expect their photos to be instantaneously displayed and shareable. We have read-only and read/write shards to separate these use cases. Spanner’s sharding flexibility allows both these use cases to be satisfied in the same database. The system has to be reliable and available for user uploads, while simultaneously ensuring that ML-based features not only perform well, but also don’t impact interactive traffic. Spanner’s sharding capabilities give us low latency worldwide, help us smooth our computational workloads, and make it easy for us to support the ever increasing set of regulatory requirements concerning data residency. Photos relies on Spanner to automatically replicate data with 99.999% availability. Photos enables users to access all of their photos at any time, reliably across the globe. Balancing our online and offline traffic is also manageable thanks to Spanner’s workload tunable replication capabilities. Thanks to Spanner’s traffic isolation, capacity management, and automatic sharding capabilities, we are able to provide a highly reliable user experience even with unpredictably bursty traffic loads. High level architecture for media processing in Google Photos using Spannerĭespite Google Photos’ size and complexity, Spanner has a number of features that make our integration easy to maintain. Photos also has dozens of large batch-processing Flume pipelines that power our most expensive workloads: AI/ML processes, data integrity management, and other types of full account or database-wide processing. The Photos backend is composed of dozens of microservices, all of which interact with Spanner in different ways, some serving user-facing traffic, and others handling batch traffic. We store all the metadata that powers Google Photos in Spanner, including both media-specific and product-specific metadata for features like album organization, search, and clustering. With Photos storing over 4 trillion photos and videos, we need a database that can handle a staggering amount of data with a wide variety of read and write patterns. All of this data can be accessed and experienced in delightful ways thanks to machine learning-powered features like search, suggested edits, suggested sharing, and Memories. Users can automatically back up, organize, edit, and share their photos and videos with friends and family. Google Photos offers a complete consumer photo workflow app for mobile and web. Spanner has proved to be exactly the database we needed. With more than one billion users and four trillion photos and videos - and with the responsibility to protect personal, private, and sensitive user data - Google Photos needs a database solution that is highly scalable, reliable, secure, and supports large scale data processing workloads conducive to AI/ML applications. Google Photos aims to be the home for all these memories, organized and brought to life so that users can share and save what matters. This has resulted in explosive growth in the number of photo and video captures, and a huge portion of these photos and videos contain private, cherished, and beloved memories - everything from small, everyday moments to life’s biggest milestones. ![]() Mobile photography has become ubiquitous over the past decade, and it’s now easier than ever to take professional quality photos with the push of a button.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |