Reimagining Photos: My Internship at Facebook

During my internship at Facebook, one of the projects I worked on was the new multi-photo story grid. I was responsible for the algorithm, its implementation on the server, and the client implementation on Android.

Before this past summer, Facebook used to show photo sets in carousels. This meant that all user interaction was based on how good the first picture looked. In the case below, for example, the first photo is of a door and the rest is of delicious food. It doesn’t make sense to just show a door in people’s newsfeeds. We wanted to show photos in a more beautiful and intuitive way than a carousel.

Old layout on the left. New layout on the right

The end result

Here are some more examples of what we managed to do:

The Problem

Display a set of photos in a way such that it increases engagement and is more visually appealing than a carousel. We could go about this by trying to dynamically generate an optimum layout for any given set of photos. However, there is no guarantee that this layout will be aesthetically pleasing. Because of this, we decided not to generate layouts on the fly, but instead to pick the optimum layout from a set of given layouts.

We also decided to preserve the ordering of the photos in an album and start with the first photo. This gives a user the ability to tell a story with their pictures.

After lots of deliberation, we came up with this initial set of layouts (which has since been modified):

Finding the Optimal Layout

How do we decide if a layout is optimal for a set of photos? In order to tune the algorithm, I wrote a simple application that rendered a set of photos in every layout that we deemed remotely useful for that set of photos. We had some engineers on the photos team play with it and look at sets of photos from their own newsfeeds. The app let them select the layouts they thought fit the best for any given set of photos.

This gave us data to compare the algorithm output to. We also spent a lot of time playing around with test builds and checking if the algorithm was giving us sensible layouts.

Layout Definition

The layouts are based on a 6 x 6 grid where each photo container is defined by its top left corner, its width, and its height. For example, we may define some layouts as follows:

[
    [ // 1 photo
        [0, 0, 6, 6]
    ],
    [ // 2 photos
        [[0, 0, 6, 3], [0, 3, 6, 3]], // Two landscape photos
        [[0, 0, 3, 3], [3, 0, 3, 3]]  // Two squares side by side
    ],
    ...
]

Each photo container is formatted as [x, y, w, h] and each row of photo containers is a layout.

The Algorithm

So how do we pick a layout? Here are some things we considered:

• Number of pixels cropped
• Faces
• Tags
• Feedback (comments, likes)
• Aspect ratios of photos vs aspect ratios of containers

After lots of testing, we settled on a score based approach for the algorithm. We iterate through all the layouts and calculate a score for each of them. We then select the layout with the highest score.

To calculate the score for a given layout, we consider many factors (such as the ones listed above). We iterate through every container in the layout and determine if it “fits” the picture that would go in that container. For example, if the aspect ratios match, we’ll increase the score based on how well they match. We may also add to the score based on other factors that make the layout more appealing (e.g., the number of visible faces). If we assume photos with lots of likes and comments are better content, we may want them to be larger. We can do this by adding another factor proportional to the size of the container and amount of feedback that the photo has.

This algorithm is implemented server-side so it can be reused across all the mobile and web clients.

Once we decide on a layout, we send the layout specification down to the client along with the set of photos to display. The client then renders the photos appropriately.

This new way of displaying photos makes the viewing experience much better and greatly increases user engagement (unfortunately, I can’t share the exact stats). Thanks to Rocky Smith, JP Ren, Christopher Chedeau, and Thai Tran for an awesome experience working on this project!

Let me know if you have any questions ([my initials]@[my name].com)!

Join the conversation on Hacker News!

Header image from Thomas Hawk