Post on 14-Jan-2017
RecSys FR: 4th Session
6th October 2016
Etienne Sanson – manager R&D Engine
About Criteo
1
3 | Copyright © 2016 Criteo
Our mission
TARGET THE RIGHT USER
AT THE RIGHT TIME
WITH THE RIGHT MESSAGE
4 | Copyright © 2016 Criteo
Key Figures
16 000 PUBLISHERS
90% RETENTION RATE2
+130COUNTRIES
LISTED ON THE
NASDAQ SINCE
OCTOBER 2013
R&D REPRESENTS 21% OF THE WORKFORCE
2000EMPLOYEES
21 BILLIONS $3
11 000 ADVERTISERS
1.19 bn€1
31OFFICES
1: REVENUE IN 20152: ANNUAL RATE 2015
3: $ OF TURNOVER GENERATED TO OUR CLIENTS - TURNOVER POST-CLICK WW FROM JANUARY TO DECEMBER 2015
5 | Copyright © 2016 Criteo
Revenue Growth
2009 2010 2011 2012 2013
22M$86M$
199M$
349M$
589M$
988M$
2014
1,3MM$
2015
6 | Copyright © 2016 Criteo
GENERAL CONCEPT
Users visit an advertiser’s website
1
Criteo identifies the users (via cookies)
2
Users leave the advertiser’s website& browse publisher on the Internet
3
Criteo identifies users on these pages(via cookie)
4
Criteo displays an advertising banner, personalized for
each user
5
Click through directlyto the advertiser’s
page
6
@
Retargeting principles
7 | Copyright © 2016 Criteo
Infrastructure Key Figures
Sunnyvale2 PoP
500 kVA1 559 Servers
New York2 PoP
930 kVA2 625 Servers
Hong Kong2 PoP
472 kVA2185 Servers
Paris4 Pop
1 800 kVA3 625 Servers
Amsterdam2 PoP
+2 500 kVA3 609 Servers
Tokyo2 PoP
455 kVA2 564 Servers
Shanghai1 PoP
200 kVA931 Servers
World Wide15 PoP
6,8 MVA17 098 Servers
> 55Gbps+ 2.5M req/s
Hosting Global Partners :
About ML@Criteo
2
9 | Copyright © 2016 Criteo
Our challenges – Product recommendation
• Select the best ~10 products to show to a user
• >1B users• Product catalog contains ~1M items, up to 1B• Time constraints: 20ms
• Combination of offline/online processing steps
• CF• Product embeddings (word2vec -> prod2vec)• CNN for image features
What products should we recommend?
10 | Copyright © 2016 Criteo
Our challenges - Bidding
How much should we bid for this display?
What is the best campaign to display?
My company
BUY! BUY! BUY!
BUY!
• Select the best campaign to display and evaluate its value in a few ms
• Large scale regression models• >1B daily displays (but few positive examples!)• >1M parameters
• Distributed optimization (SGD, L-BFGS)• Feature Engineering• Transfer learning, FFM, Policy learning• Marketplace, game theory, auction theory
11 | Copyright © 2016 Criteo
Our challenges – X-device
• Build a huge graph (Billions of nodes/edges):• Nodes = devices• Edge = the 2 devices belong to the same user
• How to connect 2 devices?• How to know the ground truth?• How to keep it stable?
• At scale & taking care about privacy
Who is the user behind the device?
12 | Copyright © 2016 Criteo
Our challenges – Testing
• We test everything!
• Offline tests / AB Tests• Infrastructure to perform large-scale tests
• >100K offline tests / year• >1K AB Tests / year• Dedicated teams
• Technical / Business Metrics• Randomization• Counterfactual evaluation
Thank you!
…and we’re hiring!