RATIONALE

Information on social media comprises of various modalities such as textual, visual and audio. NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, computational processing of Internet memes needs a hybrid approach. The growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter further suggests that we can not ignore such multimodal content anymore. To the best of our knowledge, there is not much attention towards meme emotion analysis. The objective of this proposal is to bring the attention of the research community towards the automatic processing of Internet memes. The task Memotion analysis will release 8K annotated memes - with human annotated tags namely sentiment, and type of humor that is, sarcastic, humorous, or offensive.

 

The Multimodal Social Media 
In the last few years, the growing ubiquity of Internet memes on social media platforms such as Facebook, Instagram, and Twitter has become a topic of immense interest. Memes, one of the most typed English words (Sonnad, 2018) in recent times. Memes are often derived from our prior social and cultural experiences such as TV series or a popular cartoon character (think: One Does Not Simply - a now immensely popular meme taken from the movie Lord of the Rings). These digital constructs are so deeply ingrained in our Internet culture that to understand the opinion of a community, we need to understand the type of memes it shares. (Gal et al., 2016) aptly describes them as performative acts, which involve a conscious decision to either support or reject an ongoing social discourse. 

Online Hate - A brutal Job:  The prevalence of hate speech in online social media is a nightmare and a great societal responsiblity for many social media companies. However, the latest entrant Internet memes (Williams et al., 2016) has doubled the challenge. When malicious users upload something offensive to torment or disturb people, it traditionally has to be seen and flagged by at least one human, either an user or a paid worker. Even today, companies like Facebook and Twitter rely extensively on outside human contractors from start-ups like CrowdFlower, or companies in the Philippines. But with the growing volume of multimodal social media it is becoming impossible to scale. The detection of offensive content on online social media is an ongoing struggle. OffenseEval (Zampieri et al., 2019) is a shared task which is being organized since the last two years at SemEval. But, detecting an offensive meme is more complex than detecting an offensive text – it involves visual cue and language understanding. This is one of the motivating aspects which encourages us to propose this task.

Multimodal Social Media Analysis - The Necessity:  The prevalence of hate speech in online social media is a nightmare and a great societal responsiblity for many social media companies. However, the latest entrant Internet memes (Williams et al., 2016) has doubled the challenge. When malicious users upload something offensive to torment or disturb people, it traditionally has to be seen and flagged by at least one human, either an user or a paid worker. Even today, companies like Facebook and Twitter rely extensively on outside human contractors from start-ups like CrowdFlower, or companies in the Philippines. But with the growing volume of multimodal social media it is becoming impossible to scale. The detection of offensive content on online social media is an ongoing struggle. OffenseEval (Zampieri et al., 2019) is a shared task which is being organized since the last two years at SemEval. But, detecting an offensive meme is more complex than detecting an offensive text – it involves visual cue and language understanding. This is one of the motivating aspects which encourages us to propose this task.

 

 

The Memotion Analyis Task 
Memes typically induce humor and strive to be relatable. Many of them aim to express solidarity during certain life phases and thus, to connect with their audience. Some memes are directly humorous whereas others go for sarcastic dig at daily life events. Inspired by the various humorous effects of memes, we propose three task as follows:

 

• Task A- Sentiment Classification: Given an Internet meme, the first task is to classify it as positive or negative meme. We presume that a meme is not neutral.


• Task B- Humor Classification: Given an Internet meme, the system has to identify the type of humor expressed. The categories are sarcastic, humorous, and offensive meme. If a meme does not fall under any of these categories, then it is marked as a other meme. A meme can have more than one category. For instance, Fig 3 is an offensive meme but sarcastic too.


• Task C- Scales of Semantic Classes: The third task is to quantify the extent to which a particular effect is being expressed. Details of such quantifications is reported in the Table 1. Appropriate annotated data will be provided.

 

We will release 8K human annotated Internet memes labelled with semantic dimensions namely sentiment, and type of humor that is, sarcastic, humorous, or offensive. The humor types are further quantified on a likert scale as in Table 1. The dataset will also contain the extracted captions/texts from the memes.

 

 

The Memotion Analyis Task 

Taking into account the sheer volume of photos shared each day on Facebook, Twitter, and Instagram, the number of languages supported on our global platform, and the variations of the text, the problem of understanding text in images is quite different from those solved by traditional optical character recognition (OCR) systems, which recognize the characters but don’t understand the context of the associated image (Sivakumar et al.,2018).

For instance, the caption for the Fig. 1 is sufficient to sense the sarcasm or even dislike towards a feminist man. In fact, the image has no significant role to play in this case and the provided text is good enough to sense the pun. But, in Fig. 2, the final punchline on the unavailability of Deep Learning based OCR tutorials is dependednt on the man’s expression in the meme who is trying to read a small piece of paper - the facial expression of the man aids in interpreting the struggle to find OCR tutorials online. To derive the intended meaning, someone needs to establish an association between the provided image and the text. If we solely process the caption, we will lose the humor and also the intended meaning (lack of tutorials). In Fig. 3, to establish the sense of provided caption along with the racism against the middle east woman, we need to process image, caption and dominant societal beliefs.

 

Data & Resources 

This shared task will release 8K annotated memes categorized into sentiment classes and the type of humor. As discussed earlier, both textual and visual cues are indispensable for meme emotion analysis. Thus, extracting caption from memes is an inevitable step towards automatic Meme emotion analysis. Keeping that in mind this shared task will also provide the caption from memes as a part of the dataset. The extraction is performed using the Google OCR system and then manually corrected by crowdsourcing services. Therefore, the performance of the participating team’s performances will not dependent on the OCR accuracy.

Quality: To ensure the quality of data annotation, we make sure to get each meme annotated by at least two annotators. After the SemEval2020, the labels for the test data will be released as well. We will ask the participants to submit their predictions in a specified format (within 24 hours), and the organizers will evaluate the results for each participant. We will make no distinction between constrained and unconstrained systems, but the participants will be asked to report what additional resources they have used for each submitted run.

 

Expected Impacts 

Understanding multimodal social media is the need of the hour. Internet memes are the integral part of today’s social media. The proposed task initiate the research on the given dimension by publishing qualitative annotated data, as the availability of properly annotated data is still under scarcity in this domain. Although the task will mainly be focusing on sentiment analysis and multi-class classification {sarcastic, humorous, offensive, and other}, the data will continue to serve meme research including problems such as meme generation and influence of cultural background.

 

Evaluation Ranking and the Baseline 

The metric for evaluating the participating systems will be as follows. For the Task-A, and Task-B, we will use averaged F1 score across all the classes (Positive/Negative-Sentiment and Sarcasm/Humor/Offensive/Regular- Semantic Analysis) with macro-averaged recall, since the latter has better theoretical properties than the former (Esuli and Sebastiani, 2015), and provides better consistency. For the Task-C, we will use averaged Mean Absolute Error across all the classes.

Baseline: A baseline system with clear description of the system - ML model details, features etc. will be made available through Github repository.

 

Organizers

	Dr. Amitava Das is a Lead Scientist at Wipro AI Labs, Bangalore, India. Currently, Dr. Das is actively working on Code-Mixing and Social Computing.
	Chhavi Sharma is a MS student in Computer Science at the IIIT Sri City. Her research interests lie in the field of Natural Language Processing and Computer Vision.
	Soujanya Poria is a presidentialpostdoctoral fellow at NanyangTechnological University, Sin-gapore. His main areas of re-search interest are NLP and sen-timent analysis.
	Prof. Bjorn Gamback os a Professor at NTNU Norway also Senior Advisor at RISE SICS AB in Stockholm, Sweden. He has been actively working on Machine Translation, Code-Mixing and Sentiment Analysis.
	Prof. Tanmoy Chakraborty is an Assistant Professor and a Ramanujan Fellow at Indraprastha Institute of Information Technology Delhi (IIIT-D). His broad research interests include Data Mining, Complex Networks, Social Computing, and NLP.

 

Student Organizers

	William Scott Paka is an undergraduate student at IIIT Delhi.
	Deepesh Bhageria is an undergraduate student at IIIT Sri City.

 

References

Andrea Esuli and Fabrizio Sebastiani. 2015. Optimizing text quantifiers for multivariate loss functions. ACM Trans. Knowl. Discov. Data, 9(4):27:1–27:27.

 

Jean H French. 2017. Image-based memes as sentiment predictors. In 2017 International Conference on Information Society (i-Society), pages 80–85. IEEE.

 

Noam Gal, Limor Shifman, and Zohar Kampf. 2016. it gets better: Internet memes and the construction of collective identity. New Media & Society, 18(8):1698–1714.

 

Hugo Gonc¸alo Oliveira, Diogo Costa, and Alexandre Miguel Pinto. 2016. One does not simply produce funny memes!–explorations on the automatic generation of internet humor. In Proceedings of 7th International Conference on Computational Creativity, ICCC.

 

V Peirson, L Abel, and E Meltem Tolunay. 2018. Dank learning: Generating memes using deep neural networks. arXiv preprint arXiv:1806.04510.

 

Viswanath Sivakumar, Albert Gordo, and Manohar Paluri. 2018. Rosetta: Understanding text in images and videos with machine learning.

 

Nikhil Sonnad. 2018. The worlds biggest meme is the word meme itself.

 

Amanda Williams, Clio Oliver, Katherine Aumer, and Chanel Meyers. 2016. Racial microaggressions and perceptions of internet memes. Computers in Human Behavior, 63:424–432.

 

Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Farra Noura, and Ritesh Kumar. 2019. Offenseval 2019.

 

IMPORTANT DATES

Trial data ready July 31, 2019

Training data ready September 4, 2019

Test data ready December 3, 2019

Evaluation start January 10, 2020

Evaluation end January 31, 2020

Paper submission due February 23, 2020

Notification to authors March 29, 2020

Camera ready due April 5, 2020

SemEval workshop Summer 2020