Text Mining of Twitter Data: Topic Modelling.
Abstract
Access to the Internet is becoming more affordable especially in Africa and with this the number of active social media users is also on the rise.
Twitter is a social media platform on which users post and interact with messages known as “tweets”. These tweets are usually short with a limit of 280 characters.
With over 100 million Internet users and 6 million active monthly users in Nigeria, lots of data is generated through this medium daily.
This thesis aims to gain insights from the ever-growing Nigerian data generated from twitter using Topic modelling. We use Latent Dirichlet Allocation (LDA) on Nigerian heath tweets from verified accounts covering time period of 2015 – 2019 to derive top health topics in Nigeria.
We detected the outbreaks of Ebola, Lassa fever and meningitis within this time frame. We also detected reoccurring topics of child immunization/vaccination.
Twitter data contains useful information that can give insights to individuals, organizations and the government hence it should be further explored and utilized.
Introduction
Background Of Study
There has been an exponential increase in the availability of data over the past years. According to Hal Varian, Chief Economist at Google,
“Between the dawn of civilization and 2003, we only created five exabytes; now we’re creating that amount every two days. By 2020, that figure is predicted to sit at 53 zettabytes (53 trillion gigabytes) — an increase of 50 times.”
While we generate 2.5 quintillion bytes of data every day, 90% of the worlds data has been created in the past two years alone (Winans et al., 2017).
These data are generated from the internet, social media, IoT, through communication, digital photos, videos and services. With this increase and availability of data comes the question of what we can do with it because the data growth phenomenon continues.
With smart phones and internet getting more affordable and available, the number of social media users is on the rise; this again shows an increase in data generation and availability.
Every minute; Google conducts 3,877,140 searches, 49,380 users post on Instagram, 4,333,560 videos are streamed on YouTube and 473,400 tweets are sent on Twitter (Data Never Sleeps 6.0, 2018). Based on this statistic, the question once again is how can available data be used?
A lot of these data come in unstructured and text format and are mined using special techniques like information retrieval, clustering, text summarization and topic modelling.
Insights in politics, business, entertainment and health can be derived from the loads of data available by applying topic modelling technique.
Data mining encompasses numerous techniques and processes. It can be defined as the process of gaining meaningful insight and patterns from a large data set.
Various forms of data (text, numeric, time series, structured unstructured etc.) require different techniques.
References
Aggarwal, C. C. (2015). Data Mining. In Springer; 2015 edition (1st ed.). https://doi.org/10.1007/978-3-319-14142-8
Ahuja, A., Wei, W., & Carley, K. M. (2016). Microblog Sentiment Topic Model. IEEE International Conference on Data Mining Workshops, ICDMW, 1031–1038. https://doi.org/10.1109/ICDMW.2016.0149
Andrius Velykis. (2018). tint – Tokenization and sentence splitting. Retrieved April 9, 2019, from http://tint.fbk.eu/tokenization.html
Anjaria, M., & Guddeti, R. M. R. (2014). Influence factor based opinion mining of Twitter data using supervised learning. 2014 6th International Conference on Communication Systems and Networks, COMSNETS 2014. https://doi.org/10.1109/COMSNETS.2014.6734907
Asghari, M., Sierra-Sosa, D., & Elmaghraby, A. (2019). Trends on Health in Social Media: Analysis using Twitter Topic Modeling. 2018 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), (December), 558–563. https://doi.org/10.1109/isspit.2018.8642679
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3, 993–1022. https://doi.org/10.1162/jmlr.2003.3.4-5.993
Be the first to comment