SYSTEM COMPONENTS FOR DATA EXTRACTION AND PROCESSING FROM INTERNET OF THINGS

Abstract

In the last two decades, information technology has developed at a very rapid pace as its products have become very popular and widespread due to their small size and low cost. Apart from mobile phones, laptops and computers and network equipment, many different digital components have entered almost all areas and aspects of human life and activities. Nowadays, there are numerous devices that transmit and/or receive data over the Internet, and with their participation, the Internet and other data processing devices form the modern phenomenon "Internet of Things“ - IoT. This term denotes the connection of sensors and other elements to the global network and fetching measured by the sensors physical values such as temperature, humidity, pressure. Other important components of IoT are digital tags attached to goods, vehicles even living beings, smart devices connecting and controlling processes and appliances in smart offices and smart homes. The whole complex of the listed components generate huge amounts of digital data which is stored in Big data systems.  This data can be structured, semi-structured and unstructured and is used by information systems to report, store and sometimes feedback to the objects from which this data originated. The main blocks of a system for work with IoT data can be defined as: edge IoT sensors and devices, preprocessing edge devices (routers, gateways), communication networks (sensor networks, Internet, mobile systems), brokers (systems to receive and aggregate data) and Big data systems (often build around Hadoop). We discuss all those blocks and present examples. There have been various approaches and architectures for processing Big data from IoT but one of the architectures, that is popular in many others under different names is the Lambda architecture (LA). It is a general, extensible, and fault-tolerant data processing architecture. LA is a way of processing huge amounts of data that provides access to batch processing and stream processing methods with a hybrid approach. Its main functional blocks are presented. The work also considers the most popular open source software for Big data processing – the Hadoop environment. This ecosystem for Big data processing is presented with software tools and components with functions like data extraction, data distribution, data storage, data processing, etc. Examples for performing those functions with the packets HDFS, Fulme, Storm, WiFi, Kafka, Hive, Hue, Spark and Impala are given. The components used for streaming and batch processing data, as in the LA are identified. The paper presents a simplified model using open source components, that can extract data from IoT, make the necessary transformation, store them and process that data according to the requirements of the end users. The system is scalable, flexible and extendable with other modules and components. A successful verification with different IoT data sources has been carried out.