IOT Data Access and Storage

IOT Data Access and Storage

Most of today’s device are based on exsiting legecy systems.With the development of IOT, all those devices need a bridge to pass raw data through the network to reach a cloud for the future.However, there poses a challenge that most of the exitting legacy system are not so compatible and scable.Therefore, it runs in the opposite direction of IOT. To manage the challenge, the IOT gateway provide an interoperable solution that enable effective intergration of existing devices.
OMNILab IOT gateway emphasizes on providing fast connections, easily deployment and flexibe configration.That means it makes much easier for connencting securely between devices and cloud for enabling intelligent big data analysis and data-driven decision making. In addtion ,the gateway includes standard API for dabases,such as MySQL,MongoDB.Also, it wil be flexibly configured with diffrent protocal-ready modules to communicate with end devices.IOT gateway can also connect to cloud servers through wireless 3G/WiFi, wired LAN networks。And all the communication are based on standard cloud API.

随现有的大多数设备都是基于传统的系统。随着物联网的发展,人们需要一座桥梁将设备和网络云连接起来,以便将设备产生的数据发送至云端。然而,传统 的设备系统之间互不兼容,数据格式变化多样,这给数据传输至云端带来了困难并且这和物联网的发展背道而驰。为了解决这项挑战,物联网网关提供了一种高效整 合和连接现有设备的解决方案。
OMNILab的物联网关强调快速连接、轻松部署和灵活配置。这意味着设备和云端的连接将变得更加容易,从而更加利 于大数据的分析和数据驱动的决策导向。同时,物联网关还提供诸如MySQL和MongoDB等数据库的标准连接接口。随着项目的进行,我们的物联网关还将 支持一些现有的工业协议。物联网关可以通过3G/WiFi,有线网口与云端主机进行通信。所有的通信都是基于标准的云端开放接口进行的。

Person/Organization: Nanshu Zhou, Honglun Zhang

Data Analysis Platform

Data Analysis Platform

With the rapid development of the Internet technology. Big Data and Cloud Computing has become the most popular technologies currently. Gartner Inc. gives the definition of Big Data as being high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. Data analysis is an appropriate method with statistical analysis on large amounts of data to extract useful information and conclusions for the detailed process.

Since the set up of OMNILab, we have made extensive progress towards understanding Cloud Computing and Big Data. The goal of our platform is to integrate data collection from different sources, storage and computation services together to achieve effective data analysis. The platform is build upon a core message queue system and consists of four parts: data collection, storage, computation and service modules. Practically, ten nodes of Hadoop framework are built based on CentOS with the Cloudera CDH release and the Cloudera Manager as the management tool. As for now, the storage capacity is 80TB. Data computation module consists of two primary procedures: real-time processing and batch processing (a.k.a. Lambda Architecture). We adopt Apache Storm and Spark Streaming to handle the real-time processing with the data from Kafka and leverage Apache Spark as our batch processing framework to handle big data which is a distributed, scalable analytic tool for big data. The service module comprises data services (NFS storage, MySQL, PostgreSQL) and application services (HomePage, GitLab, Wiki) for external public use.

随着互联网技术的飞速发展,大数据和云计算技术已经变成了当下最为热门的前沿技术。对于“大数据”(Big data)研究机构Gartner给出了这样的定义。“大数据”是需要新处理模式才能具有更强的决策力、洞察发现力和流程优化能力的海量、高增长率和多样 化的信息资产。数据分析是指用适当的统计分析方法对收集来的大量数据进行分析,提取有用信息和形成结论而对数据加以详细研究和概括总结的过程。
自 实验室成立以来,OMNILab的研究人员,就致力于在数据分析和云计算等前沿科技方面的研究。我们的数据分析平台的目标是为了整合数据采集、存储、计算 和服务为一体,从而能够有效地实现从不同的数据源能够以标准化的程序来对数据进行有效地分析。平台以消息队列为核心,并且主要由数据采集、存储、计算和服 务为主要的四个模块。在我们的实践中采用了10台服务器作为Hadoop集群构架,使用Apache Kafka作为消息队列;通过网络爬虫、传感器等方式采集各种时序数据(服务器日志、设备状态、网络流量)和空间数据(IOT数据,GPS数据等);采用 HDFS作为我们的数据存储模块,并且使用Cloudera Manager作为管理工具,目前我们的平台存储容量已达80TB;数据计算模块主要采用了Storm和Spark作为分布式处理大数据;数据服务模块主 要包含数据服务(诸如NFS、MySQL、PostgreSQL)和应用服务(HomePage、GitLab、Wiki)。

Person/Organization: Haiyang Wang, Jianwen Wei, Yusu Zhao, Pengfei Zhang

Cloud Computing Platform

Cloud Computing Platform

Cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a data center, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.

OMNICloud in OMNILab is an IaaS ( Infrastructure as a service) platform. Our researchers deploy a Icehouse Version OpenStack Platform for supporting us to do researching computation, service backup, and various project prototypes and so on. The platform is constructed based on six physical servers. In this architecture, one controller node is used to control the whole platform and four computing nodes for computing. We also have one network node and make two l3 agents for providing both internal and external IP service. The operating system mirrors are hosted on a reliable NFS-based storage to gain high flexibility and efficiency.

随着互联网技术的飞速发展,大数据和云计算技术已经变成了当下最为热门的前沿技术。云计算(cloud computing)是基于互联网的相关服务的增加、使用和交付模式,通常涉及通过互联网来提供动态易扩展且经常是虚拟化的资源。OpenStack是一 个云操作系统,通过数据中心,用来进行大量的计算、存储和网络资源的分配,通过一个操作控制台,能够有效地对提供的各种资源进行利用。
自实验室成 立以来,OMNILab的研究人员,就致力于在数据分析和云计算等前沿科技方面的研究。OMNICloud是一个IaaS平台,我们的研究人员部署了一个 Icehouse版本的OpenStack平台,为我们实验室研究人员用以提供基础服务,诸如计算、例程等。我们的平台总共采用了6台服务器。在这些服务 器当中,有一个控制节点,用作对于整个平台的控制,还有4个计算节点,用以提供虚机服务,还有1个网络节点,然后并且部署了2个l3 agent用以提供内部和外部ip的服务。并且使用了NFS作为贡献存储,提供存储的高可用性。

Person/Organization: Pengfei Shi, Pengfei Zhang, Xuan Luo

Mobile traffic analysis and measurement

Mobile traffic analysis and measurement

Mobile market is driven to be explosive by the prosperity of hand-held smart devices and full-experience mobile applications. Traditional problems such as networking optimization and human social analysis can be tackled in new ways. From an abstract view, users with mobile devices are sophisticated sensors of our realistic world. The convenience of accessing human-generated data via mobile equipment stimulates rapid development of ubiquitous computing technologies and human behaviour researches. For example, the success of revenue models based on subscription and advertisement relies heavily upon the knowledge of user participation and individual preferences. In mobile networks, accurate interpretation of user behaviour and experience is of use to facilitate network management for Internet service providers (ISPs), application decision optimization for content providers, and the enhancement of user privacy protection policies from a hacking view.

Since 2011, our researchers in OMNILab and academic partners have made extensive steps towards understanding UX and spatiotemporal characteristics of mobile traffic. Recently, we proposed the concept of mobile user engagement in designing human-concentrated mobile technologies and applications. We focused on a characterization and modelling to understand mobile user engagement, especially its interactions with user-perceived application performance (a.k.a. user perception) in real context. At the urban scale, limited knowledge about the spatiotemporal dependence of mobile traffic is gained. Our researchers fill this gap by making use of request-response records extracted from HTTP traffic at the city scale. The results suggest connections between spatio- temporal dependence of cellular traffic and the organization of human lives. Region differences are observed to impact traffic dependence to a great extent. Additionally, interactive knowledge between space and time enhances traffic prediction with a decrease in root-mean-square error of 2.8%~25.2%.

随着移动技术的快速发展,传统的网络优化和人类行为研究,可以借助移动技术产生的海量数据从一个新的视角进行研究。从抽象角度来看,移动网络的用户 如同一个个现实世界的传感器,数字化这我们生活中各种各样的行为。这种数据获取的便利,促进了普适计算技术(ubiquitous computing)和人类行为学的快速发展。例如,一个传统意义上基于广告推广营销的平台,对用户参与行为和个人爱好有着很强的依赖。基于移动数据的分 析便可以较小的代价获得更大的收益。在这样的应用中,对用户行为的准确解读是这类技术成功与否的关键。与此同时,这类技术对服务运营商和内容分发服务商、 甚至用户隐私的保护策略,都有着重要的价值。

自2011年以来,OMNILab的研究人员以及合作伙伴,在移动网络流量分析和用户体验测量上进行了深入的研究。我们利用移动用户参与度 (user engagement)的概念,从网络流量出发对用户角度的体验进行客观测量和分析。在我们的方法框架中,用户参与行为、网络性能以及使用场景得到了细致 的分类研究以及综合模型分析,从相关性角度找出了影响用户参与度的关键影响因素。除此以外,大范围角度的移动流量特征也是OMNILab研究人员关注的领 域。受限于传统网络上数据采集规模的限制,城市角度的流量时空特征一直未能得到充分地理解。我们基于中国某城市的移动网络流量数据,对网络流量的时间-空 间依赖性(spatiotemporal dependence)进行了系统分析。结果显示,大尺度上的移动流量分布是非随机和非线性的(如幂律分布),而且和人类的日常行为紧密联系在一起。在结 合了时空依赖信息以后,我们对移动网络流量的预测均方误差减小了2.8%~25.2%。移动网络流量分析是我们的长期研究领域,希望能和更多的合作伙伴共 享我们的技术,将研究成果用于实际当中

Person/Organization: Xiaming Chen, Siwei Qiang, Xin Yang
Project:

  • Mobile traffic mining funded by Huawei Inc.
  • HTTP traffic analysis funded by China Telecom

Papers:

  • Chen, Xiaming; Jin, Yaohui; Qiang, Siwei; Hu, Weisheng; Jiang, Kaida; Analyzing and Modeling Spatio-Temporal Dependence of Cellular Traffic at City Scale, Submission to ICC 2015.
  • Chen, Xiaming; Qiang, Siwei; Wei, Jianwen; Jiang, Kaida; Jin, Yaohui; Towards Understanding User Engagement in Mobile Networks: A Perception Perspective, Submission to Elsevier Pervasive and Mobile Computing.
  • Chen, Xiaming; Jin, Yaohui; Web Page Identification from Network Traffic with Support Vector Machine,Technical Report, 2012, Shanghai Jiao Tong Univ.
  • Yang, Xin; Chen, Xiaming; Jin, Yaohui; A high-speed real-time HTTP performance measurement architecture based on network processor, ICT Convergence (ICTC), 2011 International Conference on,744-745,2011,IEEE.
Analysis of Human Spatio-Temporal Behaviour

Analysis of Human Spatio-Temporal Behaviour

City organization and residents behavior is one of the key research in urban geography. With the rapid development of information technology, the impact of research on residents spatial and temporal behavior on urban spatial organization and structure shows a growing trend, therefore in-depth analysis of the spatio-temporal behavior of city space and urban residents have high research value. Our researchers have put great efforts in understanding the spatio-temporal behavior of urban residents through the analysis of data from urban mobile networks, campus WiFi networks and satellite position system (i.e., Beidou).

The spatio-temporal behavior patterns of people living within a city are crucial to many applications ranging from personalized location based services to city management. Facilitating data from urban mobile network and campus WiFi network, we provide an objective, large-scale measurement framework incorporating multiple-layer behaviors: network, application and user behaviors. Comprehension of the changeability helps to optimize existing mobility models in network researches accompanied by a reinforced interpretation of human behavior, and additionally paves a new way to estimate abnormal mobilities which gives assistance to, say, stolen devices’ retrieval or school security management. With the GPS data from Beidou system, we focus on three aspects: user’s semantic trajectory extraction, fine-grained user profile and spatial-temporal anomaly detection.

城市的空间组织和居民行为研究是城市地理学研究的重点,随着信息技术的快速发展,居民的时空行为对城市空间的组织和结构的影响呈现出日益增加的趋势,因此,对城市空间以及居民时空行为的深入分析具有很高的研究价值。

自2013年以来,OMNILab的研究人员通过分析隐私处理后的杭州移动数据、校园WiFi数据、北斗位置数据对用户时空行为分析进行了深入的研 究。对于杭州移动数据的研究集中于用户移动性及网络流量特性,挖掘并理解城市范围内用户的时空行为特征将会助力与包括从基于位置的个性化推荐到城市规划的 各个应用领域。校园WiFi数据挖掘主要用于研究用户轨迹模型变化,将离散的用户时空行为抽象描述成时空序列数据,并结合地点、应用语义添加相应标签,提 取用户的轨迹模型。在实际应用中,轨迹模式变化的检测,不仅有助于对已有的网络设施部署进行调整和优化,也能够透过用户的行为信息,检测出特定时空域的群 体事件,为安全管理提供方便的途径。北斗数据分析研究主要集中于:用户语义轨迹的提取、细粒度用户分类、时空行为异常检测。用户时空行为分析是我们的长期 研究领域,希望能和更多的合作伙伴共享我们的技术,将研究成果用于实际当中。

Person/Organization: Siwei Qiang, Wenjuan Gong, Xiaming Chen, Haiyang Wang
Project:

  • Hangzhou mobile network dataset analysis
  • Campus WiFi networks data mining
  • BeiDou location data Mining project

Papers:

  • Urban Spatio-temporal Behavior Analysis Based on Mobile Network Traffic Logs, CCF Big Data 2014 (recommended to Journal of Computer Research and Development).
  • Gong, Wenjuan; Chen, Xiaming; Qiang, Siwei; Jin, Yaohui; Trajectory pattern change analysis in campus WiFi networks, Proceedings of the Second ACM SIGSPATIAL International Workshop on Mobile Geographic Information Systems,1-8,2013,ACM.
Optimization of Data Center Network

Optimization of Data Center Network

Cloud Computing has become one of the most profounding technologies currently. As the core innovation infrastructure of cloud computing and next-generation network technology, data center network (DCN) has become very popular in academia and industry in recent years. Our researchers in OMNILab and academic partners have made extensive steps towards understanding Data Center Network. We are mainly involved in the study of SDN (Software Defined Network), in which including elephant flow detection and intelligent timeout master,we also doing research in dynamic bandwidth allocation for cloud network performance guarantee.

随着互联网技术的飞速发展,大数据和云计算技术已经变成了当下最为热门的前沿技术。数据中心网络作为云计算的核心基础设施,和下一代网络技术的创新平台, 数据中心网络的研究成为了近年来学术界和工业界关注的热点,其为连接数据中心大规模服务器进行分布式。OMNILab的研究人员致力于在数据中心网络的研 究。主要研究包括对SDN软件定义网络的研究,其中有对大象流的检测以及对于动态流表超时的研究;以及对通过动态带宽分配来保障云端网络性能的研究。

Person/Organization: Pengfei Zhang, Huikang Zhu, Conghui Bi
Project:
Papers:

  • Pengfei Zhang, et al. DWARF-Net: Dynamic bandWidth Allocation&Guarantee on Resource Fairness for Networks in Cloud, CFI 2013 : 8th International Conference on Future Internet Technologies
Innovative Markers

Innovative Markers

Apart from regular researches and works every day, has it ever occurred to you to try something different and interesting? Tired with boring tasks and rigid affairs from your boss or leader, why not release your imagination and enjoy the pleasure of design and creation?

We greatly encourage students to get rid of their regular thinking models, try new things never experienced before and put them into practice. We call them Markers, people who are hot for creative thinking and bold trials. Students are expected to come up with interesting ideals and try to realize them with common materials and tools, even if these ideals turn out to be of not so much business value. It is widely admitted that most students harvest a lot after taking part in Makers’ activities, from basic skills and technologies to critical and enlightening thinking.

We have established an innovative students’ organization, **Newbee Studio**, which concentrates on mobile techniques and the Internet of things, undertakes web application projects and participates in innovation competitions. Moreover, we also conduct some contests as host and welcome students from the campus to join us, such as the Marker’s Workshop, where participants were provided with Arduino devices and environmental sensors to develop meteorology monitors, and the EMC Campus Open Data Contest, which published campus e-card consumption and Wi-Fi traffic data to promote statistic mining and deep insight into students’ behaviors and lifestyles.