TraceBench

Overview

TraceBench is an open data set for trace-oriented monitoring, collected using MTracer on a HDFS system deployed in a real IaaS environment. When collecting, we considered different scenarios, involving multiple scales of clusters, different kinds of user requests, various speeds of workloads, etc. In addition to recording the traces when the HDFS runs normally, we also collected the traces under the situations with various faults injected. There are 17 faults we have injected, including function and performance faults (and real system bugs). The traces in TraceBench are clustered in different MySQL files, and each file records the traces collected under a certain situation. The total collection time of TraceBench is more than 180 hours, resulting 364 files that record more than 370,000 traces. We believe TraceBench is helpful for the research of trace-based monitoring and many other topics.

Download

Getting Start

4. Click the button "enable" to enable the file you are interested in, and you will see the related traces in the browser.

Or, you can see TraceBench on line.

Trace

A trace in TraceBench records the handling process of a user request. The trace consists of events and edges, where the event records information of executing an operation, like function, and the edge records the relationship between events, like function call. Following are some samples chosen from TraceBench.

	Request	Description	Details
1	ls	list all files in HDFS	Trace Tree | Events | Edges
2	copyFromLocal	upload a file (1 data blocks) from local to HDFS	Trace Tree | Events | Edges
3	copyToLocal	download a file (2 data block) from HDFS to local	Trace Tree | Events | Edges
4	copyFromLocal	a write requst with function fault, reflecting in Trace Tree and the Description field of Events	Trace Tree | Events | Edges
5	copyToLocal	a read requst with performance fault, reflecting in the latencies of events	Trace Tree | Events | Edges

The method of constructing a trace tree from events and edges is following:

(a) Pick out all the events and edges with TraceID.

(b) Classify the picked events to different classes with NID.

(c) Calculate the relationships of the events in each class according to the time stamps. For example, (F1.startTime < F2.startTime) and (F1.endTime > F2.endTime) mean F1 is the ancestor of F2.

(d) Construct the relationships between classes using edges. The node, identified by the fatherNID and fatherStartTime fields, is the father of all the root nodes in the class decided by the childNID field.

The computational complexity of the construction procss is O(nm+n^2/m) on average and O(n^2) in the worst case, where n is the number of the events and m is the number of edges. Following, we give the analysis process:

We first respectively analyze the computational complexity of each construction step.

• Step (a) picks out all the events and edges of the under-reconstructed trace from the database. Actually, the database querying operation is not a part of our reconstruction algorithm. Therefore, we do not consider the computational complexity of this step.

• Step (b) classifies the picked events into classes according to nid, i.e., for each event in the trace, finds the class with the same nid. Therefore, the computational complexity of step (b) is O(n * N_class), where N_class means the number of the classes.

• Step (c) calculates the relationships in each class using the time stamps, in which each event needs to compare with all the other events in the same class. Therefore, the computational complexity of step (c) is O(n * N_eventinclass), where N_eventinclass the average number of the events in a class.

• Step (d) constructs the relationships between classes using edges, i.e., finds the father event and the child class for each edge. Thus, the computational complexity of step (d) is O(m * (n + N_class)).

Therefore, the computational complexity of the whole reconstruction process is O(n * N_class)+O(n * N_eventinclass)+O(m * (n+N_class)). With the relationships of n > m, N_class = m + 1 and N_class * N_eventinclass = n, the computational complexity can be simplified to O(nm + n^2/m). In the worst case, i.e., the trace contains only one class (i.e., m = 0) or each class contains only one event (i.e., n = m + 1), the computational complexity becomes to O(n^2).

Data Storing

Traces are stored in the form of events and edges. There are following four tables in MySQL for storing data.

Table	Fields	Description
Event	TraceID	ID of the trace containing this event.
	NID	Mentioned above.
	OpName	Name of the operation recorded by this event.
	StartTime/EndTime	Timestamps of starting/finishing the operation.
	HostAddress/HostName	Host IP/Name of generating this event.
	Agent	Location of the operation inside the code, usually meaning class.
	Description	Some results of executing the operation, including exceptions.
Edge	TraceID/FatherNID/FatherStartTime/ChildNID	Mentioned above.
Trace	TraceID	Mentioned above.
	Title	Name of the trace.
	NumEvents/NumEdges	Number of events/edges included by this trace.
	FirstSeen/LastUpdated	Time of first/last receiving the data in this trace on monitor server.
	StartTime/EndTime	Timestamps of starting/finishing this event.
Operation	OpName	Name of the operation, corresponding to the OpName field in Event table.
	Num	Number of events recording this operation.
	MaxLatency/MinLatency/AverageLatency	Maximal/minimal/average latency of this operation.

Data in Trace and Operation are the statistic information, which can be extracted from the Event and Edge. In other words, the Event and Edge record all needed data of traces.

Note that, the terms used in implementation maybe a little different from above. Here are some of them: Task=Trace, TID=NID, Report=Event, Delay=Latency

Structure

The structure of TraceBench is shown in following table:

Class	Type	Fault	Workload	Variable
Normal (NM)	Clientload (CL)	-	r/w/rw/rpc/rwrpc	1,5i Clients (C)
	Datanode (DN)	-	r/w/rw	1,5i datanodes (DN)
Abnormal (AN)	Process (Proc)	killDN	r/w	0,1,2,3,4,5i FDN
		suspendDN	r/w	1,2,3,4,5i FDNs
	Network (Net)	disconnectDN	r/w	1,2,3,4,5i FDN
		slowHDFS	r/w/rpc	0,10i/2i/100i ms
		slowDN	r/w	1,2,3,4,5i FDNs
	Data	corruptBlk	r	0,1,2,3,4,5i FDN
		corruptMeta	r	0,1,2,3,4,5i FDN
		lossBlk	r	1,2,3,4,5i FDN
		lossMeta	r	1,2,3,4,5i FDN
		cutBlk	r	1,2,3,4,5i FDN
		cutMeta	r	1,2,3,4,5i FDN
	System (Sys)	panicDN	r/w	1,2,3,4,5 FDN
		deadDN	r/w	1,2,3,4,5 FDN
		readOnlyDN	w	1,2,3,4,5 FDN
Combination (COM)	Single (Sin)	Process (Proc)	rwrpc	1,2,3
		Network (Net)	rwrpc	1,2,3
		Data	rwrpc	1,2,3
		System (Sys)	rwrpc	1,2,3
		Bug	rwrpc	1,2,3
	Multiple (Mul)	AnarchyApe (AA)	rwrpc	1,2,3

In TraceBench, we name a set of traces according to the items of the columns in the table, i.e., “[Class](_[Type](_[Fault](_[Workload](_[Variable])?)?)?)?” in the form of the regular expression, where the words means the sets of items appeared in corresponding columns of the table. In addition, abbreviations given in the brackets in the table are used for compressing the names. As an example, the trace set named as Normal_Clientload_-_r, or NM_CL_r for short, contains the traces collected under the workload r in the Clientload type of Normal class.

Environment

TraceBench is collected in a real environment, which consists of more than 100 virtual machines hosted on our IaaS platform, i.e., CloudStack. Following figure shows the environment of our collection.

The environment consists of following component:

• HDFS: providing a distributed storage service, con taining 50 datanodes and one namenode.

• Clients: used to generate workloads to HDFS, to simulate the real usages of HDFS, containing 50 hosts.

• MTracer Server: receiving, storing and visualizing traces generated when HDFS processes the requests from clients.

• Controller: controlling the whole process of collection, and also being in charge of injecting faults.

• Ganglia Server [4]: monitoring the whole environment, to help for solving the unexpected issues in collection, like VMs are shut down by accident.

The MTracer Server is deployed on a VM with 4 GB memory and 8 × 1 GHz CPU, while all the rest hosts are deployed on the VMs with 2 GB memory and 4 × 1 GHz CPU. The OS that all the VMs use is CentOS 6.3.

Workload

During collection, we introduce following 5 workload:

Workload	Contained HDFS requests
r	copyToLocal
w	copyFromLocal
rpc	mkdir, touchz, mv, chmod, chown, ls, count, rmr
rw	r+w
rwrpc	r+w+rpc

Faults

We introduced 17 faults in 5 types, we explain each of them as follows

Type	Fault	Description	Category	Selected From
Process	killDN	Kill the HDFS processes on some datanodes	Functional	AnarchyApe
	suspendDN	Suspend the HDFS processes on some datanodes	Functional
Network	disconnectDN	Disconnect some datanodes from network	Functional
	slowHDFS	Slow all the HDFS nodes	Performance
	slowDN	Slow some datanodes	Performance
Data	corruptBlk	Modify all the data blocks on some datanodes	Functional
	corruptMeta	Modify all the metadata files on some datanodes	Functional
	lossBlk	Delete all the data blocks on some datanodes	Functional
	lossMeta	Delete all the metadata files on some datanodes	Functional
	cutBlk	Remove some bits in all data blocks on some datanodes	Functional
	cutMeta	Remove some bits in all metadata files on some datanodes	Functional
System	panicDN	Make the system panic on some datanodes	Functional
	deadDN	Make the system dead on some datanodes	Functional
	readOnlyDN	Make the system read-only on some datanodes	Functional
Bug	HADOOP-3257	The path in HDFS requests is limited by URI semantics	Functional	Hadoop issues repository
	HADOOP-6502	ls is very slow when listing a directory with a size of 1300	Performance
	HADOOP-7064	rmr does not properly check permissions of files	Functional

Applications

See the applications here.

Contact us

• Jingwen Zhou (jwzhou@nudt.edu.cn): PhD Student, National University of Defense Technology, Changsha 410073, China

• Zhenbang chen (zbchen@nudt.edu.cn): Lecturer, National University of Defense Technology, Changsha 410073, China

• Ji Wang: Professor, National University of Defense Technology, Changsha 410073, China

• Zibin Zheng: Associate Research Fellow, Shenzhen Research Institute, The Chinese University of Hong Kong, Shenzhen, China

• Michael R. Lyu: Professor, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China