Ceph cluster report: Nemo

Status:

Setting	Value
Collected at	2015-09-19 08:22:27
Collected at GMT	2015-09-19 08:22:27
Status	HEALTH_WARN
PG count	2752
Pool count	13
Used	11.9 G
Avail	139.6 G
Data	17.5 M
Free %	92
Mon count	3

OSD:

Setting	Value
Count	5
PG per OSD	550
Cluster net	192.168.1.0/24
Public net	192.168.0.0/24
Near full ratio	0.85
Full ratio	0.95
Backfill full ratio	0.85
Filesafe full ratio	0.97
Journal aio	true
Journal dio	true
Filestorage sync	5s

Activity:

Setting	Value
Client IO Bps	0
Client IO IOPS	0

Status messages:

Pool test has too few pgs

Host's info:

Name	Services	CPU's	RAM total	RAM free	Swap used	Load avg 5m
node-10	osd-1 mon(node-10)	1	2.9 Gi	244.3 Mi	1.7 Gi	1.21
node-11	osd-0 mon(node-11)	1	2.9 Gi	251.8 Mi	1.6 Gi	1.84
node-12	osd-2 mon(node-12)	1	2.9 Gi	238.7 Mi	1.3 Gi	1.22
node-13	osd-4	1	2.9 Gi	262.3 Mi	252 Ki	0.24
node-14	osd-3	1	2.9 Gi	279.2 Mi	356 Ki	0.12

Monitors info:

Name	Node	Role	Disk free B (%)
node-10	HEALTH_OK	None	5.4 G (36)
node-12	HEALTH_OK	None	5.5 G (37)
node-11	HEALTH_OK	None	5.5 G (37)

OSD's state:

Status	Count	ID's
up	5

OSD's info:

OSD	node	status	daemon run	weight reweight	PG count	Storage used	Storage free	Storage free %	Journal on same disk	Journal on SSD	Journal on file
0	node-11	up	yes	0.030 / 1.000	1256	2.4 G	32.9 G	93	yes	no	yes
1	node-10	up	yes	0.030 / 1.000	1224	2.4 G	33.0 G	93	yes	no	yes
2	node-12	up	yes	0.030 / 1.000	1216	2.4 G	33.0 G	93	yes	no	yes
3	node-14	up	yes	0.020 / 1.000	909	2.3 G	20.4 G	90	yes	no	yes
4	node-13	up	yes	0.020 / 1.000	899	2.3 G	20.3 G	90	yes	no	yes

OSD's load uptime average:

OSD	node	apply lat, ms	commit lat, ms	D dev	D read Bps	D write Bps	D read OPS	D write OPS	D IO time %	J dev	J read Bps	J write Bps	J read OPS	J write OPS	J IO time %
0	node-11	166	32	sda	95.3 K	195.6 K	7.5	10.3	5	sda	95.3 K	195.6 K	7.5	10.3	5
1	node-10	121	120	sda	105.0 K	207.5 K	7.8	11.1	4	sda	105.0 K	207.5 K	7.8	11.1	4
2	node-12	125	23	sda	93.0 K	197.6 K	7.4	10.7	5	sda	93.0 K	197.6 K	7.4	10.7	5
3	node-14	14	4	sda	19.5 K	11.0 K	4.9	1.2	0	sda	19.5 K	11.0 K	4.9	1.2	0
4	node-13	26	7	sda	19.5 K	11.0 K	4.9	1.2	0	sda	19.5 K	11.0 K	4.9	1.2	0

OSD's current load:

OSD	node	D dev	D read Bps	D write Bps	D read OPS	D write OPS	D lat ms	D IO time %	J dev	J read Bps	J write Bps	J read OPS	J write OPS	J lat ms	J IO time %
0	node-11	sda	111.5 K	335.5 K	10.4	14.4	9	6	sda	111.5 K	335.5 K	10.4	14.4	9	6
1	node-10	sda	143.3 K	219.4 K	10.8	14.6	6	5	sda	143.3 K	219.4 K	10.8	14.6	6	5
2	node-12	sda	68.4 K	185.6 K	7.9	12.4	10	6	sda	68.4 K	185.6 K	7.9	12.4	10	6
3	node-14	sda	19.6 K	13.6 K	4.9	1.3	1	0	sda	19.6 K	13.6 K	4.9	1.3	1	0
4	node-13	sda	19.6 K	13.0 K	4.9	1.4	6	0	sda	19.6 K	13.0 K	4.9	1.4	6	0

Pool's stats:

Pool	Id	size	min_size	obj	data	free	read	write	ruleset	PG	PGP	PG per OSD Dev %
data	0	2	1	0	0	---	0	0	0	64	64	13
metadata	1	2	1	0	0	---	0	0	0	64	64	21
rbd	2	2	1	0	0	---	0	0	0	64	64	23
images	3	2	1	5	12.7 M	---	69	13	0	256	256	12
volumes	4	2	1	0	0	---	0	0	0	256	256	23
backups	5	2	1	0	0	---	0	0	0	256	256	11
.rgw.root	6	2	1	3	840	---	24	5	0	256	256	15
.rgw.control	7	2	1	8	0	---	0	0	0	256	256	17
.rgw	8	2	1	0	0	---	0	0	0	256	256	10
.rgw.gc	9	2	1	32	0	---	152.3 K	101.5 K	0	256	256	21
.users.uid	10	2	1	0	0	---	0	0	0	256	256	21
compute	11	2	1	0	0	---	0	0	0	256	256	14
test	12	2	1	79.4 K	4.8 M	---	2	236.5 K	0	256	256	12

PG's status:

Status	Count	%
any	2752	100.00
active	2752	100.00
clean	2752	100.00

PG copy per OSD:

OSD/pool	.rgw	.rgw.control	.rgw.gc	.rgw.root	.users.uid	backups	compute	data	images	metadata	rbd	test	volumes	sum
0	105	107	130	118	127	115	121	29	121	24	22	121	116	1256
1	114	127	116	105	112	109	104	26	112	34	37	101	127	1224
2	106	113	112	119	121	111	116	29	102	27	26	112	122	1216
3	105	85	72	75	72	93	86	24	88	26	20	87	76	909
4	82	80	82	95	80	84	85	20	89	17	23	91	71	899
sum	512	512	512	512	512	512	512	128	512	128	128	512	512	5504

Current disk IO load

IOPS

host	load
node-10	sda 25.3
node-11	sda 24.8
node-12	sda 20.3
node-13	sda 6.3
node-14	sda 6.2

Read IOPS

host	load
node-10	sda 10.8
node-11	sda 10.4
node-12	sda 7.9
node-13	sda 4.9
node-14	sda 4.9

Write IOPS

host	load
node-10	sda 14.6
node-11	sda 14.4
node-12	sda 12.4
node-13	sda 1.4
node-14	sda 1.3

Bps

host	load
node-10	sda 362.7 K
node-11	sda 447.0 K
node-12	sda 254.0 K
node-13	sda 32.6 K
node-14	sda 33.2 K

Read Bps

host	load
node-10	sda 143.3 K
node-11	sda 111.5 K
node-12	sda 68.4 K
node-13	sda 19.6 K
node-14	sda 19.6 K

Write Bps

host	load
node-10	sda 219.4 K
node-11	sda 335.5 K
node-12	sda 185.6 K
node-13	sda 13.0 K
node-14	sda 13.6 K

Latency, ms

host	load
node-10	sda 6
node-11	sda 9
node-12	sda 10
node-13	sda 6
node-14	sda 1

Average QD

host	load
node-10	sda 0.2
node-11	sda 0.2
node-12	sda 0.2
node-13	sda 0.0
node-14	sda 0.0

Active time %

host	load
node-10	sda 5.0
node-11	sda 6.0
node-12	sda 6.0
node-13	sda 0.0
node-14	sda 0.0

Network load (to max dev throughput)

Send

host	public net	cluster net	hw adapter	hw adapter
node-10	811.9	97.7 K	eth0 106.2 K	eth1 1.4 K
node-11	795.2	62.1 K	eth0 79.2 K	eth1 1.4 K
node-12	818.1	57.1 K	eth0 65.1 K	eth1 1.3 K
node-13	864.3	1.7 K	eth0 2.7 K	-
node-14	799.7	4.1 K	eth0 5.1 K	-

Receive

host	public net	cluster net	hw adapter	hw adapter
node-10	708.0	65.9 K	eth0 158.5 K	eth1 3.1 K
node-11	694.3	66.1 K	eth0 186.3 K	eth1 3.0 K
node-12	709.7	55.3 K	eth0 200.5 K	eth1 3.1 K
node-13	753.5	1.8 K	eth0 260.3 K	-
node-14	695.2	4.1 K	eth0 258.0 K	-

Host's resource usage:

Hostname	Cluster net dev, ip settings	Cluster net uptime average send/recv	Cluster net current send/recv	Public net dev, ip settings	Public net uptime average send/recv	Public net current send/recv	Net	Net	Net	Net
node-10	br-storage 192.168.1.4	1004.4 / 876.5 Bps 7.8 / 7.7 Pps	811.9 / 708.0 Bps 7.4 / 7.4 Pps	br-mgmt 192.168.0.5	73.8 K / 66.7 K Bps 472.3 / 494.8 Pps	97.7 K / 65.9 K Bps 555.1 / 541.3 Pps	eth0 1.2 Gi	eth1 1.2 Gi	eth2 1.2 Gi	eth3 1.2 Gi
node-11	br-storage 192.168.1.2	1012.1 / 936.6 Bps 7.8 / 7.8 Pps	795.2 / 694.3 Bps 7.3 / 7.3 Pps	br-mgmt 192.168.0.7	72.9 K / 57.8 K Bps 407.6 / 466.3 Pps	62.1 K / 66.1 K Bps 376.1 / 468.6 Pps	eth0 1.2 Gi	eth1 1.2 Gi	eth2 1.2 Gi	eth3 1.2 Gi
node-12	br-storage 192.168.1.3	1.0 K / 903.3 Bps 7.8 / 7.8 Pps	818.1 / 709.7 Bps 7.5 / 7.5 Pps	br-mgmt 192.168.0.6	68.3 K / 61.1 K Bps 471.3 / 467.6 Pps	57.1 K / 55.3 K Bps 432.0 / 430.4 Pps	eth0 1.2 Gi	eth1 1.2 Gi	eth2 1.2 Gi	eth3 1.2 Gi
node-13	br-storage 192.168.1.1	975.2 / 863.0 Bps 7.7 / 7.7 Pps	864.3 / 753.5 Bps 7.9 / 7.9 Pps	br-mgmt 192.168.0.3	1.7 K / 1.7 K Bps 12.2 / 14.5 Pps	1.7 K / 1.8 K Bps 12.6 / 15.0 Pps	eth0 1.2 Gi	eth1 1.2 Gi	eth2 1.2 Gi	eth3 1.2 Gi
node-14	br-storage 192.168.1.5	966.5 / 852.9 Bps 7.7 / 7.7 Pps	799.7 / 695.2 Bps 7.3 / 7.3 Pps	br-mgmt 192.168.0.4	3.9 K / 3.9 K Bps 24.8 / 28.6 Pps	4.1 K / 4.1 K Bps 25.7 / 29.3 Pps	eth0 1.2 Gi	eth1 1.2 Gi	eth2 1.2 Gi	eth3 1.2 Gi

Crush weight:

PG's count: