You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
276 lines
7.7 KiB
276 lines
7.7 KiB
9 months ago
|
# Metrics
|
||
|
|
|
||
|
|
A more straightforward (and hopefully better) way to export metrics.
|
||
|
|
|
||
|
|
## Related configurations
|
||
|
|
|
||
|
|
It's possible to add a metrics inbound among inbounds.
|
||
|
|
|
||
|
|
```json
|
||
|
|
"inbounds": [
|
||
|
|
{
|
||
|
|
"listen": "127.0.0.1",
|
||
|
|
"port": 11111,
|
||
|
|
"protocol": "dokodemo-door",
|
||
|
|
"settings": {
|
||
|
|
"address": "127.0.0.1"
|
||
|
|
},
|
||
|
|
"tag": "metrics_in"
|
||
|
|
}
|
||
|
|
]
|
||
|
|
```
|
||
|
|
|
||
|
|
And add routing rules regarding the metrics inbound in the routing configuration.
|
||
|
|
|
||
|
|
```json
|
||
|
|
"routing": {
|
||
|
|
"rules": [
|
||
|
|
{
|
||
|
|
"type": "field",
|
||
|
|
"inboundTag": [
|
||
|
|
"metrics_in"
|
||
|
|
],
|
||
|
|
"outboundTag": "metrics_out"
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Then finally, enable metrics under the root object.
|
||
|
|
|
||
|
|
```json
|
||
|
|
"metrics": {
|
||
|
|
"tag": "metrics_out"
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
### pprof
|
||
|
|
|
||
|
|
Access `http://127.0.0.1:11111/debug/pprof/` or use go tool pprof to start profiling or inspect running goroutines.
|
||
|
|
|
||
|
|
### expvars
|
||
|
|
|
||
|
|
Access `http://127.0.0.1:11111/debug/vars`
|
||
|
|
|
||
|
|
Variables exported include:
|
||
|
|
* `stats` includes statistics about inbounds, outbounds and users
|
||
|
|
* `observatory` includes observatory results
|
||
|
|
|
||
|
|
for example with [luci-app-xray](https://github.com/yichya/luci-app-xray) you are likely to get a result like this (standard expvar things like `cmdline` and `memstats` are omitted)
|
||
|
|
|
||
|
|
<details><summary>Click to expand</summary><br>
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"observatory": {
|
||
|
|
"tcp_outbound": {
|
||
|
|
"alive": true,
|
||
|
|
"delay": 782,
|
||
|
|
"outbound_tag": "tcp_outbound",
|
||
|
|
"last_seen_time": 1648477189,
|
||
|
|
"last_try_time": 1648477189
|
||
|
|
},
|
||
|
|
"udp_outbound": {
|
||
|
|
"alive": true,
|
||
|
|
"delay": 779,
|
||
|
|
"outbound_tag": "udp_outbound",
|
||
|
|
"last_seen_time": 1648477191,
|
||
|
|
"last_try_time": 1648477191
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"stats": {
|
||
|
|
"inbound": {
|
||
|
|
"api": {
|
||
|
|
"downlink": 0,
|
||
|
|
"uplink": 0
|
||
|
|
},
|
||
|
|
"dns_server_inbound_5300": {
|
||
|
|
"downlink": 14286,
|
||
|
|
"uplink": 5857
|
||
|
|
},
|
||
|
|
"http_inbound": {
|
||
|
|
"downlink": 74460,
|
||
|
|
"uplink": 10231
|
||
|
|
},
|
||
|
|
"https_inbound": {
|
||
|
|
"downlink": 0,
|
||
|
|
"uplink": 0
|
||
|
|
},
|
||
|
|
"metrics": {
|
||
|
|
"downlink": 6327,
|
||
|
|
"uplink": 1347
|
||
|
|
},
|
||
|
|
"socks_inbound": {
|
||
|
|
"downlink": 19925615,
|
||
|
|
"uplink": 5512
|
||
|
|
},
|
||
|
|
"tproxy_tcp_inbound": {
|
||
|
|
"downlink": 4739161,
|
||
|
|
"uplink": 1568869
|
||
|
|
},
|
||
|
|
"tproxy_udp_inbound": {
|
||
|
|
"downlink": 0,
|
||
|
|
"uplink": 2608142
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outbound": {
|
||
|
|
"blackhole_outbound": {
|
||
|
|
"downlink": 0,
|
||
|
|
"uplink": 0
|
||
|
|
},
|
||
|
|
"direct": {
|
||
|
|
"downlink": 97714548,
|
||
|
|
"uplink": 3234617
|
||
|
|
},
|
||
|
|
"dns_server_outbound": {
|
||
|
|
"downlink": 7116,
|
||
|
|
"uplink": 2229
|
||
|
|
},
|
||
|
|
"manual_tproxy_outbound_tcp_1": {
|
||
|
|
"downlink": 0,
|
||
|
|
"uplink": 0
|
||
|
|
},
|
||
|
|
"manual_tproxy_outbound_udp_1": {
|
||
|
|
"downlink": 0,
|
||
|
|
"uplink": 0
|
||
|
|
},
|
||
|
|
"tcp_outbound": {
|
||
|
|
"downlink": 23873238,
|
||
|
|
"uplink": 1049595
|
||
|
|
},
|
||
|
|
"udp_outbound": {
|
||
|
|
"downlink": 639282,
|
||
|
|
"uplink": 74634
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"user": {}
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
</details>
|
||
|
|
|
||
|
|
To get a better view of these numbers, [Netdata](https://github.com/netdata/netdata) (with python.d plugin) is a great option:
|
||
|
|
|
||
|
|
1. Edit related configuration file (`sudo /etc/netdata/edit-config python.d/go_expvar.conf`)
|
||
|
|
2. Take the following configuration file as an example:
|
||
|
|
|
||
|
|
<details><summary>Click to expand</summary><br>
|
||
|
|
|
||
|
|
```
|
||
|
|
xray:
|
||
|
|
name: 'xray'
|
||
|
|
update_every: 2
|
||
|
|
url: 'http://127.0.0.1:11111/debug/vars'
|
||
|
|
collect_memstats: false
|
||
|
|
extra_charts:
|
||
|
|
- id: 'inbounds'
|
||
|
|
options:
|
||
|
|
name: 'inbounds'
|
||
|
|
title: 'Xray System Inbounds'
|
||
|
|
units: bytes
|
||
|
|
family: xray
|
||
|
|
context: xray.inbounds
|
||
|
|
chart_type: line
|
||
|
|
lines:
|
||
|
|
- expvar_key: stats.inbound.tproxy_tcp_inbound.downlink
|
||
|
|
id: 'tcp.downlink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.tproxy_udp_inbound.downlink
|
||
|
|
id: 'udp.downlink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.http_inbound.downlink
|
||
|
|
id: 'http.downlink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.https_inbound.downlink
|
||
|
|
id: 'https.downlink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.socks_inbound.downlink
|
||
|
|
id: 'socks.downlink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.tproxy_tcp_inbound.uplink
|
||
|
|
id: 'tcp.uplink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.tproxy_udp_inbound.uplink
|
||
|
|
id: 'udp.uplink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.http_inbound.uplink
|
||
|
|
id: 'http.uplink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.https_inbound.uplink
|
||
|
|
id: 'https.uplink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.inbound.socks_inbound.uplink
|
||
|
|
id: 'socks.uplink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- id: 'outbounds'
|
||
|
|
options:
|
||
|
|
name: 'outbounds'
|
||
|
|
title: 'Xray System Outbounds'
|
||
|
|
units: bytes
|
||
|
|
family: xray
|
||
|
|
context: xray.outbounds
|
||
|
|
chart_type: line
|
||
|
|
lines:
|
||
|
|
- expvar_key: stats.outbound.tcp_outbound.downlink
|
||
|
|
id: 'tcp.downlink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.outbound.udp_outbound.downlink
|
||
|
|
id: 'udp.downlink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.outbound.direct.downlink
|
||
|
|
id: 'direct.downlink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.outbound.tcp_outbound.uplink
|
||
|
|
id: 'tcp.uplink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.outbound.udp_outbound.uplink
|
||
|
|
id: 'udp.uplink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: stats.outbound.direct.uplink
|
||
|
|
id: 'direct.uplink'
|
||
|
|
algorithm: incremental
|
||
|
|
expvar_type: int
|
||
|
|
- id: 'observatory'
|
||
|
|
options:
|
||
|
|
name: 'observatory'
|
||
|
|
title: 'Xray Observatory Metrics'
|
||
|
|
units: milliseconds
|
||
|
|
family: xray
|
||
|
|
context: xray.observatory
|
||
|
|
chart_type: line
|
||
|
|
lines:
|
||
|
|
- expvar_key: observatory.tcp_outbound.delay
|
||
|
|
id: tcp
|
||
|
|
expvar_type: int
|
||
|
|
- expvar_key: observatory.udp_outbound.delay
|
||
|
|
id: udp
|
||
|
|
expvar_type: int
|
||
|
|
```
|
||
|
|
</details>
|
||
|
|
|
||
|
|
And you will get a nice plot like this:
|
||
|
|
|
||
|
|
|
||
|
|
|
||
|
|
### Additional
|
||
|
|
Maybe reusing the empty object `stats` in config file is better than adding `metrics` here?
|
||
|
|
|
||
|
|
**Edit:** removed prometheus related things and added usage about expvars
|