github
/
nps
mirror of https://github.com/ehang-io/nps
1
0
Fork 0
Code Issues Releases Wiki Activity

Merge branch 'dev' into new_dev

pull/123/head
ffdfgdfg 2019-12-01 22:41:59 +08:00 committed by GitHub
parent d888511f60 4381eb44b4
commit 9abe5874b7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
332 changed files with 3416 additions and 63358 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
.github/ISSUE_TEMPLATE/bug_report.md vendored Normal file
View File

@ -0,0 +1,38 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: bug
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Opening '...'
2. Click on '....'
3. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots or logs**
Add screenshots or logs to help explain your problem.
**Server (please complete the following information):**
- OS: [e.g. Centos, Windows]
- ARCH: [e.g. Amd64, Arm]
- Tunnel [e.g. TCP, HTTP]
- Version [e.g. 0.24.0]
**Client (please complete the following information):**
- OS: [e.g. Centos, Windows]
- ARCH: [e.g. Amd64, Arm]
- Tunnel [e.g. TCP, HTTP]
- Version [e.g. 0.24.0]
**Additional context**
Add any other context about the problem here.

20
.github/ISSUE_TEMPLATE/feature_request.md vendored Normal file
View File

@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: enhancement
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.

4
.gitignore vendored
View File

@ -1 +1,3 @@
.idea
.idea
nps
npc

10
Dockerfile.npc Executable file
View File

@ -0,0 +1,10 @@
FROM golang as builder
WORKDIR /go/src/github.com/cnlh/nps
COPY . .
RUN go get -d -v ./...
RUN CGO_ENABLED=0 go build -ldflags="-w -s -extldflags -static" ./cmd/npc/npc.go
FROM scratch
COPY --from=builder /go/src/github.com/cnlh/nps/npc /
VOLUME /conf
ENTRYPOINT ["/npc"]

11
Dockerfile.nps Executable file
View File

@ -0,0 +1,11 @@
FROM golang as builder
WORKDIR /go/src/github.com/cnlh/nps
COPY . .
RUN go get -d -v ./...
RUN CGO_ENABLED=0 go build -ldflags="-w -s -extldflags -static" ./cmd/nps/nps.go
FROM scratch
COPY --from=builder /go/src/github.com/cnlh/nps/nps /
COPY --from=builder /go/src/github.com/cnlh/nps/web /web
VOLUME /conf
CMD ["/nps"]

75
Makefile Normal file
View File

@ -0,0 +1,75 @@
SOURCE_FILES?=./...
TEST_PATTERN?=.
TEST_OPTIONS?=
export PATH := ./bin:$(PATH)
export GO111MODULE := on
export GOPROXY := https://gocenter.io
# Build a beta version of goreleaser
build:
go build cmd/nps/nps.go
go build cmd/npc/npc.go
.PHONY: build
# Install all the build and lint dependencies
setup:
curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh
curl -L https://git.io/misspell | sh
go mod download
.PHONY: setup
# Run all the tests
test:
go test $(TEST_OPTIONS) -failfast -race -coverpkg=./... -covermode=atomic -coverprofile=coverage.txt $(SOURCE_FILES) -run $(TEST_PATTERN) -timeout=2m
.PHONY: test
# Run all the tests and opens the coverage report
cover: test
go tool cover -html=coverage.txt
.PHONY: cover
# gofmt and goimports all go files
fmt:
find . -name '*.go' -not -wholename './vendor/*' | while read -r file; do gofmt -w -s "$$file"; goimports -w "$$file"; done
.PHONY: fmt
# Run all the linters
lint:
# TODO: fix tests and lll issues
./bin/golangci-lint run --tests=false --enable-all --disable=lll ./...
./bin/misspell -error **/*
.PHONY: lint
# Clean go.mod
go-mod-tidy:
@go mod tidy -v
@git diff HEAD
@git diff-index --quiet HEAD
.PHONY: go-mod-tidy
# Run all the tests and code checks
ci: build test lint go-mod-tidy
.PHONY: ci
# Generate the static documentation
static:
@hugo --enableGitInfo --source www
.PHONY: static
# Show to-do items per file.
todo:
@grep \
--exclude-dir=vendor \
--exclude-dir=node_modules \
--exclude=Makefile \
--text \
--color \
-nRo -E ' TODO:.*|SkipNow' .
.PHONY: todo
clean:
rm npc nps
.PHONY: clean
.DEFAULT_GOAL := build

65
README.md
View File

@ -26,6 +26,7 @@ nps是一款轻量级、高性能、功能强大的**内网穿透**代理服务
* [安装](#安装)
* [编译安装](#源码安装)
* [release安装](#release安装)
* [docker安装](#docker安装)
* [使用示例（以web主控模式为主）](#使用示例)
* [统一准备工作](#统一准备工作(必做))
* [http|https域名解析](#域名解析)
@ -112,6 +113,7 @@ nps是一款轻量级、高性能、功能强大的**内网穿透**代理服务
* [获取用户真实ip](#获取用户真实ip)
* [客户端地址显示](#客户端地址显示)
* [客户端与服务端版本对比](#客户端与服务端版本对比)
* [Linux系统限制](#Linux系统限制)
* [webAPI](#webAPI)
* [贡献](#贡献)
* [支持nps发展](#捐赠)
@ -121,7 +123,7 @@ nps是一款轻量级、高性能、功能强大的**内网穿透**代理服务
## 安装
### releases安装
### release安装
> [releases](https://github.com/cnlh/nps/releases)
下载对应的系统版本即可，服务端和客户端是单独的
@ -134,6 +136,10 @@ nps是一款轻量级、高性能、功能强大的**内网穿透**代理服务
> go build cmd/npc/npc.go
### docker安装
> [server](https://hub.docker.com/r/ffdfgdfg/nps)
> [client](https://hub.docker.com/r/ffdfgdfg/npc)
## 使用示例
### 统一准备工作（必做）
@ -145,6 +151,7 @@ nps是一款轻量级、高性能、功能强大的**内网穿透**代理服务
```shell
./npc -server=1.1.1.1:8284 -vkey=客户端的密钥
```
**注意：运行服务端后，请确保能从客户端设备上正常访问配置文件中所配置的`bridge_port`端口，telnet，netcat这类的来检查**
### 域名解析
@ -184,7 +191,7 @@ nps是一款轻量级、高性能、功能强大的**内网穿透**代理服务
**使用步骤**
- 在刚才创建的客户端的隧道管理中添加一条udp隧道，填写监听的端口（53）、内网目标ip和目标端口（10.1.50.102:53），保存。
- 修改需要使用的内网dns为127.0.0.1，则相当于使用10.1.50.202作为dns服务器
- 修改需要使用的dns地址为1.1.1.1，则相当于使用10.1.50.102作为dns服务器
### socks5代理
@ -198,6 +205,9 @@ nps是一款轻量级、高性能、功能强大的**内网穿透**代理服务
- 在刚才创建的客户端隧道管理中添加一条socks5代理，填写监听的端口（8003），保存。
- 在外网环境的本机配置socks5代理(例如使用proxifier进行全局代理)，ip为公网服务器ip（1.1.1.1），端口为填写的监听端口(8003)，即可畅享内网了
**注意**
经过socks5代理，当收到socks5数据包时socket已经是accept状态。表现是扫描端口全open，建立连接后短时间关闭。若想同内网表现一致，建议远程连接一台设备。
### http正向代理
**适用范围：** 在外网环境下使用http正向代理访问内网站点
@ -400,7 +410,13 @@ web_base_url=/nps
```
(./nps|nps.exe) install
```
安装成功后，对于linux，darwin，将会把配置文件和静态文件放置于/etc/nps/，并将可执行文件nps复制到/usr/bin/nps或者/usr/local/bin/nps，安装成功后可在任何位置执行
安装成功后，对于linux，darwin，将会把配置文件和静态文件放置于/etc/nps/，并将可执行文件nps复制到/usr/bin/nps或者/usr/local/bin/nps，安装成功后可在任何位置执行，同时也会添加systemd配置。
```
sudo systemctl enable|disable|start|stop|restart|status nps
```
systemd，带有开机自启，自动重启配置，当进程结束后15秒会启动，日志输出至/var/log/nps/nps.log。
建议采用此方式启动，能够捕获panic信息，便于排查问题。
```
nps test|start|stop|restart|status
@ -457,6 +473,27 @@ server_ip=xxx
```
./npc -config=npc配置文件路径
```
可自行添加systemd service，例如：`npc.service`
```
[Unit]
Description=npc - convenient proxy server client
Documentation=https://github.com/cnlh/nps/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
[Service]
Type=simple
KillMode=process
Restart=always
RestartSec=15s
StandardOutput=append:/var/log/nps/npc.log
ExecStartPre=/bin/echo 'Starting npc'
ExecStopPost=/bin/echo 'Stopping npc'
ExecStart=/absolutely path to/npc -server=ip:port -vkey=web界面中显示的密钥
[Install]
WantedBy=multi-user.target
```
#### 配置文件说明
[示例配置文件](https://github.com/cnlh/nps/tree/master/conf/npc.conf)
##### 全局配置
@ -563,11 +600,13 @@ vkey=123
[socks5]
mode=socks5
server_port=9004
multi_account=multi_account.conf
```
项 | 含义
---|---
mode | socks5
server_port | 在服务端的代理端口
multi_account | socks5多账号配置文件（可选),配置后使用basic_username和basic_password无法通过认证
##### 私密代理模式
```ini
@ -635,7 +674,7 @@ auto_reconnection=true
```
./npc nat
```
如果p2p双方都是Symmetic Nat，肯定不能成功，其他组合都有较大成功率。
如果p2p双方都是Symmetric Nat，肯定不能成功，其他组合都有较大成功率。
#### 状态检查
```
./npc status -config=npc配置文件路径
@ -817,6 +856,19 @@ nps支持对客户端的隧道数量进行限制，该功能默认是关闭的
nps主要通信默认基于多路复用，无需开启。
多路复用基于TCP滑动窗口原理设计，动态计算延迟以及带宽来算出应该往网络管道中打入的流量。
由于主要通信大多采用TCP协议，并无法探测其实时丢包情况，对于产生丢包重传的情况，采用较大的宽容度，
5分钟的等待时间，超时将会关闭当前隧道连接并重新建立，这将会抛弃当前所有的连接。
在Linux上，可以通过调节内核参数来适应不同应用场景。
对于需求大带宽又有一定的丢包的场景，可以保持默认参数不变，尽可能少抛弃连接
高并发下可根据[Linux系统限制](#Linux系统限制) 调整
对于延迟敏感而又有一定丢包的场景，可以适当调整TCP重传次数
`tcp_syn_retries`, `tcp_retries1`, `tcp_retries2`
高并发同上
nps会在系统主动关闭连接的时候拿到报错，进而重新建立隧道连接
### 环境变量渲染
npc支持环境变量渲染以适应在某些特殊场景下的要求。
@ -925,6 +977,11 @@ LevelInformational->6 LevelDebug->7
### 客户端与服务端版本对比
为了程序正常运行，客户端与服务端的核心版本必须一致，否则将导致客户端无法成功连接致服务端。
### Linux系统限制
默认情况下linux对连接数量有限制，对于性能好的机器完全可以调整内核参数以处理更多的连接。
`tcp_max_syn_backlog` `somaxconn`
酌情调整参数，增强网络性能
## webAPI
### webAPI验证说明

23
bridge/bridge.go
View File

@ -4,6 +4,15 @@ import (
"encoding/binary"
"errors"
"fmt"
"net"
"os"
"strconv"
"strings"
"sync"
"time"
"github.com/astaxie/beego"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/crypt"
@ -12,14 +21,6 @@ import (
"github.com/cnlh/nps/lib/version"
"github.com/cnlh/nps/server/connection"
"github.com/cnlh/nps/server/tool"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"net"
"os"
"strconv"
"strings"
"sync"
"time"
)
type Client struct {
@ -146,7 +147,7 @@ func (s *Bridge) GetHealthFromClient(id int, c *conn.Conn) {
})
}
}
s.DelClient(id, )
s.DelClient(id)
}
//验证失败，返回错误验证flag，并且关闭连接
@ -295,7 +296,7 @@ func (s *Bridge) register(c *conn.Conn) {
func (s *Bridge) SendLinkInfo(clientId int, link *conn.Link, t *file.Tunnel) (target net.Conn, err error) {
//if the proxy type is local
if link.LocalProxy {
target, err = net.Dial(link.ConnType, link.Host)
target, err = net.Dial("tcp", link.Host)
return
}
if v, ok := s.Client.Load(clientId); ok {
@ -472,6 +473,7 @@ loop:
tl.Remark = t.Remark
} else {
tl.Remark = t.Remark + "_" + strconv.Itoa(tl.Port)
tl.Target = new(file.Target)
if t.TargetAddr != "" {
tl.Target.TargetStr = t.TargetAddr + ":" + strconv.Itoa(targets[i])
} else {
@ -486,6 +488,7 @@ loop:
tl.Password = t.Password
tl.LocalPath = t.LocalPath
tl.StripPre = t.StripPre
tl.MultiAccount = t.MultiAccount
if !client.HasTunnel(tl) {
if err := file.GetDb().NewTask(tl); err != nil {
logs.Notice("Add task error ", err.Error())

18
client/client.go
View File

@ -2,17 +2,18 @@ package client
import (
"bufio"
"net"
"net/http"
"strconv"
"time"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/config"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/mux"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/vender/github.com/xtaci/kcp"
"net"
"net/http"
"strconv"
"time"
"github.com/xtaci/kcp-go"
)
type TRPClient struct {
@ -48,6 +49,11 @@ retry:
time.Sleep(time.Second * 5)
goto retry
}
if c == nil {
logs.Error("Error data from server, and will be reconnected in five seconds")
time.Sleep(time.Second * 5)
goto retry
}
logs.Info("Successful connection with server %s", s.svrAddr)
//monitor the connection
go s.ping()

7
client/client_test.go
View File

@ -1,12 +1,13 @@
package client
import (
"github.com/cnlh/nps/lib/common"
conn2 "github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"net"
"sync"
"testing"
"github.com/cnlh/nps/lib/common"
conn2 "github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
)
func TestConfig(t *testing.T) {

28
client/control.go
View File

@ -5,14 +5,6 @@ import (
"encoding/binary"
"errors"
"fmt"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/config"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/version"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/vender/github.com/xtaci/kcp"
"github.com/cnlh/nps/vender/golang.org/x/net/proxy"
"io/ioutil"
"log"
"math"
@ -26,6 +18,15 @@ import (
"strconv"
"strings"
"time"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/config"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/version"
"github.com/xtaci/kcp-go"
"golang.org/x/net/proxy"
)
func GetTaskStatus(path string) {
@ -222,8 +223,13 @@ func NewConn(tp string, vkey string, server string, connType string, proxyUrl st
if _, err := c.Write([]byte(crypt.Md5(version.GetVersion()))); err != nil {
return nil, err
}
if b, err := c.GetShortContent(32); err != nil || crypt.Md5(version.GetVersion()) != string(b) {
logs.Error("The client does not match the server version. The current version of the client is", version.GetVersion())
b, err := c.GetShortContent(32)
if err != nil {
logs.Error(err)
return nil, err
}
if crypt.Md5(version.GetVersion()) != string(b) {
logs.Error("The client does not match the server version. The current core version of the client is", version.GetVersion())
return nil, err
}
if _, err := c.Write([]byte(common.Getverifyval(vkey))); err != nil {
@ -379,7 +385,7 @@ func sendP2PTestMsg(localConn *net.UDPConn, remoteAddr1, remoteAddr2, remoteAddr
ip := common.GetIpByAddr(remoteAddr2)
go func() {
ports := getRandomPortArr(common.GetPortByAddr(remoteAddr3), common.GetPortByAddr(remoteAddr3)+interval*50)
for i := 0; i <= 50; i ++ {
for i := 0; i <= 50; i++ {
go func(port int) {
trueAddress := ip + ":" + strconv.Itoa(port)
logs.Trace("try send test packet to target %s", trueAddress)

17
client/health.go
View File

@ -2,15 +2,16 @@ package client
import (
"container/heap"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/lib/sheap"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/pkg/errors"
"net"
"net/http"
"strings"
"time"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/lib/sheap"
"github.com/pkg/errors"
)
var isStart bool
@ -70,7 +71,11 @@ func check(t *file.Health) {
var rs *http.Response
for _, v := range arr {
if t.HealthCheckType == "tcp" {
_, err = net.DialTimeout("tcp", v, time.Duration(t.HealthCheckTimeout)*time.Second);
var c net.Conn
c, err = net.DialTimeout("tcp", v, time.Duration(t.HealthCheckTimeout)*time.Second)
if err == nil {
c.Close()
}
} else {
client := &http.Client{}
client.Timeout = time.Duration(t.HealthCheckTimeout) * time.Second

15
client/local.go
View File

@ -1,6 +1,12 @@
package client
import (
"net"
"net/http"
"sync"
"time"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/config"
"github.com/cnlh/nps/lib/conn"
@ -8,12 +14,7 @@ import (
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/lib/mux"
"github.com/cnlh/nps/server/proxy"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/vender/github.com/xtaci/kcp"
"net"
"net/http"
"sync"
"time"
"github.com/xtaci/kcp-go"
)
var (
@ -116,7 +117,7 @@ func StartLocalServer(l *config.LocalServer, config *config.CommonConfig) error
func handleUdpMonitor(config *config.CommonConfig, l *config.LocalServer) {
ticker := time.NewTicker(time.Second * 1)
for{
for {
select {
case <-ticker.C:
if !udpConnStatus {

3
client/register.go
View File

@ -2,9 +2,10 @@ package client
import (
"encoding/binary"
"github.com/cnlh/nps/lib/common"
"log"
"os"
"github.com/cnlh/nps/lib/common"
)
func RegisterLocalIp(server string, vKey string, tp string, proxyUrl string, hour int) {

11
cmd/npc/npc.go
View File

@ -3,17 +3,18 @@ package main
import (
"flag"
"fmt"
"os"
"strings"
"time"
"github.com/astaxie/beego/logs"
"github.com/ccding/go-stun/stun"
"github.com/cnlh/nps/client"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/config"
"github.com/cnlh/nps/lib/daemon"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/lib/version"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/vender/github.com/ccding/go-stun/stun"
"os"
"strings"
"time"
)
var (

14
cmd/nps/nps.go
View File

@ -2,6 +2,12 @@ package main
import (
"flag"
"log"
"os"
"path/filepath"
"github.com/astaxie/beego"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/daemon"
@ -12,12 +18,8 @@ import (
"github.com/cnlh/nps/server/connection"
"github.com/cnlh/nps/server/test"
"github.com/cnlh/nps/server/tool"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/web/routers"
"log"
"os"
"path/filepath"
)
var (
@ -61,7 +63,7 @@ func main() {
logs.Error("Getting bridge_port error", err)
os.Exit(0)
}
logs.Info("the version of server is %s ,allow client version to be %s", version.VERSION, version.GetVersion())
logs.Info("the version of server is %s ,allow client core version to be %s", version.VERSION, version.GetVersion())
connection.InitConnectionService()
crypt.InitTls(filepath.Join(common.GetRunPath(), "conf", "server.pem"), filepath.Join(common.GetRunPath(), "conf", "server.key"))
tool.InitAllowPort()

2
conf/multi_account.conf Normal file
View File

@ -0,0 +1,2 @@
# key -> user | value -> pwd
npc=npc.pwd

1
conf/npc.conf
View File

@ -40,6 +40,7 @@ server_port=10000
[socks5]
mode=socks5
server_port=19009
multi_account=multi_account.conf
[file]
mode=file

29
go.mod Normal file
View File

@ -0,0 +1,29 @@
module github.com/cnlh/nps
go 1.12
require (
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d // indirect
github.com/astaxie/beego v1.12.0
github.com/belogik/goes v0.0.0-20151229125003-e54d722c3aff // indirect
github.com/ccding/go-stun v0.0.0-20180726100737-be486d185f3d
github.com/go-ole/go-ole v1.2.4 // indirect
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
github.com/klauspost/cpuid v1.2.1 // indirect
github.com/klauspost/reedsolomon v1.9.2 // indirect
github.com/onsi/gomega v1.5.0 // indirect
github.com/panjf2000/ants/v2 v2.2.2
github.com/pkg/errors v0.8.0
github.com/shiena/ansicolor v0.0.0-20151119151921-a422bbe96644 // indirect
github.com/shirou/gopsutil v2.18.12+incompatible
github.com/stretchr/testify v1.3.0 // indirect
github.com/templexxx/cpufeat v0.0.0-20180724012125-cef66df7f161 // indirect
github.com/templexxx/xor v0.0.0-20181023030647-4e92f724b73b // indirect
github.com/tjfoc/gmsm v1.0.1 // indirect
github.com/xtaci/kcp-go v5.4.4+incompatible
github.com/xtaci/lossyconn v0.0.0-20190602105132-8df528c0c9ae // indirect
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a
golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa // indirect
)
replace github.com/astaxie/beego => github.com/exfly/beego v1.12.0-export-init

92
go.sum Normal file
View File

@ -0,0 +1,92 @@
github.com/Knetic/govaluate v3.0.0+incompatible/go.mod h1:r7JcOSlj0wfOMncg0iLm8Leh48TZaKVeNIfJntJ2wa0=
github.com/OwnLocal/goes v1.0.0/go.mod h1:8rIFjBGTue3lCU0wplczcUgt9Gxgrkkrw7etMIcn8TM=
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUWq3EgK3CesDbo8upS2Vm9/P3FtgI+Jk=
github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg=
github.com/astaxie/beego v1.12.0 h1:MRhVoeeye5N+Flul5PoVfD9CslfdoH+xqC/xvSQ5u2Y=
github.com/astaxie/beego v1.12.0/go.mod h1:fysx+LZNZKnvh4GED/xND7jWtjCR6HzydR2Hh2Im57o=
github.com/beego/goyaml2 v0.0.0-20130207012346-5545475820dd/go.mod h1:1b+Y/CofkYwXMUU0OhQqGvsY2Bvgr4j6jfT699wyZKQ=
github.com/beego/x2j v0.0.0-20131220205130-a0352aadc542/go.mod h1:kSeGC/p1AbBiEp5kat81+DSQrZenVBZXklMLaELspWU=
github.com/belogik/goes v0.0.0-20151229125003-e54d722c3aff/go.mod h1:PhH1ZhyCzHKt4uAasyx+ljRCgoezetRNf59CUtwUkqY=
github.com/bradfitz/gomemcache v0.0.0-20180710155616-bc664df96737/go.mod h1:PmM6Mmwb0LSuEubjR8N7PtNe1KxZLtOUHtbeikc5h60=
github.com/casbin/casbin v1.7.0/go.mod h1:c67qKN6Oum3UF5Q1+BByfFxkwKvhwW57ITjqwtzR1KE=
github.com/ccding/go-stun v0.0.0-20180726100737-be486d185f3d h1:As4937T5NVbJ/DmZT9z33pyLEprMd6CUSfhbmMY57Io=
github.com/ccding/go-stun v0.0.0-20180726100737-be486d185f3d/go.mod h1:3FK1bMar37f7jqVY7q/63k3OMX1c47pGCufzt3X0sYE=
github.com/cloudflare/golz4 v0.0.0-20150217214814-ef862a3cdc58/go.mod h1:EOBUe0h4xcZ5GoxqC5SDxFQ8gwyZPKQoEzownBlhI80=
github.com/couchbase/go-couchbase v0.0.0-20181122212707-3e9b6e1258bb/go.mod h1:TWI8EKQMs5u5jLKW/tsb9VwauIrMIxQG1r5fMsswK5U=
github.com/couchbase/gomemcached v0.0.0-20181122193126-5125a94a666c/go.mod h1:srVSlQLB8iXBVXHgnqemxUXqN6FCvClgCMPCsjBDR7c=
github.com/couchbase/goutils v0.0.0-20180530154633-e865a1461c8a/go.mod h1:BQwMFlJzDjFDG3DJUdU0KORxn88UlsOULuxLExMh3Hs=
github.com/cupcake/rdb v0.0.0-20161107195141-43ba34106c76/go.mod h1:vYwsqCOLxGiisLwp9rITslkFNpZD5rz43tf41QFkTWY=
github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/edsrzf/mmap-go v0.0.0-20170320065105-0bce6a688712/go.mod h1:YO35OhQPt3KJa3ryjFM5Bs14WD66h8eGKpfaBNrHW5M=
github.com/elazarl/go-bindata-assetfs v1.0.0 h1:G/bYguwHIzWq9ZoyUQqrjTmJbbYn3j3CKKpKinvZLFk=
github.com/elazarl/go-bindata-assetfs v1.0.0/go.mod h1:v+YaWX3bdea5J/mo8dSETolEo7R71Vk1u8bnjau5yw4=
github.com/exfly/beego v1.12.0-export-init h1:VQNYKdXhAwZGUaFmQv8Aj921O3rQJZRIF8xeGrhsjrI=
github.com/exfly/beego v1.12.0-export-init/go.mod h1:fysx+LZNZKnvh4GED/xND7jWtjCR6HzydR2Hh2Im57o=
github.com/exfly/beego v1.12.0 h1:OXwIwngaAx35Mga+jLiZmArusBxj8/H0jYXzGDAdwOg=
github.com/exfly/beego v1.12.0/go.mod h1:fysx+LZNZKnvh4GED/xND7jWtjCR6HzydR2Hh2Im57o=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/go-ole/go-ole v1.2.4 h1:nNBDSCOigTSiarFpYE9J/KtEA1IOW4CNeqT9TQDqCxI=
github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM=
github.com/go-redis/redis v6.14.2+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA=
github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db h1:woRePGFeVFfLKN/pOkfl+p/TAqKOfFu+7KPlMVpok/w=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/gomodule/redigo v2.0.0+incompatible/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/klauspost/cpuid v1.2.1 h1:vJi+O/nMdFt0vqm8NZBI6wzALWdA2X+egi0ogNyrC/w=
github.com/klauspost/cpuid v1.2.1/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/klauspost/reedsolomon v1.9.2 h1:E9CMS2Pqbv+C7tsrYad4YC9MfhnMVWhMRsTi7U0UB18=
github.com/klauspost/reedsolomon v1.9.2/go.mod h1:CwCi+NUr9pqSVktrkN+Ondf06rkhYZ/pcNv7fu+8Un4=
github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/mattn/go-sqlite3 v1.10.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/panjf2000/ants/v2 v2.2.2 h1:TWzusBjq/IflXhy+/S6u5wmMLCBdJnB9tPIx9Zmhvok=
github.com/panjf2000/ants/v2 v2.2.2/go.mod h1:1GFm8bV8nyCQvU5K4WvBCTG1/YBFOD2VzjffD8fV55A=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
github.com/pkg/errors v0.8.0 h1:WdK/asTD0HN+q6hsWO3/vpuAkAr+tw6aNJNDFFf0+qw=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/shiena/ansicolor v0.0.0-20151119151921-a422bbe96644 h1:X+yvsM2yrEktyI+b2qND5gpH8YhURn0k8OCaeRnkINo=
github.com/shiena/ansicolor v0.0.0-20151119151921-a422bbe96644/go.mod h1:nkxAfR/5quYxwPZhyDxgasBMnRtBZd0FCEpawpjMUFg=
github.com/shirou/gopsutil v2.18.12+incompatible h1:1eaJvGomDnH74/5cF4CTmTbLHAriGFsTZppLXDX93OM=
github.com/shirou/gopsutil v2.18.12+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726/go.mod h1:3yhqj7WBBfRhbBlzyOC3gUxftwsU0u8gqevxwIHQpMw=
github.com/siddontang/ledisdb v0.0.0-20181029004158-becf5f38d373/go.mod h1:mF1DpOSOUiJRMR+FDqaqu3EBqrybQtrDDszLUZ6oxPg=
github.com/siddontang/rdb v0.0.0-20150307021120-fc89ed2e418d/go.mod h1:AMEsy7v5z92TR1JKMkLLoaOQk++LVnOKL3ScbJ8GNGA=
github.com/ssdb/gossdb v0.0.0-20180723034631-88f6b59b84ec/go.mod h1:QBvMkMya+gXctz3kmljlUCu/yB3GZ6oee+dUozsezQE=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/syndtr/goleveldb v0.0.0-20181127023241-353a9fca669c/go.mod h1:Z4AUp2Km+PwemOoO/VB5AOx9XSsIItzFjoJlOSiYmn0=
github.com/templexxx/cpufeat v0.0.0-20180724012125-cef66df7f161 h1:89CEmDvlq/F7SJEOqkIdNDGJXrQIhuIx9D2DBXjavSU=
github.com/templexxx/cpufeat v0.0.0-20180724012125-cef66df7f161/go.mod h1:wM7WEvslTq+iOEAMDLSzhVuOt5BRZ05WirO+b09GHQU=
github.com/templexxx/xor v0.0.0-20181023030647-4e92f724b73b h1:mnG1fcsIB1d/3vbkBak2MM0u+vhGhlQwpeimUi7QncM=
github.com/templexxx/xor v0.0.0-20181023030647-4e92f724b73b/go.mod h1:5XA7W9S6mni3h5uvOC75dA3m9CCCaS83lltmc0ukdi4=
github.com/tjfoc/gmsm v1.0.1 h1:R11HlqhXkDospckjZEihx9SW/2VW0RgdwrykyWMFOQU=
github.com/tjfoc/gmsm v1.0.1/go.mod h1:XxO4hdhhrzAd+G4CjDqaOkd0hUzmtPR/d3EiBBMn/wc=
github.com/wendal/errors v0.0.0-20130201093226-f66c77a7882b/go.mod h1:Q12BUT7DqIlHRmgv3RskH+UCM/4eqVMgI0EMmlSpAXc=
github.com/xtaci/kcp-go v5.4.4+incompatible h1:QIJ0a0Q0N1G20yLHL2+fpdzyy2v/Cb3PI+xiwx/KK9c=
github.com/xtaci/kcp-go v5.4.4+incompatible/go.mod h1:bN6vIwHQbfHaHtFpEssmWsN45a+AZwO7eyRCmEIbtvE=
github.com/xtaci/lossyconn v0.0.0-20190602105132-8df528c0c9ae h1:J0GxkO96kL4WF+AIT3M4mfUVinOCPgf2uUWYFUzN0sM=
github.com/xtaci/lossyconn v0.0.0-20190602105132-8df528c0c9ae/go.mod h1:gXtu8J62kEgmN++bm9BVICuT/e8yiLI2KFobd/TRFsE=
golang.org/x/crypto v0.0.0-20181127143415-eb0de9b17e85 h1:et7+NAX3lLIk5qUCTA9QelBjGE/NkhzYw/mhnr0s7nI=
golang.org/x/crypto v0.0.0-20181127143415-eb0de9b17e85/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a h1:gOpx8G595UYyvj8UK4+OFyY4rx037g3fmfhe5SasG3U=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa h1:KIDDMLT1O0Nr7TSxp8xM5tJcdn8tgyAONntO829og1M=
golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

16
lib/common/const.go
View File

@ -36,3 +36,19 @@ WWW-Authenticate: Basic realm="easyProxy"
`
)
const (
MUX_PING_FLAG uint8 = iota
MUX_NEW_CONN_OK
MUX_NEW_CONN_Fail
MUX_NEW_MSG
MUX_NEW_MSG_PART
MUX_MSG_SEND_OK
MUX_NEW_CONN
MUX_CONN_CLOSE
MUX_PING_RETURN
MUX_PING int32 = -1
MAXIMUM_SEGMENT_SIZE = PoolSizeWindow
MAXIMUM_WINDOW_SIZE = 1 << 25 // 1<<31-1 TCP slide window size is very large,
// we use 32M, reduce memory usage
)

235
lib/common/netpackager.go Normal file
View File

@ -0,0 +1,235 @@
package common
import (
"bytes"
"encoding/binary"
"encoding/json"
"errors"
"io"
"strings"
)
type NetPackager interface {
Pack(writer io.Writer) (err error)
UnPack(reader io.Reader) (err error)
}
type BasePackager struct {
Length uint16
Content []byte
}
func (Self *BasePackager) NewPac(contents ...interface{}) (err error) {
Self.clean()
for _, content := range contents {
switch content.(type) {
case nil:
Self.Content = Self.Content[:0]
case []byte:
err = Self.appendByte(content.([]byte))
case string:
err = Self.appendByte([]byte(content.(string)))
if err != nil {
return
}
err = Self.appendByte([]byte(CONN_DATA_SEQ))
default:
err = Self.marshal(content)
}
}
Self.setLength()
return
}
func (Self *BasePackager) appendByte(data []byte) (err error) {
m := len(Self.Content)
n := m + len(data)
if n <= cap(Self.Content) {
Self.Content = Self.Content[0:n] // grow the length for copy
copy(Self.Content[m:n], data)
return nil
} else {
return errors.New("pack content too large")
}
}
//似乎这里涉及到父类作用域问题，当子类调用父类的方法时，其struct仅仅为父类的
func (Self *BasePackager) Pack(writer io.Writer) (err error) {
err = binary.Write(writer, binary.LittleEndian, Self.Length)
if err != nil {
return
}
err = binary.Write(writer, binary.LittleEndian, Self.Content)
return
}
//Unpack 会导致传入的数字类型转化成float64！！
//主要原因是json unmarshal并未传入正确的数据类型
func (Self *BasePackager) UnPack(reader io.Reader) (n uint16, err error) {
Self.clean()
n += 2 // uint16
err = binary.Read(reader, binary.LittleEndian, &Self.Length)
if err != nil {
return
}
if int(Self.Length) > cap(Self.Content) {
err = errors.New("unpack err, content length too large")
}
Self.Content = Self.Content[:int(Self.Length)]
//n, err := io.ReadFull(reader, Self.Content)
//if n != int(Self.Length) {
// err = io.ErrUnexpectedEOF
//}
err = binary.Read(reader, binary.LittleEndian, Self.Content)
n += Self.Length
return
}
func (Self *BasePackager) marshal(content interface{}) (err error) {
tmp, err := json.Marshal(content)
if err != nil {
return err
}
err = Self.appendByte(tmp)
return
}
func (Self *BasePackager) Unmarshal(content interface{}) (err error) {
err = json.Unmarshal(Self.Content, content)
if err != nil {
return err
}
return
}
func (Self *BasePackager) setLength() {
Self.Length = uint16(len(Self.Content))
return
}
func (Self *BasePackager) clean() {
Self.Length = 0
Self.Content = Self.Content[:0] // reset length
}
func (Self *BasePackager) Split() (strList []string) {
n := bytes.IndexByte(Self.Content, 0)
strList = strings.Split(string(Self.Content[:n]), CONN_DATA_SEQ)
strList = strList[0 : len(strList)-1]
return
}
type ConnPackager struct { // Todo
ConnType uint8
BasePackager
}
func (Self *ConnPackager) NewPac(connType uint8, content ...interface{}) (err error) {
Self.ConnType = connType
err = Self.BasePackager.NewPac(content...)
return
}
func (Self *ConnPackager) Pack(writer io.Writer) (err error) {
err = binary.Write(writer, binary.LittleEndian, Self.ConnType)
if err != nil {
return
}
err = Self.BasePackager.Pack(writer)
return
}
func (Self *ConnPackager) UnPack(reader io.Reader) (n uint16, err error) {
err = binary.Read(reader, binary.LittleEndian, &Self.ConnType)
if err != nil && err != io.EOF {
return
}
n, err = Self.BasePackager.UnPack(reader)
n += 2
return
}
type MuxPackager struct {
Flag uint8
Id int32
Window uint32
ReadLength uint32
BasePackager
}
func (Self *MuxPackager) NewPac(flag uint8, id int32, content ...interface{}) (err error) {
Self.Flag = flag
Self.Id = id
switch flag {
case MUX_PING_FLAG, MUX_PING_RETURN, MUX_NEW_MSG, MUX_NEW_MSG_PART:
Self.Content = WindowBuff.Get()
err = Self.BasePackager.NewPac(content...)
//logs.Warn(Self.Length, string(Self.Content))
case MUX_MSG_SEND_OK:
// MUX_MSG_SEND_OK contains two data
switch content[0].(type) {
case int:
Self.Window = uint32(content[0].(int))
case uint32:
Self.Window = content[0].(uint32)
}
switch content[1].(type) {
case int:
Self.ReadLength = uint32(content[1].(int))
case uint32:
Self.ReadLength = content[1].(uint32)
}
}
return
}
func (Self *MuxPackager) Pack(writer io.Writer) (err error) {
err = binary.Write(writer, binary.LittleEndian, Self.Flag)
if err != nil {
return
}
err = binary.Write(writer, binary.LittleEndian, Self.Id)
if err != nil {
return
}
switch Self.Flag {
case MUX_NEW_MSG, MUX_NEW_MSG_PART, MUX_PING_FLAG, MUX_PING_RETURN:
err = Self.BasePackager.Pack(writer)
WindowBuff.Put(Self.Content)
case MUX_MSG_SEND_OK:
err = binary.Write(writer, binary.LittleEndian, Self.Window)
if err != nil {
return
}
err = binary.Write(writer, binary.LittleEndian, Self.ReadLength)
}
return
}
func (Self *MuxPackager) UnPack(reader io.Reader) (n uint16, err error) {
err = binary.Read(reader, binary.LittleEndian, &Self.Flag)
if err != nil {
return
}
err = binary.Read(reader, binary.LittleEndian, &Self.Id)
if err != nil {
return
}
switch Self.Flag {
case MUX_NEW_MSG, MUX_NEW_MSG_PART, MUX_PING_FLAG, MUX_PING_RETURN:
Self.Content = WindowBuff.Get() // need get a window buf from pool
Self.BasePackager.clean() // also clean the content
n, err = Self.BasePackager.UnPack(reader)
//logs.Warn("unpack", Self.Length, string(Self.Content))
case MUX_MSG_SEND_OK:
err = binary.Read(reader, binary.LittleEndian, &Self.Window)
if err != nil {
return
}
n += 4 // uint32
err = binary.Read(reader, binary.LittleEndian, &Self.ReadLength)
n += 4 // uint32
}
n += 5 //uint8 int32
return
}

199
lib/common/pool.go Normal file
View File

@ -0,0 +1,199 @@
package common
import (
"bytes"
"sync"
)
const PoolSize = 64 * 1024
const PoolSizeSmall = 100
const PoolSizeUdp = 1472
const PoolSizeCopy = 32 << 10
const PoolSizeBuffer = 4096
const PoolSizeWindow = PoolSizeBuffer - 2 - 4 - 4 - 1
var BufPool = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSize)
},
}
var BufPoolUdp = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSizeUdp)
},
}
var BufPoolMax = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSize)
},
}
var BufPoolSmall = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSizeSmall)
},
}
var BufPoolCopy = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSizeCopy)
},
}
func PutBufPoolUdp(buf []byte) {
if cap(buf) == PoolSizeUdp {
BufPoolUdp.Put(buf[:PoolSizeUdp])
}
}
func PutBufPoolCopy(buf []byte) {
if cap(buf) == PoolSizeCopy {
BufPoolCopy.Put(buf[:PoolSizeCopy])
}
}
func GetBufPoolCopy() []byte {
return (BufPoolCopy.Get().([]byte))[:PoolSizeCopy]
}
func PutBufPoolMax(buf []byte) {
if cap(buf) == PoolSize {
BufPoolMax.Put(buf[:PoolSize])
}
}
type copyBufferPool struct {
pool sync.Pool
}
func (Self *copyBufferPool) New() {
Self.pool = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSizeCopy, PoolSizeCopy)
},
}
}
func (Self *copyBufferPool) Get() []byte {
buf := Self.pool.Get().([]byte)
return buf[:PoolSizeCopy] // just like make a new slice, but data may not be 0
}
func (Self *copyBufferPool) Put(x []byte) {
if len(x) == PoolSizeCopy {
Self.pool.Put(x)
} else {
x = nil // buf is not full, not allowed, New method returns a full buf
}
}
type windowBufferPool struct {
pool sync.Pool
}
func (Self *windowBufferPool) New() {
Self.pool = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSizeWindow, PoolSizeWindow)
},
}
}
func (Self *windowBufferPool) Get() (buf []byte) {
buf = Self.pool.Get().([]byte)
return buf[:PoolSizeWindow]
}
func (Self *windowBufferPool) Put(x []byte) {
Self.pool.Put(x[:PoolSizeWindow]) // make buf to full
}
type bufferPool struct {
pool sync.Pool
}
func (Self *bufferPool) New() {
Self.pool = sync.Pool{
New: func() interface{} {
return bytes.NewBuffer(make([]byte, 0, PoolSizeBuffer))
},
}
}
func (Self *bufferPool) Get() *bytes.Buffer {
return Self.pool.Get().(*bytes.Buffer)
}
func (Self *bufferPool) Put(x *bytes.Buffer) {
x.Reset()
Self.pool.Put(x)
}
type muxPackagerPool struct {
pool sync.Pool
}
func (Self *muxPackagerPool) New() {
Self.pool = sync.Pool{
New: func() interface{} {
pack := MuxPackager{}
return &pack
},
}
}
func (Self *muxPackagerPool) Get() *MuxPackager {
return Self.pool.Get().(*MuxPackager)
}
func (Self *muxPackagerPool) Put(pack *MuxPackager) {
Self.pool.Put(pack)
}
type ListElement struct {
Buf []byte
L uint16
Part bool
}
type listElementPool struct {
pool sync.Pool
}
func (Self *listElementPool) New() {
Self.pool = sync.Pool{
New: func() interface{} {
element := ListElement{}
return &element
},
}
}
func (Self *listElementPool) Get() *ListElement {
return Self.pool.Get().(*ListElement)
}
func (Self *listElementPool) Put(element *ListElement) {
element.L = 0
element.Buf = nil
element.Part = false
Self.pool.Put(element)
}
var once = sync.Once{}
var BuffPool = bufferPool{}
var CopyBuff = copyBufferPool{}
var MuxPack = muxPackagerPool{}
var WindowBuff = windowBufferPool{}
var ListElementPool = listElementPool{}
func newPool() {
BuffPool.New()
CopyBuff.New()
MuxPack.New()
WindowBuff.New()
ListElementPool.New()
}
func init() {
once.Do(newPool)
}

20
lib/common/util.go
View File

@ -4,8 +4,6 @@ import (
"bytes"
"encoding/base64"
"encoding/binary"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/pool"
"html/template"
"io"
"io/ioutil"
@ -16,6 +14,8 @@ import (
"strconv"
"strings"
"sync"
"github.com/cnlh/nps/lib/crypt"
)
//Get the corresponding IP address through domain name
@ -108,6 +108,9 @@ func ChangeHostAndHeader(r *http.Request, host string, header string, addr strin
}
}
addr = strings.Split(addr, ":")[0]
if prior, ok := r.Header["X-Forwarded-For"]; ok {
addr = strings.Join(prior, ", ") + ", " + addr
}
r.Header.Set("X-Forwarded-For", addr)
r.Header.Set("X-Real-IP", addr)
}
@ -263,11 +266,14 @@ func GetPortByAddr(addr string) int {
return p
}
func CopyBuffer(dst io.Writer, src io.Reader) (written int64, err error) {
buf := pool.GetBufPoolCopy()
defer pool.PutBufPoolCopy(buf)
func CopyBuffer(dst io.Writer, src io.Reader, label ...string) (written int64, err error) {
buf := CopyBuff.Get()
defer CopyBuff.Put(buf)
for {
nr, er := src.Read(buf)
//if len(pr)>0 && pr[0] && nr > 50 {
// logs.Warn(string(buf[:50]))
//}
if nr > 0 {
nw, ew := dst.Write(buf[0:nr])
if nw > 0 {
@ -283,9 +289,7 @@ func CopyBuffer(dst io.Writer, src io.Reader) (written int64, err error) {
}
}
if er != nil {
if er != io.EOF {
err = er
}
err = er
break
}
}

34
lib/config/config.go
View File

@ -3,10 +3,11 @@ package config
import (
"errors"
"fmt"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/file"
"regexp"
"strings"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/file"
)
type CommonConfig struct {
@ -226,8 +227,10 @@ func dealTunnel(s string) *file.Tunnel {
t.ServerIp = item[1]
case "mode":
t.Mode = item[1]
case "target_port", "target_addr":
case "target_addr":
t.Target.TargetStr = strings.Replace(item[1], ",", "\n", -1)
case "target_port":
t.Target.TargetStr = item[1]
case "target_ip":
t.TargetAddr = item[1]
case "password":
@ -236,12 +239,37 @@ func dealTunnel(s string) *file.Tunnel {
t.LocalPath = item[1]
case "strip_pre":
t.StripPre = item[1]
case "multi_account":
t.MultiAccount = &file.MultiAccount{}
if b, err := common.ReadAllFromFile(item[1]); err != nil {
panic(err)
} else {
if content, err := common.ParseStr(string(b)); err != nil {
panic(err)
} else {
t.MultiAccount.AccountMap = dealMultiUser(content)
}
}
}
}
return t
}
func dealMultiUser(s string) map[string]string {
multiUserMap := make(map[string]string)
for _, v := range splitStr(s) {
item := strings.Split(v, "=")
if len(item) == 0 {
continue
} else if len(item) == 1 {
item = append(item, "")
}
multiUserMap[strings.TrimSpace(item[0])] = item[1]
}
return multiUserMap
}
func delLocalService(s string) *LocalServer {
l := new(LocalServer)
for _, v := range splitStr(s) {

61
lib/conn/conn.go
View File

@ -6,21 +6,22 @@ import (
"encoding/binary"
"encoding/json"
"errors"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/lib/mux"
"github.com/cnlh/nps/lib/pool"
"github.com/cnlh/nps/lib/rate"
"github.com/cnlh/nps/vender/github.com/xtaci/kcp"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/goroutine"
"io"
"net"
"net/http"
"net/url"
"strconv"
"strings"
"sync"
"time"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/lib/mux"
"github.com/cnlh/nps/lib/rate"
"github.com/xtaci/kcp-go"
)
type Conn struct {
@ -158,8 +159,8 @@ func (s *Conn) SendHealthInfo(info, status string) (int, error) {
//get health info from conn
func (s *Conn) GetHealthInfo() (info string, status bool, err error) {
var l int
buf := pool.BufPoolMax.Get().([]byte)
defer pool.PutBufPoolMax(buf)
buf := common.BufPoolMax.Get().([]byte)
defer common.PutBufPoolMax(buf)
if l, err = s.GetLen(); err != nil {
return
} else if _, err = s.ReadLen(l, buf); err != nil {
@ -232,8 +233,8 @@ func (s *Conn) SendInfo(t interface{}, flag string) (int, error) {
//get task info
func (s *Conn) getInfo(t interface{}) (err error) {
var l int
buf := pool.BufPoolMax.Get().([]byte)
defer pool.PutBufPoolMax(buf)
buf := common.BufPoolMax.Get().([]byte)
defer common.PutBufPoolMax(buf)
if l, err = s.GetLen(); err != nil {
return
} else if _, err = s.ReadLen(l, buf); err != nil {
@ -350,30 +351,34 @@ func SetUdpSession(sess *kcp.UDPSession) {
//conn1 mux conn
func CopyWaitGroup(conn1, conn2 net.Conn, crypt bool, snappy bool, rate *rate.Rate, flow *file.Flow, isServer bool, rb []byte) {
var in, out int64
var wg sync.WaitGroup
//var in, out int64
//var wg sync.WaitGroup
connHandle := GetConn(conn1, crypt, snappy, rate, isServer)
if rb != nil {
connHandle.Write(rb)
}
go func(in *int64) {
wg.Add(1)
*in, _ = common.CopyBuffer(connHandle, conn2)
connHandle.Close()
conn2.Close()
wg.Done()
}(&in)
out, _ = common.CopyBuffer(conn2, connHandle)
connHandle.Close()
conn2.Close()
wg.Wait()
if flow != nil {
flow.Add(in, out)
//go func(in *int64) {
// wg.Add(1)
// *in, _ = common.CopyBuffer(connHandle, conn2)
// connHandle.Close()
// conn2.Close()
// wg.Done()
//}(&in)
//out, _ = common.CopyBuffer(conn2, connHandle)
//connHandle.Close()
//conn2.Close()
//wg.Wait()
//if flow != nil {
// flow.Add(in, out)
//}
err := goroutine.CopyConnsPool.Invoke(goroutine.NewConns(connHandle, conn2, flow))
if err != nil {
logs.Error(err)
}
}
//get crypt or snappy conn
func GetConn(conn net.Conn, cpt, snappy bool, rt *rate.Rate, isServer bool) (io.ReadWriteCloser) {
func GetConn(conn net.Conn, cpt, snappy bool, rt *rate.Rate, isServer bool) io.ReadWriteCloser {
if cpt {
if isServer {
return rate.NewRateConn(crypt.NewTlsServerConn(conn), rt)

12
lib/conn/listener.go
View File

@ -1,10 +1,11 @@
package conn
import (
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/vender/github.com/xtaci/kcp"
"net"
"strings"
"github.com/astaxie/beego/logs"
"github.com/xtaci/kcp-go"
)
func NewTcpListenerAndProcess(addr string, f func(c net.Conn), listener *net.Listener) error {
@ -42,9 +43,16 @@ func Accept(l net.Listener, f func(c net.Conn)) {
if strings.Contains(err.Error(), "use of closed network connection") {
break
}
if strings.Contains(err.Error(), "the mux has closed") {
break
}
logs.Warn(err)
continue
}
if c == nil {
logs.Warn("nil connection")
break
}
go f(c)
}
}

9
lib/conn/snappy.go
View File

@ -1,9 +1,10 @@
package conn
import (
"github.com/cnlh/nps/lib/pool"
"github.com/cnlh/nps/vender/github.com/golang/snappy"
"io"
"github.com/cnlh/nps/lib/common"
"github.com/golang/snappy"
)
type SnappyConn struct {
@ -31,8 +32,8 @@ func (s *SnappyConn) Write(b []byte) (n int, err error) {
//snappy压缩读
func (s *SnappyConn) Read(b []byte) (n int, err error) {
buf := pool.BufPool.Get().([]byte)
defer pool.BufPool.Put(buf)
buf := common.BufPool.Get().([]byte)
defer common.BufPool.Put(buf)
if n, err = s.r.Read(buf); err != nil {
return
}

3
lib/crypt/tls.go
View File

@ -2,9 +2,10 @@ package crypt
import (
"crypto/tls"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"net"
"os"
"github.com/astaxie/beego/logs"
)
var pemPath, keyPath string

3
lib/daemon/daemon.go
View File

@ -1,7 +1,6 @@
package daemon
import (
"github.com/cnlh/nps/lib/common"
"io/ioutil"
"log"
"os"
@ -9,6 +8,8 @@ import (
"path/filepath"
"strconv"
"strings"
"github.com/cnlh/nps/lib/common"
)
func InitDaemon(f string, runPath string, pidPath string) {

5
lib/daemon/reload.go
View File

@ -3,12 +3,13 @@
package daemon
import (
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"os"
"os/signal"
"path/filepath"
"syscall"
"github.com/astaxie/beego"
"github.com/cnlh/nps/lib/common"
)
func init() {

7
lib/file/db.go
View File

@ -3,14 +3,15 @@ package file
import (
"errors"
"fmt"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/rate"
"net/http"
"regexp"
"sort"
"strings"
"sync"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/rate"
)
type DbUtils struct {

5
lib/file/file.go
View File

@ -3,13 +3,14 @@ package file
import (
"encoding/json"
"errors"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/rate"
"os"
"path/filepath"
"strings"
"sync"
"sync/atomic"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/rate"
)
func NewJsonDb(runPath string) *JsonDb {

42
lib/file/obj.go
View File

@ -1,12 +1,13 @@
package file
import (
"github.com/cnlh/nps/lib/rate"
"github.com/pkg/errors"
"strings"
"sync"
"sync/atomic"
"time"
"github.com/cnlh/nps/lib/rate"
"github.com/pkg/errors"
)
type Flow struct {
@ -123,22 +124,23 @@ func (s *Client) HasHost(h *Host) bool {
}
type Tunnel struct {
Id int
Port int
ServerIp string
Mode string
Status bool
RunStatus bool
Client *Client
Ports string
Flow *Flow
Password string
Remark string
TargetAddr string
NoStore bool
LocalPath string
StripPre string
Target *Target
Id int
Port int
ServerIp string
Mode string
Status bool
RunStatus bool
Client *Client
Ports string
Flow *Flow
Password string
Remark string
TargetAddr string
NoStore bool
LocalPath string
StripPre string
Target *Target
MultiAccount *MultiAccount
Health
sync.RWMutex
}
@ -183,6 +185,10 @@ type Target struct {
sync.RWMutex
}
type MultiAccount struct {
AccountMap map[string]string // multi account and pwd
}
func (s *Target) GetRandomTarget() (string, error) {
if s.TargetArr == nil {
s.TargetArr = strings.Split(s.TargetStr, "\n")

73
lib/goroutine/pool.go Normal file
View File

@ -0,0 +1,73 @@
package goroutine
import (
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/file"
"github.com/panjf2000/ants/v2"
"io"
"net"
"sync"
)
type connGroup struct {
src io.ReadWriteCloser
dst io.ReadWriteCloser
wg *sync.WaitGroup
n *int64
}
func newConnGroup(dst, src io.ReadWriteCloser, wg *sync.WaitGroup, n *int64) connGroup {
return connGroup{
src: src,
dst: dst,
wg: wg,
n: n,
}
}
func copyConnGroup(group interface{}) {
cg, ok := group.(connGroup)
if !ok {
return
}
var err error
*cg.n, err = common.CopyBuffer(cg.dst, cg.src)
if err != nil {
cg.src.Close()
cg.dst.Close()
//logs.Warn("close npc by copy from nps", err, c.connId)
}
cg.wg.Done()
}
type Conns struct {
conn1 io.ReadWriteCloser // mux connection
conn2 net.Conn // outside connection
flow *file.Flow
}
func NewConns(c1 io.ReadWriteCloser, c2 net.Conn, flow *file.Flow) Conns {
return Conns{
conn1: c1,
conn2: c2,
flow: flow,
}
}
func copyConns(group interface{}) {
conns := group.(Conns)
wg := new(sync.WaitGroup)
wg.Add(2)
var in, out int64
_ = connCopyPool.Invoke(newConnGroup(conns.conn1, conns.conn2, wg, &in))
// outside to mux : incoming
_ = connCopyPool.Invoke(newConnGroup(conns.conn2, conns.conn1, wg, &out))
// mux to outside : outgoing
wg.Wait()
if conns.flow != nil {
conns.flow.Add(in, out)
}
}
var connCopyPool, _ = ants.NewPoolWithFunc(200000, copyConnGroup, ants.WithNonblocking(false))
var CopyConnsPool, _ = ants.NewPoolWithFunc(100000, copyConns, ants.WithNonblocking(false))

45
lib/install/install.go
View File

@ -3,15 +3,34 @@ package install
import (
"errors"
"fmt"
"github.com/cnlh/nps/lib/common"
"io"
"io/ioutil"
"log"
"os"
"path/filepath"
"strings"
"github.com/cnlh/nps/lib/common"
)
func InstallNps() {
unit := `[Unit]
Description=nps - convenient proxy server
Documentation=https://github.com/cnlh/nps/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target`
service := `[Service]
Type=simple
KillMode=process
Restart=always
RestartSec=15s
StandardOutput=append:/var/log/nps/nps.log
ExecStartPre=/bin/echo 'Starting nps'
ExecStopPost=/bin/echo 'Stopping nps'
ExecStart=`
install := `[Install]
WantedBy=multi-user.target`
path := common.GetInstallPath()
if common.FileExists(path) {
log.Fatalf("the path %s has exist, does not support install", path)
@ -33,22 +52,36 @@ func InstallNps() {
if _, err := copyFile(filepath.Join(common.GetAppPath(), "nps"), "/usr/local/bin/nps"); err != nil {
log.Fatalln(err)
} else {
os.Chmod("/usr/local/bin/nps", 0777)
os.Chmod("/usr/local/bin/nps", 0755)
service += "/usr/local/bin/nps"
log.Println("Executable files have been copied to", "/usr/local/bin/nps")
}
} else {
os.Chmod("/usr/bin/nps", 0777)
os.Chmod("/usr/bin/nps", 0755)
service += "/usr/bin/nps"
log.Println("Executable files have been copied to", "/usr/bin/nps")
}
systemd := unit + "\n\n" + service + "\n\n" + install
_ = os.Remove("/usr/lib/systemd/system/nps.service")
err := ioutil.WriteFile("/usr/lib/systemd/system/nps.service", []byte(systemd), 0644)
if err != nil {
log.Println("Write systemd service err ", err)
}
_ = os.Mkdir("/var/log/nps", 644)
}
log.Println("install ok!")
log.Println("Static files and configuration files in the current directory will be useless")
log.Println("The new configuration file is located in", path, "you can edit them")
if !common.IsWindows() {
log.Println("You can start with nps test|start|stop|restart|status anywhere")
log.Println(`You can start with:
sudo systemctl enable|disable|start|stop|restart|status nps
or:
nps test|start|stop|restart|status
anywhere!`)
} else {
log.Println("You can copy executable files to any directory and start working with nps.exe test|start|stop|restart|status")
log.Println(`You can copy executable files to any directory and start working with:
nps.exe test|start|stop|restart|status
now!`)
}
}
func MkidrDirAll(path string, v ...string) {

2
lib/mux/bytes.go
View File

@ -20,7 +20,7 @@ func WriteLenBytes(buf []byte, w io.Writer) (int, error) {
//read bytes by length
func ReadLenBytes(buf []byte, r io.Reader) (int, error) {
var l int32
var l uint32
var err error
if binary.Read(r, binary.LittleEndian, &l) != nil {
return 0, err

690
lib/mux/conn.go
View File

@ -2,11 +2,15 @@ package mux
import (
"errors"
"github.com/cnlh/nps/lib/pool"
"io"
"math"
"net"
"runtime"
"sync"
"sync/atomic"
"time"
"github.com/cnlh/nps/lib/common"
)
type conn struct {
@ -14,34 +18,36 @@ type conn struct {
getStatusCh chan struct{}
connStatusOkCh chan struct{}
connStatusFailCh chan struct{}
readTimeOut time.Time
writeTimeOut time.Time
readBuffer []byte
startRead int //now read position
endRead int //now end read
readFlag bool
readCh chan struct{}
waitQueue *sliceEntry
stopWrite bool
connId int32
isClose bool
readWait bool
hasWrite int
mux *Mux
closeFlag bool // close conn flag
receiveWindow *ReceiveWindow
sendWindow *SendWindow
once sync.Once
//label string
}
var connPool = sync.Pool{}
func NewConn(connId int32, mux *Mux) *conn {
func NewConn(connId int32, mux *Mux, label ...string) *conn {
c := &conn{
readCh: make(chan struct{}),
getStatusCh: make(chan struct{}),
connStatusOkCh: make(chan struct{}),
connStatusFailCh: make(chan struct{}),
waitQueue: NewQueue(),
connId: connId,
mux: mux,
receiveWindow: new(ReceiveWindow),
sendWindow: new(SendWindow),
once: sync.Once{},
}
//if len(label) > 0 {
// c.label = label[0]
//}
c.receiveWindow.New(mux)
c.sendWindow.New(mux)
//logm := &connLog{
// startTime: time.Now(),
// isClose: false,
// logs: []string{c.label + "new conn success"},
//}
//setM(label[0], int(connId), logm)
return c
}
@ -49,135 +55,581 @@ func (s *conn) Read(buf []byte) (n int, err error) {
if s.isClose || buf == nil {
return 0, errors.New("the conn has closed")
}
if s.endRead-s.startRead == 0 { //read finish or start
if s.waitQueue.Size() == 0 {
s.readWait = true
if t := s.readTimeOut.Sub(time.Now()); t > 0 {
timer := time.NewTimer(t)
defer timer.Stop()
select {
case <-timer.C:
s.readWait = false
return 0, errors.New("read timeout")
case <-s.readCh:
}
} else {
<-s.readCh
}
}
if s.isClose { //If the connection is closed instead of continuing command
return 0, errors.New("the conn has closed")
}
if node, err := s.waitQueue.Pop(); err != nil {
s.Close()
return 0, io.EOF
} else {
pool.PutBufPoolCopy(s.readBuffer)
s.readBuffer = node.val
s.endRead = node.l
s.startRead = 0
}
}
if len(buf) < s.endRead-s.startRead {
n = copy(buf, s.readBuffer[s.startRead:s.startRead+len(buf)])
s.startRead += n
} else {
n = copy(buf, s.readBuffer[s.startRead:s.endRead])
s.startRead += n
s.mux.sendInfo(MUX_MSG_SEND_OK, s.connId, nil)
if len(buf) == 0 {
return 0, nil
}
// waiting for takeout from receive window finish or timeout
//now := time.Now()
n, err = s.receiveWindow.Read(buf, s.connId)
//t := time.Now().Sub(now)
//if t.Seconds() > 0.5 {
//logs.Warn("conn read long", n, t.Seconds())
//}
//var errstr string
//if err == nil {
// errstr = "err:nil"
//} else {
// errstr = err.Error()
//}
//d := getM(s.label, int(s.connId))
//d.logs = append(d.logs, s.label+"read "+strconv.Itoa(n)+" "+errstr+" "+string(buf[:100]))
//setM(s.label, int(s.connId), d)
return
}
func (s *conn) Write(buf []byte) (int, error) {
func (s *conn) Write(buf []byte) (n int, err error) {
if s.isClose {
return 0, errors.New("the conn has closed")
}
ch := make(chan struct{})
go s.write(buf, ch)
if t := s.writeTimeOut.Sub(time.Now()); t > 0 {
timer := time.NewTimer(t)
defer timer.Stop()
select {
case <-timer.C:
return 0, errors.New("write timeout")
case <-ch:
}
} else {
<-ch
if s.closeFlag {
//s.Close()
return 0, errors.New("io: write on closed conn")
}
if s.isClose {
return 0, io.EOF
if len(buf) == 0 {
return 0, nil
}
return len(buf), nil
}
func (s *conn) write(buf []byte, ch chan struct{}) {
start := 0
l := len(buf)
for {
if s.hasWrite > 50 {
<-s.getStatusCh
}
s.hasWrite++
if l-start > pool.PoolSizeCopy {
s.mux.sendInfo(MUX_NEW_MSG, s.connId, buf[start:start+pool.PoolSizeCopy])
start += pool.PoolSizeCopy
} else {
s.mux.sendInfo(MUX_NEW_MSG, s.connId, buf[start:l])
break
}
}
ch <- struct{}{}
//logs.Warn("write buf", len(buf))
//now := time.Now()
n, err = s.sendWindow.WriteFull(buf, s.connId)
//t := time.Now().Sub(now)
//if t.Seconds() > 0.5 {
// logs.Warn("conn write long", n, t.Seconds())
//}
return
}
func (s *conn) Close() error {
if s.isClose {
return errors.New("the conn has closed")
}
times := 0
retry:
if s.waitQueue.Size() > 0 && times < 600 {
time.Sleep(time.Millisecond * 100)
times++
goto retry
}
if s.isClose {
return errors.New("the conn has closed")
}
func (s *conn) Close() (err error) {
s.once.Do(s.closeProcess)
return
}
func (s *conn) closeProcess() {
s.isClose = true
pool.PutBufPoolCopy(s.readBuffer)
if s.readWait {
s.readCh <- struct{}{}
s.receiveWindow.mux.connMap.Delete(s.connId)
if !s.receiveWindow.mux.IsClose {
// if server or user close the conn while reading, will get a io.EOF
// and this Close method will be invoke, send this signal to close other side
s.receiveWindow.mux.sendInfo(common.MUX_CONN_CLOSE, s.connId, nil)
}
s.waitQueue.Clear()
s.mux.connMap.Delete(s.connId)
if !s.mux.IsClose {
s.mux.sendInfo(MUX_CONN_CLOSE, s.connId, nil)
}
connPool.Put(s)
return nil
s.sendWindow.CloseWindow()
s.receiveWindow.CloseWindow()
//d := getM(s.label, int(s.connId))
//d.isClose = true
//d.logs = append(d.logs, s.label+"close "+time.Now().String())
//setM(s.label, int(s.connId), d)
return
}
func (s *conn) LocalAddr() net.Addr {
return s.mux.conn.LocalAddr()
return s.receiveWindow.mux.conn.LocalAddr()
}
func (s *conn) RemoteAddr() net.Addr {
return s.mux.conn.RemoteAddr()
return s.receiveWindow.mux.conn.RemoteAddr()
}
func (s *conn) SetDeadline(t time.Time) error {
s.readTimeOut = t
s.writeTimeOut = t
_ = s.SetReadDeadline(t)
_ = s.SetWriteDeadline(t)
return nil
}
func (s *conn) SetReadDeadline(t time.Time) error {
s.readTimeOut = t
s.receiveWindow.SetTimeOut(t)
return nil
}
func (s *conn) SetWriteDeadline(t time.Time) error {
s.writeTimeOut = t
s.sendWindow.SetTimeOut(t)
return nil
}
type window struct {
remainingWait uint64 // 64bit alignment
off uint32
maxSize uint32
closeOp bool
closeOpCh chan struct{}
mux *Mux
}
func (Self *window) unpack(ptrs uint64) (remaining, wait uint32) {
const mask = 1<<dequeueBits - 1
remaining = uint32((ptrs >> dequeueBits) & mask)
wait = uint32(ptrs & mask)
return
}
func (Self *window) pack(remaining, wait uint32) uint64 {
const mask = 1<<dequeueBits - 1
return (uint64(remaining) << dequeueBits) |
uint64(wait&mask)
}
func (Self *window) New() {
Self.closeOpCh = make(chan struct{}, 2)
}
func (Self *window) CloseWindow() {
if !Self.closeOp {
Self.closeOp = true
Self.closeOpCh <- struct{}{}
Self.closeOpCh <- struct{}{}
}
}
type ReceiveWindow struct {
window
bufQueue ReceiveWindowQueue
element *common.ListElement
count int8
once sync.Once
}
func (Self *ReceiveWindow) New(mux *Mux) {
// initial a window for receive
Self.bufQueue.New()
Self.element = common.ListElementPool.Get()
Self.maxSize = common.MAXIMUM_SEGMENT_SIZE * 10
Self.mux = mux
Self.window.New()
}
func (Self *ReceiveWindow) remainingSize(delta uint16) (n uint32) {
// receive window remaining
l := int64(atomic.LoadUint32(&Self.maxSize)) - int64(Self.bufQueue.Len())
l -= int64(delta)
if l > 0 {
n = uint32(l)
}
return
}
func (Self *ReceiveWindow) calcSize() {
// calculating maximum receive window size
if Self.count == 0 {
//logs.Warn("ping, bw", Self.mux.latency, Self.bw.Get())
conns := Self.mux.connMap.Size()
n := uint32(math.Float64frombits(atomic.LoadUint64(&Self.mux.latency)) *
Self.mux.bw.Get() / float64(conns))
if n < common.MAXIMUM_SEGMENT_SIZE*10 {
n = common.MAXIMUM_SEGMENT_SIZE * 10
}
bufLen := Self.bufQueue.Len()
if n < bufLen {
n = bufLen
}
if n < Self.maxSize/2 {
n = Self.maxSize / 2
}
// set the minimal size
if n > 2*Self.maxSize {
n = 2 * Self.maxSize
}
if n > (common.MAXIMUM_WINDOW_SIZE / uint32(conns)) {
n = common.MAXIMUM_WINDOW_SIZE / uint32(conns)
}
// set the maximum size
//logs.Warn("n", n)
atomic.StoreUint32(&Self.maxSize, n)
Self.count = -10
}
Self.count += 1
return
}
func (Self *ReceiveWindow) Write(buf []byte, l uint16, part bool, id int32) (err error) {
if Self.closeOp {
return errors.New("conn.receiveWindow: write on closed window")
}
element, err := NewListElement(buf, l, part)
//logs.Warn("push the buf", len(buf), l, (&element).l)
if err != nil {
return
}
Self.calcSize() // calculate the max window size
var wait uint32
start:
ptrs := atomic.LoadUint64(&Self.remainingWait)
_, wait = Self.unpack(ptrs)
newRemaining := Self.remainingSize(l)
// calculate the remaining window size now, plus the element we will push
if newRemaining == 0 {
//logs.Warn("window full true", remaining)
wait = 1
}
if !atomic.CompareAndSwapUint64(&Self.remainingWait, ptrs, Self.pack(0, wait)) {
goto start
// another goroutine change the status, make sure shall we need wait
}
Self.bufQueue.Push(element)
// status check finish, now we can push the element into the queue
if wait == 0 {
Self.mux.sendInfo(common.MUX_MSG_SEND_OK, id, Self.maxSize, newRemaining)
// send the remaining window size, not including zero size
}
return nil
}
func (Self *ReceiveWindow) Read(p []byte, id int32) (n int, err error) {
if Self.closeOp {
return 0, io.EOF // receive close signal, returns eof
}
pOff := 0
l := 0
//logs.Warn("receive window read off, element.l", Self.off, Self.element.l)
copyData:
if Self.off == uint32(Self.element.L) {
// on the first Read method invoked, Self.off and Self.element.l
// both zero value
common.ListElementPool.Put(Self.element)
if Self.closeOp {
return 0, io.EOF
}
Self.element, err = Self.bufQueue.Pop()
// if the queue is empty, Pop method will wait until one element push
// into the queue successful, or timeout.
// timer start on timeout parameter is set up ,
// reset to 60s if timeout and data still available
Self.off = 0
if err != nil {
return // queue receive stop or time out, break the loop and return
}
//logs.Warn("pop element", Self.element.l, Self.element.part)
}
l = copy(p[pOff:], Self.element.Buf[Self.off:Self.element.L])
pOff += l
Self.off += uint32(l)
//logs.Warn("window read length buf len", Self.readLength, Self.bufQueue.Len())
n += l
l = 0
if Self.off == uint32(Self.element.L) {
//logs.Warn("put the element end ", string(Self.element.buf[:15]))
common.WindowBuff.Put(Self.element.Buf)
Self.sendStatus(id, Self.element.L)
// check the window full status
}
if pOff < len(p) && Self.element.Part {
// element is a part of the segments, trying to fill up buf p
goto copyData
}
return // buf p is full or all of segments in buf, return
}
func (Self *ReceiveWindow) sendStatus(id int32, l uint16) {
var remaining, wait uint32
for {
ptrs := atomic.LoadUint64(&Self.remainingWait)
remaining, wait = Self.unpack(ptrs)
remaining += uint32(l)
if atomic.CompareAndSwapUint64(&Self.remainingWait, ptrs, Self.pack(remaining, 0)) {
break
}
runtime.Gosched()
// another goroutine change remaining or wait status, make sure
// we need acknowledge other side
}
// now we get the current window status success
if wait == 1 {
//logs.Warn("send the wait status", remaining)
Self.mux.sendInfo(common.MUX_MSG_SEND_OK, id, atomic.LoadUint32(&Self.maxSize), remaining)
}
return
}
func (Self *ReceiveWindow) SetTimeOut(t time.Time) {
// waiting for FIFO queue Pop method
Self.bufQueue.SetTimeOut(t)
}
func (Self *ReceiveWindow) Stop() {
// queue has no more data to push, so unblock pop method
Self.once.Do(Self.bufQueue.Stop)
}
func (Self *ReceiveWindow) CloseWindow() {
Self.window.CloseWindow()
Self.Stop()
Self.release()
}
func (Self *ReceiveWindow) release() {
//if Self.element != nil {
// if Self.element.Buf != nil {
// common.WindowBuff.Put(Self.element.Buf)
// }
// common.ListElementPool.Put(Self.element)
//}
for {
Self.element = Self.bufQueue.TryPop()
if Self.element == nil {
return
}
if Self.element.Buf != nil {
common.WindowBuff.Put(Self.element.Buf)
}
common.ListElementPool.Put(Self.element)
} // release resource
}
type SendWindow struct {
window
buf []byte
setSizeCh chan struct{}
timeout time.Time
}
func (Self *SendWindow) New(mux *Mux) {
Self.setSizeCh = make(chan struct{})
Self.maxSize = common.MAXIMUM_SEGMENT_SIZE * 10
atomic.AddUint64(&Self.remainingWait, uint64(common.MAXIMUM_SEGMENT_SIZE*10)<<dequeueBits)
Self.mux = mux
Self.window.New()
}
func (Self *SendWindow) SetSendBuf(buf []byte) {
// send window buff from conn write method, set it to send window
Self.buf = buf
Self.off = 0
}
func (Self *SendWindow) SetSize(windowSize, newRemaining uint32) (closed bool) {
// set the window size from receive window
defer func() {
if recover() != nil {
closed = true
}
}()
if Self.closeOp {
close(Self.setSizeCh)
return true
}
//logs.Warn("set send window size to ", windowSize, newRemaining)
var remaining, wait, newWait uint32
for {
ptrs := atomic.LoadUint64(&Self.remainingWait)
remaining, wait = Self.unpack(ptrs)
if remaining == newRemaining {
//logs.Warn("waiting for another window size")
return false // waiting for receive another usable window size
}
if newRemaining == 0 && wait == 1 {
newWait = 1 // keep the wait status,
// also if newRemaining is not zero, change wait to 0
}
if atomic.CompareAndSwapUint64(&Self.remainingWait, ptrs, Self.pack(newRemaining, newWait)) {
break
}
// anther goroutine change wait status or window size
}
if wait == 1 {
// send window into the wait status, need notice the channel
//logs.Warn("send window remaining size is 0")
Self.allow()
}
// send window not into the wait status, so just do slide
return false
}
func (Self *SendWindow) allow() {
select {
case Self.setSizeCh <- struct{}{}:
//logs.Warn("send window remaining size is 0 finish")
return
case <-Self.closeOpCh:
close(Self.setSizeCh)
return
}
}
func (Self *SendWindow) sent(sentSize uint32) {
atomic.AddUint64(&Self.remainingWait, ^(uint64(sentSize)<<dequeueBits - 1))
}
func (Self *SendWindow) WriteTo() (p []byte, sendSize uint32, part bool, err error) {
// returns buf segments, return only one segments, need a loop outside
// until err = io.EOF
if Self.closeOp {
return nil, 0, false, errors.New("conn.writeWindow: window closed")
}
if Self.off == uint32(len(Self.buf)) {
return nil, 0, false, io.EOF
// send window buff is drain, return eof and get another one
}
var remaining uint32
start:
ptrs := atomic.LoadUint64(&Self.remainingWait)
remaining, _ = Self.unpack(ptrs)
if remaining == 0 {
if !atomic.CompareAndSwapUint64(&Self.remainingWait, ptrs, Self.pack(0, 1)) {
goto start // another goroutine change the window, try again
}
// into the wait status
//logs.Warn("send window into wait status")
err = Self.waitReceiveWindow()
if err != nil {
return nil, 0, false, err
}
//logs.Warn("rem into wait finish")
goto start
}
// there are still remaining window
//logs.Warn("rem", remaining)
if len(Self.buf[Self.off:]) > common.MAXIMUM_SEGMENT_SIZE {
sendSize = common.MAXIMUM_SEGMENT_SIZE
//logs.Warn("cut buf by mss")
} else {
sendSize = uint32(len(Self.buf[Self.off:]))
}
if remaining < sendSize {
// usable window size is small than
// window MAXIMUM_SEGMENT_SIZE or send buf left
sendSize = remaining
//logs.Warn("cut buf by remainingsize", sendSize, len(Self.buf[Self.off:]))
}
//logs.Warn("send size", sendSize)
if sendSize < uint32(len(Self.buf[Self.off:])) {
part = true
}
p = Self.buf[Self.off : sendSize+Self.off]
Self.off += sendSize
Self.sent(sendSize)
return
}
func (Self *SendWindow) waitReceiveWindow() (err error) {
t := Self.timeout.Sub(time.Now())
if t < 0 {
t = time.Minute * 5
}
timer := time.NewTimer(t)
defer timer.Stop()
// waiting for receive usable window size, or timeout
select {
case _, ok := <-Self.setSizeCh:
if !ok {
return errors.New("conn.writeWindow: window closed")
}
return nil
case <-timer.C:
return errors.New("conn.writeWindow: write to time out")
case <-Self.closeOpCh:
return errors.New("conn.writeWindow: window closed")
}
}
func (Self *SendWindow) WriteFull(buf []byte, id int32) (n int, err error) {
Self.SetSendBuf(buf) // set the buf to send window
//logs.Warn("set the buf to send window")
var bufSeg []byte
var part bool
var l uint32
for {
bufSeg, l, part, err = Self.WriteTo()
//logs.Warn("buf seg", len(bufSeg), part, err)
// get the buf segments from send window
if bufSeg == nil && part == false && err == io.EOF {
// send window is drain, break the loop
err = nil
break
}
if err != nil {
break
}
n += int(l)
l = 0
if part {
Self.mux.sendInfo(common.MUX_NEW_MSG_PART, id, bufSeg)
} else {
Self.mux.sendInfo(common.MUX_NEW_MSG, id, bufSeg)
//logs.Warn("buf seg sent", len(bufSeg), part, err)
}
// send to other side, not send nil data to other side
}
//logs.Warn("buf seg write success")
return
}
func (Self *SendWindow) SetTimeOut(t time.Time) {
// waiting for receive a receive window size
Self.timeout = t
}
//type bandwidth struct {
// readStart time.Time
// lastReadStart time.Time
// readEnd time.Time
// lastReadEnd time.Time
// bufLength int
// lastBufLength int
// count int8
// readBW float64
// writeBW float64
// readBandwidth float64
//}
//
//func (Self *bandwidth) StartRead() {
// Self.lastReadStart, Self.readStart = Self.readStart, time.Now()
// if !Self.lastReadStart.IsZero() {
// if Self.count == -5 {
// Self.calcBandWidth()
// }
// }
//}
//
//func (Self *bandwidth) EndRead() {
// Self.lastReadEnd, Self.readEnd = Self.readEnd, time.Now()
// if Self.count == -5 {
// Self.calcWriteBandwidth()
// }
// if Self.count == 0 {
// Self.calcReadBandwidth()
// Self.count = -6
// }
// Self.count += 1
//}
//
//func (Self *bandwidth) SetCopySize(n int) {
// // must be invoke between StartRead and EndRead
// Self.lastBufLength, Self.bufLength = Self.bufLength, n
//}
//// calculating
//// start end start end
//// read read
//// write
//
//func (Self *bandwidth) calcBandWidth() {
// t := Self.readStart.Sub(Self.lastReadStart)
// if Self.lastBufLength >= 32768 {
// Self.readBandwidth = float64(Self.lastBufLength) / t.Seconds()
// }
//}
//
//func (Self *bandwidth) calcReadBandwidth() {
// // Bandwidth between nps and npc
// readTime := Self.readEnd.Sub(Self.readStart)
// Self.readBW = float64(Self.bufLength) / readTime.Seconds()
// //logs.Warn("calc read bw", Self.readBW, Self.bufLength, readTime.Seconds())
//}
//
//func (Self *bandwidth) calcWriteBandwidth() {
// // Bandwidth between nps and user, npc and application
// writeTime := Self.readStart.Sub(Self.lastReadEnd)
// Self.writeBW = float64(Self.lastBufLength) / writeTime.Seconds()
// //logs.Warn("calc write bw", Self.writeBW, Self.bufLength, writeTime.Seconds())
//}
//
//func (Self *bandwidth) Get() (bw float64) {
// // The zero value, 0 for numeric types
// if Self.writeBW == 0 && Self.readBW == 0 {
// //logs.Warn("bw both 0")
// return 100
// }
// if Self.writeBW == 0 && Self.readBW != 0 {
// return Self.readBW
// }
// if Self.readBW == 0 && Self.writeBW != 0 {
// return Self.writeBW
// }
// return Self.readBandwidth
//}

65
lib/mux/map.go
View File

@ -2,28 +2,35 @@ package mux
import (
"sync"
"time"
)
type connMap struct {
connMap map[int32]*conn
closeCh chan struct{}
//closeCh chan struct{}
sync.RWMutex
}
func NewConnMap() *connMap {
connMap := &connMap{
connMap: make(map[int32]*conn),
closeCh: make(chan struct{}),
//closeCh: make(chan struct{}),
}
go connMap.clean()
//go connMap.clean()
return connMap
}
func (s *connMap) Size() (n int) {
s.Lock()
n = len(s.connMap)
s.Unlock()
return
}
func (s *connMap) Get(id int32) (*conn, bool) {
s.Lock()
defer s.Unlock()
if v, ok := s.connMap[id]; ok && v != nil {
v, ok := s.connMap[id]
s.Unlock()
if ok && v != nil {
return v, true
}
return nil, false
@ -31,40 +38,38 @@ func (s *connMap) Get(id int32) (*conn, bool) {
func (s *connMap) Set(id int32, v *conn) {
s.Lock()
defer s.Unlock()
s.connMap[id] = v
s.Unlock()
}
func (s *connMap) Close() {
s.Lock()
defer s.Unlock()
//s.closeCh <- struct{}{} // stop the clean goroutine first
for _, v := range s.connMap {
v.isClose = true
v.Close() // close all the connections in the mux
}
s.closeCh <- struct{}{}
}
func (s *connMap) Delete(id int32) {
s.Lock()
defer s.Unlock()
delete(s.connMap, id)
s.Unlock()
}
func (s *connMap) clean() {
ticker := time.NewTimer(time.Minute * 1)
for {
select {
case <-ticker.C:
s.Lock()
for _, v := range s.connMap {
if v.isClose {
delete(s.connMap, v.connId)
}
}
s.Unlock()
case <-s.closeCh:
ticker.Stop()
return
}
}
}
//func (s *connMap) clean() {
// ticker := time.NewTimer(time.Minute * 1)
// for {
// select {
// case <-ticker.C:
// s.Lock()
// for _, v := range s.connMap {
// if v.isClose {
// delete(s.connMap, v.connId)
// }
// }
// s.Unlock()
// case <-s.closeCh:
// ticker.Stop()
// return
// }
// }
//}

527
lib/mux/mux.go
View File

@ -1,56 +1,59 @@
package mux
import (
"bytes"
"encoding/binary"
"errors"
"github.com/cnlh/nps/lib/pool"
"io"
"math"
"net"
"sync"
"sync/atomic"
"time"
)
const (
MUX_PING_FLAG int32 = iota
MUX_NEW_CONN_OK
MUX_NEW_CONN_Fail
MUX_NEW_MSG
MUX_MSG_SEND_OK
MUX_NEW_CONN
MUX_PING
MUX_CONN_CLOSE
MUX_PING_RETURN
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
)
type Mux struct {
latency uint64 // we store latency in bits, but it's float64
net.Listener
conn net.Conn
connMap *connMap
newConnCh chan *conn
id int32
closeChan chan struct{}
IsClose bool
pingOk int
connType string
sync.Mutex
conn net.Conn
connMap *connMap
newConnCh chan *conn
id int32
closeChan chan struct{}
IsClose bool
pingOk uint32
counter *latencyCounter
bw *bandwidth
pingCh chan []byte
pingCheckTime uint32
connType string
writeQueue PriorityQueue
newConnQueue ConnQueue
}
func NewMux(c net.Conn, connType string) *Mux {
//c.(*net.TCPConn).SetReadBuffer(0)
//c.(*net.TCPConn).SetWriteBuffer(0)
m := &Mux{
conn: c,
connMap: NewConnMap(),
id: 0,
closeChan: make(chan struct{}),
closeChan: make(chan struct{}, 1),
newConnCh: make(chan *conn),
bw: new(bandwidth),
IsClose: false,
connType: connType,
pingCh: make(chan []byte),
counter: newLatencyCounter(),
}
m.writeQueue.New()
m.newConnQueue.New()
//read session by flag
go m.readSession()
m.readSession()
//ping
go m.ping()
m.ping()
m.pingReturn()
m.writeSession()
return m
}
@ -58,12 +61,10 @@ func (s *Mux) NewConn() (*conn, error) {
if s.IsClose {
return nil, errors.New("the mux has closed")
}
conn := NewConn(s.getId(), s)
conn := NewConn(s.getId(), s, "nps ")
//it must be set before send
s.connMap.Set(conn.connId, conn)
if err := s.sendInfo(MUX_NEW_CONN, conn.connId, nil); err != nil {
return nil, err
}
s.sendInfo(common.MUX_NEW_CONN, conn.connId, nil)
//set a timer timeout 30 second
timer := time.NewTimer(time.Minute * 2)
defer timer.Stop()
@ -78,7 +79,7 @@ func (s *Mux) NewConn() (*conn, error) {
func (s *Mux) Accept() (net.Conn, error) {
if s.IsClose {
return nil, errors.New("accpet error,the conn has closed")
return nil, errors.New("accpet error,the mux has closed")
}
conn := <-s.newConnCh
if conn == nil {
@ -91,129 +92,417 @@ func (s *Mux) Addr() net.Addr {
return s.conn.LocalAddr()
}
func (s *Mux) sendInfo(flag int32, id int32, content []byte) error {
raw := bytes.NewBuffer([]byte{})
binary.Write(raw, binary.LittleEndian, flag)
binary.Write(raw, binary.LittleEndian, id)
if content != nil && len(content) > 0 {
binary.Write(raw, binary.LittleEndian, int32(len(content)))
binary.Write(raw, binary.LittleEndian, content)
func (s *Mux) sendInfo(flag uint8, id int32, data ...interface{}) {
if s.IsClose {
return
}
if _, err := s.conn.Write(raw.Bytes()); err != nil {
var err error
pack := common.MuxPack.Get()
err = pack.NewPac(flag, id, data...)
if err != nil {
common.MuxPack.Put(pack)
logs.Error("mux: new pack err", err)
s.Close()
return err
return
}
return nil
s.writeQueue.Push(pack)
return
}
func (s *Mux) writeSession() {
go s.packBuf()
//go s.writeBuf()
}
func (s *Mux) packBuf() {
//buffer := common.BuffPool.Get()
for {
if s.IsClose {
break
}
//buffer.Reset()
pack := s.writeQueue.Pop()
if s.IsClose {
break
}
//buffer := common.BuffPool.Get()
err := pack.Pack(s.conn)
common.MuxPack.Put(pack)
if err != nil {
logs.Error("mux: pack err", err)
//common.BuffPool.Put(buffer)
s.Close()
break
}
//logs.Warn(buffer.String())
//s.bufQueue.Push(buffer)
//l := buffer.Len()
//n, err := buffer.WriteTo(s.conn)
//common.BuffPool.Put(buffer)
//if err != nil || int(n) != l {
// logs.Error("mux: close from write session fail ", err, n, l)
// s.Close()
// break
//}
}
}
//func (s *Mux) writeBuf() {
// for {
// if s.IsClose {
// break
// }
// buffer, err := s.bufQueue.Pop()
// if err != nil {
// break
// }
// l := buffer.Len()
// n, err := buffer.WriteTo(s.conn)
// common.BuffPool.Put(buffer)
// if err != nil || int(n) != l {
// logs.Warn("close from write session fail ", err, n, l)
// s.Close()
// break
// }
// }
//}
func (s *Mux) ping() {
go func() {
ticker := time.NewTicker(time.Second * 1)
now, _ := time.Now().UTC().MarshalText()
s.sendInfo(common.MUX_PING_FLAG, common.MUX_PING, now)
// send the ping flag and get the latency first
ticker := time.NewTicker(time.Second * 5)
for {
if s.IsClose {
ticker.Stop()
break
}
select {
case <-ticker.C:
}
//Avoid going beyond the scope
if (math.MaxInt32 - s.id) < 10000 {
s.id = 0
if atomic.LoadUint32(&s.pingCheckTime) >= 60 {
logs.Error("mux: ping time out")
s.Close()
// more than 5 minutes not receive the ping return package,
// mux conn is damaged, maybe a packet drop, close it
break
}
if err := s.sendInfo(MUX_PING_FLAG, MUX_PING, nil); err != nil || (s.pingOk > 10 && s.connType == "kcp") {
now, _ := time.Now().UTC().MarshalText()
s.sendInfo(common.MUX_PING_FLAG, common.MUX_PING, now)
atomic.AddUint32(&s.pingCheckTime, 1)
if atomic.LoadUint32(&s.pingOk) > 10 && s.connType == "kcp" {
logs.Error("mux: kcp ping err")
s.Close()
break
}
s.pingOk++
atomic.AddUint32(&s.pingOk, 1)
}
}()
}
func (s *Mux) pingReturn() {
go func() {
var now time.Time
var data []byte
for {
if s.IsClose {
break
}
select {
case data = <-s.pingCh:
atomic.StoreUint32(&s.pingCheckTime, 0)
case <-s.closeChan:
break
}
_ = now.UnmarshalText(data)
latency := time.Now().UTC().Sub(now).Seconds() / 2
if latency > 0 {
atomic.StoreUint64(&s.latency, math.Float64bits(s.counter.Latency(latency)))
// convert float64 to bits, store it atomic
}
//logs.Warn("latency", math.Float64frombits(atomic.LoadUint64(&s.latency)))
if cap(data) > 0 {
common.WindowBuff.Put(data)
}
}
}()
select {
case <-s.closeChan:
}
}
func (s *Mux) readSession() {
var buf []byte
go func() {
var connection *conn
for {
var flag, i int32
var n int
var err error
if binary.Read(s.conn, binary.LittleEndian, &flag) == nil {
if binary.Read(s.conn, binary.LittleEndian, &i) != nil {
break
}
s.pingOk = 0
switch flag {
case MUX_NEW_CONN: //new conn
conn := NewConn(i, s)
s.connMap.Set(i, conn) //it has been set before send ok
s.newConnCh <- conn
s.sendInfo(MUX_NEW_CONN_OK, i, nil)
continue
case MUX_PING_FLAG: //ping
s.sendInfo(MUX_PING_RETURN, MUX_PING, nil)
continue
case MUX_PING_RETURN:
continue
case MUX_NEW_MSG:
buf = pool.GetBufPoolCopy()
if n, err = ReadLenBytes(buf, s.conn); err != nil {
break
}
}
if conn, ok := s.connMap.Get(i); ok && !conn.isClose {
switch flag {
case MUX_NEW_MSG: //new msg from remote conn
//insert wait queue
conn.waitQueue.Push(NewBufNode(buf, n))
//judge len if >xxx ,send stop
if conn.readWait {
conn.readWait = false
conn.readCh <- struct{}{}
}
case MUX_MSG_SEND_OK: //the remote has read
select {
case conn.getStatusCh <- struct{}{}:
default:
}
conn.hasWrite --
case MUX_NEW_CONN_OK: //conn ok
conn.connStatusOkCh <- struct{}{}
case MUX_NEW_CONN_Fail:
conn.connStatusFailCh <- struct{}{}
case MUX_CONN_CLOSE: //close the connection
go conn.Close()
s.connMap.Delete(i)
}
} else if flag == MUX_NEW_MSG {
pool.PutBufPoolCopy(buf)
}
} else {
if s.IsClose {
break
}
connection = s.newConnQueue.Pop()
if s.IsClose {
break // make sure that is closed
}
s.connMap.Set(connection.connId, connection) //it has been set before send ok
s.newConnCh <- connection
s.sendInfo(common.MUX_NEW_CONN_OK, connection.connId, nil)
}
}()
go func() {
pack := common.MuxPack.Get()
var l uint16
var err error
for {
if s.IsClose {
break
}
pack = common.MuxPack.Get()
s.bw.StartRead()
if l, err = pack.UnPack(s.conn); err != nil {
logs.Error("mux: read session unpack from connection err", err)
s.Close()
break
}
s.bw.SetCopySize(l)
atomic.StoreUint32(&s.pingOk, 0)
switch pack.Flag {
case common.MUX_NEW_CONN: //new connection
connection := NewConn(pack.Id, s)
s.newConnQueue.Push(connection)
continue
case common.MUX_PING_FLAG: //ping
s.sendInfo(common.MUX_PING_RETURN, common.MUX_PING, pack.Content)
common.WindowBuff.Put(pack.Content)
continue
case common.MUX_PING_RETURN:
//go func(content []byte) {
s.pingCh <- pack.Content
//}(pack.Content)
continue
}
if connection, ok := s.connMap.Get(pack.Id); ok && !connection.isClose {
switch pack.Flag {
case common.MUX_NEW_MSG, common.MUX_NEW_MSG_PART: //new msg from remote connection
err = s.newMsg(connection, pack)
if err != nil {
logs.Error("mux: read session connection new msg err", err)
connection.Close()
}
continue
case common.MUX_NEW_CONN_OK: //connection ok
connection.connStatusOkCh <- struct{}{}
continue
case common.MUX_NEW_CONN_Fail:
connection.connStatusFailCh <- struct{}{}
continue
case common.MUX_MSG_SEND_OK:
if connection.isClose {
continue
}
connection.sendWindow.SetSize(pack.Window, pack.ReadLength)
continue
case common.MUX_CONN_CLOSE: //close the connection
connection.closeFlag = true
//s.connMap.Delete(pack.Id)
//go func(connection *conn) {
connection.receiveWindow.Stop() // close signal to receive window
//}(connection)
continue
}
} else if pack.Flag == common.MUX_CONN_CLOSE {
continue
}
common.MuxPack.Put(pack)
}
common.MuxPack.Put(pack)
s.Close()
}()
select {
case <-s.closeChan:
}
}
func (s *Mux) Close() error {
func (s *Mux) newMsg(connection *conn, pack *common.MuxPackager) (err error) {
if connection.isClose {
err = io.ErrClosedPipe
return
}
//logs.Warn("read session receive new msg", pack.Length)
//go func(connection *conn, pack *common.MuxPackager) { // do not block read session
//insert into queue
if pack.Flag == common.MUX_NEW_MSG_PART {
err = connection.receiveWindow.Write(pack.Content, pack.Length, true, pack.Id)
}
if pack.Flag == common.MUX_NEW_MSG {
err = connection.receiveWindow.Write(pack.Content, pack.Length, false, pack.Id)
}
//logs.Warn("read session write success", pack.Length)
return
}
func (s *Mux) Close() (err error) {
logs.Warn("close mux")
if s.IsClose {
return errors.New("the mux has closed")
}
s.IsClose = true
s.connMap.Close()
select {
case s.closeChan <- struct{}{}:
}
select {
case s.closeChan <- struct{}{}:
}
s.connMap = nil
//s.bufQueue.Stop()
s.closeChan <- struct{}{}
close(s.newConnCh)
return s.conn.Close()
err = s.conn.Close()
s.release()
return
}
func (s *Mux) release() {
for {
pack := s.writeQueue.TryPop()
if pack == nil {
break
}
if pack.BasePackager.Content != nil {
common.WindowBuff.Put(pack.BasePackager.Content)
}
common.MuxPack.Put(pack)
}
for {
connection := s.newConnQueue.TryPop()
if connection == nil {
break
}
connection = nil
}
s.writeQueue.Stop()
s.newConnQueue.Stop()
}
//get new connId as unique flag
func (s *Mux) getId() int32 {
return atomic.AddInt32(&s.id, 1)
func (s *Mux) getId() (id int32) {
//Avoid going beyond the scope
if (math.MaxInt32 - s.id) < 10000 {
atomic.StoreInt32(&s.id, 0)
}
id = atomic.AddInt32(&s.id, 1)
if _, ok := s.connMap.Get(id); ok {
return s.getId()
}
return
}
type bandwidth struct {
readBandwidth uint64 // store in bits, but it's float64
readStart time.Time
lastReadStart time.Time
bufLength uint32
}
func (Self *bandwidth) StartRead() {
if Self.readStart.IsZero() {
Self.readStart = time.Now()
}
if Self.bufLength >= common.MAXIMUM_SEGMENT_SIZE*300 {
Self.lastReadStart, Self.readStart = Self.readStart, time.Now()
Self.calcBandWidth()
}
}
func (Self *bandwidth) SetCopySize(n uint16) {
Self.bufLength += uint32(n)
}
func (Self *bandwidth) calcBandWidth() {
t := Self.readStart.Sub(Self.lastReadStart)
atomic.StoreUint64(&Self.readBandwidth, math.Float64bits(float64(Self.bufLength)/t.Seconds()))
Self.bufLength = 0
}
func (Self *bandwidth) Get() (bw float64) {
// The zero value, 0 for numeric types
bw = math.Float64frombits(atomic.LoadUint64(&Self.readBandwidth))
if bw <= 0 {
bw = 100
}
//logs.Warn(bw)
return
}
const counterBits = 4
const counterMask = 1<<counterBits - 1
func newLatencyCounter() *latencyCounter {
return &latencyCounter{
buf: make([]float64, 1<<counterBits, 1<<counterBits),
headMin: 0,
}
}
type latencyCounter struct {
buf []float64 //buf is a fixed length ring buffer,
// if buffer is full, new value will replace the oldest one.
headMin uint8 //head indicate the head in ring buffer,
// in meaning, slot in list will be replaced;
// min indicate this slot value is minimal in list.
}
func (Self *latencyCounter) unpack(idxs uint8) (head, min uint8) {
head = uint8((idxs >> counterBits) & counterMask)
// we set head is 4 bits
min = uint8(idxs & counterMask)
return
}
func (Self *latencyCounter) pack(head, min uint8) uint8 {
return uint8(head<<counterBits) |
uint8(min&counterMask)
}
func (Self *latencyCounter) add(value float64) {
head, min := Self.unpack(Self.headMin)
Self.buf[head] = value
if head == min {
min = Self.minimal()
//if head equals min, means the min slot already be replaced,
// so we need to find another minimal value in the list,
// and change the min indicator
}
if Self.buf[min] > value {
min = head
}
head++
Self.headMin = Self.pack(head, min)
}
func (Self *latencyCounter) minimal() (min uint8) {
var val float64
var i uint8
for i = 0; i < counterMask; i++ {
if Self.buf[i] > 0 {
if val > Self.buf[i] {
val = Self.buf[i]
min = i
}
}
}
return
}
func (Self *latencyCounter) Latency(value float64) (latency float64) {
Self.add(value)
_, min := Self.unpack(Self.headMin)
latency = Self.buf[min] * Self.countSuccess()
return
}
const lossRatio = 1.6
func (Self *latencyCounter) countSuccess() (successRate float64) {
var success, loss, i uint8
_, min := Self.unpack(Self.headMin)
for i = 0; i < counterMask; i++ {
if Self.buf[i] > lossRatio*Self.buf[min] && Self.buf[i] > 0 {
loss++
}
if Self.buf[i] <= lossRatio*Self.buf[min] && Self.buf[i] > 0 {
success++
}
}
// counting all the data in the ring buf, except zero
successRate = float64(success) / float64(loss+success)
return
}

396
lib/mux/mux_test.go
View File

@ -1,15 +1,22 @@
package mux
import (
"bufio"
"fmt"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/pool"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/goroutine"
"io"
"log"
"net"
"net/http"
"net/http/httputil"
_ "net/http/pprof"
"strconv"
"testing"
"time"
"unsafe"
"github.com/astaxie/beego/logs"
)
var conn1 net.Conn
@ -23,24 +30,54 @@ func TestNewMux(t *testing.T) {
logs.SetLogFuncCallDepth(3)
server()
client()
//poolConnCopy, _ := ants.NewPoolWithFunc(200000, common.copyConn, ants.WithNonblocking(false))
time.Sleep(time.Second * 3)
go func() {
m2 := NewMux(conn2, "tcp")
for {
//logs.Warn("npc starting accept")
c, err := m2.Accept()
if err != nil {
log.Fatalln(err)
logs.Warn(err)
continue
}
go func(c net.Conn) {
c2, err := net.Dial("tcp", "127.0.0.1:8082")
if err != nil {
log.Fatalln(err)
}
go common.CopyBuffer(c2, c)
common.CopyBuffer(c, c2)
//logs.Warn("npc accept success ")
c2, err := net.Dial("tcp", "127.0.0.1:80")
if err != nil {
logs.Warn(err)
c.Close()
c2.Close()
}(c)
continue
}
//c2.(*net.TCPConn).SetReadBuffer(0)
//c2.(*net.TCPConn).SetReadBuffer(0)
_ = goroutine.CopyConnsPool.Invoke(goroutine.NewConns(c, c2, nil))
//go func(c2 net.Conn, c *conn) {
// wg := new(sync.WaitGroup)
// wg.Add(2)
// _ = poolConnCopy.Invoke(common.newConnGroup(c2, c, wg))
// //go func() {
// // _, err = common.CopyBuffer(c2, c)
// // if err != nil {
// // c2.Close()
// // c.Close()
// // //logs.Warn("close npc by copy from nps", err, c.connId)
// // }
// // wg.Done()
// //}()
// //wg.Add(1)
// _ = poolConnCopy.Invoke(common.newConnGroup(c, c2, wg))
// //go func() {
// // _, err = common.CopyBuffer(c, c2)
// // if err != nil {
// // c2.Close()
// // c.Close()
// // //logs.Warn("close npc by copy from server", err, c.connId)
// // }
// // wg.Done()
// //}()
// //logs.Warn("npc wait")
// wg.Wait()
//}(c2, c.(*conn))
}
}()
@ -48,26 +85,58 @@ func TestNewMux(t *testing.T) {
m1 := NewMux(conn1, "tcp")
l, err := net.Listen("tcp", "127.0.0.1:7777")
if err != nil {
log.Fatalln(err)
logs.Warn(err)
}
for {
conn, err := l.Accept()
//logs.Warn("nps starting accept")
conns, err := l.Accept()
if err != nil {
log.Fatalln(err)
logs.Warn(err)
continue
}
go func(conn net.Conn) {
tmpCpnn, err := m1.NewConn()
if err != nil {
log.Fatalln(err)
}
go common.CopyBuffer(tmpCpnn, conn)
common.CopyBuffer(conn, tmpCpnn)
conn.Close()
tmpCpnn.Close()
}(conn)
//conns.(*net.TCPConn).SetReadBuffer(0)
//conns.(*net.TCPConn).SetReadBuffer(0)
//logs.Warn("nps accept success starting new conn")
tmpCpnn, err := m1.NewConn()
if err != nil {
logs.Warn("nps new conn err ", err)
continue
}
//logs.Warn("nps new conn success ", tmpCpnn.connId)
_ = goroutine.CopyConnsPool.Invoke(goroutine.NewConns(tmpCpnn, conns, nil))
//go func(tmpCpnn *conn, conns net.Conn) {
// wg := new(sync.WaitGroup)
// wg.Add(2)
// _ = poolConnCopy.Invoke(common.newConnGroup(tmpCpnn, conns, wg))
// //go func() {
// // _, err := common.CopyBuffer(tmpCpnn, conns)
// // if err != nil {
// // conns.Close()
// // tmpCpnn.Close()
// // //logs.Warn("close nps by copy from user", tmpCpnn.connId, err)
// // }
// //}()
// //wg.Add(1)
// _ = poolConnCopy.Invoke(common.newConnGroup(conns, tmpCpnn, wg))
// //time.Sleep(time.Second)
// //_, err = common.CopyBuffer(conns, tmpCpnn)
// //if err != nil {
// // conns.Close()
// // tmpCpnn.Close()
// // //logs.Warn("close nps by copy from npc ", tmpCpnn.connId, err)
// //}
// wg.Wait()
//}(tmpCpnn, conns)
}
}()
//go NewLogServer()
time.Sleep(time.Second * 5)
//for i := 0; i < 1; i++ {
// go test_raw(i)
//}
//test_request()
for {
time.Sleep(time.Second * 5)
}
@ -77,12 +146,12 @@ func server() {
var err error
l, err := net.Listen("tcp", "127.0.0.1:9999")
if err != nil {
log.Fatalln(err)
logs.Warn(err)
}
go func() {
conn1, err = l.Accept()
if err != nil {
log.Fatalln(err)
logs.Warn(err)
}
}()
return
@ -92,12 +161,79 @@ func client() {
var err error
conn2, err = net.Dial("tcp", "127.0.0.1:9999")
if err != nil {
log.Fatalln(err)
logs.Warn(err)
}
}
func test_request() {
conn, _ := net.Dial("tcp", "127.0.0.1:7777")
for i := 0; i < 1000; i++ {
conn.Write([]byte(`GET / HTTP/1.1
Host: 127.0.0.1:7777
Connection: keep-alive
`))
r, err := http.ReadResponse(bufio.NewReader(conn), nil)
if err != nil {
logs.Warn("close by read response err", err)
break
}
logs.Warn("read response success", r)
b, err := httputil.DumpResponse(r, true)
if err != nil {
logs.Warn("close by dump response err", err)
break
}
fmt.Println(string(b[:20]), err)
//time.Sleep(time.Second)
}
logs.Warn("finish")
}
func test_raw(k int) {
for i := 0; i < 1000; i++ {
ti := time.Now()
conn, err := net.Dial("tcp", "127.0.0.1:7777")
if err != nil {
logs.Warn("conn dial err", err)
}
tid := time.Now()
conn.Write([]byte(`GET /videojs5/video.js HTTP/1.1
Host: 127.0.0.1:7777
`))
tiw := time.Now()
buf := make([]byte, 3572)
n, err := io.ReadFull(conn, buf)
//n, err := conn.Read(buf)
if err != nil {
logs.Warn("close by read response err", err)
break
}
logs.Warn(n, string(buf[:50]), "\n--------------\n", string(buf[n-50:n]))
//time.Sleep(time.Second)
err = conn.Close()
if err != nil {
logs.Warn("close conn err ", err)
}
now := time.Now()
du := now.Sub(ti).Seconds()
dud := now.Sub(tid).Seconds()
duw := now.Sub(tiw).Seconds()
if du > 1 {
logs.Warn("duration long", du, dud, duw, k, i)
}
if n != 3572 {
logs.Warn("n loss", n, string(buf))
}
}
logs.Warn("finish")
}
func TestNewConn(t *testing.T) {
buf := pool.GetBufPoolCopy()
buf := common.GetBufPoolCopy()
logs.Warn(len(buf), cap(buf))
//b := pool.GetBufPoolCopy()
//b[0] = 1
@ -107,3 +243,205 @@ func TestNewConn(t *testing.T) {
logs.Warn(copy(buf[:3], b), len(buf), cap(buf))
logs.Warn(len(buf), buf[0])
}
func TestDQueue(t *testing.T) {
logs.EnableFuncCallDepth(true)
logs.SetLogFuncCallDepth(3)
d := new(bufDequeue)
d.vals = make([]unsafe.Pointer, 8)
go func() {
time.Sleep(time.Second)
for i := 0; i < 10; i++ {
logs.Warn(i)
logs.Warn(d.popTail())
}
}()
go func() {
time.Sleep(time.Second)
for i := 0; i < 10; i++ {
data := "test"
go logs.Warn(i, unsafe.Pointer(&data), d.pushHead(unsafe.Pointer(&data)))
}
}()
time.Sleep(time.Second * 3)
}
func TestChain(t *testing.T) {
go func() {
log.Println(http.ListenAndServe("0.0.0.0:8889", nil))
}()
logs.EnableFuncCallDepth(true)
logs.SetLogFuncCallDepth(3)
time.Sleep(time.Second * 5)
d := new(bufChain)
d.new(256)
go func() {
time.Sleep(time.Second)
for i := 0; i < 30000; i++ {
unsa, ok := d.popTail()
str := (*string)(unsa)
if ok {
fmt.Println(i, str, *str, ok)
//logs.Warn(i, str, *str, ok)
} else {
fmt.Println("nil", i, ok)
//logs.Warn("nil", i, ok)
}
}
}()
go func() {
time.Sleep(time.Second)
for i := 0; i < 3000; i++ {
go func(i int) {
for n := 0; n < 10; n++ {
data := "test " + strconv.Itoa(i) + strconv.Itoa(n)
fmt.Println(data, unsafe.Pointer(&data))
//logs.Warn(data, unsafe.Pointer(&data))
d.pushHead(unsafe.Pointer(&data))
}
}(i)
}
}()
time.Sleep(time.Second * 100000)
}
func TestFIFO(t *testing.T) {
go func() {
log.Println(http.ListenAndServe("0.0.0.0:8889", nil))
}()
logs.EnableFuncCallDepth(true)
logs.SetLogFuncCallDepth(3)
time.Sleep(time.Second * 5)
d := new(ReceiveWindowQueue)
d.New()
go func() {
time.Sleep(time.Second)
for i := 0; i < 1001; i++ {
data, err := d.Pop()
if err == nil {
//fmt.Println(i, string(data.buf), err)
logs.Warn(i, string(data.Buf), err)
common.ListElementPool.Put(data)
} else {
//fmt.Println("err", err)
logs.Warn("err", err)
}
//logs.Warn(d.Len())
}
logs.Warn("pop finish")
}()
go func() {
time.Sleep(time.Second * 10)
for i := 0; i < 1000; i++ {
by := []byte("test " + strconv.Itoa(i) + " ") //
data, _ := NewListElement(by, uint16(len(by)), true)
//fmt.Println(string((*data).buf), data)
//logs.Warn(string((*data).buf), data)
d.Push(data)
}
}()
time.Sleep(time.Second * 100000)
}
func TestPriority(t *testing.T) {
go func() {
log.Println(http.ListenAndServe("0.0.0.0:8889", nil))
}()
logs.EnableFuncCallDepth(true)
logs.SetLogFuncCallDepth(3)
time.Sleep(time.Second * 5)
d := new(PriorityQueue)
d.New()
go func() {
time.Sleep(time.Second)
for i := 0; i < 360050; i++ {
data := d.Pop()
//fmt.Println(i, string(data.buf), err)
logs.Warn(i, string(data.Content), data)
}
logs.Warn("pop finish")
}()
go func() {
time.Sleep(time.Second * 10)
for i := 0; i < 30000; i++ {
go func(i int) {
for n := 0; n < 10; n++ {
data := new(common.MuxPackager)
by := []byte("test " + strconv.Itoa(i) + strconv.Itoa(n))
_ = data.NewPac(common.MUX_NEW_MSG_PART, int32(i), by)
//fmt.Println(string((*data).buf), data)
logs.Warn(string((*data).Content), data)
d.Push(data)
}
}(i)
go func(i int) {
data := new(common.MuxPackager)
_ = data.NewPac(common.MUX_NEW_CONN, int32(i), nil)
//fmt.Println(string((*data).buf), data)
logs.Warn(data)
d.Push(data)
}(i)
go func(i int) {
data := new(common.MuxPackager)
_ = data.NewPac(common.MUX_NEW_CONN_OK, int32(i), nil)
//fmt.Println(string((*data).buf), data)
logs.Warn(data)
d.Push(data)
}(i)
}
}()
time.Sleep(time.Second * 100000)
}
//func TestReceive(t *testing.T) {
// go func() {
// log.Println(http.ListenAndServe("0.0.0.0:8889", nil))
// }()
// logs.EnableFuncCallDepth(true)
// logs.SetLogFuncCallDepth(3)
// time.Sleep(time.Second * 5)
// mux := new(Mux)
// mux.bw.readBandwidth = float64(1*1024*1024)
// mux.latency = float64(1/1000)
// wind := new(ReceiveWindow)
// wind.New(mux)
// wind.
// go func() {
// time.Sleep(time.Second)
// for i := 0; i < 36000; i++ {
// data := d.Pop()
// //fmt.Println(i, string(data.buf), err)
// logs.Warn(i, string(data.Content), data)
// }
// }()
// go func() {
// time.Sleep(time.Second*10)
// for i := 0; i < 3000; i++ {
// go func(i int) {
// for n := 0; n < 10; n++{
// data := new(common.MuxPackager)
// by := []byte("test " + strconv.Itoa(i) + strconv.Itoa(n))
// _ = data.NewPac(common.MUX_NEW_MSG_PART, int32(i), by)
// //fmt.Println(string((*data).buf), data)
// logs.Warn(string((*data).Content), data)
// d.Push(data)
// }
// }(i)
// go func(i int) {
// data := new(common.MuxPackager)
// _ = data.NewPac(common.MUX_NEW_CONN, int32(i), nil)
// //fmt.Println(string((*data).buf), data)
// logs.Warn(data)
// d.Push(data)
// }(i)
// go func(i int) {
// data := new(common.MuxPackager)
// _ = data.NewPac(common.MUX_NEW_CONN_OK, int32(i), nil)
// //fmt.Println(string((*data).buf), data)
// logs.Warn(data)
// d.Push(data)
// }(i)
// }
// }()
// time.Sleep(time.Second * 100000)
//}

7
lib/mux/pmux.go
View File

@ -5,15 +5,16 @@ package mux
import (
"bufio"
"bytes"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/pkg/errors"
"io"
"net"
"os"
"strconv"
"strings"
"time"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
"github.com/pkg/errors"
)
const (

5
lib/mux/pmux_test.go
View File

@ -1,9 +1,10 @@
package mux
import (
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"testing"
"time"
"github.com/astaxie/beego/logs"
)
func TestPortMux_Close(t *testing.T) {
@ -11,7 +12,7 @@ func TestPortMux_Close(t *testing.T) {
logs.EnableFuncCallDepth(true)
logs.SetLogFuncCallDepth(3)
pMux := NewPortMux(8888,"Ds")
pMux := NewPortMux(8888, "Ds")
go func() {
if pMux.Start() != nil {
logs.Warn("Error")

610
lib/mux/queue.go
View File

@ -2,81 +2,577 @@ package mux
import (
"errors"
"github.com/cnlh/nps/lib/pool"
"github.com/cnlh/nps/lib/common"
"io"
"math"
"runtime"
"sync"
"sync/atomic"
"time"
"unsafe"
)
type Element *bufNode
type bufNode struct {
val []byte //buf value
l int //length
type PriorityQueue struct {
highestChain *bufChain
middleChain *bufChain
lowestChain *bufChain
starving uint8
stop bool
cond *sync.Cond
}
func NewBufNode(buf []byte, l int) *bufNode {
return &bufNode{
val: buf,
l: l,
func (Self *PriorityQueue) New() {
Self.highestChain = new(bufChain)
Self.highestChain.new(4)
Self.middleChain = new(bufChain)
Self.middleChain.new(32)
Self.lowestChain = new(bufChain)
Self.lowestChain.new(256)
locker := new(sync.Mutex)
Self.cond = sync.NewCond(locker)
}
func (Self *PriorityQueue) Push(packager *common.MuxPackager) {
//logs.Warn("push start")
Self.push(packager)
Self.cond.Broadcast()
//logs.Warn("push finish")
return
}
func (Self *PriorityQueue) push(packager *common.MuxPackager) {
switch packager.Flag {
case common.MUX_PING_FLAG, common.MUX_PING_RETURN:
Self.highestChain.pushHead(unsafe.Pointer(packager))
// the ping package need highest priority
// prevent ping calculation error
case common.MUX_NEW_CONN, common.MUX_NEW_CONN_OK, common.MUX_NEW_CONN_Fail:
// the new conn package need some priority too
Self.middleChain.pushHead(unsafe.Pointer(packager))
default:
Self.lowestChain.pushHead(unsafe.Pointer(packager))
}
}
type Queue interface {
Push(e Element) //向队列中添加元素
Pop() Element //移除队列中最前面的元素
Clear() bool //清空队列
Size() int //获取队列的元素个数
IsEmpty() bool //判断队列是否是空
}
const maxStarving uint8 = 8
type sliceEntry struct {
element []Element
sync.Mutex
}
func NewQueue() *sliceEntry {
return &sliceEntry{}
}
//向队列中添加元素
func (entry *sliceEntry) Push(e Element) {
entry.Lock()
defer entry.Unlock()
entry.element = append(entry.element, e)
}
//移除队列中最前面的额元素
func (entry *sliceEntry) Pop() (Element, error) {
if entry.IsEmpty() {
return nil, errors.New("queue is empty!")
func (Self *PriorityQueue) Pop() (packager *common.MuxPackager) {
var iter bool
for {
packager = Self.TryPop()
if packager != nil {
return
}
if Self.stop {
return
}
if iter {
break
// trying to pop twice
}
iter = true
runtime.Gosched()
}
entry.Lock()
defer entry.Unlock()
firstElement := entry.element[0]
entry.element = entry.element[1:]
return firstElement, nil
Self.cond.L.Lock()
defer Self.cond.L.Unlock()
for packager = Self.TryPop(); packager == nil; {
if Self.stop {
return
}
//logs.Warn("queue into wait")
Self.cond.Wait()
// wait for it with no more iter
packager = Self.TryPop()
//logs.Warn("queue wait finish", packager)
}
return
}
func (entry *sliceEntry) Clear() bool {
entry.Lock()
defer entry.Unlock()
if entry.IsEmpty() {
func (Self *PriorityQueue) TryPop() (packager *common.MuxPackager) {
ptr, ok := Self.highestChain.popTail()
if ok {
packager = (*common.MuxPackager)(ptr)
return
}
if Self.starving < maxStarving {
// not pop too much, lowestChain will wait too long
ptr, ok = Self.middleChain.popTail()
if ok {
packager = (*common.MuxPackager)(ptr)
Self.starving++
return
}
}
ptr, ok = Self.lowestChain.popTail()
if ok {
packager = (*common.MuxPackager)(ptr)
if Self.starving > 0 {
Self.starving = uint8(Self.starving / 2)
}
return
}
if Self.starving > 0 {
ptr, ok = Self.middleChain.popTail()
if ok {
packager = (*common.MuxPackager)(ptr)
Self.starving++
return
}
}
return
}
func (Self *PriorityQueue) Stop() {
Self.stop = true
Self.cond.Broadcast()
}
type ConnQueue struct {
chain *bufChain
starving uint8
stop bool
cond *sync.Cond
}
func (Self *ConnQueue) New() {
Self.chain = new(bufChain)
Self.chain.new(32)
locker := new(sync.Mutex)
Self.cond = sync.NewCond(locker)
}
func (Self *ConnQueue) Push(connection *conn) {
Self.chain.pushHead(unsafe.Pointer(connection))
Self.cond.Broadcast()
return
}
func (Self *ConnQueue) Pop() (connection *conn) {
var iter bool
for {
connection = Self.TryPop()
if connection != nil {
return
}
if Self.stop {
return
}
if iter {
break
// trying to pop twice
}
iter = true
runtime.Gosched()
}
Self.cond.L.Lock()
defer Self.cond.L.Unlock()
for connection = Self.TryPop(); connection == nil; {
if Self.stop {
return
}
//logs.Warn("queue into wait")
Self.cond.Wait()
// wait for it with no more iter
connection = Self.TryPop()
//logs.Warn("queue wait finish", packager)
}
return
}
func (Self *ConnQueue) TryPop() (connection *conn) {
ptr, ok := Self.chain.popTail()
if ok {
connection = (*conn)(ptr)
return
}
return
}
func (Self *ConnQueue) Stop() {
Self.stop = true
Self.cond.Broadcast()
}
func NewListElement(buf []byte, l uint16, part bool) (element *common.ListElement, err error) {
if uint16(len(buf)) != l {
err = errors.New("ListElement: buf length not match")
return
}
//if l == 0 {
// logs.Warn("push zero")
//}
element = common.ListElementPool.Get()
element.Buf = buf
element.L = l
element.Part = part
return
}
type ReceiveWindowQueue struct {
lengthWait uint64
chain *bufChain
stopOp chan struct{}
readOp chan struct{}
timeout time.Time
}
func (Self *ReceiveWindowQueue) New() {
Self.readOp = make(chan struct{})
Self.chain = new(bufChain)
Self.chain.new(64)
Self.stopOp = make(chan struct{}, 2)
}
func (Self *ReceiveWindowQueue) Push(element *common.ListElement) {
var length, wait uint32
for {
ptrs := atomic.LoadUint64(&Self.lengthWait)
length, wait = Self.chain.head.unpack(ptrs)
length += uint32(element.L)
if atomic.CompareAndSwapUint64(&Self.lengthWait, ptrs, Self.chain.head.pack(length, 0)) {
break
}
// another goroutine change the length or into wait, make sure
}
//logs.Warn("window push before", Self.Len(), uint32(element.l), len(element.buf))
Self.chain.pushHead(unsafe.Pointer(element))
//logs.Warn("window push", Self.Len())
if wait == 1 {
Self.allowPop()
}
return
}
func (Self *ReceiveWindowQueue) Pop() (element *common.ListElement, err error) {
var length uint32
startPop:
ptrs := atomic.LoadUint64(&Self.lengthWait)
length, _ = Self.chain.head.unpack(ptrs)
if length == 0 {
if !atomic.CompareAndSwapUint64(&Self.lengthWait, ptrs, Self.chain.head.pack(0, 1)) {
goto startPop // another goroutine is pushing
}
err = Self.waitPush()
// there is no more data in queue, wait for it
if err != nil {
return
}
goto startPop // wait finish, trying to get the new status
}
// length is not zero, so try to pop
for {
element = Self.TryPop()
if element != nil {
return
}
runtime.Gosched() // another goroutine is still pushing
}
}
func (Self *ReceiveWindowQueue) TryPop() (element *common.ListElement) {
ptr, ok := Self.chain.popTail()
if ok {
//logs.Warn("window pop before", Self.Len())
element = (*common.ListElement)(ptr)
atomic.AddUint64(&Self.lengthWait, ^(uint64(element.L)<<dequeueBits - 1))
//logs.Warn("window pop", Self.Len(), uint32(element.l))
return
}
return nil
}
func (Self *ReceiveWindowQueue) allowPop() (closed bool) {
//logs.Warn("allow pop", Self.Len())
select {
case Self.readOp <- struct{}{}:
return false
case <-Self.stopOp:
return true
}
for i := 0; i < entry.Size(); i++ {
pool.PutBufPoolCopy(entry.element[i].val)
entry.element[i] = nil
}
func (Self *ReceiveWindowQueue) waitPush() (err error) {
//logs.Warn("wait push")
//defer logs.Warn("wait push finish")
t := Self.timeout.Sub(time.Now())
if t <= 0 {
t = time.Minute * 5
}
entry.element = nil
timer := time.NewTimer(t)
defer timer.Stop()
//logs.Warn("queue into wait")
select {
case <-Self.readOp:
//logs.Warn("queue wait finish")
return nil
case <-Self.stopOp:
err = io.EOF
return
case <-timer.C:
err = errors.New("mux.queue: read time out")
return
}
}
func (Self *ReceiveWindowQueue) Len() (n uint32) {
ptrs := atomic.LoadUint64(&Self.lengthWait)
n, _ = Self.chain.head.unpack(ptrs)
return
}
func (Self *ReceiveWindowQueue) Stop() {
Self.stopOp <- struct{}{}
Self.stopOp <- struct{}{}
}
func (Self *ReceiveWindowQueue) SetTimeOut(t time.Time) {
Self.timeout = t
}
// https://golang.org/src/sync/poolqueue.go
type bufDequeue struct {
// headTail packs together a 32-bit head index and a 32-bit
// tail index. Both are indexes into vals modulo len(vals)-1.
//
// tail = index of oldest data in queue
// head = index of next slot to fill
//
// Slots in the range [tail, head) are owned by consumers.
// A consumer continues to own a slot outside this range until
// it nils the slot, at which point ownership passes to the
// producer.
//
// The head index is stored in the most-significant bits so
// that we can atomically add to it and the overflow is
// harmless.
headTail uint64
// vals is a ring buffer of interface{} values stored in this
// dequeue. The size of this must be a power of 2.
//
// A slot is still in use until *both* the tail
// index has moved beyond it and typ has been set to nil. This
// is set to nil atomically by the consumer and read
// atomically by the producer.
vals []unsafe.Pointer
starving uint32
}
const dequeueBits = 32
// dequeueLimit is the maximum size of a bufDequeue.
//
// This must be at most (1<<dequeueBits)/2 because detecting fullness
// depends on wrapping around the ring buffer without wrapping around
// the index. We divide by 4 so this fits in an int on 32-bit.
const dequeueLimit = (1 << dequeueBits) / 4
func (d *bufDequeue) unpack(ptrs uint64) (head, tail uint32) {
const mask = 1<<dequeueBits - 1
head = uint32((ptrs >> dequeueBits) & mask)
tail = uint32(ptrs & mask)
return
}
func (d *bufDequeue) pack(head, tail uint32) uint64 {
const mask = 1<<dequeueBits - 1
return (uint64(head) << dequeueBits) |
uint64(tail&mask)
}
// pushHead adds val at the head of the queue. It returns false if the
// queue is full.
func (d *bufDequeue) pushHead(val unsafe.Pointer) bool {
var slot *unsafe.Pointer
var starve uint8
if atomic.LoadUint32(&d.starving) > 0 {
runtime.Gosched()
}
for {
ptrs := atomic.LoadUint64(&d.headTail)
head, tail := d.unpack(ptrs)
if (tail+uint32(len(d.vals)))&(1<<dequeueBits-1) == head {
// Queue is full.
return false
}
ptrs2 := d.pack(head+1, tail)
if atomic.CompareAndSwapUint64(&d.headTail, ptrs, ptrs2) {
slot = &d.vals[head&uint32(len(d.vals)-1)]
if starve >= 3 && atomic.LoadUint32(&d.starving) > 0 {
atomic.StoreUint32(&d.starving, 0)
}
break
}
starve++
if starve >= 3 {
atomic.StoreUint32(&d.starving, 1)
}
}
// The head slot is free, so we own it.
*slot = val
return true
}
func (entry *sliceEntry) Size() int {
return len(entry.element)
// popTail removes and returns the element at the tail of the queue.
// It returns false if the queue is empty. It may be called by any
// number of consumers.
func (d *bufDequeue) popTail() (unsafe.Pointer, bool) {
ptrs := atomic.LoadUint64(&d.headTail)
head, tail := d.unpack(ptrs)
if tail == head {
// Queue is empty.
return nil, false
}
slot := &d.vals[tail&uint32(len(d.vals)-1)]
var val unsafe.Pointer
for {
val = atomic.LoadPointer(slot)
if val != nil {
// We now own slot.
break
}
// Another goroutine is still pushing data on the tail.
}
// Tell pushHead that we're done with this slot. Zeroing the
// slot is also important so we don't leave behind references
// that could keep this object live longer than necessary.
//
// We write to val first and then publish that we're done with
atomic.StorePointer(slot, nil)
// At this point pushHead owns the slot.
if tail < math.MaxUint32 {
atomic.AddUint64(&d.headTail, 1)
} else {
atomic.AddUint64(&d.headTail, ^uint64(math.MaxUint32-1))
}
return val, true
}
func (entry *sliceEntry) IsEmpty() bool {
if len(entry.element) == 0 {
return true
}
return false
// bufChain is a dynamically-sized version of bufDequeue.
//
// This is implemented as a doubly-linked list queue of poolDequeues
// where each dequeue is double the size of the previous one. Once a
// dequeue fills up, this allocates a new one and only ever pushes to
// the latest dequeue. Pops happen from the other end of the list and
// once a dequeue is exhausted, it gets removed from the list.
type bufChain struct {
// head is the bufDequeue to push to. This is only accessed
// by the producer, so doesn't need to be synchronized.
head *bufChainElt
// tail is the bufDequeue to popTail from. This is accessed
// by consumers, so reads and writes must be atomic.
tail *bufChainElt
newChain uint32
}
type bufChainElt struct {
bufDequeue
// next and prev link to the adjacent poolChainElts in this
// bufChain.
//
// next is written atomically by the producer and read
// atomically by the consumer. It only transitions from nil to
// non-nil.
//
// prev is written atomically by the consumer and read
// atomically by the producer. It only transitions from
// non-nil to nil.
next, prev *bufChainElt
}
func storePoolChainElt(pp **bufChainElt, v *bufChainElt) {
atomic.StorePointer((*unsafe.Pointer)(unsafe.Pointer(pp)), unsafe.Pointer(v))
}
func loadPoolChainElt(pp **bufChainElt) *bufChainElt {
return (*bufChainElt)(atomic.LoadPointer((*unsafe.Pointer)(unsafe.Pointer(pp))))
}
func (c *bufChain) new(initSize int) {
// Initialize the chain.
// initSize must be a power of 2
d := new(bufChainElt)
d.vals = make([]unsafe.Pointer, initSize)
storePoolChainElt(&c.head, d)
storePoolChainElt(&c.tail, d)
}
func (c *bufChain) pushHead(val unsafe.Pointer) {
startPush:
for {
if atomic.LoadUint32(&c.newChain) > 0 {
runtime.Gosched()
} else {
break
}
}
d := loadPoolChainElt(&c.head)
if d.pushHead(val) {
return
}
// The current dequeue is full. Allocate a new one of twice
// the size.
if atomic.CompareAndSwapUint32(&c.newChain, 0, 1) {
newSize := len(d.vals) * 2
if newSize >= dequeueLimit {
// Can't make it any bigger.
newSize = dequeueLimit
}
d2 := &bufChainElt{prev: d}
d2.vals = make([]unsafe.Pointer, newSize)
d2.pushHead(val)
storePoolChainElt(&c.head, d2)
storePoolChainElt(&d.next, d2)
atomic.StoreUint32(&c.newChain, 0)
return
}
goto startPush
}
func (c *bufChain) popTail() (unsafe.Pointer, bool) {
d := loadPoolChainElt(&c.tail)
if d == nil {
return nil, false
}
for {
// It's important that we load the next pointer
// *before* popping the tail. In general, d may be
// transiently empty, but if next is non-nil before
// the TryPop and the TryPop fails, then d is permanently
// empty, which is the only condition under which it's
// safe to drop d from the chain.
d2 := loadPoolChainElt(&d.next)
if val, ok := d.popTail(); ok {
return val, ok
}
if d2 == nil {
// This is the only dequeue. It's empty right
// now, but could be pushed to in the future.
return nil, false
}
// The tail of the chain has been drained, so move on
// to the next dequeue. Try to drop it from the chain
// so the next TryPop doesn't have to look at the empty
// dequeue again.
if atomic.CompareAndSwapPointer((*unsafe.Pointer)(unsafe.Pointer(&c.tail)), unsafe.Pointer(d), unsafe.Pointer(d2)) {
// We won the race. Clear the prev pointer so
// the garbage collector can collect the empty
// dequeue and so popHead doesn't back up
// further than necessary.
storePoolChainElt(&d2.prev, nil)
}
d = d2
}
}

154
lib/mux/web.go Normal file
View File

@ -0,0 +1,154 @@
package mux
import (
"fmt"
"github.com/astaxie/beego/logs"
"net/http"
"sort"
"strconv"
"strings"
"sync"
"time"
)
type connLog struct {
startTime time.Time
isClose bool
logs []string
}
var logms map[int]*connLog
var logmc map[int]*connLog
var copyMaps map[int]*connLog
var copyMapc map[int]*connLog
var stashTimeNow time.Time
var mutex sync.Mutex
func deepCopyMaps() {
copyMaps = make(map[int]*connLog)
for k, v := range logms {
copyMaps[k] = &connLog{
startTime: v.startTime,
isClose: v.isClose,
logs: v.logs,
}
}
}
func deepCopyMapc() {
copyMapc = make(map[int]*connLog)
for k, v := range logmc {
copyMapc[k] = &connLog{
startTime: v.startTime,
isClose: v.isClose,
logs: v.logs,
}
}
}
func init() {
logms = make(map[int]*connLog)
logmc = make(map[int]*connLog)
}
type IntSlice []int
func (s IntSlice) Len() int { return len(s) }
func (s IntSlice) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
func (s IntSlice) Less(i, j int) bool { return s[i] < s[j] }
func NewLogServer() {
http.HandleFunc("/", index)
http.HandleFunc("/detail", detail)
http.HandleFunc("/stash", stash)
fmt.Println(http.ListenAndServe(":8899", nil))
}
func stash(w http.ResponseWriter, r *http.Request) {
stashTimeNow = time.Now()
deepCopyMaps()
deepCopyMapc()
w.Write([]byte("ok"))
}
func getM(label string, id int) (cL *connLog) {
label = strings.TrimSpace(label)
mutex.Lock()
defer mutex.Unlock()
if label == "nps" {
cL = logms[id]
}
if label == "npc" {
cL = logmc[id]
}
return
}
func setM(label string, id int, cL *connLog) {
label = strings.TrimSpace(label)
mutex.Lock()
defer mutex.Unlock()
if label == "nps" {
logms[id] = cL
}
if label == "npc" {
logmc[id] = cL
}
}
func index(w http.ResponseWriter, r *http.Request) {
var keys []int
for k := range copyMaps {
keys = append(keys, k)
}
sort.Sort(IntSlice(keys))
var s string
s += "<h1>nps</h1>"
for _, v := range keys {
connL := copyMaps[v]
s += "<a href='/detail?id=" + strconv.Itoa(v) + "&label=nps" + "'>" + strconv.Itoa(v) + "</a>----------"
s += strconv.Itoa(int(stashTimeNow.Sub(connL.startTime).Milliseconds())) + "ms----------"
s += strconv.FormatBool(connL.isClose)
s += "<br>"
}
keys = keys[:0]
s += "<h1>npc</h1>"
for k := range copyMapc {
keys = append(keys, k)
}
sort.Sort(IntSlice(keys))
for _, v := range keys {
connL := copyMapc[v]
s += "<a href='/detail?id=" + strconv.Itoa(v) + "&label=npc" + "'>" + strconv.Itoa(v) + "</a>----------"
s += strconv.Itoa(int(stashTimeNow.Sub(connL.startTime).Milliseconds())) + "ms----------"
s += strconv.FormatBool(connL.isClose)
s += "<br>"
}
w.Write([]byte(s))
}
func detail(w http.ResponseWriter, r *http.Request) {
id := r.FormValue("id")
label := r.FormValue("label")
logs.Warn(label)
i, _ := strconv.Atoi(id)
var v *connLog
if label == "nps" {
v, _ = copyMaps[i]
}
if label == "npc" {
v, _ = copyMapc[i]
}
var s string
if v != nil {
for i, vv := range v.logs {
s += "<p>" + strconv.Itoa(i+1) + ":" + vv + "</p>"
}
}
w.Write([]byte(s))
}

7
lib/mux/web_test.go Normal file
View File

@ -0,0 +1,7 @@
package mux
import "testing"
func TestWeb(t *testing.T) {
NewLogServer()
}

59
lib/pool/pool.go
View File

@ -1,59 +0,0 @@
package pool
import (
"sync"
)
const PoolSize = 64 * 1024
const PoolSizeSmall = 100
const PoolSizeUdp = 1472
const PoolSizeCopy = 32 << 10
var BufPool = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSize)
},
}
var BufPoolUdp = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSizeUdp)
},
}
var BufPoolMax = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSize)
},
}
var BufPoolSmall = sync.Pool{
New: func() interface{} {
return make([]byte, PoolSizeSmall)
},
}
var BufPoolCopy = sync.Pool{
New: func() interface{} {
buf := make([]byte, PoolSizeCopy)
return &buf
},
}
func PutBufPoolUdp(buf []byte) {
if cap(buf) == PoolSizeUdp {
BufPoolUdp.Put(buf[:PoolSizeUdp])
}
}
func PutBufPoolCopy(buf []byte) {
if cap(buf) == PoolSizeCopy {
BufPoolCopy.Put(&buf)
}
}
func GetBufPoolCopy() ([]byte) {
return (*BufPoolCopy.Get().(*[]byte))[:PoolSizeCopy]
}
func PutBufPoolMax(buf []byte) {
if cap(buf) == PoolSize {
BufPoolMax.Put(buf[:PoolSize])
}
}

4
lib/version/version.go
View File

@ -1,8 +1,8 @@
package version
const VERSION = "0.23.1"
const VERSION = "0.24.0"
// Compulsory minimum version, Minimum downward compatibility to this version
func GetVersion() string {
return "0.21.0"
return "0.24.0"
}

7
server/connection/connection.go
View File

@ -1,12 +1,13 @@
package connection
import (
"github.com/cnlh/nps/lib/mux"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"net"
"os"
"strconv"
"github.com/astaxie/beego"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/mux"
)
var pMux *mux.PortMux

9
server/proxy/base.go
View File

@ -2,14 +2,15 @@ package proxy
import (
"errors"
"net"
"net/http"
"sync"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/bridge"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"net"
"net/http"
"sync"
)
type Service interface {

194
server/proxy/http.go
View File

@ -3,13 +3,6 @@ package proxy
import (
"bufio"
"crypto/tls"
"github.com/cnlh/nps/bridge"
"github.com/cnlh/nps/lib/cache"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/server/connection"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"io"
"net"
"net/http"
@ -19,6 +12,14 @@ import (
"strconv"
"strings"
"sync"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/bridge"
"github.com/cnlh/nps/lib/cache"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/server/connection"
)
type httpServer struct {
@ -108,12 +109,11 @@ func (s *httpServer) handleTunneling(w http.ResponseWriter, r *http.Request) {
if err != nil {
http.Error(w, err.Error(), http.StatusServiceUnavailable)
}
s.httpHandle(conn.NewConn(c), r)
s.handleHttp(conn.NewConn(c), r)
}
func (s *httpServer) httpHandle(c *conn.Conn, r *http.Request) {
func (s *httpServer) handleHttp(c *conn.Conn, r *http.Request) {
var (
isConn = false
host *file.Host
target net.Conn
lastHost *file.Host
@ -122,89 +122,80 @@ func (s *httpServer) httpHandle(c *conn.Conn, r *http.Request) {
scheme = r.URL.Scheme
lk *conn.Link
targetAddr string
readReq bool
reqCh = make(chan *http.Request)
lenConn *conn.LenConn
isReset bool
wg sync.WaitGroup
)
defer func() {
if connClient != nil {
s.writeConnFail(c.Conn)
connClient.Close()
}
c.Close()
}()
if host, err = file.GetDb().GetInfoByHost(r.Host, r); err != nil {
logs.Notice("the url %s %s %s can't be parsed!", r.URL.Scheme, r.Host, r.RequestURI)
goto end
return
}
if err := s.CheckFlowAndConnNum(host.Client); err != nil {
logs.Warn("client id %d, host id %d, error %s, when https connection", host.Client.Id, host.Id, err.Error())
c.Close()
return
}
defer host.Client.AddConn()
if err = s.auth(r, c, host.Client.Cnf.U, host.Client.Cnf.P); err != nil {
logs.Warn("auth error", err, r.RemoteAddr)
return
}
reset:
if targetAddr, err = host.Target.GetRandomTarget(); err != nil {
logs.Warn(err.Error())
return
}
lk = conn.NewLink("http", targetAddr, host.Client.Cnf.Crypt, host.Client.Cnf.Compress, r.RemoteAddr, host.Target.LocalProxy)
if target, err = s.bridge.SendLinkInfo(host.Client.Id, lk, nil); err != nil {
logs.Notice("connect to target %s error %s", lk.Host, err)
return
}
connClient = conn.GetConn(target, lk.Crypt, lk.Compress, host.Client.Rate, true)
lastHost = host
for {
start:
if isConn {
if err = s.auth(r, c, host.Client.Cnf.U, host.Client.Cnf.P); err != nil {
logs.Warn("auth error", err, r.RemoteAddr)
break
//read from inc-client
go func() {
wg.Add(1)
isReset = false
defer connClient.Close()
defer func() {
wg.Done()
if !isReset {
c.Close()
}
if targetAddr, err = host.Target.GetRandomTarget(); err != nil {
logs.Warn(err.Error())
break
}
lk = conn.NewLink("http", targetAddr, host.Client.Cnf.Crypt, host.Client.Cnf.Compress, r.RemoteAddr, host.Target.LocalProxy)
if target, err = s.bridge.SendLinkInfo(host.Client.Id, lk, nil); err != nil {
logs.Notice("connect to target %s error %s", lk.Host, err)
break
}
connClient = conn.GetConn(target, lk.Crypt, lk.Compress, host.Client.Rate, true)
isConn = false
go func() {
defer connClient.Close()
defer c.Close()
for {
if resp, err := http.ReadResponse(bufio.NewReader(connClient), r); err != nil {
}()
for {
if resp, err := http.ReadResponse(bufio.NewReader(connClient), r); err != nil {
return
} else {
//if the cache is start and the response is in the extension,store the response to the cache list
if s.useCache && strings.Contains(r.URL.Path, ".") {
b, err := httputil.DumpResponse(resp, true)
if err != nil {
return
} else {
r := <-reqCh
//if the cache is start and the response is in the extension,store the response to the cache list
if s.useCache && strings.Contains(r.URL.Path, ".") {
b, err := httputil.DumpResponse(resp, true)
if err != nil {
return
}
c.Write(b)
host.Flow.Add(0, int64(len(b)))
s.cache.Add(filepath.Join(host.Host, r.URL.Path), b)
} else {
lenConn := conn.NewLenConn(c)
if err := resp.Write(lenConn); err != nil {
logs.Error(err)
return
}
host.Flow.Add(0, int64(lenConn.Len))
}
}
c.Write(b)
host.Flow.Add(0, int64(len(b)))
s.cache.Add(filepath.Join(host.Host, r.URL.Path), b)
} else {
lenConn := conn.NewLenConn(c)
if err := resp.Write(lenConn); err != nil {
logs.Error(err)
return
}
host.Flow.Add(0, int64(lenConn.Len))
}
}()
} else if readReq {
r, err = http.ReadRequest(bufio.NewReader(c))
if err != nil {
break
}
r.URL.Scheme = scheme
//What happened ，Why one character less???
if r.Method == "ET" {
r.Method = "GET"
}
if r.Method == "OST" {
r.Method = "POST"
}
if hostTmp, err := file.GetDb().GetInfoByHost(r.Host, r); err != nil {
logs.Notice("the url %s %s %s can't be parsed!", r.URL.Scheme, r.Host, r.RequestURI)
break
} else if host != lastHost {
host = hostTmp
lastHost = host
isConn = true
goto start
}
}
}()
for {
//if the cache start and the request is in the cache list, return the cache
if s.useCache {
if v, ok := s.cache.Get(filepath.Join(host.Host, r.URL.Path)); ok {
@ -215,39 +206,54 @@ func (s *httpServer) httpHandle(c *conn.Conn, r *http.Request) {
logs.Trace("%s request, method %s, host %s, url %s, remote address %s, return cache", r.URL.Scheme, r.Method, r.Host, r.URL.Path, c.RemoteAddr().String())
host.Flow.Add(0, int64(n))
//if return cache and does not create a new conn with client and Connection is not set or close, close the connection.
if connClient == nil && (strings.ToLower(r.Header.Get("Connection")) == "close" || strings.ToLower(r.Header.Get("Connection")) == "") {
c.Close()
if strings.ToLower(r.Header.Get("Connection")) == "close" || strings.ToLower(r.Header.Get("Connection")) == "" {
break
}
readReq = true
goto start
goto readReq
}
}
if connClient == nil {
isConn = true
goto start
}
readReq = true
//change the host and header and set proxy setting
common.ChangeHostAndHeader(r, host.HostChange, host.HeaderChange, c.Conn.RemoteAddr().String())
logs.Trace("%s request, method %s, host %s, url %s, remote address %s, target %s", r.URL.Scheme, r.Method, r.Host, r.URL.Path, c.RemoteAddr().String(), lk.Host)
//write
lenConn := conn.NewLenConn(connClient)
lenConn = conn.NewLenConn(connClient)
if err := r.Write(lenConn); err != nil {
logs.Error(err)
break
}
host.Flow.Add(int64(lenConn.Len), 0)
reqCh <- r
readReq:
//read req from connection
if r, err = http.ReadRequest(bufio.NewReader(c)); err != nil {
break
}
r.URL.Scheme = scheme
//What happened ，Why one character less???
r.Method = resetReqMethod(r.Method)
if hostTmp, err := file.GetDb().GetInfoByHost(r.Host, r); err != nil {
logs.Notice("the url %s %s %s can't be parsed!", r.URL.Scheme, r.Host, r.RequestURI)
break
} else if host != lastHost {
host = hostTmp
lastHost = host
isReset = true
connClient.Close()
goto reset
}
}
end:
if !readReq {
s.writeConnFail(c.Conn)
wg.Wait()
}
func resetReqMethod(method string) string {
if method == "ET" {
return "GET"
}
c.Close()
if target != nil {
target.Close()
if method == "OST" {
return "POST"
}
return method
}
func (s *httpServer) NewServer(port int, scheme string) *http.Server {

13
server/proxy/https.go
View File

@ -1,18 +1,19 @@
package proxy
import (
"net"
"net/http"
"net/url"
"sync"
"github.com/astaxie/beego"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/cache"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/crypt"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/pkg/errors"
"net"
"net/http"
"net/url"
"sync"
)
type HttpsServer struct {

8
server/proxy/p2p.go
View File

@ -1,12 +1,12 @@
package proxy
import (
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/pool"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"net"
"strings"
"time"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
)
type P2PServer struct {
@ -36,7 +36,7 @@ func (s *P2PServer) Start() error {
return err
}
for {
buf := pool.BufPoolUdp.Get().([]byte)
buf := common.BufPoolUdp.Get().([]byte)
n, addr, err := s.listener.ReadFromUDP(buf)
if err != nil {
if strings.Contains(err.Error(), "use of closed network connection") {

30
server/proxy/socks5.go
View File

@ -3,13 +3,14 @@ package proxy
import (
"encoding/binary"
"errors"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"io"
"net"
"strconv"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
)
const (
@ -198,7 +199,7 @@ func (s *Sock5ModeServer) handleConn(c net.Conn) {
c.Close()
return
}
if s.task.Client.Cnf.U != "" && s.task.Client.Cnf.P != "" {
if (s.task.Client.Cnf.U != "" && s.task.Client.Cnf.P != "") || (s.task.MultiAccount != nil && len(s.task.MultiAccount.AccountMap) > 0) {
buf[1] = UserPassAuth
c.Write(buf)
if err := s.Auth(c); err != nil {
@ -235,7 +236,22 @@ func (s *Sock5ModeServer) Auth(c net.Conn) error {
if _, err := io.ReadAtLeast(c, pass, passLen); err != nil {
return err
}
if string(user) == s.task.Client.Cnf.U && string(pass) == s.task.Client.Cnf.P {
var U, P string
if s.task.MultiAccount != nil {
// enable multi user auth
U = string(user)
var ok bool
P, ok = s.task.MultiAccount.AccountMap[U]
if !ok {
return errors.New("验证不通过")
}
} else {
U = s.task.Client.Cnf.U
P = s.task.Client.Cnf.P
}
if string(user) == U && string(pass) == P {
if _, err := c.Write([]byte{userAuthVersion, authSuccess}); err != nil {
return err
}
@ -273,4 +289,4 @@ func NewSock5ModeServer(bridge NetBridge, task *file.Tunnel) *Sock5ModeServer {
//close
func (s *Sock5ModeServer) Close() error {
return s.listener.Close()
}
}

13
server/proxy/tcp.go
View File

@ -2,17 +2,18 @@ package proxy
import (
"errors"
"net"
"net/http"
"path/filepath"
"strconv"
"github.com/astaxie/beego"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/bridge"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/server/connection"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"net"
"net/http"
"path/filepath"
"strconv"
)
type TunnelModeServer struct {

5
server/proxy/transport.go
View File

@ -3,11 +3,12 @@
package proxy
import (
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"net"
"strconv"
"syscall"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
)
func HandleTrans(c *conn.Conn, s *TunnelModeServer) error {

14
server/proxy/udp.go
View File

@ -1,14 +1,14 @@
package proxy
import (
"net"
"strings"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/bridge"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/conn"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/lib/pool"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"net"
"strings"
)
type UdpModeServer struct {
@ -33,7 +33,7 @@ func (s *UdpModeServer) Start() error {
if err != nil {
return err
}
buf := pool.BufPoolUdp.Get().([]byte)
buf := common.BufPoolUdp.Get().([]byte)
for {
n, addr, err := s.listener.ReadFromUDP(buf)
if err != nil {
@ -59,8 +59,8 @@ func (s *UdpModeServer) process(addr *net.UDPAddr, data []byte) {
return
} else {
s.task.Flow.Add(int64(len(data)), 0)
buf := pool.BufPoolUdp.Get().([]byte)
defer pool.BufPoolUdp.Put(buf)
buf := common.BufPoolUdp.Get().([]byte)
defer common.BufPoolUdp.Put(buf)
target.Write(data)
s.task.Flow.Add(int64(len(data)), 0)
if n, err := target.Read(buf); err != nil {

25
server/server.go
View File

@ -2,22 +2,23 @@ package server
import (
"errors"
"github.com/cnlh/nps/bridge"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/server/proxy"
"github.com/cnlh/nps/server/tool"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/shirou/gopsutil/cpu"
"github.com/shirou/gopsutil/load"
"github.com/shirou/gopsutil/mem"
"github.com/shirou/gopsutil/net"
"math"
"os"
"strconv"
"strings"
"time"
"github.com/astaxie/beego"
"github.com/astaxie/beego/logs"
"github.com/cnlh/nps/bridge"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/server/proxy"
"github.com/cnlh/nps/server/tool"
"github.com/shirou/gopsutil/cpu"
"github.com/shirou/gopsutil/load"
"github.com/shirou/gopsutil/mem"
"github.com/shirou/gopsutil/net"
)
var (
@ -359,7 +360,7 @@ func GetDashboardData() map[string]interface{} {
case "tcp":
tcp += 1
case "socks5":
udp += 1
socks5 += 1
case "httpProxy":
http += 1
case "udp":

7
server/test/test.go
View File

@ -1,12 +1,13 @@
package test
import (
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/file"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"log"
"path/filepath"
"strconv"
"github.com/astaxie/beego"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/lib/file"
)
func TestServerConfig() {

9
server/tool/utils.go
View File

@ -1,15 +1,16 @@
package tool
import (
"math"
"strconv"
"time"
"github.com/astaxie/beego"
"github.com/cnlh/nps/lib/common"
"github.com/cnlh/nps/vender/github.com/astaxie/beego"
"github.com/shirou/gopsutil/cpu"
"github.com/shirou/gopsutil/load"
"github.com/shirou/gopsutil/mem"
"github.com/shirou/gopsutil/net"
"math"
"strconv"
"time"
)
var (

6
vender/github.com/astaxie/beego/.gitignore vendored
View File

@ -1,6 +0,0 @@
.idea
.vscode
.DS_Store
*.swp
*.swo
beego.iml

4
vender/github.com/astaxie/beego/.gosimpleignore
View File

@ -1,4 +0,0 @@
github.com/cnlh/nps/vender/github.com/astaxie/beego/*/*:S1012
github.com/cnlh/nps/vender/github.com/astaxie/beego/*:S1012
github.com/cnlh/nps/vender/github.com/astaxie/beego/*/*:S1007
github.com/cnlh/nps/vender/github.com/astaxie/beego/*:S1007

63
vender/github.com/astaxie/beego/.travis.yml
View File

@ -1,63 +0,0 @@
language: go
go:
- "1.9.7"
- "1.10.3"
- "1.11"
services:
- redis-server
- mysql
- postgresql
- memcached
env:
- ORM_DRIVER=sqlite3 ORM_SOURCE=$TRAVIS_BUILD_DIR/orm_test.db
- ORM_DRIVER=postgres ORM_SOURCE="user=postgres dbname=orm_test sslmode=disable"
before_install:
- git clone git://github.com/ideawu/ssdb.git
- cd ssdb
- make
- cd ..
install:
- go get github.com/lib/pq
- go get github.com/go-sql-driver/mysql
- go get github.com/mattn/go-sqlite3
- go get github.com/bradfitz/gomemcache/memcache
- go get github.com/gomodule/redigo/redis
- go get github.com/beego/x2j
- go get github.com/couchbase/go-couchbase
- go get github.com/beego/goyaml2
- go get gopkg.in/yaml.v2
- go get github.com/belogik/goes
- go get github.com/siddontang/ledisdb/config
- go get github.com/siddontang/ledisdb/ledis
- go get github.com/ssdb/gossdb/ssdb
- go get github.com/cloudflare/golz4
- go get github.com/gogo/protobuf/proto
- go get github.com/Knetic/govaluate
- go get github.com/casbin/casbin
- go get -u honnef.co/go/tools/cmd/gosimple
- go get -u github.com/mdempsky/unconvert
- go get -u github.com/gordonklaus/ineffassign
- go get -u github.com/golang/lint/golint
- go get -u github.com/go-redis/redis
before_script:
- psql --version
- sh -c "if [ '$ORM_DRIVER' = 'postgres' ]; then psql -c 'create database orm_test;' -U postgres; fi"
- sh -c "if [ '$ORM_DRIVER' = 'mysql' ]; then mysql -u root -e 'create database orm_test;'; fi"
- sh -c "if [ '$ORM_DRIVER' = 'sqlite' ]; then touch $TRAVIS_BUILD_DIR/orm_test.db; fi"
- sh -c "go get github.com/golang/lint/golint; golint ./...;"
- sh -c "go list ./... | grep -v vendor | xargs go vet -v"
- mkdir -p res/var
- ./ssdb/ssdb-server ./ssdb/ssdb.conf -d
after_script:
-killall -w ssdb-server
- rm -rf ./res/var/*
script:
- go test -v ./...
- gosimple -ignore "$(cat .gosimpleignore)" $(go list ./... | grep -v /vendor/)
- unconvert $(go list ./... | grep -v /vendor/)
- ineffassign .
- find . ! \( -path './vendor' -prune \) -type f -name '*.go' -print0 | xargs -0 gofmt -l -s
- golint ./...
addons:
postgresql: "9.6"

52
vender/github.com/astaxie/beego/CONTRIBUTING.md
View File

@ -1,52 +0,0 @@
# Contributing to beego
beego is an open source project.
It is the work of hundreds of contributors. We appreciate your help!
Here are instructions to get you started. They are probably not perfect,
please let us know if anything feels wrong or incomplete.
## Contribution guidelines
### Pull requests
First of all. beego follow the gitflow. So please send you pull request
to **develop** branch. We will close the pull request to master branch.
We are always happy to receive pull requests, and do our best to
review them as fast as possible. Not sure if that typo is worth a pull
request? Do it! We will appreciate it.
If your pull request is not accepted on the first try, don't be
discouraged! Sometimes we can make a mistake, please do more explaining
for us. We will appreciate it.
We're trying very hard to keep beego simple and fast. We don't want it
to do everything for everybody. This means that we might decide against
incorporating a new feature. But we will give you some advice on how to
do it in other way.
### Create issues
Any significant improvement should be documented as [a GitHub
issue](https://github.com/cnlh/nps/vender/github.com/astaxie/beego/issues) before anybody
starts working on it.
Also when filing an issue, make sure to answer these five questions:
- What version of beego are you using (bee version)?
- What operating system and processor architecture are you using?
- What did you do?
- What did you expect to see?
- What did you see instead?
### but check existing issues and docs first!
Please take a moment to check that an issue doesn't already exist
documenting your bug report or improvement proposal. If it does, it
never hurts to add a quick "+1" or "I have this problem too". This will
help prioritize the most common problems and requests.
Also if you don't know how to use it. please make sure you have read though
the docs in http://beego.me/docs

13
vender/github.com/astaxie/beego/LICENSE
View File

@ -1,13 +0,0 @@
Copyright 2014 astaxie
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

63
vender/github.com/astaxie/beego/README.md
View File

@ -1,63 +0,0 @@
# Beego [![Build Status](https://travis-ci.org/astaxie/beego.svg?branch=master)](https://travis-ci.org/astaxie/beego) [![GoDoc](http://godoc.org/github.com/cnlh/nps/vender/github.com/astaxie/beego?status.svg)](http://godoc.org/github.com/cnlh/nps/vender/github.com/astaxie/beego) [![Foundation](https://img.shields.io/badge/Golang-Foundation-green.svg)](http://golangfoundation.org) [![Go Report Card](https://goreportcard.com/badge/github.com/cnlh/nps/vender/github.com/astaxie/beego)](https://goreportcard.com/report/github.com/cnlh/nps/vender/github.com/astaxie/beego)
beego is used for rapid development of RESTful APIs, web apps and backend services in Go.
It is inspired by Tornado, Sinatra and Flask. beego has some Go-specific features such as interfaces and struct embedding.
Response time ranking: [web-frameworks](https://github.com/the-benchmarker/web-frameworks).
###### More info at [beego.me](http://beego.me).
## Quick Start
#### Download and install
go get github.com/cnlh/nps/vender/github.com/astaxie/beego
#### Create file `hello.go`
```go
package main
import "github.com/cnlh/nps/vender/github.com/astaxie/beego"
func main(){
beego.Run()
}
```
#### Build and run
go build hello.go
./hello
#### Go to [http://localhost:8080](http://localhost:8080)
Congratulations! You've just built your first **beego** app.
###### Please see [Documentation](http://beego.me/docs) for more.
## Features
* RESTful support
* MVC architecture
* Modularity
* Auto API documents
* Annotation router
* Namespace
* Powerful development tools
* Full stack for Web & API
## Documentation
* [English](http://beego.me/docs/intro/)
* [中文文档](http://beego.me/docs/intro/)
* [Русский](http://beego.me/docs/intro/)
## Community
* [http://beego.me/community](http://beego.me/community)
* Welcome to join us in Slack: [https://beego.slack.com](https://beego.slack.com), you can get invited from [here](https://github.com/beego/beedoc/issues/232)
## License
beego source code is licensed under the Apache Licence, Version 2.0
(http://www.apache.org/licenses/LICENSE-2.0.html).

416
vender/github.com/astaxie/beego/admin.go
View File

@ -1,416 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package beego
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"os"
"text/template"
"time"
"reflect"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/grace"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/toolbox"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/utils"
)
// BeeAdminApp is the default adminApp used by admin module.
var beeAdminApp *adminApp
// FilterMonitorFunc is default monitor filter when admin module is enable.
// if this func returns, admin module records qbs for this request by condition of this function logic.
// usage:
// func MyFilterMonitor(method, requestPath string, t time.Duration, pattern string, statusCode int) bool {
// if method == "POST" {
// return false
// }
// if t.Nanoseconds() < 100 {
// return false
// }
// if strings.HasPrefix(requestPath, "/astaxie") {
// return false
// }
// return true
// }
// beego.FilterMonitorFunc = MyFilterMonitor.
var FilterMonitorFunc func(string, string, time.Duration, string, int) bool
func init() {
beeAdminApp = &adminApp{
routers: make(map[string]http.HandlerFunc),
}
beeAdminApp.Route("/", adminIndex)
beeAdminApp.Route("/qps", qpsIndex)
beeAdminApp.Route("/prof", profIndex)
beeAdminApp.Route("/healthcheck", healthcheck)
beeAdminApp.Route("/task", taskStatus)
beeAdminApp.Route("/listconf", listConf)
FilterMonitorFunc = func(string, string, time.Duration, string, int) bool { return true }
}
// AdminIndex is the default http.Handler for admin module.
// it matches url pattern "/".
func adminIndex(rw http.ResponseWriter, _ *http.Request) {
execTpl(rw, map[interface{}]interface{}{}, indexTpl, defaultScriptsTpl)
}
// QpsIndex is the http.Handler for writing qbs statistics map result info in http.ResponseWriter.
// it's registered with url pattern "/qbs" in admin module.
func qpsIndex(rw http.ResponseWriter, _ *http.Request) {
data := make(map[interface{}]interface{})
data["Content"] = toolbox.StatisticsMap.GetMap()
// do html escape before display path, avoid xss
if content, ok := (data["Content"]).(M); ok {
if resultLists, ok := (content["Data"]).([][]string); ok {
for i := range resultLists {
if len(resultLists[i]) > 0 {
resultLists[i][0] = template.HTMLEscapeString(resultLists[i][0])
}
}
}
}
execTpl(rw, data, qpsTpl, defaultScriptsTpl)
}
// ListConf is the http.Handler of displaying all beego configuration values as key/value pair.
// it's registered with url pattern "/listconf" in admin module.
func listConf(rw http.ResponseWriter, r *http.Request) {
r.ParseForm()
command := r.Form.Get("command")
if command == "" {
rw.Write([]byte("command not support"))
return
}
data := make(map[interface{}]interface{})
switch command {
case "conf":
m := make(M)
list("BConfig", BConfig, m)
m["AppConfigPath"] = appConfigPath
m["AppConfigProvider"] = appConfigProvider
tmpl := template.Must(template.New("dashboard").Parse(dashboardTpl))
tmpl = template.Must(tmpl.Parse(configTpl))
tmpl = template.Must(tmpl.Parse(defaultScriptsTpl))
data["Content"] = m
tmpl.Execute(rw, data)
case "router":
content := PrintTree()
content["Fields"] = []string{
"Router Pattern",
"Methods",
"Controller",
}
data["Content"] = content
data["Title"] = "Routers"
execTpl(rw, data, routerAndFilterTpl, defaultScriptsTpl)
case "filter":
var (
content = M{
"Fields": []string{
"Router Pattern",
"Filter Function",
},
}
filterTypes = []string{}
filterTypeData = make(M)
)
if BeeApp.Handlers.enableFilter {
var filterType string
for k, fr := range map[int]string{
BeforeStatic: "Before Static",
BeforeRouter: "Before Router",
BeforeExec: "Before Exec",
AfterExec: "After Exec",
FinishRouter: "Finish Router"} {
if bf := BeeApp.Handlers.filters[k]; len(bf) > 0 {
filterType = fr
filterTypes = append(filterTypes, filterType)
resultList := new([][]string)
for _, f := range bf {
var result = []string{
f.pattern,
utils.GetFuncName(f.filterFunc),
}
*resultList = append(*resultList, result)
}
filterTypeData[filterType] = resultList
}
}
}
content["Data"] = filterTypeData
content["Methods"] = filterTypes
data["Content"] = content
data["Title"] = "Filters"
execTpl(rw, data, routerAndFilterTpl, defaultScriptsTpl)
default:
rw.Write([]byte("command not support"))
}
}
func list(root string, p interface{}, m M) {
pt := reflect.TypeOf(p)
pv := reflect.ValueOf(p)
if pt.Kind() == reflect.Ptr {
pt = pt.Elem()
pv = pv.Elem()
}
for i := 0; i < pv.NumField(); i++ {
var key string
if root == "" {
key = pt.Field(i).Name
} else {
key = root + "." + pt.Field(i).Name
}
if pv.Field(i).Kind() == reflect.Struct {
list(key, pv.Field(i).Interface(), m)
} else {
m[key] = pv.Field(i).Interface()
}
}
}
// PrintTree prints all registered routers.
func PrintTree() M {
var (
content = M{}
methods = []string{}
methodsData = make(M)
)
for method, t := range BeeApp.Handlers.routers {
resultList := new([][]string)
printTree(resultList, t)
methods = append(methods, method)
methodsData[method] = resultList
}
content["Data"] = methodsData
content["Methods"] = methods
return content
}
func printTree(resultList *[][]string, t *Tree) {
for _, tr := range t.fixrouters {
printTree(resultList, tr)
}
if t.wildcard != nil {
printTree(resultList, t.wildcard)
}
for _, l := range t.leaves {
if v, ok := l.runObject.(*ControllerInfo); ok {
if v.routerType == routerTypeBeego {
var result = []string{
v.pattern,
fmt.Sprintf("%s", v.methods),
v.controllerType.String(),
}
*resultList = append(*resultList, result)
} else if v.routerType == routerTypeRESTFul {
var result = []string{
v.pattern,
fmt.Sprintf("%s", v.methods),
"",
}
*resultList = append(*resultList, result)
} else if v.routerType == routerTypeHandler {
var result = []string{
v.pattern,
"",
"",
}
*resultList = append(*resultList, result)
}
}
}
}
// ProfIndex is a http.Handler for showing profile command.
// it's in url pattern "/prof" in admin module.
func profIndex(rw http.ResponseWriter, r *http.Request) {
r.ParseForm()
command := r.Form.Get("command")
if command == "" {
return
}
var (
format = r.Form.Get("format")
data = make(map[interface{}]interface{})
result bytes.Buffer
)
toolbox.ProcessInput(command, &result)
data["Content"] = result.String()
if format == "json" && command == "gc summary" {
dataJSON, err := json.Marshal(data)
if err != nil {
http.Error(rw, err.Error(), http.StatusInternalServerError)
return
}
rw.Header().Set("Content-Type", "application/json")
rw.Write(dataJSON)
return
}
data["Title"] = command
defaultTpl := defaultScriptsTpl
if command == "gc summary" {
defaultTpl = gcAjaxTpl
}
execTpl(rw, data, profillingTpl, defaultTpl)
}
// Healthcheck is a http.Handler calling health checking and showing the result.
// it's in "/healthcheck" pattern in admin module.
func healthcheck(rw http.ResponseWriter, _ *http.Request) {
var (
result []string
data = make(map[interface{}]interface{})
resultList = new([][]string)
content = M{
"Fields": []string{"Name", "Message", "Status"},
}
)
for name, h := range toolbox.AdminCheckList {
if err := h.Check(); err != nil {
result = []string{
"error",
name,
err.Error(),
}
} else {
result = []string{
"success",
name,
"OK",
}
}
*resultList = append(*resultList, result)
}
content["Data"] = resultList
data["Content"] = content
data["Title"] = "Health Check"
execTpl(rw, data, healthCheckTpl, defaultScriptsTpl)
}
// TaskStatus is a http.Handler with running task status (task name, status and the last execution).
// it's in "/task" pattern in admin module.
func taskStatus(rw http.ResponseWriter, req *http.Request) {
data := make(map[interface{}]interface{})
// Run Task
req.ParseForm()
taskname := req.Form.Get("taskname")
if taskname != "" {
if t, ok := toolbox.AdminTaskList[taskname]; ok {
if err := t.Run(); err != nil {
data["Message"] = []string{"error", fmt.Sprintf("%s", err)}
}
data["Message"] = []string{"success", fmt.Sprintf("%s run success,Now the Status is <br>%s", taskname, t.GetStatus())}
} else {
data["Message"] = []string{"warning", fmt.Sprintf("there's no task which named: %s", taskname)}
}
}
// List Tasks
content := make(M)
resultList := new([][]string)
var fields = []string{
"Task Name",
"Task Spec",
"Task Status",
"Last Time",
"",
}
for tname, tk := range toolbox.AdminTaskList {
result := []string{
tname,
tk.GetSpec(),
tk.GetStatus(),
tk.GetPrev().String(),
}
*resultList = append(*resultList, result)
}
content["Fields"] = fields
content["Data"] = resultList
data["Content"] = content
data["Title"] = "Tasks"
execTpl(rw, data, tasksTpl, defaultScriptsTpl)
}
func execTpl(rw http.ResponseWriter, data map[interface{}]interface{}, tpls ...string) {
tmpl := template.Must(template.New("dashboard").Parse(dashboardTpl))
for _, tpl := range tpls {
tmpl = template.Must(tmpl.Parse(tpl))
}
tmpl.Execute(rw, data)
}
// adminApp is an http.HandlerFunc map used as beeAdminApp.
type adminApp struct {
routers map[string]http.HandlerFunc
}
// Route adds http.HandlerFunc to adminApp with url pattern.
func (admin *adminApp) Route(pattern string, f http.HandlerFunc) {
admin.routers[pattern] = f
}
// Run adminApp http server.
// Its addr is defined in configuration file as adminhttpaddr and adminhttpport.
func (admin *adminApp) Run() {
if len(toolbox.AdminTaskList) > 0 {
toolbox.StartTask()
}
addr := BConfig.Listen.AdminAddr
if BConfig.Listen.AdminPort != 0 {
addr = fmt.Sprintf("%s:%d", BConfig.Listen.AdminAddr, BConfig.Listen.AdminPort)
}
for p, f := range admin.routers {
http.Handle(p, f)
}
logs.Info("Admin server Running on %s", addr)
var err error
if BConfig.Listen.Graceful {
err = grace.ListenAndServe(addr, nil)
} else {
err = http.ListenAndServe(addr, nil)
}
if err != nil {
logs.Critical("Admin ListenAndServe: ", err, fmt.Sprintf("%d", os.Getpid()))
}
}

75
vender/github.com/astaxie/beego/admin_test.go
View File

@ -1,75 +0,0 @@
package beego
import (
"fmt"
"testing"
)
func TestList_01(t *testing.T) {
m := make(M)
list("BConfig", BConfig, m)
t.Log(m)
om := oldMap()
for k, v := range om {
if fmt.Sprint(m[k]) != fmt.Sprint(v) {
t.Log(k, "old-key", v, "new-key", m[k])
t.FailNow()
}
}
}
func oldMap() M {
m := make(M)
m["BConfig.AppName"] = BConfig.AppName
m["BConfig.RunMode"] = BConfig.RunMode
m["BConfig.RouterCaseSensitive"] = BConfig.RouterCaseSensitive
m["BConfig.ServerName"] = BConfig.ServerName
m["BConfig.RecoverPanic"] = BConfig.RecoverPanic
m["BConfig.CopyRequestBody"] = BConfig.CopyRequestBody
m["BConfig.EnableGzip"] = BConfig.EnableGzip
m["BConfig.MaxMemory"] = BConfig.MaxMemory
m["BConfig.EnableErrorsShow"] = BConfig.EnableErrorsShow
m["BConfig.Listen.Graceful"] = BConfig.Listen.Graceful
m["BConfig.Listen.ServerTimeOut"] = BConfig.Listen.ServerTimeOut
m["BConfig.Listen.ListenTCP4"] = BConfig.Listen.ListenTCP4
m["BConfig.Listen.EnableHTTP"] = BConfig.Listen.EnableHTTP
m["BConfig.Listen.HTTPAddr"] = BConfig.Listen.HTTPAddr
m["BConfig.Listen.HTTPPort"] = BConfig.Listen.HTTPPort
m["BConfig.Listen.EnableHTTPS"] = BConfig.Listen.EnableHTTPS
m["BConfig.Listen.HTTPSAddr"] = BConfig.Listen.HTTPSAddr
m["BConfig.Listen.HTTPSPort"] = BConfig.Listen.HTTPSPort
m["BConfig.Listen.HTTPSCertFile"] = BConfig.Listen.HTTPSCertFile
m["BConfig.Listen.HTTPSKeyFile"] = BConfig.Listen.HTTPSKeyFile
m["BConfig.Listen.EnableAdmin"] = BConfig.Listen.EnableAdmin
m["BConfig.Listen.AdminAddr"] = BConfig.Listen.AdminAddr
m["BConfig.Listen.AdminPort"] = BConfig.Listen.AdminPort
m["BConfig.Listen.EnableFcgi"] = BConfig.Listen.EnableFcgi
m["BConfig.Listen.EnableStdIo"] = BConfig.Listen.EnableStdIo
m["BConfig.WebConfig.AutoRender"] = BConfig.WebConfig.AutoRender
m["BConfig.WebConfig.EnableDocs"] = BConfig.WebConfig.EnableDocs
m["BConfig.WebConfig.FlashName"] = BConfig.WebConfig.FlashName
m["BConfig.WebConfig.FlashSeparator"] = BConfig.WebConfig.FlashSeparator
m["BConfig.WebConfig.DirectoryIndex"] = BConfig.WebConfig.DirectoryIndex
m["BConfig.WebConfig.StaticDir"] = BConfig.WebConfig.StaticDir
m["BConfig.WebConfig.StaticExtensionsToGzip"] = BConfig.WebConfig.StaticExtensionsToGzip
m["BConfig.WebConfig.TemplateLeft"] = BConfig.WebConfig.TemplateLeft
m["BConfig.WebConfig.TemplateRight"] = BConfig.WebConfig.TemplateRight
m["BConfig.WebConfig.ViewsPath"] = BConfig.WebConfig.ViewsPath
m["BConfig.WebConfig.EnableXSRF"] = BConfig.WebConfig.EnableXSRF
m["BConfig.WebConfig.XSRFExpire"] = BConfig.WebConfig.XSRFExpire
m["BConfig.WebConfig.Session.SessionOn"] = BConfig.WebConfig.Session.SessionOn
m["BConfig.WebConfig.Session.SessionProvider"] = BConfig.WebConfig.Session.SessionProvider
m["BConfig.WebConfig.Session.SessionName"] = BConfig.WebConfig.Session.SessionName
m["BConfig.WebConfig.Session.SessionGCMaxLifetime"] = BConfig.WebConfig.Session.SessionGCMaxLifetime
m["BConfig.WebConfig.Session.SessionProviderConfig"] = BConfig.WebConfig.Session.SessionProviderConfig
m["BConfig.WebConfig.Session.SessionCookieLifeTime"] = BConfig.WebConfig.Session.SessionCookieLifeTime
m["BConfig.WebConfig.Session.SessionAutoSetCookie"] = BConfig.WebConfig.Session.SessionAutoSetCookie
m["BConfig.WebConfig.Session.SessionDomain"] = BConfig.WebConfig.Session.SessionDomain
m["BConfig.WebConfig.Session.SessionDisableHTTPOnly"] = BConfig.WebConfig.Session.SessionDisableHTTPOnly
m["BConfig.Log.AccessLogs"] = BConfig.Log.AccessLogs
m["BConfig.Log.EnableStaticLogs"] = BConfig.Log.EnableStaticLogs
m["BConfig.Log.AccessLogsFormat"] = BConfig.Log.AccessLogsFormat
m["BConfig.Log.FileLineNum"] = BConfig.Log.FileLineNum
m["BConfig.Log.Outputs"] = BConfig.Log.Outputs
return m
}

356
vender/github.com/astaxie/beego/adminui.go
View File

File diff suppressed because one or more lines are too long

497
vender/github.com/astaxie/beego/app.go
View File

@ -1,497 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package beego
import (
"crypto/tls"
"crypto/x509"
"fmt"
"io/ioutil"
"net"
"net/http"
"net/http/fcgi"
"os"
"path"
"strings"
"time"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/grace"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/utils"
"golang.org/x/crypto/acme/autocert"
)
var (
// BeeApp is an application instance
BeeApp *App
)
func init() {
// create beego application
BeeApp = NewApp()
}
// App defines beego application with a new PatternServeMux.
type App struct {
Handlers *ControllerRegister
Server *http.Server
}
// NewApp returns a new beego application.
func NewApp() *App {
cr := NewControllerRegister()
app := &App{Handlers: cr, Server: &http.Server{}}
return app
}
// MiddleWare function for http.Handler
type MiddleWare func(http.Handler) http.Handler
// Run beego application.
func (app *App) Run(mws ...MiddleWare) {
addr := BConfig.Listen.HTTPAddr
if BConfig.Listen.HTTPPort != 0 {
addr = fmt.Sprintf("%s:%d", BConfig.Listen.HTTPAddr, BConfig.Listen.HTTPPort)
}
var (
err error
l net.Listener
endRunning = make(chan bool, 1)
)
// run cgi server
if BConfig.Listen.EnableFcgi {
if BConfig.Listen.EnableStdIo {
if err = fcgi.Serve(nil, app.Handlers); err == nil { // standard I/O
logs.Info("Use FCGI via standard I/O")
} else {
logs.Critical("Cannot use FCGI via standard I/O", err)
}
return
}
if BConfig.Listen.HTTPPort == 0 {
// remove the Socket file before start
if utils.FileExists(addr) {
os.Remove(addr)
}
l, err = net.Listen("unix", addr)
} else {
l, err = net.Listen("tcp", addr)
}
if err != nil {
logs.Critical("Listen: ", err)
}
if err = fcgi.Serve(l, app.Handlers); err != nil {
logs.Critical("fcgi.Serve: ", err)
}
return
}
app.Server.Handler = app.Handlers
for i := len(mws) - 1; i >= 0; i-- {
if mws[i] == nil {
continue
}
app.Server.Handler = mws[i](app.Server.Handler)
}
app.Server.ReadTimeout = time.Duration(BConfig.Listen.ServerTimeOut) * time.Second
app.Server.WriteTimeout = time.Duration(BConfig.Listen.ServerTimeOut) * time.Second
app.Server.ErrorLog = logs.GetLogger("HTTP")
// run graceful mode
if BConfig.Listen.Graceful {
httpsAddr := BConfig.Listen.HTTPSAddr
app.Server.Addr = httpsAddr
if BConfig.Listen.EnableHTTPS || BConfig.Listen.EnableMutualHTTPS {
go func() {
time.Sleep(1000 * time.Microsecond)
if BConfig.Listen.HTTPSPort != 0 {
httpsAddr = fmt.Sprintf("%s:%d", BConfig.Listen.HTTPSAddr, BConfig.Listen.HTTPSPort)
app.Server.Addr = httpsAddr
}
server := grace.NewServer(httpsAddr, app.Handlers)
server.Server.ReadTimeout = app.Server.ReadTimeout
server.Server.WriteTimeout = app.Server.WriteTimeout
if BConfig.Listen.EnableMutualHTTPS {
if err := server.ListenAndServeMutualTLS(BConfig.Listen.HTTPSCertFile, BConfig.Listen.HTTPSKeyFile, BConfig.Listen.TrustCaFile); err != nil {
logs.Critical("ListenAndServeTLS: ", err, fmt.Sprintf("%d", os.Getpid()))
time.Sleep(100 * time.Microsecond)
endRunning <- true
}
} else {
if BConfig.Listen.AutoTLS {
m := autocert.Manager{
Prompt: autocert.AcceptTOS,
HostPolicy: autocert.HostWhitelist(BConfig.Listen.Domains...),
Cache: autocert.DirCache(BConfig.Listen.TLSCacheDir),
}
app.Server.TLSConfig = &tls.Config{GetCertificate: m.GetCertificate}
BConfig.Listen.HTTPSCertFile, BConfig.Listen.HTTPSKeyFile = "", ""
}
if err := server.ListenAndServeTLS(BConfig.Listen.HTTPSCertFile, BConfig.Listen.HTTPSKeyFile); err != nil {
logs.Critical("ListenAndServeTLS: ", err, fmt.Sprintf("%d", os.Getpid()))
time.Sleep(100 * time.Microsecond)
endRunning <- true
}
}
}()
}
if BConfig.Listen.EnableHTTP {
go func() {
server := grace.NewServer(addr, app.Handlers)
server.Server.ReadTimeout = app.Server.ReadTimeout
server.Server.WriteTimeout = app.Server.WriteTimeout
if BConfig.Listen.ListenTCP4 {
server.Network = "tcp4"
}
if err := server.ListenAndServe(); err != nil {
logs.Critical("ListenAndServe: ", err, fmt.Sprintf("%d", os.Getpid()))
time.Sleep(100 * time.Microsecond)
endRunning <- true
}
}()
}
<-endRunning
return
}
// run normal mode
if BConfig.Listen.EnableHTTPS || BConfig.Listen.EnableMutualHTTPS {
go func() {
time.Sleep(1000 * time.Microsecond)
if BConfig.Listen.HTTPSPort != 0 {
app.Server.Addr = fmt.Sprintf("%s:%d", BConfig.Listen.HTTPSAddr, BConfig.Listen.HTTPSPort)
} else if BConfig.Listen.EnableHTTP {
BeeLogger.Info("Start https server error, conflict with http. Please reset https port")
return
}
logs.Info("https server Running on https://%s", app.Server.Addr)
if BConfig.Listen.AutoTLS {
m := autocert.Manager{
Prompt: autocert.AcceptTOS,
HostPolicy: autocert.HostWhitelist(BConfig.Listen.Domains...),
Cache: autocert.DirCache(BConfig.Listen.TLSCacheDir),
}
app.Server.TLSConfig = &tls.Config{GetCertificate: m.GetCertificate}
BConfig.Listen.HTTPSCertFile, BConfig.Listen.HTTPSKeyFile = "", ""
} else if BConfig.Listen.EnableMutualHTTPS {
pool := x509.NewCertPool()
data, err := ioutil.ReadFile(BConfig.Listen.TrustCaFile)
if err != nil {
BeeLogger.Info("MutualHTTPS should provide TrustCaFile")
return
}
pool.AppendCertsFromPEM(data)
app.Server.TLSConfig = &tls.Config{
ClientCAs: pool,
ClientAuth: tls.RequireAndVerifyClientCert,
}
}
if err := app.Server.ListenAndServeTLS(BConfig.Listen.HTTPSCertFile, BConfig.Listen.HTTPSKeyFile); err != nil {
logs.Critical("ListenAndServeTLS: ", err)
time.Sleep(100 * time.Microsecond)
endRunning <- true
}
}()
}
if BConfig.Listen.EnableHTTP {
go func() {
app.Server.Addr = addr
logs.Info("http server Running on http://%s", app.Server.Addr)
if BConfig.Listen.ListenTCP4 {
ln, err := net.Listen("tcp4", app.Server.Addr)
if err != nil {
logs.Critical("ListenAndServe: ", err)
time.Sleep(100 * time.Microsecond)
endRunning <- true
return
}
if err = app.Server.Serve(ln); err != nil {
logs.Critical("ListenAndServe: ", err)
time.Sleep(100 * time.Microsecond)
endRunning <- true
return
}
} else {
if err := app.Server.ListenAndServe(); err != nil {
logs.Critical("ListenAndServe: ", err)
time.Sleep(100 * time.Microsecond)
endRunning <- true
}
}
}()
}
<-endRunning
}
// Router adds a patterned controller handler to BeeApp.
// it's an alias method of App.Router.
// usage:
// simple router
// beego.Router("/admin", &admin.UserController{})
// beego.Router("/admin/index", &admin.ArticleController{})
//
// regex router
//
// beego.Router("/api/:id([0-9]+)", &controllers.RController{})
//
// custom rules
// beego.Router("/api/list",&RestController{},"*:ListFood")
// beego.Router("/api/create",&RestController{},"post:CreateFood")
// beego.Router("/api/update",&RestController{},"put:UpdateFood")
// beego.Router("/api/delete",&RestController{},"delete:DeleteFood")
func Router(rootpath string, c ControllerInterface, mappingMethods ...string) *App {
BeeApp.Handlers.Add(rootpath, c, mappingMethods...)
return BeeApp
}
// UnregisterFixedRoute unregisters the route with the specified fixedRoute. It is particularly useful
// in web applications that inherit most routes from a base webapp via the underscore
// import, and aim to overwrite only certain paths.
// The method parameter can be empty or "*" for all HTTP methods, or a particular
// method type (e.g. "GET" or "POST") for selective removal.
//
// Usage (replace "GET" with "*" for all methods):
// beego.UnregisterFixedRoute("/yourpreviouspath", "GET")
// beego.Router("/yourpreviouspath", yourControllerAddress, "get:GetNewPage")
func UnregisterFixedRoute(fixedRoute string, method string) *App {
subPaths := splitPath(fixedRoute)
if method == "" || method == "*" {
for m := range HTTPMETHOD {
if _, ok := BeeApp.Handlers.routers[m]; !ok {
continue
}
if BeeApp.Handlers.routers[m].prefix == strings.Trim(fixedRoute, "/ ") {
findAndRemoveSingleTree(BeeApp.Handlers.routers[m])
continue
}
findAndRemoveTree(subPaths, BeeApp.Handlers.routers[m], m)
}
return BeeApp
}
// Single HTTP method
um := strings.ToUpper(method)
if _, ok := BeeApp.Handlers.routers[um]; ok {
if BeeApp.Handlers.routers[um].prefix == strings.Trim(fixedRoute, "/ ") {
findAndRemoveSingleTree(BeeApp.Handlers.routers[um])
return BeeApp
}
findAndRemoveTree(subPaths, BeeApp.Handlers.routers[um], um)
}
return BeeApp
}
func findAndRemoveTree(paths []string, entryPointTree *Tree, method string) {
for i := range entryPointTree.fixrouters {
if entryPointTree.fixrouters[i].prefix == paths[0] {
if len(paths) == 1 {
if len(entryPointTree.fixrouters[i].fixrouters) > 0 {
// If the route had children subtrees, remove just the functional leaf,
// to allow children to function as before
if len(entryPointTree.fixrouters[i].leaves) > 0 {
entryPointTree.fixrouters[i].leaves[0] = nil
entryPointTree.fixrouters[i].leaves = entryPointTree.fixrouters[i].leaves[1:]
}
} else {
// Remove the *Tree from the fixrouters slice
entryPointTree.fixrouters[i] = nil
if i == len(entryPointTree.fixrouters)-1 {
entryPointTree.fixrouters = entryPointTree.fixrouters[:i]
} else {
entryPointTree.fixrouters = append(entryPointTree.fixrouters[:i], entryPointTree.fixrouters[i+1:len(entryPointTree.fixrouters)]...)
}
}
return
}
findAndRemoveTree(paths[1:], entryPointTree.fixrouters[i], method)
}
}
}
func findAndRemoveSingleTree(entryPointTree *Tree) {
if entryPointTree == nil {
return
}
if len(entryPointTree.fixrouters) > 0 {
// If the route had children subtrees, remove just the functional leaf,
// to allow children to function as before
if len(entryPointTree.leaves) > 0 {
entryPointTree.leaves[0] = nil
entryPointTree.leaves = entryPointTree.leaves[1:]
}
}
}
// Include will generate router file in the router/xxx.go from the controller's comments
// usage:
// beego.Include(&BankAccount{}, &OrderController{},&RefundController{},&ReceiptController{})
// type BankAccount struct{
// beego.Controller
// }
//
// register the function
// func (b *BankAccount)Mapping(){
// b.Mapping("ShowAccount" , b.ShowAccount)
// b.Mapping("ModifyAccount", b.ModifyAccount)
//}
//
// //@router /account/:id [get]
// func (b *BankAccount) ShowAccount(){
// //logic
// }
//
//
// //@router /account/:id [post]
// func (b *BankAccount) ModifyAccount(){
// //logic
// }
//
// the comments @router url methodlist
// url support all the function Router's pattern
// methodlist [get post head put delete options *]
func Include(cList ...ControllerInterface) *App {
BeeApp.Handlers.Include(cList...)
return BeeApp
}
// RESTRouter adds a restful controller handler to BeeApp.
// its' controller implements beego.ControllerInterface and
// defines a param "pattern/:objectId" to visit each resource.
func RESTRouter(rootpath string, c ControllerInterface) *App {
Router(rootpath, c)
Router(path.Join(rootpath, ":objectId"), c)
return BeeApp
}
// AutoRouter adds defined controller handler to BeeApp.
// it's same to App.AutoRouter.
// if beego.AddAuto(&MainContorlller{}) and MainController has methods List and Page,
// visit the url /main/list to exec List function or /main/page to exec Page function.
func AutoRouter(c ControllerInterface) *App {
BeeApp.Handlers.AddAuto(c)
return BeeApp
}
// AutoPrefix adds controller handler to BeeApp with prefix.
// it's same to App.AutoRouterWithPrefix.
// if beego.AutoPrefix("/admin",&MainContorlller{}) and MainController has methods List and Page,
// visit the url /admin/main/list to exec List function or /admin/main/page to exec Page function.
func AutoPrefix(prefix string, c ControllerInterface) *App {
BeeApp.Handlers.AddAutoPrefix(prefix, c)
return BeeApp
}
// Get used to register router for Get method
// usage:
// beego.Get("/", func(ctx *context.Context){
// ctx.Output.Body("hello world")
// })
func Get(rootpath string, f FilterFunc) *App {
BeeApp.Handlers.Get(rootpath, f)
return BeeApp
}
// Post used to register router for Post method
// usage:
// beego.Post("/api", func(ctx *context.Context){
// ctx.Output.Body("hello world")
// })
func Post(rootpath string, f FilterFunc) *App {
BeeApp.Handlers.Post(rootpath, f)
return BeeApp
}
// Delete used to register router for Delete method
// usage:
// beego.Delete("/api", func(ctx *context.Context){
// ctx.Output.Body("hello world")
// })
func Delete(rootpath string, f FilterFunc) *App {
BeeApp.Handlers.Delete(rootpath, f)
return BeeApp
}
// Put used to register router for Put method
// usage:
// beego.Put("/api", func(ctx *context.Context){
// ctx.Output.Body("hello world")
// })
func Put(rootpath string, f FilterFunc) *App {
BeeApp.Handlers.Put(rootpath, f)
return BeeApp
}
// Head used to register router for Head method
// usage:
// beego.Head("/api", func(ctx *context.Context){
// ctx.Output.Body("hello world")
// })
func Head(rootpath string, f FilterFunc) *App {
BeeApp.Handlers.Head(rootpath, f)
return BeeApp
}
// Options used to register router for Options method
// usage:
// beego.Options("/api", func(ctx *context.Context){
// ctx.Output.Body("hello world")
// })
func Options(rootpath string, f FilterFunc) *App {
BeeApp.Handlers.Options(rootpath, f)
return BeeApp
}
// Patch used to register router for Patch method
// usage:
// beego.Patch("/api", func(ctx *context.Context){
// ctx.Output.Body("hello world")
// })
func Patch(rootpath string, f FilterFunc) *App {
BeeApp.Handlers.Patch(rootpath, f)
return BeeApp
}
// Any used to register router for all methods
// usage:
// beego.Any("/api", func(ctx *context.Context){
// ctx.Output.Body("hello world")
// })
func Any(rootpath string, f FilterFunc) *App {
BeeApp.Handlers.Any(rootpath, f)
return BeeApp
}
// Handler used to register a Handler router
// usage:
// beego.Handler("/api", http.HandlerFunc(func (w http.ResponseWriter, r *http.Request) {
// fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
// }))
func Handler(rootpath string, h http.Handler, options ...interface{}) *App {
BeeApp.Handlers.Handler(rootpath, h, options...)
return BeeApp
}
// InsertFilter adds a FilterFunc with pattern condition and action constant.
// The pos means action constant including
// beego.BeforeStatic, beego.BeforeRouter, beego.BeforeExec, beego.AfterExec and beego.FinishRouter.
// The bool params is for setting the returnOnOutput value (false allows multiple filters to execute)
func InsertFilter(pattern string, pos int, filter FilterFunc, params ...bool) *App {
BeeApp.Handlers.InsertFilter(pattern, pos, filter, params...)
return BeeApp
}

123
vender/github.com/astaxie/beego/beego.go
View File

@ -1,123 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package beego
import (
"os"
"path/filepath"
"strconv"
"strings"
)
const (
// VERSION represent beego web framework version.
VERSION = "1.10.1"
// DEV is for develop
DEV = "dev"
// PROD is for production
PROD = "prod"
)
// Map shortcut
type M map[string]interface{}
// Hook function to run
type hookfunc func() error
var (
hooks = make([]hookfunc, 0) //hook function slice to store the hookfunc
)
// AddAPPStartHook is used to register the hookfunc
// The hookfuncs will run in beego.Run()
// such as initiating session , starting middleware , building template, starting admin control and so on.
func AddAPPStartHook(hf ...hookfunc) {
hooks = append(hooks, hf...)
}
// Run beego application.
// beego.Run() default run on HttpPort
// beego.Run("localhost")
// beego.Run(":8089")
// beego.Run("127.0.0.1:8089")
func Run(params ...string) {
InitBeforeHTTPRun()
if len(params) > 0 && params[0] != "" {
strs := strings.Split(params[0], ":")
if len(strs) > 0 && strs[0] != "" {
BConfig.Listen.HTTPAddr = strs[0]
}
if len(strs) > 1 && strs[1] != "" {
BConfig.Listen.HTTPPort, _ = strconv.Atoi(strs[1])
}
BConfig.Listen.Domains = params
}
BeeApp.Run()
}
// RunWithMiddleWares Run beego application with middlewares.
func RunWithMiddleWares(addr string, mws ...MiddleWare) {
InitBeforeHTTPRun()
strs := strings.Split(addr, ":")
if len(strs) > 0 && strs[0] != "" {
BConfig.Listen.HTTPAddr = strs[0]
BConfig.Listen.Domains = []string{strs[0]}
}
if len(strs) > 1 && strs[1] != "" {
BConfig.Listen.HTTPPort, _ = strconv.Atoi(strs[1])
}
BeeApp.Run(mws...)
}
func InitBeforeHTTPRun() {
//init hooks
AddAPPStartHook(
registerMime,
registerDefaultErrorHandler,
registerSession,
registerTemplate,
registerAdmin,
registerGzip,
)
for _, hk := range hooks {
if err := hk(); err != nil {
panic(err)
}
}
}
// TestBeegoInit is for test package init
func TestBeegoInit(ap string) {
path := filepath.Join(ap, "conf", "app.conf")
os.Chdir(ap)
InitBeegoBeforeTest(path)
}
// InitBeegoBeforeTest is for test package init
func InitBeegoBeforeTest(appConfigPath string) {
if err := LoadAppConfig(appConfigProvider, appConfigPath); err != nil {
panic(err)
}
BConfig.RunMode = "test"
InitBeforeHTTPRun()
}

510
vender/github.com/astaxie/beego/config.go
View File

@ -1,510 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package beego
import (
"fmt"
"os"
"path/filepath"
"reflect"
"runtime"
"strings"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/context"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/session"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/utils"
)
// Config is the main struct for BConfig
type Config struct {
AppName string //Application name
RunMode string //Running Mode: dev | prod
RouterCaseSensitive bool
ServerName string
RecoverPanic bool
RecoverFunc func(*context.Context)
CopyRequestBody bool
EnableGzip bool
MaxMemory int64
EnableErrorsShow bool
EnableErrorsRender bool
Listen Listen
WebConfig WebConfig
Log LogConfig
}
// Listen holds for http and https related config
type Listen struct {
Graceful bool // Graceful means use graceful module to start the server
ServerTimeOut int64
ListenTCP4 bool
EnableHTTP bool
HTTPAddr string
HTTPPort int
AutoTLS bool
Domains []string
TLSCacheDir string
EnableHTTPS bool
EnableMutualHTTPS bool
HTTPSAddr string
HTTPSPort int
HTTPSCertFile string
HTTPSKeyFile string
TrustCaFile string
EnableAdmin bool
AdminAddr string
AdminPort int
EnableFcgi bool
EnableStdIo bool // EnableStdIo works with EnableFcgi Use FCGI via standard I/O
}
// WebConfig holds web related config
type WebConfig struct {
AutoRender bool
EnableDocs bool
FlashName string
FlashSeparator string
DirectoryIndex bool
StaticDir map[string]string
StaticExtensionsToGzip []string
TemplateLeft string
TemplateRight string
ViewsPath string
EnableXSRF bool
XSRFKey string
XSRFExpire int
Session SessionConfig
}
// SessionConfig holds session related config
type SessionConfig struct {
SessionOn bool
SessionProvider string
SessionName string
SessionGCMaxLifetime int64
SessionProviderConfig string
SessionCookieLifeTime int
SessionAutoSetCookie bool
SessionDomain string
SessionDisableHTTPOnly bool // used to allow for cross domain cookies/javascript cookies.
SessionEnableSidInHTTPHeader bool // enable store/get the sessionId into/from http headers
SessionNameInHTTPHeader string
SessionEnableSidInURLQuery bool // enable get the sessionId from Url Query params
}
// LogConfig holds Log related config
type LogConfig struct {
AccessLogs bool
EnableStaticLogs bool //log static files requests default: false
AccessLogsFormat string //access log format: JSON_FORMAT, APACHE_FORMAT or empty string
FileLineNum bool
Outputs map[string]string // Store Adaptor : config
}
var (
// BConfig is the default config for Application
BConfig *Config
// AppConfig is the instance of Config, store the config information from file
AppConfig *beegoAppConfig
// AppPath is the absolute path to the app
AppPath string
// GlobalSessions is the instance for the session manager
GlobalSessions *session.Manager
// appConfigPath is the path to the config files
appConfigPath string
// appConfigProvider is the provider for the config, default is ini
appConfigProvider = "ini"
)
func init() {
BConfig = newBConfig()
var err error
if AppPath, err = filepath.Abs(filepath.Dir(os.Args[0])); err != nil {
panic(err)
}
workPath, err := os.Getwd()
if err != nil {
panic(err)
}
var filename = "app.conf"
if os.Getenv("BEEGO_RUNMODE") != "" {
filename = os.Getenv("BEEGO_RUNMODE") + ".app.conf"
}
appConfigPath = filepath.Join(workPath, "conf", filename)
if !utils.FileExists(appConfigPath) {
appConfigPath = filepath.Join(AppPath, "conf", filename)
if !utils.FileExists(appConfigPath) {
AppConfig = &beegoAppConfig{innerConfig: config.NewFakeConfig()}
return
}
}
if err = parseConfig(appConfigPath); err != nil {
panic(err)
}
}
func recoverPanic(ctx *context.Context) {
if err := recover(); err != nil {
if err == ErrAbort {
return
}
if !BConfig.RecoverPanic {
panic(err)
}
if BConfig.EnableErrorsShow {
if _, ok := ErrorMaps[fmt.Sprint(err)]; ok {
exception(fmt.Sprint(err), ctx)
return
}
}
var stack string
logs.Critical("the request url is ", ctx.Input.URL())
logs.Critical("Handler crashed with error", err)
for i := 1; ; i++ {
_, file, line, ok := runtime.Caller(i)
if !ok {
break
}
logs.Critical(fmt.Sprintf("%s:%d", file, line))
stack = stack + fmt.Sprintln(fmt.Sprintf("%s:%d", file, line))
}
if BConfig.RunMode == DEV && BConfig.EnableErrorsRender {
showErr(err, ctx, stack)
}
if ctx.Output.Status != 0 {
ctx.ResponseWriter.WriteHeader(ctx.Output.Status)
} else {
ctx.ResponseWriter.WriteHeader(500)
}
}
}
func newBConfig() *Config {
return &Config{
AppName: "beego",
RunMode: PROD,
RouterCaseSensitive: true,
ServerName: "beegoServer:" + VERSION,
RecoverPanic: true,
RecoverFunc: recoverPanic,
CopyRequestBody: false,
EnableGzip: false,
MaxMemory: 1 << 26, //64MB
EnableErrorsShow: true,
EnableErrorsRender: true,
Listen: Listen{
Graceful: false,
ServerTimeOut: 0,
ListenTCP4: false,
EnableHTTP: true,
AutoTLS: false,
Domains: []string{},
TLSCacheDir: ".",
HTTPAddr: "",
HTTPPort: 8080,
EnableHTTPS: false,
HTTPSAddr: "",
HTTPSPort: 10443,
HTTPSCertFile: "",
HTTPSKeyFile: "",
EnableAdmin: false,
AdminAddr: "",
AdminPort: 8088,
EnableFcgi: false,
EnableStdIo: false,
},
WebConfig: WebConfig{
AutoRender: true,
EnableDocs: false,
FlashName: "BEEGO_FLASH",
FlashSeparator: "BEEGOFLASH",
DirectoryIndex: false,
StaticDir: map[string]string{"/static": "static"},
StaticExtensionsToGzip: []string{".css", ".js"},
TemplateLeft: "{{",
TemplateRight: "}}",
ViewsPath: "views",
EnableXSRF: false,
XSRFKey: "beegoxsrf",
XSRFExpire: 0,
Session: SessionConfig{
SessionOn: false,
SessionProvider: "memory",
SessionName: "beegosessionID",
SessionGCMaxLifetime: 3600,
SessionProviderConfig: "",
SessionDisableHTTPOnly: false,
SessionCookieLifeTime: 0, //set cookie default is the browser life
SessionAutoSetCookie: true,
SessionDomain: "",
SessionEnableSidInHTTPHeader: false, // enable store/get the sessionId into/from http headers
SessionNameInHTTPHeader: "Beegosessionid",
SessionEnableSidInURLQuery: false, // enable get the sessionId from Url Query params
},
},
Log: LogConfig{
AccessLogs: false,
EnableStaticLogs: false,
AccessLogsFormat: "APACHE_FORMAT",
FileLineNum: true,
Outputs: map[string]string{"console": ""},
},
}
}
// now only support ini, next will support json.
func parseConfig(appConfigPath string) (err error) {
AppConfig, err = newAppConfig(appConfigProvider, appConfigPath)
if err != nil {
return err
}
return assignConfig(AppConfig)
}
func assignConfig(ac config.Configer) error {
for _, i := range []interface{}{BConfig, &BConfig.Listen, &BConfig.WebConfig, &BConfig.Log, &BConfig.WebConfig.Session} {
assignSingleConfig(i, ac)
}
// set the run mode first
if envRunMode := os.Getenv("BEEGO_RUNMODE"); envRunMode != "" {
BConfig.RunMode = envRunMode
} else if runMode := ac.String("RunMode"); runMode != "" {
BConfig.RunMode = runMode
}
if sd := ac.String("StaticDir"); sd != "" {
BConfig.WebConfig.StaticDir = map[string]string{}
sds := strings.Fields(sd)
for _, v := range sds {
if url2fsmap := strings.SplitN(v, ":", 2); len(url2fsmap) == 2 {
BConfig.WebConfig.StaticDir["/"+strings.Trim(url2fsmap[0], "/")] = url2fsmap[1]
} else {
BConfig.WebConfig.StaticDir["/"+strings.Trim(url2fsmap[0], "/")] = url2fsmap[0]
}
}
}
if sgz := ac.String("StaticExtensionsToGzip"); sgz != "" {
extensions := strings.Split(sgz, ",")
fileExts := []string{}
for _, ext := range extensions {
ext = strings.TrimSpace(ext)
if ext == "" {
continue
}
if !strings.HasPrefix(ext, ".") {
ext = "." + ext
}
fileExts = append(fileExts, ext)
}
if len(fileExts) > 0 {
BConfig.WebConfig.StaticExtensionsToGzip = fileExts
}
}
if lo := ac.String("LogOutputs"); lo != "" {
// if lo is not nil or empty
// means user has set his own LogOutputs
// clear the default setting to BConfig.Log.Outputs
BConfig.Log.Outputs = make(map[string]string)
los := strings.Split(lo, ";")
for _, v := range los {
if logType2Config := strings.SplitN(v, ",", 2); len(logType2Config) == 2 {
BConfig.Log.Outputs[logType2Config[0]] = logType2Config[1]
} else {
continue
}
}
}
//init log
//logs.Reset()
//for adaptor, config := range BConfig.Log.Outputs {
// err := logs.SetLogger(adaptor, config)
// if err != nil {
// fmt.Fprintln(os.Stderr, fmt.Sprintf("%s with the config %q got err:%s", adaptor, config, err.Error()))
// }
//}
//logs.SetLogFuncCall(BConfig.Log.FileLineNum)
return nil
}
func assignSingleConfig(p interface{}, ac config.Configer) {
pt := reflect.TypeOf(p)
if pt.Kind() != reflect.Ptr {
return
}
pt = pt.Elem()
if pt.Kind() != reflect.Struct {
return
}
pv := reflect.ValueOf(p).Elem()
for i := 0; i < pt.NumField(); i++ {
pf := pv.Field(i)
if !pf.CanSet() {
continue
}
name := pt.Field(i).Name
switch pf.Kind() {
case reflect.String:
pf.SetString(ac.DefaultString(name, pf.String()))
case reflect.Int, reflect.Int64:
pf.SetInt(ac.DefaultInt64(name, pf.Int()))
case reflect.Bool:
pf.SetBool(ac.DefaultBool(name, pf.Bool()))
case reflect.Struct:
default:
//do nothing here
}
}
}
// LoadAppConfig allow developer to apply a config file
func LoadAppConfig(adapterName, configPath string) error {
absConfigPath, err := filepath.Abs(configPath)
if err != nil {
return err
}
if !utils.FileExists(absConfigPath) {
return fmt.Errorf("the target config file: %s don't exist", configPath)
}
appConfigPath = absConfigPath
appConfigProvider = adapterName
return parseConfig(appConfigPath)
}
type beegoAppConfig struct {
innerConfig config.Configer
}
func newAppConfig(appConfigProvider, appConfigPath string) (*beegoAppConfig, error) {
ac, err := config.NewConfig(appConfigProvider, appConfigPath)
if err != nil {
return nil, err
}
return &beegoAppConfig{ac}, nil
}
func (b *beegoAppConfig) Set(key, val string) error {
if err := b.innerConfig.Set(BConfig.RunMode+"::"+key, val); err != nil {
return err
}
return b.innerConfig.Set(key, val)
}
func (b *beegoAppConfig) String(key string) string {
if v := b.innerConfig.String(BConfig.RunMode + "::" + key); v != "" {
return v
}
return b.innerConfig.String(key)
}
func (b *beegoAppConfig) Strings(key string) []string {
if v := b.innerConfig.Strings(BConfig.RunMode + "::" + key); len(v) > 0 {
return v
}
return b.innerConfig.Strings(key)
}
func (b *beegoAppConfig) Int(key string) (int, error) {
if v, err := b.innerConfig.Int(BConfig.RunMode + "::" + key); err == nil {
return v, nil
}
return b.innerConfig.Int(key)
}
func (b *beegoAppConfig) Int64(key string) (int64, error) {
if v, err := b.innerConfig.Int64(BConfig.RunMode + "::" + key); err == nil {
return v, nil
}
return b.innerConfig.Int64(key)
}
func (b *beegoAppConfig) Bool(key string) (bool, error) {
if v, err := b.innerConfig.Bool(BConfig.RunMode + "::" + key); err == nil {
return v, nil
}
return b.innerConfig.Bool(key)
}
func (b *beegoAppConfig) Float(key string) (float64, error) {
if v, err := b.innerConfig.Float(BConfig.RunMode + "::" + key); err == nil {
return v, nil
}
return b.innerConfig.Float(key)
}
func (b *beegoAppConfig) DefaultString(key string, defaultVal string) string {
if v := b.String(key); v != "" {
return v
}
return defaultVal
}
func (b *beegoAppConfig) DefaultStrings(key string, defaultVal []string) []string {
if v := b.Strings(key); len(v) != 0 {
return v
}
return defaultVal
}
func (b *beegoAppConfig) DefaultInt(key string, defaultVal int) int {
if v, err := b.Int(key); err == nil {
return v
}
return defaultVal
}
func (b *beegoAppConfig) DefaultInt64(key string, defaultVal int64) int64 {
if v, err := b.Int64(key); err == nil {
return v
}
return defaultVal
}
func (b *beegoAppConfig) DefaultBool(key string, defaultVal bool) bool {
if v, err := b.Bool(key); err == nil {
return v
}
return defaultVal
}
func (b *beegoAppConfig) DefaultFloat(key string, defaultVal float64) float64 {
if v, err := b.Float(key); err == nil {
return v
}
return defaultVal
}
func (b *beegoAppConfig) DIY(key string) (interface{}, error) {
return b.innerConfig.DIY(key)
}
func (b *beegoAppConfig) GetSection(section string) (map[string]string, error) {
return b.innerConfig.GetSection(section)
}
func (b *beegoAppConfig) SaveConfigFile(filename string) error {
return b.innerConfig.SaveConfigFile(filename)
}

242
vender/github.com/astaxie/beego/config/config.go
View File

@ -1,242 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package config is used to parse config.
// Usage:
// import "github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
//Examples.
//
// cnf, err := config.NewConfig("ini", "config.conf")
//
// cnf APIS:
//
// cnf.Set(key, val string) error
// cnf.String(key string) string
// cnf.Strings(key string) []string
// cnf.Int(key string) (int, error)
// cnf.Int64(key string) (int64, error)
// cnf.Bool(key string) (bool, error)
// cnf.Float(key string) (float64, error)
// cnf.DefaultString(key string, defaultVal string) string
// cnf.DefaultStrings(key string, defaultVal []string) []string
// cnf.DefaultInt(key string, defaultVal int) int
// cnf.DefaultInt64(key string, defaultVal int64) int64
// cnf.DefaultBool(key string, defaultVal bool) bool
// cnf.DefaultFloat(key string, defaultVal float64) float64
// cnf.DIY(key string) (interface{}, error)
// cnf.GetSection(section string) (map[string]string, error)
// cnf.SaveConfigFile(filename string) error
//More docs http://beego.me/docs/module/config.md
package config
import (
"fmt"
"os"
"reflect"
"time"
)
// Configer defines how to get and set value from configuration raw data.
type Configer interface {
Set(key, val string) error //support section::key type in given key when using ini type.
String(key string) string //support section::key type in key string when using ini and json type; Int,Int64,Bool,Float,DIY are same.
Strings(key string) []string //get string slice
Int(key string) (int, error)
Int64(key string) (int64, error)
Bool(key string) (bool, error)
Float(key string) (float64, error)
DefaultString(key string, defaultVal string) string // support section::key type in key string when using ini and json type; Int,Int64,Bool,Float,DIY are same.
DefaultStrings(key string, defaultVal []string) []string //get string slice
DefaultInt(key string, defaultVal int) int
DefaultInt64(key string, defaultVal int64) int64
DefaultBool(key string, defaultVal bool) bool
DefaultFloat(key string, defaultVal float64) float64
DIY(key string) (interface{}, error)
GetSection(section string) (map[string]string, error)
SaveConfigFile(filename string) error
}
// Config is the adapter interface for parsing config file to get raw data to Configer.
type Config interface {
Parse(key string) (Configer, error)
ParseData(data []byte) (Configer, error)
}
var adapters = make(map[string]Config)
// Register makes a config adapter available by the adapter name.
// If Register is called twice with the same name or if driver is nil,
// it panics.
func Register(name string, adapter Config) {
if adapter == nil {
panic("config: Register adapter is nil")
}
if _, ok := adapters[name]; ok {
panic("config: Register called twice for adapter " + name)
}
adapters[name] = adapter
}
// NewConfig adapterName is ini/json/xml/yaml.
// filename is the config file path.
func NewConfig(adapterName, filename string) (Configer, error) {
adapter, ok := adapters[adapterName]
if !ok {
return nil, fmt.Errorf("config: unknown adaptername %q (forgotten import?)", adapterName)
}
return adapter.Parse(filename)
}
// NewConfigData adapterName is ini/json/xml/yaml.
// data is the config data.
func NewConfigData(adapterName string, data []byte) (Configer, error) {
adapter, ok := adapters[adapterName]
if !ok {
return nil, fmt.Errorf("config: unknown adaptername %q (forgotten import?)", adapterName)
}
return adapter.ParseData(data)
}
// ExpandValueEnvForMap convert all string value with environment variable.
func ExpandValueEnvForMap(m map[string]interface{}) map[string]interface{} {
for k, v := range m {
switch value := v.(type) {
case string:
m[k] = ExpandValueEnv(value)
case map[string]interface{}:
m[k] = ExpandValueEnvForMap(value)
case map[string]string:
for k2, v2 := range value {
value[k2] = ExpandValueEnv(v2)
}
m[k] = value
}
}
return m
}
// ExpandValueEnv returns value of convert with environment variable.
//
// Return environment variable if value start with "${" and end with "}".
// Return default value if environment variable is empty or not exist.
//
// It accept value formats "${env}" , "${env||}}" , "${env||defaultValue}" , "defaultvalue".
// Examples:
// v1 := config.ExpandValueEnv("${GOPATH}") // return the GOPATH environment variable.
// v2 := config.ExpandValueEnv("${GOAsta||/usr/local/go}") // return the default value "/usr/local/go/".
// v3 := config.ExpandValueEnv("Astaxie") // return the value "Astaxie".
func ExpandValueEnv(value string) (realValue string) {
realValue = value
vLen := len(value)
// 3 = ${}
if vLen < 3 {
return
}
// Need start with "${" and end with "}", then return.
if value[0] != '$' || value[1] != '{' || value[vLen-1] != '}' {
return
}
key := ""
defaultV := ""
// value start with "${"
for i := 2; i < vLen; i++ {
if value[i] == '|' && (i+1 < vLen && value[i+1] == '|') {
key = value[2:i]
defaultV = value[i+2 : vLen-1] // other string is default value.
break
} else if value[i] == '}' {
key = value[2:i]
break
}
}
realValue = os.Getenv(key)
if realValue == "" {
realValue = defaultV
}
return
}
// ParseBool returns the boolean value represented by the string.
//
// It accepts 1, 1.0, t, T, TRUE, true, True, YES, yes, Yes,Y, y, ON, on, On,
// 0, 0.0, f, F, FALSE, false, False, NO, no, No, N,n, OFF, off, Off.
// Any other value returns an error.
func ParseBool(val interface{}) (value bool, err error) {
if val != nil {
switch v := val.(type) {
case bool:
return v, nil
case string:
switch v {
case "1", "t", "T", "true", "TRUE", "True", "YES", "yes", "Yes", "Y", "y", "ON", "on", "On":
return true, nil
case "0", "f", "F", "false", "FALSE", "False", "NO", "no", "No", "N", "n", "OFF", "off", "Off":
return false, nil
}
case int8, int32, int64:
strV := fmt.Sprintf("%d", v)
if strV == "1" {
return true, nil
} else if strV == "0" {
return false, nil
}
case float64:
if v == 1.0 {
return true, nil
} else if v == 0.0 {
return false, nil
}
}
return false, fmt.Errorf("parsing %q: invalid syntax", val)
}
return false, fmt.Errorf("parsing <nil>: invalid syntax")
}
// ToString converts values of any type to string.
func ToString(x interface{}) string {
switch y := x.(type) {
// Handle dates with special logic
// This needs to come above the fmt.Stringer
// test since time.Time's have a .String()
// method
case time.Time:
return y.Format("A Monday")
// Handle type string
case string:
return y
// Handle type with .String() method
case fmt.Stringer:
return y.String()
// Handle type with .Error() method
case error:
return y.Error()
}
// Handle named string type
if v := reflect.ValueOf(x); v.Kind() == reflect.String {
return v.String()
}
// Fallback to fmt package for anything else like numeric types
return fmt.Sprint(x)
}

55
vender/github.com/astaxie/beego/config/config_test.go
View File

@ -1,55 +0,0 @@
// Copyright 2016 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package config
import (
"os"
"testing"
)
func TestExpandValueEnv(t *testing.T) {
testCases := []struct {
item string
want string
}{
{"", ""},
{"$", "$"},
{"{", "{"},
{"{}", "{}"},
{"${}", ""},
{"${|}", ""},
{"${}", ""},
{"${{}}", ""},
{"${{||}}", "}"},
{"${pwd||}", ""},
{"${pwd||}", ""},
{"${pwd||}", ""},
{"${pwd||}}", "}"},
{"${pwd||{{||}}}", "{{||}}"},
{"${GOPATH}", os.Getenv("GOPATH")},
{"${GOPATH||}", os.Getenv("GOPATH")},
{"${GOPATH||root}", os.Getenv("GOPATH")},
{"${GOPATH_NOT||root}", "root"},
{"${GOPATH_NOT||||root}", "||root"},
}
for _, c := range testCases {
if got := ExpandValueEnv(c.item); got != c.want {
t.Errorf("expand value error, item %q want %q, got %q", c.item, c.want, got)
}
}
}

87
vender/github.com/astaxie/beego/config/env/env.go vendored
View File

@ -1,87 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
// Copyright 2017 Faissal Elamraoui. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package env is used to parse environment.
package env
import (
"fmt"
"os"
"strings"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/utils"
)
var env *utils.BeeMap
func init() {
env = utils.NewBeeMap()
for _, e := range os.Environ() {
splits := strings.Split(e, "=")
env.Set(splits[0], os.Getenv(splits[0]))
}
}
// Get returns a value by key.
// If the key does not exist, the default value will be returned.
func Get(key string, defVal string) string {
if val := env.Get(key); val != nil {
return val.(string)
}
return defVal
}
// MustGet returns a value by key.
// If the key does not exist, it will return an error.
func MustGet(key string) (string, error) {
if val := env.Get(key); val != nil {
return val.(string), nil
}
return "", fmt.Errorf("no env variable with %s", key)
}
// Set sets a value in the ENV copy.
// This does not affect the child process environment.
func Set(key string, value string) {
env.Set(key, value)
}
// MustSet sets a value in the ENV copy and the child process environment.
// It returns an error in case the set operation failed.
func MustSet(key string, value string) error {
err := os.Setenv(key, value)
if err != nil {
return err
}
env.Set(key, value)
return nil
}
// GetAll returns all keys/values in the current child process environment.
func GetAll() map[string]string {
items := env.Items()
envs := make(map[string]string, env.Count())
for key, val := range items {
switch key := key.(type) {
case string:
switch val := val.(type) {
case string:
envs[key] = val
}
}
}
return envs
}

75
vender/github.com/astaxie/beego/config/env/env_test.go vendored
View File

@ -1,75 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
// Copyright 2017 Faissal Elamraoui. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package env
import (
"os"
"testing"
)
func TestEnvGet(t *testing.T) {
gopath := Get("GOPATH", "")
if gopath != os.Getenv("GOPATH") {
t.Error("expected GOPATH not empty.")
}
noExistVar := Get("NOEXISTVAR", "foo")
if noExistVar != "foo" {
t.Errorf("expected NOEXISTVAR to equal foo, got %s.", noExistVar)
}
}
func TestEnvMustGet(t *testing.T) {
gopath, err := MustGet("GOPATH")
if err != nil {
t.Error(err)
}
if gopath != os.Getenv("GOPATH") {
t.Errorf("expected GOPATH to be the same, got %s.", gopath)
}
_, err = MustGet("NOEXISTVAR")
if err == nil {
t.Error("expected error to be non-nil")
}
}
func TestEnvSet(t *testing.T) {
Set("MYVAR", "foo")
myVar := Get("MYVAR", "bar")
if myVar != "foo" {
t.Errorf("expected MYVAR to equal foo, got %s.", myVar)
}
}
func TestEnvMustSet(t *testing.T) {
err := MustSet("FOO", "bar")
if err != nil {
t.Error(err)
}
fooVar := os.Getenv("FOO")
if fooVar != "bar" {
t.Errorf("expected FOO variable to equal bar, got %s.", fooVar)
}
}
func TestEnvGetAll(t *testing.T) {
envMap := GetAll()
if len(envMap) == 0 {
t.Error("expected environment not empty.")
}
}

134
vender/github.com/astaxie/beego/config/fake.go
View File

@ -1,134 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package config
import (
"errors"
"strconv"
"strings"
)
type fakeConfigContainer struct {
data map[string]string
}
func (c *fakeConfigContainer) getData(key string) string {
return c.data[strings.ToLower(key)]
}
func (c *fakeConfigContainer) Set(key, val string) error {
c.data[strings.ToLower(key)] = val
return nil
}
func (c *fakeConfigContainer) String(key string) string {
return c.getData(key)
}
func (c *fakeConfigContainer) DefaultString(key string, defaultval string) string {
v := c.String(key)
if v == "" {
return defaultval
}
return v
}
func (c *fakeConfigContainer) Strings(key string) []string {
v := c.String(key)
if v == "" {
return nil
}
return strings.Split(v, ";")
}
func (c *fakeConfigContainer) DefaultStrings(key string, defaultval []string) []string {
v := c.Strings(key)
if v == nil {
return defaultval
}
return v
}
func (c *fakeConfigContainer) Int(key string) (int, error) {
return strconv.Atoi(c.getData(key))
}
func (c *fakeConfigContainer) DefaultInt(key string, defaultval int) int {
v, err := c.Int(key)
if err != nil {
return defaultval
}
return v
}
func (c *fakeConfigContainer) Int64(key string) (int64, error) {
return strconv.ParseInt(c.getData(key), 10, 64)
}
func (c *fakeConfigContainer) DefaultInt64(key string, defaultval int64) int64 {
v, err := c.Int64(key)
if err != nil {
return defaultval
}
return v
}
func (c *fakeConfigContainer) Bool(key string) (bool, error) {
return ParseBool(c.getData(key))
}
func (c *fakeConfigContainer) DefaultBool(key string, defaultval bool) bool {
v, err := c.Bool(key)
if err != nil {
return defaultval
}
return v
}
func (c *fakeConfigContainer) Float(key string) (float64, error) {
return strconv.ParseFloat(c.getData(key), 64)
}
func (c *fakeConfigContainer) DefaultFloat(key string, defaultval float64) float64 {
v, err := c.Float(key)
if err != nil {
return defaultval
}
return v
}
func (c *fakeConfigContainer) DIY(key string) (interface{}, error) {
if v, ok := c.data[strings.ToLower(key)]; ok {
return v, nil
}
return nil, errors.New("key not find")
}
func (c *fakeConfigContainer) GetSection(section string) (map[string]string, error) {
return nil, errors.New("not implement in the fakeConfigContainer")
}
func (c *fakeConfigContainer) SaveConfigFile(filename string) error {
return errors.New("not implement in the fakeConfigContainer")
}
var _ Configer = new(fakeConfigContainer)
// NewFakeConfig return a fake Configer
func NewFakeConfig() Configer {
return &fakeConfigContainer{
data: make(map[string]string),
}
}

504
vender/github.com/astaxie/beego/config/ini.go
View File

@ -1,504 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package config
import (
"bufio"
"bytes"
"errors"
"io"
"io/ioutil"
"os"
"os/user"
"path/filepath"
"strconv"
"strings"
"sync"
)
var (
defaultSection = "default" // default section means if some ini items not in a section, make them in default section,
bNumComment = []byte{'#'} // number signal
bSemComment = []byte{';'} // semicolon signal
bEmpty = []byte{}
bEqual = []byte{'='} // equal signal
bDQuote = []byte{'"'} // quote signal
sectionStart = []byte{'['} // section start signal
sectionEnd = []byte{']'} // section end signal
lineBreak = "\n"
)
// IniConfig implements Config to parse ini file.
type IniConfig struct {
}
// Parse creates a new Config and parses the file configuration from the named file.
func (ini *IniConfig) Parse(name string) (Configer, error) {
return ini.parseFile(name)
}
func (ini *IniConfig) parseFile(name string) (*IniConfigContainer, error) {
data, err := ioutil.ReadFile(name)
if err != nil {
return nil, err
}
return ini.parseData(filepath.Dir(name), data)
}
func (ini *IniConfig) parseData(dir string, data []byte) (*IniConfigContainer, error) {
cfg := &IniConfigContainer{
data: make(map[string]map[string]string),
sectionComment: make(map[string]string),
keyComment: make(map[string]string),
RWMutex: sync.RWMutex{},
}
cfg.Lock()
defer cfg.Unlock()
var comment bytes.Buffer
buf := bufio.NewReader(bytes.NewBuffer(data))
// check the BOM
head, err := buf.Peek(3)
if err == nil && head[0] == 239 && head[1] == 187 && head[2] == 191 {
for i := 1; i <= 3; i++ {
buf.ReadByte()
}
}
section := defaultSection
tmpBuf := bytes.NewBuffer(nil)
for {
tmpBuf.Reset()
shouldBreak := false
for {
tmp, isPrefix, err := buf.ReadLine()
if err == io.EOF {
shouldBreak = true
break
}
//It might be a good idea to throw a error on all unknonw errors?
if _, ok := err.(*os.PathError); ok {
return nil, err
}
tmpBuf.Write(tmp)
if isPrefix {
continue
}
if !isPrefix {
break
}
}
if shouldBreak {
break
}
line := tmpBuf.Bytes()
line = bytes.TrimSpace(line)
if bytes.Equal(line, bEmpty) {
continue
}
var bComment []byte
switch {
case bytes.HasPrefix(line, bNumComment):
bComment = bNumComment
case bytes.HasPrefix(line, bSemComment):
bComment = bSemComment
}
if bComment != nil {
line = bytes.TrimLeft(line, string(bComment))
// Need append to a new line if multi-line comments.
if comment.Len() > 0 {
comment.WriteByte('\n')
}
comment.Write(line)
continue
}
if bytes.HasPrefix(line, sectionStart) && bytes.HasSuffix(line, sectionEnd) {
section = strings.ToLower(string(line[1 : len(line)-1])) // section name case insensitive
if comment.Len() > 0 {
cfg.sectionComment[section] = comment.String()
comment.Reset()
}
if _, ok := cfg.data[section]; !ok {
cfg.data[section] = make(map[string]string)
}
continue
}
if _, ok := cfg.data[section]; !ok {
cfg.data[section] = make(map[string]string)
}
keyValue := bytes.SplitN(line, bEqual, 2)
key := string(bytes.TrimSpace(keyValue[0])) // key name case insensitive
key = strings.ToLower(key)
// handle include "other.conf"
if len(keyValue) == 1 && strings.HasPrefix(key, "include") {
includefiles := strings.Fields(key)
if includefiles[0] == "include" && len(includefiles) == 2 {
otherfile := strings.Trim(includefiles[1], "\"")
if !filepath.IsAbs(otherfile) {
otherfile = filepath.Join(dir, otherfile)
}
i, err := ini.parseFile(otherfile)
if err != nil {
return nil, err
}
for sec, dt := range i.data {
if _, ok := cfg.data[sec]; !ok {
cfg.data[sec] = make(map[string]string)
}
for k, v := range dt {
cfg.data[sec][k] = v
}
}
for sec, comm := range i.sectionComment {
cfg.sectionComment[sec] = comm
}
for k, comm := range i.keyComment {
cfg.keyComment[k] = comm
}
continue
}
}
if len(keyValue) != 2 {
return nil, errors.New("read the content error: \"" + string(line) + "\", should key = val")
}
val := bytes.TrimSpace(keyValue[1])
if bytes.HasPrefix(val, bDQuote) {
val = bytes.Trim(val, `"`)
}
cfg.data[section][key] = ExpandValueEnv(string(val))
if comment.Len() > 0 {
cfg.keyComment[section+"."+key] = comment.String()
comment.Reset()
}
}
return cfg, nil
}
// ParseData parse ini the data
// When include other.conf,other.conf is either absolute directory
// or under beego in default temporary directory(/tmp/beego[-username]).
func (ini *IniConfig) ParseData(data []byte) (Configer, error) {
dir := "beego"
currentUser, err := user.Current()
if err == nil {
dir = "beego-" + currentUser.Username
}
dir = filepath.Join(os.TempDir(), dir)
if err = os.MkdirAll(dir, os.ModePerm); err != nil {
return nil, err
}
return ini.parseData(dir, data)
}
// IniConfigContainer A Config represents the ini configuration.
// When set and get value, support key as section:name type.
type IniConfigContainer struct {
data map[string]map[string]string // section=> key:val
sectionComment map[string]string // section : comment
keyComment map[string]string // id: []{comment, key...}; id 1 is for main comment.
sync.RWMutex
}
// Bool returns the boolean value for a given key.
func (c *IniConfigContainer) Bool(key string) (bool, error) {
return ParseBool(c.getdata(key))
}
// DefaultBool returns the boolean value for a given key.
// if err != nil return defaultval
func (c *IniConfigContainer) DefaultBool(key string, defaultval bool) bool {
v, err := c.Bool(key)
if err != nil {
return defaultval
}
return v
}
// Int returns the integer value for a given key.
func (c *IniConfigContainer) Int(key string) (int, error) {
return strconv.Atoi(c.getdata(key))
}
// DefaultInt returns the integer value for a given key.
// if err != nil return defaultval
func (c *IniConfigContainer) DefaultInt(key string, defaultval int) int {
v, err := c.Int(key)
if err != nil {
return defaultval
}
return v
}
// Int64 returns the int64 value for a given key.
func (c *IniConfigContainer) Int64(key string) (int64, error) {
return strconv.ParseInt(c.getdata(key), 10, 64)
}
// DefaultInt64 returns the int64 value for a given key.
// if err != nil return defaultval
func (c *IniConfigContainer) DefaultInt64(key string, defaultval int64) int64 {
v, err := c.Int64(key)
if err != nil {
return defaultval
}
return v
}
// Float returns the float value for a given key.
func (c *IniConfigContainer) Float(key string) (float64, error) {
return strconv.ParseFloat(c.getdata(key), 64)
}
// DefaultFloat returns the float64 value for a given key.
// if err != nil return defaultval
func (c *IniConfigContainer) DefaultFloat(key string, defaultval float64) float64 {
v, err := c.Float(key)
if err != nil {
return defaultval
}
return v
}
// String returns the string value for a given key.
func (c *IniConfigContainer) String(key string) string {
return c.getdata(key)
}
// DefaultString returns the string value for a given key.
// if err != nil return defaultval
func (c *IniConfigContainer) DefaultString(key string, defaultval string) string {
v := c.String(key)
if v == "" {
return defaultval
}
return v
}
// Strings returns the []string value for a given key.
// Return nil if config value does not exist or is empty.
func (c *IniConfigContainer) Strings(key string) []string {
v := c.String(key)
if v == "" {
return nil
}
return strings.Split(v, ";")
}
// DefaultStrings returns the []string value for a given key.
// if err != nil return defaultval
func (c *IniConfigContainer) DefaultStrings(key string, defaultval []string) []string {
v := c.Strings(key)
if v == nil {
return defaultval
}
return v
}
// GetSection returns map for the given section
func (c *IniConfigContainer) GetSection(section string) (map[string]string, error) {
if v, ok := c.data[section]; ok {
return v, nil
}
return nil, errors.New("not exist section")
}
// SaveConfigFile save the config into file.
//
// BUG(env): The environment variable config item will be saved with real value in SaveConfigFile Function.
func (c *IniConfigContainer) SaveConfigFile(filename string) (err error) {
// Write configuration file by filename.
f, err := os.Create(filename)
if err != nil {
return err
}
defer f.Close()
// Get section or key comments. Fixed #1607
getCommentStr := func(section, key string) string {
var (
comment string
ok bool
)
if len(key) == 0 {
comment, ok = c.sectionComment[section]
} else {
comment, ok = c.keyComment[section+"."+key]
}
if ok {
// Empty comment
if len(comment) == 0 || len(strings.TrimSpace(comment)) == 0 {
return string(bNumComment)
}
prefix := string(bNumComment)
// Add the line head character "#"
return prefix + strings.Replace(comment, lineBreak, lineBreak+prefix, -1)
}
return ""
}
buf := bytes.NewBuffer(nil)
// Save default section at first place
if dt, ok := c.data[defaultSection]; ok {
for key, val := range dt {
if key != " " {
// Write key comments.
if v := getCommentStr(defaultSection, key); len(v) > 0 {
if _, err = buf.WriteString(v + lineBreak); err != nil {
return err
}
}
// Write key and value.
if _, err = buf.WriteString(key + string(bEqual) + val + lineBreak); err != nil {
return err
}
}
}
// Put a line between sections.
if _, err = buf.WriteString(lineBreak); err != nil {
return err
}
}
// Save named sections
for section, dt := range c.data {
if section != defaultSection {
// Write section comments.
if v := getCommentStr(section, ""); len(v) > 0 {
if _, err = buf.WriteString(v + lineBreak); err != nil {
return err
}
}
// Write section name.
if _, err = buf.WriteString(string(sectionStart) + section + string(sectionEnd) + lineBreak); err != nil {
return err
}
for key, val := range dt {
if key != " " {
// Write key comments.
if v := getCommentStr(section, key); len(v) > 0 {
if _, err = buf.WriteString(v + lineBreak); err != nil {
return err
}
}
// Write key and value.
if _, err = buf.WriteString(key + string(bEqual) + val + lineBreak); err != nil {
return err
}
}
}
// Put a line between sections.
if _, err = buf.WriteString(lineBreak); err != nil {
return err
}
}
}
_, err = buf.WriteTo(f)
return err
}
// Set writes a new value for key.
// if write to one section, the key need be "section::key".
// if the section is not existed, it panics.
func (c *IniConfigContainer) Set(key, value string) error {
c.Lock()
defer c.Unlock()
if len(key) == 0 {
return errors.New("key is empty")
}
var (
section, k string
sectionKey = strings.Split(strings.ToLower(key), "::")
)
if len(sectionKey) >= 2 {
section = sectionKey[0]
k = sectionKey[1]
} else {
section = defaultSection
k = sectionKey[0]
}
if _, ok := c.data[section]; !ok {
c.data[section] = make(map[string]string)
}
c.data[section][k] = value
return nil
}
// DIY returns the raw value by a given key.
func (c *IniConfigContainer) DIY(key string) (v interface{}, err error) {
if v, ok := c.data[strings.ToLower(key)]; ok {
return v, nil
}
return v, errors.New("key not find")
}
// section.key or key
func (c *IniConfigContainer) getdata(key string) string {
if len(key) == 0 {
return ""
}
c.RLock()
defer c.RUnlock()
var (
section, k string
sectionKey = strings.Split(strings.ToLower(key), "::")
)
if len(sectionKey) >= 2 {
section = sectionKey[0]
k = sectionKey[1]
} else {
section = defaultSection
k = sectionKey[0]
}
if v, ok := c.data[section]; ok {
if vv, ok := v[k]; ok {
return vv
}
}
return ""
}
func init() {
Register("ini", &IniConfig{})
}

190
vender/github.com/astaxie/beego/config/ini_test.go
View File

@ -1,190 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package config
import (
"fmt"
"io/ioutil"
"os"
"strings"
"testing"
)
func TestIni(t *testing.T) {
var (
inicontext = `
;comment one
#comment two
appname = beeapi
httpport = 8080
mysqlport = 3600
PI = 3.1415976
runmode = "dev"
autorender = false
copyrequestbody = true
session= on
cookieon= off
newreg = OFF
needlogin = ON
enableSession = Y
enableCookie = N
flag = 1
path1 = ${GOPATH}
path2 = ${GOPATH||/home/go}
[demo]
key1="asta"
key2 = "xie"
CaseInsensitive = true
peers = one;two;three
password = ${GOPATH}
`
keyValue = map[string]interface{}{
"appname": "beeapi",
"httpport": 8080,
"mysqlport": int64(3600),
"pi": 3.1415976,
"runmode": "dev",
"autorender": false,
"copyrequestbody": true,
"session": true,
"cookieon": false,
"newreg": false,
"needlogin": true,
"enableSession": true,
"enableCookie": false,
"flag": true,
"path1": os.Getenv("GOPATH"),
"path2": os.Getenv("GOPATH"),
"demo::key1": "asta",
"demo::key2": "xie",
"demo::CaseInsensitive": true,
"demo::peers": []string{"one", "two", "three"},
"demo::password": os.Getenv("GOPATH"),
"null": "",
"demo2::key1": "",
"error": "",
"emptystrings": []string{},
}
)
f, err := os.Create("testini.conf")
if err != nil {
t.Fatal(err)
}
_, err = f.WriteString(inicontext)
if err != nil {
f.Close()
t.Fatal(err)
}
f.Close()
defer os.Remove("testini.conf")
iniconf, err := NewConfig("ini", "testini.conf")
if err != nil {
t.Fatal(err)
}
for k, v := range keyValue {
var err error
var value interface{}
switch v.(type) {
case int:
value, err = iniconf.Int(k)
case int64:
value, err = iniconf.Int64(k)
case float64:
value, err = iniconf.Float(k)
case bool:
value, err = iniconf.Bool(k)
case []string:
value = iniconf.Strings(k)
case string:
value = iniconf.String(k)
default:
value, err = iniconf.DIY(k)
}
if err != nil {
t.Fatalf("get key %q value fail,err %s", k, err)
} else if fmt.Sprintf("%v", v) != fmt.Sprintf("%v", value) {
t.Fatalf("get key %q value, want %v got %v .", k, v, value)
}
}
if err = iniconf.Set("name", "astaxie"); err != nil {
t.Fatal(err)
}
if iniconf.String("name") != "astaxie" {
t.Fatal("get name error")
}
}
func TestIniSave(t *testing.T) {
const (
inicontext = `
app = app
;comment one
#comment two
# comment three
appname = beeapi
httpport = 8080
# DB Info
# enable db
[dbinfo]
# db type name
# suport mysql,sqlserver
name = mysql
`
saveResult = `
app=app
#comment one
#comment two
# comment three
appname=beeapi
httpport=8080
# DB Info
# enable db
[dbinfo]
# db type name
# suport mysql,sqlserver
name=mysql
`
)
cfg, err := NewConfigData("ini", []byte(inicontext))
if err != nil {
t.Fatal(err)
}
name := "newIniConfig.ini"
if err := cfg.SaveConfigFile(name); err != nil {
t.Fatal(err)
}
defer os.Remove(name)
if data, err := ioutil.ReadFile(name); err != nil {
t.Fatal(err)
} else {
cfgData := string(data)
datas := strings.Split(saveResult, "\n")
for _, line := range datas {
if !strings.Contains(cfgData, line+"\n") {
t.Fatalf("different after save ini config file. need contains %q", line)
}
}
}
}

266
vender/github.com/astaxie/beego/config/json.go
View File

@ -1,266 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package config
import (
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"os"
"strings"
"sync"
)
// JSONConfig is a json config parser and implements Config interface.
type JSONConfig struct {
}
// Parse returns a ConfigContainer with parsed json config map.
func (js *JSONConfig) Parse(filename string) (Configer, error) {
file, err := os.Open(filename)
if err != nil {
return nil, err
}
defer file.Close()
content, err := ioutil.ReadAll(file)
if err != nil {
return nil, err
}
return js.ParseData(content)
}
// ParseData returns a ConfigContainer with json string
func (js *JSONConfig) ParseData(data []byte) (Configer, error) {
x := &JSONConfigContainer{
data: make(map[string]interface{}),
}
err := json.Unmarshal(data, &x.data)
if err != nil {
var wrappingArray []interface{}
err2 := json.Unmarshal(data, &wrappingArray)
if err2 != nil {
return nil, err
}
x.data["rootArray"] = wrappingArray
}
x.data = ExpandValueEnvForMap(x.data)
return x, nil
}
// JSONConfigContainer A Config represents the json configuration.
// Only when get value, support key as section:name type.
type JSONConfigContainer struct {
data map[string]interface{}
sync.RWMutex
}
// Bool returns the boolean value for a given key.
func (c *JSONConfigContainer) Bool(key string) (bool, error) {
val := c.getData(key)
if val != nil {
return ParseBool(val)
}
return false, fmt.Errorf("not exist key: %q", key)
}
// DefaultBool return the bool value if has no error
// otherwise return the defaultval
func (c *JSONConfigContainer) DefaultBool(key string, defaultval bool) bool {
if v, err := c.Bool(key); err == nil {
return v
}
return defaultval
}
// Int returns the integer value for a given key.
func (c *JSONConfigContainer) Int(key string) (int, error) {
val := c.getData(key)
if val != nil {
if v, ok := val.(float64); ok {
return int(v), nil
}
return 0, errors.New("not int value")
}
return 0, errors.New("not exist key:" + key)
}
// DefaultInt returns the integer value for a given key.
// if err != nil return defaultval
func (c *JSONConfigContainer) DefaultInt(key string, defaultval int) int {
if v, err := c.Int(key); err == nil {
return v
}
return defaultval
}
// Int64 returns the int64 value for a given key.
func (c *JSONConfigContainer) Int64(key string) (int64, error) {
val := c.getData(key)
if val != nil {
if v, ok := val.(float64); ok {
return int64(v), nil
}
return 0, errors.New("not int64 value")
}
return 0, errors.New("not exist key:" + key)
}
// DefaultInt64 returns the int64 value for a given key.
// if err != nil return defaultval
func (c *JSONConfigContainer) DefaultInt64(key string, defaultval int64) int64 {
if v, err := c.Int64(key); err == nil {
return v
}
return defaultval
}
// Float returns the float value for a given key.
func (c *JSONConfigContainer) Float(key string) (float64, error) {
val := c.getData(key)
if val != nil {
if v, ok := val.(float64); ok {
return v, nil
}
return 0.0, errors.New("not float64 value")
}
return 0.0, errors.New("not exist key:" + key)
}
// DefaultFloat returns the float64 value for a given key.
// if err != nil return defaultval
func (c *JSONConfigContainer) DefaultFloat(key string, defaultval float64) float64 {
if v, err := c.Float(key); err == nil {
return v
}
return defaultval
}
// String returns the string value for a given key.
func (c *JSONConfigContainer) String(key string) string {
val := c.getData(key)
if val != nil {
if v, ok := val.(string); ok {
return v
}
}
return ""
}
// DefaultString returns the string value for a given key.
// if err != nil return defaultval
func (c *JSONConfigContainer) DefaultString(key string, defaultval string) string {
// TODO FIXME should not use "" to replace non existence
if v := c.String(key); v != "" {
return v
}
return defaultval
}
// Strings returns the []string value for a given key.
func (c *JSONConfigContainer) Strings(key string) []string {
stringVal := c.String(key)
if stringVal == "" {
return nil
}
return strings.Split(c.String(key), ";")
}
// DefaultStrings returns the []string value for a given key.
// if err != nil return defaultval
func (c *JSONConfigContainer) DefaultStrings(key string, defaultval []string) []string {
if v := c.Strings(key); v != nil {
return v
}
return defaultval
}
// GetSection returns map for the given section
func (c *JSONConfigContainer) GetSection(section string) (map[string]string, error) {
if v, ok := c.data[section]; ok {
return v.(map[string]string), nil
}
return nil, errors.New("nonexist section " + section)
}
// SaveConfigFile save the config into file
func (c *JSONConfigContainer) SaveConfigFile(filename string) (err error) {
// Write configuration file by filename.
f, err := os.Create(filename)
if err != nil {
return err
}
defer f.Close()
b, err := json.MarshalIndent(c.data, "", " ")
if err != nil {
return err
}
_, err = f.Write(b)
return err
}
// Set writes a new value for key.
func (c *JSONConfigContainer) Set(key, val string) error {
c.Lock()
defer c.Unlock()
c.data[key] = val
return nil
}
// DIY returns the raw value by a given key.
func (c *JSONConfigContainer) DIY(key string) (v interface{}, err error) {
val := c.getData(key)
if val != nil {
return val, nil
}
return nil, errors.New("not exist key")
}
// section.key or key
func (c *JSONConfigContainer) getData(key string) interface{} {
if len(key) == 0 {
return nil
}
c.RLock()
defer c.RUnlock()
sectionKeys := strings.Split(key, "::")
if len(sectionKeys) >= 2 {
curValue, ok := c.data[sectionKeys[0]]
if !ok {
return nil
}
for _, key := range sectionKeys[1:] {
if v, ok := curValue.(map[string]interface{}); ok {
if curValue, ok = v[key]; !ok {
return nil
}
}
}
return curValue
}
if v, ok := c.data[key]; ok {
return v
}
return nil
}
func init() {
Register("json", &JSONConfig{})
}

222
vender/github.com/astaxie/beego/config/json_test.go
View File

@ -1,222 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package config
import (
"fmt"
"os"
"testing"
)
func TestJsonStartsWithArray(t *testing.T) {
const jsoncontextwitharray = `[
{
"url": "user",
"serviceAPI": "http://www.test.com/user"
},
{
"url": "employee",
"serviceAPI": "http://www.test.com/employee"
}
]`
f, err := os.Create("testjsonWithArray.conf")
if err != nil {
t.Fatal(err)
}
_, err = f.WriteString(jsoncontextwitharray)
if err != nil {
f.Close()
t.Fatal(err)
}
f.Close()
defer os.Remove("testjsonWithArray.conf")
jsonconf, err := NewConfig("json", "testjsonWithArray.conf")
if err != nil {
t.Fatal(err)
}
rootArray, err := jsonconf.DIY("rootArray")
if err != nil {
t.Error("array does not exist as element")
}
rootArrayCasted := rootArray.([]interface{})
if rootArrayCasted == nil {
t.Error("array from root is nil")
} else {
elem := rootArrayCasted[0].(map[string]interface{})
if elem["url"] != "user" || elem["serviceAPI"] != "http://www.test.com/user" {
t.Error("array[0] values are not valid")
}
elem2 := rootArrayCasted[1].(map[string]interface{})
if elem2["url"] != "employee" || elem2["serviceAPI"] != "http://www.test.com/employee" {
t.Error("array[1] values are not valid")
}
}
}
func TestJson(t *testing.T) {
var (
jsoncontext = `{
"appname": "beeapi",
"testnames": "foo;bar",
"httpport": 8080,
"mysqlport": 3600,
"PI": 3.1415976,
"runmode": "dev",
"autorender": false,
"copyrequestbody": true,
"session": "on",
"cookieon": "off",
"newreg": "OFF",
"needlogin": "ON",
"enableSession": "Y",
"enableCookie": "N",
"flag": 1,
"path1": "${GOPATH}",
"path2": "${GOPATH||/home/go}",
"database": {
"host": "host",
"port": "port",
"database": "database",
"username": "username",
"password": "${GOPATH}",
"conns":{
"maxconnection":12,
"autoconnect":true,
"connectioninfo":"info",
"root": "${GOPATH}"
}
}
}`
keyValue = map[string]interface{}{
"appname": "beeapi",
"testnames": []string{"foo", "bar"},
"httpport": 8080,
"mysqlport": int64(3600),
"PI": 3.1415976,
"runmode": "dev",
"autorender": false,
"copyrequestbody": true,
"session": true,
"cookieon": false,
"newreg": false,
"needlogin": true,
"enableSession": true,
"enableCookie": false,
"flag": true,
"path1": os.Getenv("GOPATH"),
"path2": os.Getenv("GOPATH"),
"database::host": "host",
"database::port": "port",
"database::database": "database",
"database::password": os.Getenv("GOPATH"),
"database::conns::maxconnection": 12,
"database::conns::autoconnect": true,
"database::conns::connectioninfo": "info",
"database::conns::root": os.Getenv("GOPATH"),
"unknown": "",
}
)
f, err := os.Create("testjson.conf")
if err != nil {
t.Fatal(err)
}
_, err = f.WriteString(jsoncontext)
if err != nil {
f.Close()
t.Fatal(err)
}
f.Close()
defer os.Remove("testjson.conf")
jsonconf, err := NewConfig("json", "testjson.conf")
if err != nil {
t.Fatal(err)
}
for k, v := range keyValue {
var err error
var value interface{}
switch v.(type) {
case int:
value, err = jsonconf.Int(k)
case int64:
value, err = jsonconf.Int64(k)
case float64:
value, err = jsonconf.Float(k)
case bool:
value, err = jsonconf.Bool(k)
case []string:
value = jsonconf.Strings(k)
case string:
value = jsonconf.String(k)
default:
value, err = jsonconf.DIY(k)
}
if err != nil {
t.Fatalf("get key %q value fatal,%v err %s", k, v, err)
} else if fmt.Sprintf("%v", v) != fmt.Sprintf("%v", value) {
t.Fatalf("get key %q value, want %v got %v .", k, v, value)
}
}
if err = jsonconf.Set("name", "astaxie"); err != nil {
t.Fatal(err)
}
if jsonconf.String("name") != "astaxie" {
t.Fatal("get name error")
}
if db, err := jsonconf.DIY("database"); err != nil {
t.Fatal(err)
} else if m, ok := db.(map[string]interface{}); !ok {
t.Log(db)
t.Fatal("db not map[string]interface{}")
} else {
if m["host"].(string) != "host" {
t.Fatal("get host err")
}
}
if _, err := jsonconf.Int("unknown"); err == nil {
t.Error("unknown keys should return an error when expecting an Int")
}
if _, err := jsonconf.Int64("unknown"); err == nil {
t.Error("unknown keys should return an error when expecting an Int64")
}
if _, err := jsonconf.Float("unknown"); err == nil {
t.Error("unknown keys should return an error when expecting a Float")
}
if _, err := jsonconf.DIY("unknown"); err == nil {
t.Error("unknown keys should return an error when expecting an interface{}")
}
if val := jsonconf.String("unknown"); val != "" {
t.Error("unknown keys should return an empty string when expecting a String")
}
if _, err := jsonconf.Bool("unknown"); err == nil {
t.Error("unknown keys should return an error when expecting a Bool")
}
if !jsonconf.DefaultBool("unknown", true) {
t.Error("unknown keys with default value wrong")
}
}

228
vender/github.com/astaxie/beego/config/xml/xml.go
View File

@ -1,228 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package xml for config provider.
//
// depend on github.com/beego/x2j.
//
// go install github.com/beego/x2j.
//
// Usage:
// import(
// _ "github.com/cnlh/nps/vender/github.com/astaxie/beego/config/xml"
// "github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
// )
//
// cnf, err := config.NewConfig("xml", "config.xml")
//
//More docs http://beego.me/docs/module/config.md
package xml
import (
"encoding/xml"
"errors"
"fmt"
"io/ioutil"
"os"
"strconv"
"strings"
"sync"
"github.com/beego/x2j"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
)
// Config is a xml config parser and implements Config interface.
// xml configurations should be included in <config></config> tag.
// only support key/value pair as <key>value</key> as each item.
type Config struct{}
// Parse returns a ConfigContainer with parsed xml config map.
func (xc *Config) Parse(filename string) (config.Configer, error) {
context, err := ioutil.ReadFile(filename)
if err != nil {
return nil, err
}
return xc.ParseData(context)
}
// ParseData xml data
func (xc *Config) ParseData(data []byte) (config.Configer, error) {
x := &ConfigContainer{data: make(map[string]interface{})}
d, err := x2j.DocToMap(string(data))
if err != nil {
return nil, err
}
x.data = config.ExpandValueEnvForMap(d["config"].(map[string]interface{}))
return x, nil
}
// ConfigContainer A Config represents the xml configuration.
type ConfigContainer struct {
data map[string]interface{}
sync.Mutex
}
// Bool returns the boolean value for a given key.
func (c *ConfigContainer) Bool(key string) (bool, error) {
if v := c.data[key]; v != nil {
return config.ParseBool(v)
}
return false, fmt.Errorf("not exist key: %q", key)
}
// DefaultBool return the bool value if has no error
// otherwise return the defaultval
func (c *ConfigContainer) DefaultBool(key string, defaultval bool) bool {
v, err := c.Bool(key)
if err != nil {
return defaultval
}
return v
}
// Int returns the integer value for a given key.
func (c *ConfigContainer) Int(key string) (int, error) {
return strconv.Atoi(c.data[key].(string))
}
// DefaultInt returns the integer value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultInt(key string, defaultval int) int {
v, err := c.Int(key)
if err != nil {
return defaultval
}
return v
}
// Int64 returns the int64 value for a given key.
func (c *ConfigContainer) Int64(key string) (int64, error) {
return strconv.ParseInt(c.data[key].(string), 10, 64)
}
// DefaultInt64 returns the int64 value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultInt64(key string, defaultval int64) int64 {
v, err := c.Int64(key)
if err != nil {
return defaultval
}
return v
}
// Float returns the float value for a given key.
func (c *ConfigContainer) Float(key string) (float64, error) {
return strconv.ParseFloat(c.data[key].(string), 64)
}
// DefaultFloat returns the float64 value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultFloat(key string, defaultval float64) float64 {
v, err := c.Float(key)
if err != nil {
return defaultval
}
return v
}
// String returns the string value for a given key.
func (c *ConfigContainer) String(key string) string {
if v, ok := c.data[key].(string); ok {
return v
}
return ""
}
// DefaultString returns the string value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultString(key string, defaultval string) string {
v := c.String(key)
if v == "" {
return defaultval
}
return v
}
// Strings returns the []string value for a given key.
func (c *ConfigContainer) Strings(key string) []string {
v := c.String(key)
if v == "" {
return nil
}
return strings.Split(v, ";")
}
// DefaultStrings returns the []string value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultStrings(key string, defaultval []string) []string {
v := c.Strings(key)
if v == nil {
return defaultval
}
return v
}
// GetSection returns map for the given section
func (c *ConfigContainer) GetSection(section string) (map[string]string, error) {
if v, ok := c.data[section].(map[string]interface{}); ok {
mapstr := make(map[string]string)
for k, val := range v {
mapstr[k] = config.ToString(val)
}
return mapstr, nil
}
return nil, fmt.Errorf("section '%s' not found", section)
}
// SaveConfigFile save the config into file
func (c *ConfigContainer) SaveConfigFile(filename string) (err error) {
// Write configuration file by filename.
f, err := os.Create(filename)
if err != nil {
return err
}
defer f.Close()
b, err := xml.MarshalIndent(c.data, " ", " ")
if err != nil {
return err
}
_, err = f.Write(b)
return err
}
// Set writes a new value for key.
func (c *ConfigContainer) Set(key, val string) error {
c.Lock()
defer c.Unlock()
c.data[key] = val
return nil
}
// DIY returns the raw value by a given key.
func (c *ConfigContainer) DIY(key string) (v interface{}, err error) {
if v, ok := c.data[key]; ok {
return v, nil
}
return nil, errors.New("not exist key")
}
func init() {
config.Register("xml", &Config{})
}

125
vender/github.com/astaxie/beego/config/xml/xml_test.go
View File

@ -1,125 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package xml
import (
"fmt"
"os"
"testing"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
)
func TestXML(t *testing.T) {
var (
//xml parse should incluce in <config></config> tags
xmlcontext = `<?xml version="1.0" encoding="UTF-8"?>
<config>
<appname>beeapi</appname>
<httpport>8080</httpport>
<mysqlport>3600</mysqlport>
<PI>3.1415976</PI>
<runmode>dev</runmode>
<autorender>false</autorender>
<copyrequestbody>true</copyrequestbody>
<path1>${GOPATH}</path1>
<path2>${GOPATH||/home/go}</path2>
<mysection>
<id>1</id>
<name>MySection</name>
</mysection>
</config>
`
keyValue = map[string]interface{}{
"appname": "beeapi",
"httpport": 8080,
"mysqlport": int64(3600),
"PI": 3.1415976,
"runmode": "dev",
"autorender": false,
"copyrequestbody": true,
"path1": os.Getenv("GOPATH"),
"path2": os.Getenv("GOPATH"),
"error": "",
"emptystrings": []string{},
}
)
f, err := os.Create("testxml.conf")
if err != nil {
t.Fatal(err)
}
_, err = f.WriteString(xmlcontext)
if err != nil {
f.Close()
t.Fatal(err)
}
f.Close()
defer os.Remove("testxml.conf")
xmlconf, err := config.NewConfig("xml", "testxml.conf")
if err != nil {
t.Fatal(err)
}
var xmlsection map[string]string
xmlsection, err = xmlconf.GetSection("mysection")
if err != nil {
t.Fatal(err)
}
if len(xmlsection) == 0 {
t.Error("section should not be empty")
}
for k, v := range keyValue {
var (
value interface{}
err error
)
switch v.(type) {
case int:
value, err = xmlconf.Int(k)
case int64:
value, err = xmlconf.Int64(k)
case float64:
value, err = xmlconf.Float(k)
case bool:
value, err = xmlconf.Bool(k)
case []string:
value = xmlconf.Strings(k)
case string:
value = xmlconf.String(k)
default:
value, err = xmlconf.DIY(k)
}
if err != nil {
t.Errorf("get key %q value fatal,%v err %s", k, v, err)
} else if fmt.Sprintf("%v", v) != fmt.Sprintf("%v", value) {
t.Errorf("get key %q value, want %v got %v .", k, v, value)
}
}
if err = xmlconf.Set("name", "astaxie"); err != nil {
t.Fatal(err)
}
if xmlconf.String("name") != "astaxie" {
t.Fatal("get name error")
}
}

316
vender/github.com/astaxie/beego/config/yaml/yaml.go
View File

@ -1,316 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package yaml for config provider
//
// depend on github.com/beego/goyaml2
//
// go install github.com/beego/goyaml2
//
// Usage:
// import(
// _ "github.com/cnlh/nps/vender/github.com/astaxie/beego/config/yaml"
// "github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
// )
//
// cnf, err := config.NewConfig("yaml", "config.yaml")
//
//More docs http://beego.me/docs/module/config.md
package yaml
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io/ioutil"
"log"
"os"
"strings"
"sync"
"github.com/beego/goyaml2"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
)
// Config is a yaml config parser and implements Config interface.
type Config struct{}
// Parse returns a ConfigContainer with parsed yaml config map.
func (yaml *Config) Parse(filename string) (y config.Configer, err error) {
cnf, err := ReadYmlReader(filename)
if err != nil {
return
}
y = &ConfigContainer{
data: cnf,
}
return
}
// ParseData parse yaml data
func (yaml *Config) ParseData(data []byte) (config.Configer, error) {
cnf, err := parseYML(data)
if err != nil {
return nil, err
}
return &ConfigContainer{
data: cnf,
}, nil
}
// ReadYmlReader Read yaml file to map.
// if json like, use json package, unless goyaml2 package.
func ReadYmlReader(path string) (cnf map[string]interface{}, err error) {
buf, err := ioutil.ReadFile(path)
if err != nil {
return
}
return parseYML(buf)
}
// parseYML parse yaml formatted []byte to map.
func parseYML(buf []byte) (cnf map[string]interface{}, err error) {
if len(buf) < 3 {
return
}
if string(buf[0:1]) == "{" {
log.Println("Look like a Json, try json umarshal")
err = json.Unmarshal(buf, &cnf)
if err == nil {
log.Println("It is Json Map")
return
}
}
data, err := goyaml2.Read(bytes.NewBuffer(buf))
if err != nil {
log.Println("Goyaml2 ERR>", string(buf), err)
return
}
if data == nil {
log.Println("Goyaml2 output nil? Pls report bug\n" + string(buf))
return
}
cnf, ok := data.(map[string]interface{})
if !ok {
log.Println("Not a Map? >> ", string(buf), data)
cnf = nil
}
cnf = config.ExpandValueEnvForMap(cnf)
return
}
// ConfigContainer A Config represents the yaml configuration.
type ConfigContainer struct {
data map[string]interface{}
sync.RWMutex
}
// Bool returns the boolean value for a given key.
func (c *ConfigContainer) Bool(key string) (bool, error) {
v, err := c.getData(key)
if err != nil {
return false, err
}
return config.ParseBool(v)
}
// DefaultBool return the bool value if has no error
// otherwise return the defaultval
func (c *ConfigContainer) DefaultBool(key string, defaultval bool) bool {
v, err := c.Bool(key)
if err != nil {
return defaultval
}
return v
}
// Int returns the integer value for a given key.
func (c *ConfigContainer) Int(key string) (int, error) {
if v, err := c.getData(key); err != nil {
return 0, err
} else if vv, ok := v.(int); ok {
return vv, nil
} else if vv, ok := v.(int64); ok {
return int(vv), nil
}
return 0, errors.New("not int value")
}
// DefaultInt returns the integer value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultInt(key string, defaultval int) int {
v, err := c.Int(key)
if err != nil {
return defaultval
}
return v
}
// Int64 returns the int64 value for a given key.
func (c *ConfigContainer) Int64(key string) (int64, error) {
if v, err := c.getData(key); err != nil {
return 0, err
} else if vv, ok := v.(int64); ok {
return vv, nil
}
return 0, errors.New("not bool value")
}
// DefaultInt64 returns the int64 value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultInt64(key string, defaultval int64) int64 {
v, err := c.Int64(key)
if err != nil {
return defaultval
}
return v
}
// Float returns the float value for a given key.
func (c *ConfigContainer) Float(key string) (float64, error) {
if v, err := c.getData(key); err != nil {
return 0.0, err
} else if vv, ok := v.(float64); ok {
return vv, nil
} else if vv, ok := v.(int); ok {
return float64(vv), nil
} else if vv, ok := v.(int64); ok {
return float64(vv), nil
}
return 0.0, errors.New("not float64 value")
}
// DefaultFloat returns the float64 value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultFloat(key string, defaultval float64) float64 {
v, err := c.Float(key)
if err != nil {
return defaultval
}
return v
}
// String returns the string value for a given key.
func (c *ConfigContainer) String(key string) string {
if v, err := c.getData(key); err == nil {
if vv, ok := v.(string); ok {
return vv
}
}
return ""
}
// DefaultString returns the string value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultString(key string, defaultval string) string {
v := c.String(key)
if v == "" {
return defaultval
}
return v
}
// Strings returns the []string value for a given key.
func (c *ConfigContainer) Strings(key string) []string {
v := c.String(key)
if v == "" {
return nil
}
return strings.Split(v, ";")
}
// DefaultStrings returns the []string value for a given key.
// if err != nil return defaultval
func (c *ConfigContainer) DefaultStrings(key string, defaultval []string) []string {
v := c.Strings(key)
if v == nil {
return defaultval
}
return v
}
// GetSection returns map for the given section
func (c *ConfigContainer) GetSection(section string) (map[string]string, error) {
if v, ok := c.data[section]; ok {
return v.(map[string]string), nil
}
return nil, errors.New("not exist section")
}
// SaveConfigFile save the config into file
func (c *ConfigContainer) SaveConfigFile(filename string) (err error) {
// Write configuration file by filename.
f, err := os.Create(filename)
if err != nil {
return err
}
defer f.Close()
err = goyaml2.Write(f, c.data)
return err
}
// Set writes a new value for key.
func (c *ConfigContainer) Set(key, val string) error {
c.Lock()
defer c.Unlock()
c.data[key] = val
return nil
}
// DIY returns the raw value by a given key.
func (c *ConfigContainer) DIY(key string) (v interface{}, err error) {
return c.getData(key)
}
func (c *ConfigContainer) getData(key string) (interface{}, error) {
if len(key) == 0 {
return nil, errors.New("key is empty")
}
c.RLock()
defer c.RUnlock()
keys := strings.Split(key, ".")
tmpData := c.data
for idx, k := range keys {
if v, ok := tmpData[k]; ok {
switch v.(type) {
case map[string]interface{}:
{
tmpData = v.(map[string]interface{})
if idx == len(keys)-1 {
return tmpData, nil
}
}
default:
{
return v, nil
}
}
}
}
return nil, fmt.Errorf("not exist key %q", key)
}
func init() {
config.Register("yaml", &Config{})
}

115
vender/github.com/astaxie/beego/config/yaml/yaml_test.go
View File

@ -1,115 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package yaml
import (
"fmt"
"os"
"testing"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
)
func TestYaml(t *testing.T) {
var (
yamlcontext = `
"appname": beeapi
"httpport": 8080
"mysqlport": 3600
"PI": 3.1415976
"runmode": dev
"autorender": false
"copyrequestbody": true
"PATH": GOPATH
"path1": ${GOPATH}
"path2": ${GOPATH||/home/go}
"empty": ""
`
keyValue = map[string]interface{}{
"appname": "beeapi",
"httpport": 8080,
"mysqlport": int64(3600),
"PI": 3.1415976,
"runmode": "dev",
"autorender": false,
"copyrequestbody": true,
"PATH": "GOPATH",
"path1": os.Getenv("GOPATH"),
"path2": os.Getenv("GOPATH"),
"error": "",
"emptystrings": []string{},
}
)
f, err := os.Create("testyaml.conf")
if err != nil {
t.Fatal(err)
}
_, err = f.WriteString(yamlcontext)
if err != nil {
f.Close()
t.Fatal(err)
}
f.Close()
defer os.Remove("testyaml.conf")
yamlconf, err := config.NewConfig("yaml", "testyaml.conf")
if err != nil {
t.Fatal(err)
}
if yamlconf.String("appname") != "beeapi" {
t.Fatal("appname not equal to beeapi")
}
for k, v := range keyValue {
var (
value interface{}
err error
)
switch v.(type) {
case int:
value, err = yamlconf.Int(k)
case int64:
value, err = yamlconf.Int64(k)
case float64:
value, err = yamlconf.Float(k)
case bool:
value, err = yamlconf.Bool(k)
case []string:
value = yamlconf.Strings(k)
case string:
value = yamlconf.String(k)
default:
value, err = yamlconf.DIY(k)
}
if err != nil {
t.Errorf("get key %q value fatal,%v err %s", k, v, err)
} else if fmt.Sprintf("%v", v) != fmt.Sprintf("%v", value) {
t.Errorf("get key %q value, want %v got %v .", k, v, value)
}
}
if err = yamlconf.Set("name", "astaxie"); err != nil {
t.Fatal(err)
}
if yamlconf.String("name") != "astaxie" {
t.Fatal("get name error")
}
}

138
vender/github.com/astaxie/beego/config_test.go
View File

@ -1,138 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package beego
import (
"encoding/json"
"reflect"
"testing"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/config"
)
func TestDefaults(t *testing.T) {
if BConfig.WebConfig.FlashName != "BEEGO_FLASH" {
t.Errorf("FlashName was not set to default.")
}
if BConfig.WebConfig.FlashSeparator != "BEEGOFLASH" {
t.Errorf("FlashName was not set to default.")
}
}
func TestAssignConfig_01(t *testing.T) {
_BConfig := &Config{}
_BConfig.AppName = "beego_test"
jcf := &config.JSONConfig{}
ac, _ := jcf.ParseData([]byte(`{"AppName":"beego_json"}`))
assignSingleConfig(_BConfig, ac)
if _BConfig.AppName != "beego_json" {
t.Log(_BConfig)
t.FailNow()
}
}
func TestAssignConfig_02(t *testing.T) {
_BConfig := &Config{}
bs, _ := json.Marshal(newBConfig())
jsonMap := M{}
json.Unmarshal(bs, &jsonMap)
configMap := M{}
for k, v := range jsonMap {
if reflect.TypeOf(v).Kind() == reflect.Map {
for k1, v1 := range v.(M) {
if reflect.TypeOf(v1).Kind() == reflect.Map {
for k2, v2 := range v1.(M) {
configMap[k2] = v2
}
} else {
configMap[k1] = v1
}
}
} else {
configMap[k] = v
}
}
configMap["MaxMemory"] = 1024
configMap["Graceful"] = true
configMap["XSRFExpire"] = 32
configMap["SessionProviderConfig"] = "file"
configMap["FileLineNum"] = true
jcf := &config.JSONConfig{}
bs, _ = json.Marshal(configMap)
ac, _ := jcf.ParseData(bs)
for _, i := range []interface{}{_BConfig, &_BConfig.Listen, &_BConfig.WebConfig, &_BConfig.Log, &_BConfig.WebConfig.Session} {
assignSingleConfig(i, ac)
}
if _BConfig.MaxMemory != 1024 {
t.Log(_BConfig.MaxMemory)
t.FailNow()
}
if !_BConfig.Listen.Graceful {
t.Log(_BConfig.Listen.Graceful)
t.FailNow()
}
if _BConfig.WebConfig.XSRFExpire != 32 {
t.Log(_BConfig.WebConfig.XSRFExpire)
t.FailNow()
}
if _BConfig.WebConfig.Session.SessionProviderConfig != "file" {
t.Log(_BConfig.WebConfig.Session.SessionProviderConfig)
t.FailNow()
}
if !_BConfig.Log.FileLineNum {
t.Log(_BConfig.Log.FileLineNum)
t.FailNow()
}
}
func TestAssignConfig_03(t *testing.T) {
jcf := &config.JSONConfig{}
ac, _ := jcf.ParseData([]byte(`{"AppName":"beego"}`))
ac.Set("AppName", "test_app")
ac.Set("RunMode", "online")
ac.Set("StaticDir", "download:down download2:down2")
ac.Set("StaticExtensionsToGzip", ".css,.js,.html,.jpg,.png")
assignConfig(ac)
t.Logf("%#v", BConfig)
if BConfig.AppName != "test_app" {
t.FailNow()
}
if BConfig.RunMode != "online" {
t.FailNow()
}
if BConfig.WebConfig.StaticDir["/download"] != "down" {
t.FailNow()
}
if BConfig.WebConfig.StaticDir["/download2"] != "down2" {
t.FailNow()
}
if len(BConfig.WebConfig.StaticExtensionsToGzip) != 5 {
t.FailNow()
}
}

232
vender/github.com/astaxie/beego/context/acceptencoder.go
View File

@ -1,232 +0,0 @@
// Copyright 2015 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package context
import (
"bytes"
"compress/flate"
"compress/gzip"
"compress/zlib"
"io"
"net/http"
"os"
"strconv"
"strings"
"sync"
)
var (
//Default size==20B same as nginx
defaultGzipMinLength = 20
//Content will only be compressed if content length is either unknown or greater than gzipMinLength.
gzipMinLength = defaultGzipMinLength
//The compression level used for deflate compression. (0-9).
gzipCompressLevel int
//List of HTTP methods to compress. If not set, only GET requests are compressed.
includedMethods map[string]bool
getMethodOnly bool
)
// InitGzip init the gzipcompress
func InitGzip(minLength, compressLevel int, methods []string) {
if minLength >= 0 {
gzipMinLength = minLength
}
gzipCompressLevel = compressLevel
if gzipCompressLevel < flate.NoCompression || gzipCompressLevel > flate.BestCompression {
gzipCompressLevel = flate.BestSpeed
}
getMethodOnly = (len(methods) == 0) || (len(methods) == 1 && strings.ToUpper(methods[0]) == "GET")
includedMethods = make(map[string]bool, len(methods))
for _, v := range methods {
includedMethods[strings.ToUpper(v)] = true
}
}
type resetWriter interface {
io.Writer
Reset(w io.Writer)
}
type nopResetWriter struct {
io.Writer
}
func (n nopResetWriter) Reset(w io.Writer) {
//do nothing
}
type acceptEncoder struct {
name string
levelEncode func(int) resetWriter
customCompressLevelPool *sync.Pool
bestCompressionPool *sync.Pool
}
func (ac acceptEncoder) encode(wr io.Writer, level int) resetWriter {
if ac.customCompressLevelPool == nil || ac.bestCompressionPool == nil {
return nopResetWriter{wr}
}
var rwr resetWriter
switch level {
case flate.BestSpeed:
rwr = ac.customCompressLevelPool.Get().(resetWriter)
case flate.BestCompression:
rwr = ac.bestCompressionPool.Get().(resetWriter)
default:
rwr = ac.levelEncode(level)
}
rwr.Reset(wr)
return rwr
}
func (ac acceptEncoder) put(wr resetWriter, level int) {
if ac.customCompressLevelPool == nil || ac.bestCompressionPool == nil {
return
}
wr.Reset(nil)
//notice
//compressionLevel==BestCompression DOES NOT MATTER
//sync.Pool will not memory leak
switch level {
case gzipCompressLevel:
ac.customCompressLevelPool.Put(wr)
case flate.BestCompression:
ac.bestCompressionPool.Put(wr)
}
}
var (
noneCompressEncoder = acceptEncoder{"", nil, nil, nil}
gzipCompressEncoder = acceptEncoder{
name: "gzip",
levelEncode: func(level int) resetWriter { wr, _ := gzip.NewWriterLevel(nil, level); return wr },
customCompressLevelPool: &sync.Pool{New: func() interface{} { wr, _ := gzip.NewWriterLevel(nil, gzipCompressLevel); return wr }},
bestCompressionPool: &sync.Pool{New: func() interface{} { wr, _ := gzip.NewWriterLevel(nil, flate.BestCompression); return wr }},
}
//according to the sec :http://tools.ietf.org/html/rfc2616#section-3.5 ,the deflate compress in http is zlib indeed
//deflate
//The "zlib" format defined in RFC 1950 [31] in combination with
//the "deflate" compression mechanism described in RFC 1951 [29].
deflateCompressEncoder = acceptEncoder{
name: "deflate",
levelEncode: func(level int) resetWriter { wr, _ := zlib.NewWriterLevel(nil, level); return wr },
customCompressLevelPool: &sync.Pool{New: func() interface{} { wr, _ := zlib.NewWriterLevel(nil, gzipCompressLevel); return wr }},
bestCompressionPool: &sync.Pool{New: func() interface{} { wr, _ := zlib.NewWriterLevel(nil, flate.BestCompression); return wr }},
}
)
var (
encoderMap = map[string]acceptEncoder{ // all the other compress methods will ignore
"gzip": gzipCompressEncoder,
"deflate": deflateCompressEncoder,
"*": gzipCompressEncoder, // * means any compress will accept,we prefer gzip
"identity": noneCompressEncoder, // identity means none-compress
}
)
// WriteFile reads from file and writes to writer by the specific encoding(gzip/deflate)
func WriteFile(encoding string, writer io.Writer, file *os.File) (bool, string, error) {
return writeLevel(encoding, writer, file, flate.BestCompression)
}
// WriteBody reads writes content to writer by the specific encoding(gzip/deflate)
func WriteBody(encoding string, writer io.Writer, content []byte) (bool, string, error) {
if encoding == "" || len(content) < gzipMinLength {
_, err := writer.Write(content)
return false, "", err
}
return writeLevel(encoding, writer, bytes.NewReader(content), gzipCompressLevel)
}
// writeLevel reads from reader,writes to writer by specific encoding and compress level
// the compress level is defined by deflate package
func writeLevel(encoding string, writer io.Writer, reader io.Reader, level int) (bool, string, error) {
var outputWriter resetWriter
var err error
var ce = noneCompressEncoder
if cf, ok := encoderMap[encoding]; ok {
ce = cf
}
encoding = ce.name
outputWriter = ce.encode(writer, level)
defer ce.put(outputWriter, level)
_, err = io.Copy(outputWriter, reader)
if err != nil {
return false, "", err
}
switch outputWriter.(type) {
case io.WriteCloser:
outputWriter.(io.WriteCloser).Close()
}
return encoding != "", encoding, nil
}
// ParseEncoding will extract the right encoding for response
// the Accept-Encoding's sec is here:
// http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
func ParseEncoding(r *http.Request) string {
if r == nil {
return ""
}
if (getMethodOnly && r.Method == "GET") || includedMethods[r.Method] {
return parseEncoding(r)
}
return ""
}
type q struct {
name string
value float64
}
func parseEncoding(r *http.Request) string {
acceptEncoding := r.Header.Get("Accept-Encoding")
if acceptEncoding == "" {
return ""
}
var lastQ q
for _, v := range strings.Split(acceptEncoding, ",") {
v = strings.TrimSpace(v)
if v == "" {
continue
}
vs := strings.Split(v, ";")
var cf acceptEncoder
var ok bool
if cf, ok = encoderMap[vs[0]]; !ok {
continue
}
if len(vs) == 1 {
return cf.name
}
if len(vs) == 2 {
f, _ := strconv.ParseFloat(strings.Replace(vs[1], "q=", "", -1), 64)
if f == 0 {
continue
}
if f > lastQ.value {
lastQ = q{cf.name, f}
}
}
}
return lastQ.name
}

59
vender/github.com/astaxie/beego/context/acceptencoder_test.go
View File

@ -1,59 +0,0 @@
// Copyright 2015 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package context
import (
"net/http"
"testing"
)
func Test_ExtractEncoding(t *testing.T) {
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"gzip,deflate"}}}) != "gzip" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"deflate,gzip"}}}) != "deflate" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"gzip;q=.5,deflate"}}}) != "deflate" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"gzip;q=.5,deflate;q=0.3"}}}) != "gzip" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"gzip;q=0,deflate"}}}) != "deflate" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"deflate;q=0.5,gzip;q=0.5,identity"}}}) != "" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"*"}}}) != "gzip" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"x,gzip,deflate"}}}) != "gzip" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"gzip,x,deflate"}}}) != "gzip" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"gzip;q=0.5,x,deflate"}}}) != "deflate" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"x"}}}) != "" {
t.Fail()
}
if parseEncoding(&http.Request{Header: map[string][]string{"Accept-Encoding": {"gzip;q=0.5,x;q=0.8"}}}) != "gzip" {
t.Fail()
}
}

262
vender/github.com/astaxie/beego/context/context.go
View File

@ -1,262 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package context provide the context utils
// Usage:
//
// import "github.com/cnlh/nps/vender/github.com/astaxie/beego/context"
//
// ctx := context.Context{Request:req,ResponseWriter:rw}
//
// more docs http://beego.me/docs/module/context.md
package context
import (
"bufio"
"crypto/hmac"
"crypto/sha1"
"encoding/base64"
"errors"
"fmt"
"net"
"net/http"
"strconv"
"strings"
"time"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/utils"
)
//commonly used mime-types
const (
ApplicationJSON = "application/json"
ApplicationXML = "application/xml"
ApplicationYAML = "application/x-yaml"
TextXML = "text/xml"
)
// NewContext return the Context with Input and Output
func NewContext() *Context {
return &Context{
Input: NewInput(),
Output: NewOutput(),
}
}
// Context Http request context struct including BeegoInput, BeegoOutput, http.Request and http.ResponseWriter.
// BeegoInput and BeegoOutput provides some api to operate request and response more easily.
type Context struct {
Input *BeegoInput
Output *BeegoOutput
Request *http.Request
ResponseWriter *Response
_xsrfToken string
}
// Reset init Context, BeegoInput and BeegoOutput
func (ctx *Context) Reset(rw http.ResponseWriter, r *http.Request) {
ctx.Request = r
if ctx.ResponseWriter == nil {
ctx.ResponseWriter = &Response{}
}
ctx.ResponseWriter.reset(rw)
ctx.Input.Reset(ctx)
ctx.Output.Reset(ctx)
ctx._xsrfToken = ""
}
// Redirect does redirection to localurl with http header status code.
func (ctx *Context) Redirect(status int, localurl string) {
http.Redirect(ctx.ResponseWriter, ctx.Request, localurl, status)
}
// Abort stops this request.
// if beego.ErrorMaps exists, panic body.
func (ctx *Context) Abort(status int, body string) {
ctx.Output.SetStatus(status)
panic(body)
}
// WriteString Write string to response body.
// it sends response body.
func (ctx *Context) WriteString(content string) {
ctx.ResponseWriter.Write([]byte(content))
}
// GetCookie Get cookie from request by a given key.
// It's alias of BeegoInput.Cookie.
func (ctx *Context) GetCookie(key string) string {
return ctx.Input.Cookie(key)
}
// SetCookie Set cookie for response.
// It's alias of BeegoOutput.Cookie.
func (ctx *Context) SetCookie(name string, value string, others ...interface{}) {
ctx.Output.Cookie(name, value, others...)
}
// GetSecureCookie Get secure cookie from request by a given key.
func (ctx *Context) GetSecureCookie(Secret, key string) (string, bool) {
val := ctx.Input.Cookie(key)
if val == "" {
return "", false
}
parts := strings.SplitN(val, "|", 3)
if len(parts) != 3 {
return "", false
}
vs := parts[0]
timestamp := parts[1]
sig := parts[2]
h := hmac.New(sha1.New, []byte(Secret))
fmt.Fprintf(h, "%s%s", vs, timestamp)
if fmt.Sprintf("%02x", h.Sum(nil)) != sig {
return "", false
}
res, _ := base64.URLEncoding.DecodeString(vs)
return string(res), true
}
// SetSecureCookie Set Secure cookie for response.
func (ctx *Context) SetSecureCookie(Secret, name, value string, others ...interface{}) {
vs := base64.URLEncoding.EncodeToString([]byte(value))
timestamp := strconv.FormatInt(time.Now().UnixNano(), 10)
h := hmac.New(sha1.New, []byte(Secret))
fmt.Fprintf(h, "%s%s", vs, timestamp)
sig := fmt.Sprintf("%02x", h.Sum(nil))
cookie := strings.Join([]string{vs, timestamp, sig}, "|")
ctx.Output.Cookie(name, cookie, others...)
}
// XSRFToken creates a xsrf token string and returns.
func (ctx *Context) XSRFToken(key string, expire int64) string {
if ctx._xsrfToken == "" {
token, ok := ctx.GetSecureCookie(key, "_xsrf")
if !ok {
token = string(utils.RandomCreateBytes(32))
ctx.SetSecureCookie(key, "_xsrf", token, expire)
}
ctx._xsrfToken = token
}
return ctx._xsrfToken
}
// CheckXSRFCookie checks xsrf token in this request is valid or not.
// the token can provided in request header "X-Xsrftoken" and "X-CsrfToken"
// or in form field value named as "_xsrf".
func (ctx *Context) CheckXSRFCookie() bool {
token := ctx.Input.Query("_xsrf")
if token == "" {
token = ctx.Request.Header.Get("X-Xsrftoken")
}
if token == "" {
token = ctx.Request.Header.Get("X-Csrftoken")
}
if token == "" {
ctx.Abort(403, "'_xsrf' argument missing from POST")
return false
}
if ctx._xsrfToken != token {
ctx.Abort(403, "XSRF cookie does not match POST argument")
return false
}
return true
}
// RenderMethodResult renders the return value of a controller method to the output
func (ctx *Context) RenderMethodResult(result interface{}) {
if result != nil {
renderer, ok := result.(Renderer)
if !ok {
err, ok := result.(error)
if ok {
renderer = errorRenderer(err)
} else {
renderer = jsonRenderer(result)
}
}
renderer.Render(ctx)
}
}
//Response is a wrapper for the http.ResponseWriter
//started set to true if response was written to then don't execute other handler
type Response struct {
http.ResponseWriter
Started bool
Status int
}
func (r *Response) reset(rw http.ResponseWriter) {
r.ResponseWriter = rw
r.Status = 0
r.Started = false
}
// Write writes the data to the connection as part of an HTTP reply,
// and sets `started` to true.
// started means the response has sent out.
func (r *Response) Write(p []byte) (int, error) {
r.Started = true
return r.ResponseWriter.Write(p)
}
// WriteHeader sends an HTTP response header with status code,
// and sets `started` to true.
func (r *Response) WriteHeader(code int) {
if r.Status > 0 {
//prevent multiple response.WriteHeader calls
return
}
r.Status = code
r.Started = true
r.ResponseWriter.WriteHeader(code)
}
// Hijack hijacker for http
func (r *Response) Hijack() (net.Conn, *bufio.ReadWriter, error) {
hj, ok := r.ResponseWriter.(http.Hijacker)
if !ok {
return nil, nil, errors.New("webserver doesn't support hijacking")
}
return hj.Hijack()
}
// Flush http.Flusher
func (r *Response) Flush() {
if f, ok := r.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
// CloseNotify http.CloseNotifier
func (r *Response) CloseNotify() <-chan bool {
if cn, ok := r.ResponseWriter.(http.CloseNotifier); ok {
return cn.CloseNotify()
}
return nil
}
// Pusher http.Pusher
func (r *Response) Pusher() (pusher http.Pusher) {
if pusher, ok := r.ResponseWriter.(http.Pusher); ok {
return pusher
}
return nil
}

47
vender/github.com/astaxie/beego/context/context_test.go
View File

@ -1,47 +0,0 @@
// Copyright 2016 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package context
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestXsrfReset_01(t *testing.T) {
r := &http.Request{}
c := NewContext()
c.Request = r
c.ResponseWriter = &Response{}
c.ResponseWriter.reset(httptest.NewRecorder())
c.Output.Reset(c)
c.Input.Reset(c)
c.XSRFToken("key", 16)
if c._xsrfToken == "" {
t.FailNow()
}
token := c._xsrfToken
c.Reset(&Response{ResponseWriter: httptest.NewRecorder()}, r)
if c._xsrfToken != "" {
t.FailNow()
}
c.XSRFToken("key", 16)
if c._xsrfToken == "" {
t.FailNow()
}
if token == c._xsrfToken {
t.FailNow()
}
}

668
vender/github.com/astaxie/beego/context/input.go
View File

@ -1,668 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package context
import (
"bytes"
"compress/gzip"
"errors"
"io"
"io/ioutil"
"net"
"net/http"
"net/url"
"reflect"
"regexp"
"strconv"
"strings"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/session"
)
// Regexes for checking the accept headers
// TODO make sure these are correct
var (
acceptsHTMLRegex = regexp.MustCompile(`(text/html|application/xhtml\+xml)(?:,|$)`)
acceptsXMLRegex = regexp.MustCompile(`(application/xml|text/xml)(?:,|$)`)
acceptsJSONRegex = regexp.MustCompile(`(application/json)(?:,|$)`)
acceptsYAMLRegex = regexp.MustCompile(`(application/x-yaml)(?:,|$)`)
maxParam = 50
)
// BeegoInput operates the http request header, data, cookie and body.
// it also contains router params and current session.
type BeegoInput struct {
Context *Context
CruSession session.Store
pnames []string
pvalues []string
data map[interface{}]interface{} // store some values in this context when calling context in filter or controller.
RequestBody []byte
RunMethod string
RunController reflect.Type
}
// NewInput return BeegoInput generated by Context.
func NewInput() *BeegoInput {
return &BeegoInput{
pnames: make([]string, 0, maxParam),
pvalues: make([]string, 0, maxParam),
data: make(map[interface{}]interface{}),
}
}
// Reset init the BeegoInput
func (input *BeegoInput) Reset(ctx *Context) {
input.Context = ctx
input.CruSession = nil
input.pnames = input.pnames[:0]
input.pvalues = input.pvalues[:0]
input.data = nil
input.RequestBody = []byte{}
}
// Protocol returns request protocol name, such as HTTP/1.1 .
func (input *BeegoInput) Protocol() string {
return input.Context.Request.Proto
}
// URI returns full request url with query string, fragment.
func (input *BeegoInput) URI() string {
return input.Context.Request.RequestURI
}
// URL returns request url path (without query string, fragment).
func (input *BeegoInput) URL() string {
return input.Context.Request.URL.Path
}
// Site returns base site url as scheme://domain type.
func (input *BeegoInput) Site() string {
return input.Scheme() + "://" + input.Domain()
}
// Scheme returns request scheme as "http" or "https".
func (input *BeegoInput) Scheme() string {
if scheme := input.Header("X-Forwarded-Proto"); scheme != "" {
return scheme
}
if input.Context.Request.URL.Scheme != "" {
return input.Context.Request.URL.Scheme
}
if input.Context.Request.TLS == nil {
return "http"
}
return "https"
}
// Domain returns host name.
// Alias of Host method.
func (input *BeegoInput) Domain() string {
return input.Host()
}
// Host returns host name.
// if no host info in request, return localhost.
func (input *BeegoInput) Host() string {
if input.Context.Request.Host != "" {
if hostPart, _, err := net.SplitHostPort(input.Context.Request.Host); err == nil {
return hostPart
}
return input.Context.Request.Host
}
return "localhost"
}
// Method returns http request method.
func (input *BeegoInput) Method() string {
return input.Context.Request.Method
}
// Is returns boolean of this request is on given method, such as Is("POST").
func (input *BeegoInput) Is(method string) bool {
return input.Method() == method
}
// IsGet Is this a GET method request?
func (input *BeegoInput) IsGet() bool {
return input.Is("GET")
}
// IsPost Is this a POST method request?
func (input *BeegoInput) IsPost() bool {
return input.Is("POST")
}
// IsHead Is this a Head method request?
func (input *BeegoInput) IsHead() bool {
return input.Is("HEAD")
}
// IsOptions Is this a OPTIONS method request?
func (input *BeegoInput) IsOptions() bool {
return input.Is("OPTIONS")
}
// IsPut Is this a PUT method request?
func (input *BeegoInput) IsPut() bool {
return input.Is("PUT")
}
// IsDelete Is this a DELETE method request?
func (input *BeegoInput) IsDelete() bool {
return input.Is("DELETE")
}
// IsPatch Is this a PATCH method request?
func (input *BeegoInput) IsPatch() bool {
return input.Is("PATCH")
}
// IsAjax returns boolean of this request is generated by ajax.
func (input *BeegoInput) IsAjax() bool {
return input.Header("X-Requested-With") == "XMLHttpRequest"
}
// IsSecure returns boolean of this request is in https.
func (input *BeegoInput) IsSecure() bool {
return input.Scheme() == "https"
}
// IsWebsocket returns boolean of this request is in webSocket.
func (input *BeegoInput) IsWebsocket() bool {
return input.Header("Upgrade") == "websocket"
}
// IsUpload returns boolean of whether file uploads in this request or not..
func (input *BeegoInput) IsUpload() bool {
return strings.Contains(input.Header("Content-Type"), "multipart/form-data")
}
// AcceptsHTML Checks if request accepts html response
func (input *BeegoInput) AcceptsHTML() bool {
return acceptsHTMLRegex.MatchString(input.Header("Accept"))
}
// AcceptsXML Checks if request accepts xml response
func (input *BeegoInput) AcceptsXML() bool {
return acceptsXMLRegex.MatchString(input.Header("Accept"))
}
// AcceptsJSON Checks if request accepts json response
func (input *BeegoInput) AcceptsJSON() bool {
return acceptsJSONRegex.MatchString(input.Header("Accept"))
}
// AcceptsYAML Checks if request accepts json response
func (input *BeegoInput) AcceptsYAML() bool {
return acceptsYAMLRegex.MatchString(input.Header("Accept"))
}
// IP returns request client ip.
// if in proxy, return first proxy id.
// if error, return RemoteAddr.
func (input *BeegoInput) IP() string {
ips := input.Proxy()
if len(ips) > 0 && ips[0] != "" {
rip, _, err := net.SplitHostPort(ips[0])
if err != nil {
rip = ips[0]
}
return rip
}
if ip, _, err := net.SplitHostPort(input.Context.Request.RemoteAddr); err == nil {
return ip
}
return input.Context.Request.RemoteAddr
}
// Proxy returns proxy client ips slice.
func (input *BeegoInput) Proxy() []string {
if ips := input.Header("X-Forwarded-For"); ips != "" {
return strings.Split(ips, ",")
}
return []string{}
}
// Referer returns http referer header.
func (input *BeegoInput) Referer() string {
return input.Header("Referer")
}
// Refer returns http referer header.
func (input *BeegoInput) Refer() string {
return input.Referer()
}
// SubDomains returns sub domain string.
// if aa.bb.domain.com, returns aa.bb .
func (input *BeegoInput) SubDomains() string {
parts := strings.Split(input.Host(), ".")
if len(parts) >= 3 {
return strings.Join(parts[:len(parts)-2], ".")
}
return ""
}
// Port returns request client port.
// when error or empty, return 80.
func (input *BeegoInput) Port() int {
if _, portPart, err := net.SplitHostPort(input.Context.Request.Host); err == nil {
port, _ := strconv.Atoi(portPart)
return port
}
return 80
}
// UserAgent returns request client user agent string.
func (input *BeegoInput) UserAgent() string {
return input.Header("User-Agent")
}
// ParamsLen return the length of the params
func (input *BeegoInput) ParamsLen() int {
return len(input.pnames)
}
// Param returns router param by a given key.
func (input *BeegoInput) Param(key string) string {
for i, v := range input.pnames {
if v == key && i <= len(input.pvalues) {
return input.pvalues[i]
}
}
return ""
}
// Params returns the map[key]value.
func (input *BeegoInput) Params() map[string]string {
m := make(map[string]string)
for i, v := range input.pnames {
if i <= len(input.pvalues) {
m[v] = input.pvalues[i]
}
}
return m
}
// SetParam will set the param with key and value
func (input *BeegoInput) SetParam(key, val string) {
// check if already exists
for i, v := range input.pnames {
if v == key && i <= len(input.pvalues) {
input.pvalues[i] = val
return
}
}
input.pvalues = append(input.pvalues, val)
input.pnames = append(input.pnames, key)
}
// ResetParams clears any of the input's Params
// This function is used to clear parameters so they may be reset between filter
// passes.
func (input *BeegoInput) ResetParams() {
input.pnames = input.pnames[:0]
input.pvalues = input.pvalues[:0]
}
// Query returns input data item string by a given string.
func (input *BeegoInput) Query(key string) string {
if val := input.Param(key); val != "" {
return val
}
if input.Context.Request.Form == nil {
input.Context.Request.ParseForm()
}
return input.Context.Request.Form.Get(key)
}
// Header returns request header item string by a given string.
// if non-existed, return empty string.
func (input *BeegoInput) Header(key string) string {
return input.Context.Request.Header.Get(key)
}
// Cookie returns request cookie item string by a given key.
// if non-existed, return empty string.
func (input *BeegoInput) Cookie(key string) string {
ck, err := input.Context.Request.Cookie(key)
if err != nil {
return ""
}
return ck.Value
}
// Session returns current session item value by a given key.
// if non-existed, return nil.
func (input *BeegoInput) Session(key interface{}) interface{} {
return input.CruSession.Get(key)
}
// CopyBody returns the raw request body data as bytes.
func (input *BeegoInput) CopyBody(MaxMemory int64) []byte {
if input.Context.Request.Body == nil {
return []byte{}
}
var requestbody []byte
safe := &io.LimitedReader{R: input.Context.Request.Body, N: MaxMemory}
if input.Header("Content-Encoding") == "gzip" {
reader, err := gzip.NewReader(safe)
if err != nil {
return nil
}
requestbody, _ = ioutil.ReadAll(reader)
} else {
requestbody, _ = ioutil.ReadAll(safe)
}
input.Context.Request.Body.Close()
bf := bytes.NewBuffer(requestbody)
input.Context.Request.Body = http.MaxBytesReader(input.Context.ResponseWriter, ioutil.NopCloser(bf), MaxMemory)
input.RequestBody = requestbody
return requestbody
}
// Data return the implicit data in the input
func (input *BeegoInput) Data() map[interface{}]interface{} {
if input.data == nil {
input.data = make(map[interface{}]interface{})
}
return input.data
}
// GetData returns the stored data in this context.
func (input *BeegoInput) GetData(key interface{}) interface{} {
if v, ok := input.data[key]; ok {
return v
}
return nil
}
// SetData stores data with given key in this context.
// This data are only available in this context.
func (input *BeegoInput) SetData(key, val interface{}) {
if input.data == nil {
input.data = make(map[interface{}]interface{})
}
input.data[key] = val
}
// ParseFormOrMulitForm parseForm or parseMultiForm based on Content-type
func (input *BeegoInput) ParseFormOrMulitForm(maxMemory int64) error {
// Parse the body depending on the content type.
if strings.Contains(input.Header("Content-Type"), "multipart/form-data") {
if err := input.Context.Request.ParseMultipartForm(maxMemory); err != nil {
return errors.New("Error parsing request body:" + err.Error())
}
} else if err := input.Context.Request.ParseForm(); err != nil {
return errors.New("Error parsing request body:" + err.Error())
}
return nil
}
// Bind data from request.Form[key] to dest
// like /?id=123&isok=true&ft=1.2&ol[0]=1&ol[1]=2&ul[]=str&ul[]=array&user.Name=astaxie
// var id int beegoInput.Bind(&id, "id") id ==123
// var isok bool beegoInput.Bind(&isok, "isok") isok ==true
// var ft float64 beegoInput.Bind(&ft, "ft") ft ==1.2
// ol := make([]int, 0, 2) beegoInput.Bind(&ol, "ol") ol ==[1 2]
// ul := make([]string, 0, 2) beegoInput.Bind(&ul, "ul") ul ==[str array]
// user struct{Name} beegoInput.Bind(&user, "user") user == {Name:"astaxie"}
func (input *BeegoInput) Bind(dest interface{}, key string) error {
value := reflect.ValueOf(dest)
if value.Kind() != reflect.Ptr {
return errors.New("beego: non-pointer passed to Bind: " + key)
}
value = value.Elem()
if !value.CanSet() {
return errors.New("beego: non-settable variable passed to Bind: " + key)
}
typ := value.Type()
// Get real type if dest define with interface{}.
// e.g var dest interface{} dest=1.0
if value.Kind() == reflect.Interface {
typ = value.Elem().Type()
}
rv := input.bind(key, typ)
if !rv.IsValid() {
return errors.New("beego: reflect value is empty")
}
value.Set(rv)
return nil
}
func (input *BeegoInput) bind(key string, typ reflect.Type) reflect.Value {
if input.Context.Request.Form == nil {
input.Context.Request.ParseForm()
}
rv := reflect.Zero(typ)
switch typ.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
val := input.Query(key)
if len(val) == 0 {
return rv
}
rv = input.bindInt(val, typ)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
val := input.Query(key)
if len(val) == 0 {
return rv
}
rv = input.bindUint(val, typ)
case reflect.Float32, reflect.Float64:
val := input.Query(key)
if len(val) == 0 {
return rv
}
rv = input.bindFloat(val, typ)
case reflect.String:
val := input.Query(key)
if len(val) == 0 {
return rv
}
rv = input.bindString(val, typ)
case reflect.Bool:
val := input.Query(key)
if len(val) == 0 {
return rv
}
rv = input.bindBool(val, typ)
case reflect.Slice:
rv = input.bindSlice(&input.Context.Request.Form, key, typ)
case reflect.Struct:
rv = input.bindStruct(&input.Context.Request.Form, key, typ)
case reflect.Ptr:
rv = input.bindPoint(key, typ)
case reflect.Map:
rv = input.bindMap(&input.Context.Request.Form, key, typ)
}
return rv
}
func (input *BeegoInput) bindValue(val string, typ reflect.Type) reflect.Value {
rv := reflect.Zero(typ)
switch typ.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
rv = input.bindInt(val, typ)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
rv = input.bindUint(val, typ)
case reflect.Float32, reflect.Float64:
rv = input.bindFloat(val, typ)
case reflect.String:
rv = input.bindString(val, typ)
case reflect.Bool:
rv = input.bindBool(val, typ)
case reflect.Slice:
rv = input.bindSlice(&url.Values{"": {val}}, "", typ)
case reflect.Struct:
rv = input.bindStruct(&url.Values{"": {val}}, "", typ)
case reflect.Ptr:
rv = input.bindPoint(val, typ)
case reflect.Map:
rv = input.bindMap(&url.Values{"": {val}}, "", typ)
}
return rv
}
func (input *BeegoInput) bindInt(val string, typ reflect.Type) reflect.Value {
intValue, err := strconv.ParseInt(val, 10, 64)
if err != nil {
return reflect.Zero(typ)
}
pValue := reflect.New(typ)
pValue.Elem().SetInt(intValue)
return pValue.Elem()
}
func (input *BeegoInput) bindUint(val string, typ reflect.Type) reflect.Value {
uintValue, err := strconv.ParseUint(val, 10, 64)
if err != nil {
return reflect.Zero(typ)
}
pValue := reflect.New(typ)
pValue.Elem().SetUint(uintValue)
return pValue.Elem()
}
func (input *BeegoInput) bindFloat(val string, typ reflect.Type) reflect.Value {
floatValue, err := strconv.ParseFloat(val, 64)
if err != nil {
return reflect.Zero(typ)
}
pValue := reflect.New(typ)
pValue.Elem().SetFloat(floatValue)
return pValue.Elem()
}
func (input *BeegoInput) bindString(val string, typ reflect.Type) reflect.Value {
return reflect.ValueOf(val)
}
func (input *BeegoInput) bindBool(val string, typ reflect.Type) reflect.Value {
val = strings.TrimSpace(strings.ToLower(val))
switch val {
case "true", "on", "1":
return reflect.ValueOf(true)
}
return reflect.ValueOf(false)
}
type sliceValue struct {
index int // Index extracted from brackets. If -1, no index was provided.
value reflect.Value // the bound value for this slice element.
}
func (input *BeegoInput) bindSlice(params *url.Values, key string, typ reflect.Type) reflect.Value {
maxIndex := -1
numNoIndex := 0
sliceValues := []sliceValue{}
for reqKey, vals := range *params {
if !strings.HasPrefix(reqKey, key+"[") {
continue
}
// Extract the index, and the index where a sub-key starts. (e.g. field[0].subkey)
index := -1
leftBracket, rightBracket := len(key), strings.Index(reqKey[len(key):], "]")+len(key)
if rightBracket > leftBracket+1 {
index, _ = strconv.Atoi(reqKey[leftBracket+1 : rightBracket])
}
subKeyIndex := rightBracket + 1
// Handle the indexed case.
if index > -1 {
if index > maxIndex {
maxIndex = index
}
sliceValues = append(sliceValues, sliceValue{
index: index,
value: input.bind(reqKey[:subKeyIndex], typ.Elem()),
})
continue
}
// It's an un-indexed element. (e.g. element[])
numNoIndex += len(vals)
for _, val := range vals {
// Unindexed values can only be direct-bound.
sliceValues = append(sliceValues, sliceValue{
index: -1,
value: input.bindValue(val, typ.Elem()),
})
}
}
resultArray := reflect.MakeSlice(typ, maxIndex+1, maxIndex+1+numNoIndex)
for _, sv := range sliceValues {
if sv.index != -1 {
resultArray.Index(sv.index).Set(sv.value)
} else {
resultArray = reflect.Append(resultArray, sv.value)
}
}
return resultArray
}
func (input *BeegoInput) bindStruct(params *url.Values, key string, typ reflect.Type) reflect.Value {
result := reflect.New(typ).Elem()
fieldValues := make(map[string]reflect.Value)
for reqKey, val := range *params {
var fieldName string
if strings.HasPrefix(reqKey, key+".") {
fieldName = reqKey[len(key)+1:]
} else if strings.HasPrefix(reqKey, key+"[") && reqKey[len(reqKey)-1] == ']' {
fieldName = reqKey[len(key)+1 : len(reqKey)-1]
} else {
continue
}
if _, ok := fieldValues[fieldName]; !ok {
// Time to bind this field. Get it and make sure we can set it.
fieldValue := result.FieldByName(fieldName)
if !fieldValue.IsValid() {
continue
}
if !fieldValue.CanSet() {
continue
}
boundVal := input.bindValue(val[0], fieldValue.Type())
fieldValue.Set(boundVal)
fieldValues[fieldName] = boundVal
}
}
return result
}
func (input *BeegoInput) bindPoint(key string, typ reflect.Type) reflect.Value {
return input.bind(key, typ.Elem()).Addr()
}
func (input *BeegoInput) bindMap(params *url.Values, key string, typ reflect.Type) reflect.Value {
var (
result = reflect.MakeMap(typ)
keyType = typ.Key()
valueType = typ.Elem()
)
for paramName, values := range *params {
if !strings.HasPrefix(paramName, key+"[") || paramName[len(paramName)-1] != ']' {
continue
}
key := paramName[len(key)+1 : len(paramName)-1]
result.SetMapIndex(input.bindValue(key, keyType), input.bindValue(values[0], valueType))
}
return result
}

207
vender/github.com/astaxie/beego/context/input_test.go
View File

@ -1,207 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package context
import (
"net/http"
"net/http/httptest"
"reflect"
"testing"
)
func TestBind(t *testing.T) {
type testItem struct {
field string
empty interface{}
want interface{}
}
type Human struct {
ID int
Nick string
Pwd string
Ms bool
}
cases := []struct {
request string
valueGp []testItem
}{
{"/?p=str", []testItem{{"p", interface{}(""), interface{}("str")}}},
{"/?p=", []testItem{{"p", "", ""}}},
{"/?p=str", []testItem{{"p", "", "str"}}},
{"/?p=123", []testItem{{"p", 0, 123}}},
{"/?p=123", []testItem{{"p", uint(0), uint(123)}}},
{"/?p=1.0", []testItem{{"p", 0.0, 1.0}}},
{"/?p=1", []testItem{{"p", false, true}}},
{"/?p=true", []testItem{{"p", false, true}}},
{"/?p=ON", []testItem{{"p", false, true}}},
{"/?p=on", []testItem{{"p", false, true}}},
{"/?p=1", []testItem{{"p", false, true}}},
{"/?p=2", []testItem{{"p", false, false}}},
{"/?p=false", []testItem{{"p", false, false}}},
{"/?p[a]=1&p[b]=2&p[c]=3", []testItem{{"p", map[string]int{}, map[string]int{"a": 1, "b": 2, "c": 3}}}},
{"/?p[a]=v1&p[b]=v2&p[c]=v3", []testItem{{"p", map[string]string{}, map[string]string{"a": "v1", "b": "v2", "c": "v3"}}}},
{"/?p[]=8&p[]=9&p[]=10", []testItem{{"p", []int{}, []int{8, 9, 10}}}},
{"/?p[0]=8&p[1]=9&p[2]=10", []testItem{{"p", []int{}, []int{8, 9, 10}}}},
{"/?p[0]=8&p[1]=9&p[2]=10&p[5]=14", []testItem{{"p", []int{}, []int{8, 9, 10, 0, 0, 14}}}},
{"/?p[0]=8.0&p[1]=9.0&p[2]=10.0", []testItem{{"p", []float64{}, []float64{8.0, 9.0, 10.0}}}},
{"/?p[]=10&p[]=9&p[]=8", []testItem{{"p", []string{}, []string{"10", "9", "8"}}}},
{"/?p[0]=8&p[1]=9&p[2]=10", []testItem{{"p", []string{}, []string{"8", "9", "10"}}}},
{"/?p[0]=true&p[1]=false&p[2]=true&p[5]=1&p[6]=ON&p[7]=other", []testItem{{"p", []bool{}, []bool{true, false, true, false, false, true, true, false}}}},
{"/?human.Nick=astaxie", []testItem{{"human", Human{}, Human{Nick: "astaxie"}}}},
{"/?human.ID=888&human.Nick=astaxie&human.Ms=true&human[Pwd]=pass", []testItem{{"human", Human{}, Human{ID: 888, Nick: "astaxie", Ms: true, Pwd: "pass"}}}},
{"/?human[0].ID=888&human[0].Nick=astaxie&human[0].Ms=true&human[0][Pwd]=pass01&human[1].ID=999&human[1].Nick=ysqi&human[1].Ms=On&human[1].Pwd=pass02",
[]testItem{{"human", []Human{}, []Human{
{ID: 888, Nick: "astaxie", Ms: true, Pwd: "pass01"},
{ID: 999, Nick: "ysqi", Ms: true, Pwd: "pass02"},
}}}},
{
"/?id=123&isok=true&ft=1.2&ol[0]=1&ol[1]=2&ul[]=str&ul[]=array&human.Nick=astaxie",
[]testItem{
{"id", 0, 123},
{"isok", false, true},
{"ft", 0.0, 1.2},
{"ol", []int{}, []int{1, 2}},
{"ul", []string{}, []string{"str", "array"}},
{"human", Human{}, Human{Nick: "astaxie"}},
},
},
}
for _, c := range cases {
r, _ := http.NewRequest("GET", c.request, nil)
beegoInput := NewInput()
beegoInput.Context = NewContext()
beegoInput.Context.Reset(httptest.NewRecorder(), r)
for _, item := range c.valueGp {
got := item.empty
err := beegoInput.Bind(&got, item.field)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(got, item.want) {
t.Fatalf("Bind %q error,should be:\n%#v \ngot:\n%#v", item.field, item.want, got)
}
}
}
}
func TestSubDomain(t *testing.T) {
r, _ := http.NewRequest("GET", "http://www.example.com/?id=123&isok=true&ft=1.2&ol[0]=1&ol[1]=2&ul[]=str&ul[]=array&user.Name=astaxie", nil)
beegoInput := NewInput()
beegoInput.Context = NewContext()
beegoInput.Context.Reset(httptest.NewRecorder(), r)
subdomain := beegoInput.SubDomains()
if subdomain != "www" {
t.Fatal("Subdomain parse error, got" + subdomain)
}
r, _ = http.NewRequest("GET", "http://localhost/", nil)
beegoInput.Context.Request = r
if beegoInput.SubDomains() != "" {
t.Fatal("Subdomain parse error, should be empty, got " + beegoInput.SubDomains())
}
r, _ = http.NewRequest("GET", "http://aa.bb.example.com/", nil)
beegoInput.Context.Request = r
if beegoInput.SubDomains() != "aa.bb" {
t.Fatal("Subdomain parse error, got " + beegoInput.SubDomains())
}
/* TODO Fix this
r, _ = http.NewRequest("GET", "http://127.0.0.1/", nil)
beegoInput.Context.Request = r
if beegoInput.SubDomains() != "" {
t.Fatal("Subdomain parse error, got " + beegoInput.SubDomains())
}
*/
r, _ = http.NewRequest("GET", "http://example.com/", nil)
beegoInput.Context.Request = r
if beegoInput.SubDomains() != "" {
t.Fatal("Subdomain parse error, got " + beegoInput.SubDomains())
}
r, _ = http.NewRequest("GET", "http://aa.bb.cc.dd.example.com/", nil)
beegoInput.Context.Request = r
if beegoInput.SubDomains() != "aa.bb.cc.dd" {
t.Fatal("Subdomain parse error, got " + beegoInput.SubDomains())
}
}
func TestParams(t *testing.T) {
inp := NewInput()
inp.SetParam("p1", "val1_ver1")
inp.SetParam("p2", "val2_ver1")
inp.SetParam("p3", "val3_ver1")
if l := inp.ParamsLen(); l != 3 {
t.Fatalf("Input.ParamsLen wrong value: %d, expected %d", l, 3)
}
if val := inp.Param("p1"); val != "val1_ver1" {
t.Fatalf("Input.Param wrong value: %s, expected %s", val, "val1_ver1")
}
if val := inp.Param("p3"); val != "val3_ver1" {
t.Fatalf("Input.Param wrong value: %s, expected %s", val, "val3_ver1")
}
vals := inp.Params()
expected := map[string]string{
"p1": "val1_ver1",
"p2": "val2_ver1",
"p3": "val3_ver1",
}
if !reflect.DeepEqual(vals, expected) {
t.Fatalf("Input.Params wrong value: %s, expected %s", vals, expected)
}
// overwriting existing params
inp.SetParam("p1", "val1_ver2")
inp.SetParam("p2", "val2_ver2")
expected = map[string]string{
"p1": "val1_ver2",
"p2": "val2_ver2",
"p3": "val3_ver1",
}
vals = inp.Params()
if !reflect.DeepEqual(vals, expected) {
t.Fatalf("Input.Params wrong value: %s, expected %s", vals, expected)
}
if l := inp.ParamsLen(); l != 3 {
t.Fatalf("Input.ParamsLen wrong value: %d, expected %d", l, 3)
}
if val := inp.Param("p1"); val != "val1_ver2" {
t.Fatalf("Input.Param wrong value: %s, expected %s", val, "val1_ver2")
}
if val := inp.Param("p2"); val != "val2_ver2" {
t.Fatalf("Input.Param wrong value: %s, expected %s", val, "val1_ver2")
}
}

395
vender/github.com/astaxie/beego/context/output.go
View File

@ -1,395 +0,0 @@
// Copyright 2014 beego Author. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package context
import (
"bytes"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"html/template"
"io"
"mime"
"net/http"
"net/url"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"gopkg.in/yaml.v2"
)
// BeegoOutput does work for sending response header.
type BeegoOutput struct {
Context *Context
Status int
EnableGzip bool
}
// NewOutput returns new BeegoOutput.
// it contains nothing now.
func NewOutput() *BeegoOutput {
return &BeegoOutput{}
}
// Reset init BeegoOutput
func (output *BeegoOutput) Reset(ctx *Context) {
output.Context = ctx
output.Status = 0
}
// Header sets response header item string via given key.
func (output *BeegoOutput) Header(key, val string) {
output.Context.ResponseWriter.Header().Set(key, val)
}
// Body sets response body content.
// if EnableGzip, compress content string.
// it sends out response body directly.
func (output *BeegoOutput) Body(content []byte) error {
var encoding string
var buf = &bytes.Buffer{}
if output.EnableGzip {
encoding = ParseEncoding(output.Context.Request)
}
if b, n, _ := WriteBody(encoding, buf, content); b {
output.Header("Content-Encoding", n)
output.Header("Content-Length", strconv.Itoa(buf.Len()))
} else {
output.Header("Content-Length", strconv.Itoa(len(content)))
}
// Write status code if it has been set manually
// Set it to 0 afterwards to prevent "multiple response.WriteHeader calls"
if output.Status != 0 {
output.Context.ResponseWriter.WriteHeader(output.Status)
output.Status = 0
} else {
output.Context.ResponseWriter.Started = true
}
io.Copy(output.Context.ResponseWriter, buf)
return nil
}
// Cookie sets cookie value via given key.
// others are ordered as cookie's max age time, path,domain, secure and httponly.
func (output *BeegoOutput) Cookie(name string, value string, others ...interface{}) {
var b bytes.Buffer
fmt.Fprintf(&b, "%s=%s", sanitizeName(name), sanitizeValue(value))
//fix cookie not work in IE
if len(others) > 0 {
var maxAge int64
switch v := others[0].(type) {
case int:
maxAge = int64(v)
case int32:
maxAge = int64(v)
case int64:
maxAge = v
}
switch {
case maxAge > 0:
fmt.Fprintf(&b, "; Expires=%s; Max-Age=%d", time.Now().Add(time.Duration(maxAge)*time.Second).UTC().Format(time.RFC1123), maxAge)
case maxAge < 0:
fmt.Fprintf(&b, "; Max-Age=0")
}
}
// the settings below
// Path, Domain, Secure, HttpOnly
// can use nil skip set
// default "/"
if len(others) > 1 {
if v, ok := others[1].(string); ok && len(v) > 0 {
fmt.Fprintf(&b, "; Path=%s", sanitizeValue(v))
}
} else {
fmt.Fprintf(&b, "; Path=%s", "/")
}
// default empty
if len(others) > 2 {
if v, ok := others[2].(string); ok && len(v) > 0 {
fmt.Fprintf(&b, "; Domain=%s", sanitizeValue(v))
}
}
// default empty
if len(others) > 3 {
var secure bool
switch v := others[3].(type) {
case bool:
secure = v
default:
if others[3] != nil {
secure = true
}
}
if secure {
fmt.Fprintf(&b, "; Secure")
}
}
// default false. for session cookie default true
if len(others) > 4 {
if v, ok := others[4].(bool); ok && v {
fmt.Fprintf(&b, "; HttpOnly")
}
}
output.Context.ResponseWriter.Header().Add("Set-Cookie", b.String())
}
var cookieNameSanitizer = strings.NewReplacer("\n", "-", "\r", "-")
func sanitizeName(n string) string {
return cookieNameSanitizer.Replace(n)
}
var cookieValueSanitizer = strings.NewReplacer("\n", " ", "\r", " ", ";", " ")
func sanitizeValue(v string) string {
return cookieValueSanitizer.Replace(v)
}
func jsonRenderer(value interface{}) Renderer {
return rendererFunc(func(ctx *Context) {
ctx.Output.JSON(value, false, false)
})
}
func errorRenderer(err error) Renderer {
return rendererFunc(func(ctx *Context) {
ctx.Output.SetStatus(500)
ctx.Output.Body([]byte(err.Error()))
})
}
// JSON writes json to response body.
// if encoding is true, it converts utf-8 to \u0000 type.
func (output *BeegoOutput) JSON(data interface{}, hasIndent bool, encoding bool) error {
output.Header("Content-Type", "application/json; charset=utf-8")
var content []byte
var err error
if hasIndent {
content, err = json.MarshalIndent(data, "", " ")
} else {
content, err = json.Marshal(data)
}
if err != nil {
http.Error(output.Context.ResponseWriter, err.Error(), http.StatusInternalServerError)
return err
}
if encoding {
content = []byte(stringsToJSON(string(content)))
}
return output.Body(content)
}
// YAML writes yaml to response body.
func (output *BeegoOutput) YAML(data interface{}) error {
output.Header("Content-Type", "application/x-yaml; charset=utf-8")
var content []byte
var err error
content, err = yaml.Marshal(data)
if err != nil {
http.Error(output.Context.ResponseWriter, err.Error(), http.StatusInternalServerError)
return err
}
return output.Body(content)
}
// JSONP writes jsonp to response body.
func (output *BeegoOutput) JSONP(data interface{}, hasIndent bool) error {
output.Header("Content-Type", "application/javascript; charset=utf-8")
var content []byte
var err error
if hasIndent {
content, err = json.MarshalIndent(data, "", " ")
} else {
content, err = json.Marshal(data)
}
if err != nil {
http.Error(output.Context.ResponseWriter, err.Error(), http.StatusInternalServerError)
return err
}
callback := output.Context.Input.Query("callback")
if callback == "" {
return errors.New(`"callback" parameter required`)
}
callback = template.JSEscapeString(callback)
callbackContent := bytes.NewBufferString(" if(window." + callback + ")" + callback)
callbackContent.WriteString("(")
callbackContent.Write(content)
callbackContent.WriteString(");\r\n")
return output.Body(callbackContent.Bytes())
}
// XML writes xml string to response body.
func (output *BeegoOutput) XML(data interface{}, hasIndent bool) error {
output.Header("Content-Type", "application/xml; charset=utf-8")
var content []byte
var err error
if hasIndent {
content, err = xml.MarshalIndent(data, "", " ")
} else {
content, err = xml.Marshal(data)
}
if err != nil {
http.Error(output.Context.ResponseWriter, err.Error(), http.StatusInternalServerError)
return err
}
return output.Body(content)
}
// ServeFormatted serve YAML, XML OR JSON, depending on the value of the Accept header
func (output *BeegoOutput) ServeFormatted(data interface{}, hasIndent bool, hasEncode ...bool) {
accept := output.Context.Input.Header("Accept")
switch accept {
case ApplicationYAML:
output.YAML(data)
case ApplicationXML, TextXML:
output.XML(data, hasIndent)
default:
output.JSON(data, hasIndent, len(hasEncode) > 0 && hasEncode[0])
}
}
// Download forces response for download file.
// it prepares the download response header automatically.
func (output *BeegoOutput) Download(file string, filename ...string) {
// check get file error, file not found or other error.
if _, err := os.Stat(file); err != nil {
http.ServeFile(output.Context.ResponseWriter, output.Context.Request, file)
return
}
var fName string
if len(filename) > 0 && filename[0] != "" {
fName = filename[0]
} else {
fName = filepath.Base(file)
}
output.Header("Content-Disposition", "attachment; filename="+url.PathEscape(fName))
output.Header("Content-Description", "File Transfer")
output.Header("Content-Type", "application/octet-stream")
output.Header("Content-Transfer-Encoding", "binary")
output.Header("Expires", "0")
output.Header("Cache-Control", "must-revalidate")
output.Header("Pragma", "public")
http.ServeFile(output.Context.ResponseWriter, output.Context.Request, file)
}
// ContentType sets the content type from ext string.
// MIME type is given in mime package.
func (output *BeegoOutput) ContentType(ext string) {
if !strings.HasPrefix(ext, ".") {
ext = "." + ext
}
ctype := mime.TypeByExtension(ext)
if ctype != "" {
output.Header("Content-Type", ctype)
}
}
// SetStatus sets response status code.
// It writes response header directly.
func (output *BeegoOutput) SetStatus(status int) {
output.Status = status
}
// IsCachable returns boolean of this request is cached.
// HTTP 304 means cached.
func (output *BeegoOutput) IsCachable() bool {
return output.Status >= 200 && output.Status < 300 || output.Status == 304
}
// IsEmpty returns boolean of this request is empty.
// HTTP 201，204 and 304 means empty.
func (output *BeegoOutput) IsEmpty() bool {
return output.Status == 201 || output.Status == 204 || output.Status == 304
}
// IsOk returns boolean of this request runs well.
// HTTP 200 means ok.
func (output *BeegoOutput) IsOk() bool {
return output.Status == 200
}
// IsSuccessful returns boolean of this request runs successfully.
// HTTP 2xx means ok.
func (output *BeegoOutput) IsSuccessful() bool {
return output.Status >= 200 && output.Status < 300
}
// IsRedirect returns boolean of this request is redirection header.
// HTTP 301,302,307 means redirection.
func (output *BeegoOutput) IsRedirect() bool {
return output.Status == 301 || output.Status == 302 || output.Status == 303 || output.Status == 307
}
// IsForbidden returns boolean of this request is forbidden.
// HTTP 403 means forbidden.
func (output *BeegoOutput) IsForbidden() bool {
return output.Status == 403
}
// IsNotFound returns boolean of this request is not found.
// HTTP 404 means not found.
func (output *BeegoOutput) IsNotFound() bool {
return output.Status == 404
}
// IsClientError returns boolean of this request client sends error data.
// HTTP 4xx means client error.
func (output *BeegoOutput) IsClientError() bool {
return output.Status >= 400 && output.Status < 500
}
// IsServerError returns boolean of this server handler errors.
// HTTP 5xx means server internal error.
func (output *BeegoOutput) IsServerError() bool {
return output.Status >= 500 && output.Status < 600
}
func stringsToJSON(str string) string {
var jsons bytes.Buffer
for _, r := range str {
rint := int(r)
if rint < 128 {
jsons.WriteRune(r)
} else {
jsons.WriteString("\\u")
if rint < 0x100 {
jsons.WriteString("00")
} else if rint < 0x1000 {
jsons.WriteString("0")
}
jsons.WriteString(strconv.FormatInt(int64(rint), 16))
}
}
return jsons.String()
}
// Session sets session item value with given key.
func (output *BeegoOutput) Session(name interface{}, value interface{}) {
output.Context.Input.CruSession.Set(name, value)
}

78
vender/github.com/astaxie/beego/context/param/conv.go
View File

@ -1,78 +0,0 @@
package param
import (
"fmt"
"reflect"
beecontext "github.com/cnlh/nps/vender/github.com/astaxie/beego/context"
"github.com/cnlh/nps/vender/github.com/astaxie/beego/logs"
)
// ConvertParams converts http method params to values that will be passed to the method controller as arguments
func ConvertParams(methodParams []*MethodParam, methodType reflect.Type, ctx *beecontext.Context) (result []reflect.Value) {
result = make([]reflect.Value, 0, len(methodParams))
for i := 0; i < len(methodParams); i++ {
reflectValue := convertParam(methodParams[i], methodType.In(i), ctx)
result = append(result, reflectValue)
}
return
}
func convertParam(param *MethodParam, paramType reflect.Type, ctx *beecontext.Context) (result reflect.Value) {
paramValue := getParamValue(param, ctx)
if paramValue == "" {
if param.required {
ctx.Abort(400, fmt.Sprintf("Missing parameter %s", param.name))
} else {
paramValue = param.defaultValue
}
}
reflectValue, err := parseValue(param, paramValue, paramType)
if err != nil {
logs.Debug(fmt.Sprintf("Error converting param %s to type %s. Value: %v, Error: %s", param.name, paramType, paramValue, err))
ctx.Abort(400, fmt.Sprintf("Invalid parameter %s. Can not convert %v to type %s", param.name, paramValue, paramType))
}
return reflectValue
}
func getParamValue(param *MethodParam, ctx *beecontext.Context) string {
switch param.in {
case body:
return string(ctx.Input.RequestBody)
case header:
return ctx.Input.Header(param.name)
case path:
return ctx.Input.Query(":" + param.name)
default:
return ctx.Input.Query(param.name)
}
}
func parseValue(param *MethodParam, paramValue string, paramType reflect.Type) (result reflect.Value, err error) {
if paramValue == "" {
return reflect.Zero(paramType), nil
}
parser := getParser(param, paramType)
value, err := parser.parse(paramValue, paramType)
if err != nil {
return result, err
}
return safeConvert(reflect.ValueOf(value), paramType)
}
func safeConvert(value reflect.Value, t reflect.Type) (result reflect.Value, err error) {
defer func() {
if r := recover(); r != nil {
var ok bool
err, ok = r.(error)
if !ok {
err = fmt.Errorf("%v", r)
}
}
}()
result = value.Convert(t)
return
}

69
vender/github.com/astaxie/beego/context/param/methodparams.go
View File

@ -1,69 +0,0 @@
package param
import (
"fmt"
"strings"
)
//MethodParam keeps param information to be auto passed to controller methods
type MethodParam struct {
name string
in paramType
required bool
defaultValue string
}
type paramType byte
const (
param paramType = iota
path
body
header
)
//New creates a new MethodParam with name and specific options
func New(name string, opts ...MethodParamOption) *MethodParam {
return newParam(name, nil, opts)
}
func newParam(name string, parser paramParser, opts []MethodParamOption) (param *MethodParam) {
param = &MethodParam{name: name}
for _, option := range opts {
option(param)
}
return
}
//Make creates an array of MethodParmas or an empty array
func Make(list ...*MethodParam) []*MethodParam {
if len(list) > 0 {
return list
}
return nil
}
func (mp *MethodParam) String() string {
options := []string{}
result := "param.New(\"" + mp.name + "\""
if mp.required {
options = append(options, "param.IsRequired")
}
switch mp.in {
case path:
options = append(options, "param.InPath")
case body:
options = append(options, "param.InBody")
case header:
options = append(options, "param.InHeader")
}
if mp.defaultValue != "" {
options = append(options, fmt.Sprintf(`param.Default("%s")`, mp.defaultValue))
}
if len(options) > 0 {
result += ", "
}
result += strings.Join(options, ", ")
result += ")"
return result
}

37
vender/github.com/astaxie/beego/context/param/options.go
View File

@ -1,37 +0,0 @@
package param
import (
"fmt"
)
// MethodParamOption defines a func which apply options on a MethodParam
type MethodParamOption func(*MethodParam)
// IsRequired indicates that this param is required and can not be omitted from the http request
var IsRequired MethodParamOption = func(p *MethodParam) {
p.required = true
}
// InHeader indicates that this param is passed via an http header
var InHeader MethodParamOption = func(p *MethodParam) {
p.in = header
}
// InPath indicates that this param is part of the URL path
var InPath MethodParamOption = func(p *MethodParam) {
p.in = path
}
// InBody indicates that this param is passed as an http request body
var InBody MethodParamOption = func(p *MethodParam) {
p.in = body
}
// Default provides a default value for the http param
func Default(defaultValue interface{}) MethodParamOption {
return func(p *MethodParam) {
if defaultValue != nil {
p.defaultValue = fmt.Sprint(defaultValue)
}
}
}

149
vender/github.com/astaxie/beego/context/param/parsers.go
View File

@ -1,149 +0,0 @@
package param
import (
"encoding/json"
"reflect"
"strconv"
"strings"
"time"
)
type paramParser interface {
parse(value string, toType reflect.Type) (interface{}, error)
}
func getParser(param *MethodParam, t reflect.Type) paramParser {
switch t.Kind() {
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
return intParser{}
case reflect.Slice:
if t.Elem().Kind() == reflect.Uint8 { //treat []byte as string
return stringParser{}
}
if param.in == body {
return jsonParser{}
}
elemParser := getParser(param, t.Elem())
if elemParser == (jsonParser{}) {
return elemParser
}
return sliceParser(elemParser)
case reflect.Bool:
return boolParser{}
case reflect.String:
return stringParser{}
case reflect.Float32, reflect.Float64:
return floatParser{}
case reflect.Ptr:
elemParser := getParser(param, t.Elem())
if elemParser == (jsonParser{}) {
return elemParser
}
return ptrParser(elemParser)
default:
if t.PkgPath() == "time" && t.Name() == "Time" {
return timeParser{}
}
return jsonParser{}
}
}
type parserFunc func(value string, toType reflect.Type) (interface{}, error)
func (f parserFunc) parse(value string, toType reflect.Type) (interface{}, error) {
return f(value, toType)
}
type boolParser struct {
}
func (p boolParser) parse(value string, toType reflect.Type) (interface{}, error) {
return strconv.ParseBool(value)
}
type stringParser struct {
}
func (p stringParser) parse(value string, toType reflect.Type) (interface{}, error) {
return value, nil
}
type intParser struct {
}
func (p intParser) parse(value string, toType reflect.Type) (interface{}, error) {
return strconv.Atoi(value)
}
type floatParser struct {
}
func (p floatParser) parse(value string, toType reflect.Type) (interface{}, error) {
if toType.Kind() == reflect.Float32 {
res, err := strconv.ParseFloat(value, 32)
if err != nil {
return nil, err
}
return float32(res), nil
}
return strconv.ParseFloat(value, 64)
}
type timeParser struct {
}
func (p timeParser) parse(value string, toType reflect.Type) (result interface{}, err error) {
result, err = time.Parse(time.RFC3339, value)
if err != nil {
result, err = time.Parse("2006-01-02", value)
}
return
}
type jsonParser struct {
}
func (p jsonParser) parse(value string, toType reflect.Type) (interface{}, error) {
pResult := reflect.New(toType)
v := pResult.Interface()
err := json.Unmarshal([]byte(value), v)
if err != nil {
return nil, err
}
return pResult.Elem().Interface(), nil
}
func sliceParser(elemParser paramParser) paramParser {
return parserFunc(func(value string, toType reflect.Type) (interface{}, error) {
values := strings.Split(value, ",")
result := reflect.MakeSlice(toType, 0, len(values))
elemType := toType.Elem()
for _, v := range values {
parsedValue, err := elemParser.parse(v, elemType)
if err != nil {
return nil, err
}
result = reflect.Append(result, reflect.ValueOf(parsedValue))
}
return result.Interface(), nil
})
}
func ptrParser(elemParser paramParser) paramParser {
return parserFunc(func(value string, toType reflect.Type) (interface{}, error) {
parsedValue, err := elemParser.parse(value, toType.Elem())
if err != nil {
return nil, err
}
newValPtr := reflect.New(toType.Elem())
newVal := reflect.Indirect(newValPtr)
convertedVal, err := safeConvert(reflect.ValueOf(parsedValue), toType.Elem())
if err != nil {
return nil, err
}
newVal.Set(convertedVal)
return newValPtr.Interface(), nil
})
}

84
vender/github.com/astaxie/beego/context/param/parsers_test.go
View File

@ -1,84 +0,0 @@
package param
import "testing"
import "reflect"
import "time"
type testDefinition struct {
strValue string
expectedValue interface{}
expectedParser paramParser
}
func Test_Parsers(t *testing.T) {
//ints
checkParser(testDefinition{"1", 1, intParser{}}, t)
checkParser(testDefinition{"-1", int64(-1), intParser{}}, t)
checkParser(testDefinition{"1", uint64(1), intParser{}}, t)
//floats
checkParser(testDefinition{"1.0", float32(1.0), floatParser{}}, t)
checkParser(testDefinition{"-1.0", float64(-1.0), floatParser{}}, t)
//strings
checkParser(testDefinition{"AB", "AB", stringParser{}}, t)
checkParser(testDefinition{"AB", []byte{65, 66}, stringParser{}}, t)
//bools
checkParser(testDefinition{"true", true, boolParser{}}, t)
checkParser(testDefinition{"0", false, boolParser{}}, t)
//timeParser
checkParser(testDefinition{"2017-05-30T13:54:53Z", time.Date(2017, 5, 30, 13, 54, 53, 0, time.UTC), timeParser{}}, t)
checkParser(testDefinition{"2017-05-30", time.Date(2017, 5, 30, 0, 0, 0, 0, time.UTC), timeParser{}}, t)
//json
checkParser(testDefinition{`{"X": 5, "Y":"Z"}`, struct {
X int
Y string
}{5, "Z"}, jsonParser{}}, t)
//slice in query is parsed as comma delimited
checkParser(testDefinition{`1,2`, []int{1, 2}, sliceParser(intParser{})}, t)
//slice in body is parsed as json
checkParser(testDefinition{`["a","b"]`, []string{"a", "b"}, jsonParser{}}, t, MethodParam{in: body})
//pointers
var someInt = 1
checkParser(testDefinition{`1`, &someInt, ptrParser(intParser{})}, t)
var someStruct = struct{ X int }{5}
checkParser(testDefinition{`{"X": 5}`, &someStruct, jsonParser{}}, t)
}
func checkParser(def testDefinition, t *testing.T, methodParam ...MethodParam) {
toType := reflect.TypeOf(def.expectedValue)
var mp MethodParam
if len(methodParam) == 0 {
mp = MethodParam{}
} else {
mp = methodParam[0]
}
parser := getParser(&mp, toType)
if reflect.TypeOf(parser) != reflect.TypeOf(def.expectedParser) {
t.Errorf("Invalid parser for value %v. Expected: %v, actual: %v", def.strValue, reflect.TypeOf(def.expectedParser).Name(), reflect.TypeOf(parser).Name())
return
}
result, err := parser.parse(def.strValue, toType)
if err != nil {
t.Errorf("Parsing error for value %v. Expected result: %v, error: %v", def.strValue, def.expectedValue, err)
return
}
convResult, err := safeConvert(reflect.ValueOf(result), toType)
if err != nil {
t.Errorf("Conversion error for %v. from value: %v, toType: %v, error: %v", def.strValue, result, toType, err)
return
}
if !reflect.DeepEqual(convResult.Interface(), def.expectedValue) {
t.Errorf("Parsing error for value %v. Expected result: %v, actual: %v", def.strValue, def.expectedValue, result)
}
}

12
vender/github.com/astaxie/beego/context/renderer.go
View File

@ -1,12 +0,0 @@
package context
// Renderer defines an http response renderer
type Renderer interface {
Render(ctx *Context)
}
type rendererFunc func(ctx *Context)
func (f rendererFunc) Render(ctx *Context) {
f(ctx)
}

27
vender/github.com/astaxie/beego/context/response.go
View File

@ -1,27 +0,0 @@
package context
import (
"strconv"
"net/http"
)
const (
//BadRequest indicates http error 400
BadRequest StatusCode = http.StatusBadRequest
//NotFound indicates http error 404
NotFound StatusCode = http.StatusNotFound
)
// StatusCode sets the http response status code
type StatusCode int
func (s StatusCode) Error() string {
return strconv.Itoa(int(s))
}
// Render sets the http status code
func (s StatusCode) Render(ctx *Context) {
ctx.Output.SetStatus(int(s))
}

Some files were not shown because too many files have changed in this diff Show More