jumpserver/apps/ops/ws.py

107 lines
3.9 KiB
Python
Raw Normal View History

2022-11-16 06:27:50 +00:00
import asyncio
2019-10-10 04:47:21 +00:00
import os
2022-11-16 06:27:50 +00:00
import aiofiles
from asgiref.sync import sync_to_async
2022-11-16 06:27:50 +00:00
from channels.generic.websocket import AsyncJsonWebsocketConsumer
from common.db.utils import close_old_connections
2022-11-16 06:27:50 +00:00
from common.utils import get_logger
from .ansible.utils import get_ansible_task_log_path
2022-11-16 06:27:50 +00:00
from .celery.utils import get_celery_task_log_path
v3.0.0-rc1 (#9322) * perf:automation * pref: 修改账号推送 * perf: 修改 assets * perf: 修改 accounts * feat: 优化代码 * fix: 修复 ObjectRelatedField 获取 value attr 时先判断是否有 attr 属性 * perf: 增加翻译 * feat: 增加部分翻译 * feat: 去除无用列 * perf: ticket remove app * fix: 修复创建账号备份任务失败的问题 * perf: 添加 accounts app * perf: ticket type serializer (#9252) Co-authored-by: feng <1304903146@qq.com> * perf: ticket * perf: 修改 accounts api * perf: 优化 AssetPermissionSerializer fields 顺序 * perf: 修改 accounts * feat: 限制常用用户名api返回长度 * feat: 限制常用用户名api返回长度 * perf: 修改 LoginAssetACL 序列类,增加 users_username_group, accounts_username_group... 字段 * perf: 修改 CommandFilterACLSerializer 增加 command_groups_amount 字段 * perf: 修改rbac API啥的 (#9254) * perf: migrate * perf: 修改 AssetPermedSerializer domain 字段类型 * perf: 放开push account 权限位 * perf: 修改 accounts * perf: 修改 LoginACLSerializer 字段类型 * pref: 修改数据库 migrations * perf: filter asset systemuser * perf: 修改 SessionSerializer 字段类型 * pref: 修改 applet host * perf: 修改 SessionCommandSerializer 字段类型 * perf: 修改 accounts import * perf: 修改 celery datetime * perf: 修改 asset serializer * pref: 修改 labeled field * feat: 修改翻译 * perf: 修改 JobSerializer 字段类型 * feat: 支持使用 ws 发送终断任务 * perf: add AccessTokenAuthentication * perf: 修改 BaseStorageSerializer 字段类型 * perf: 修改 AppletHostSerializer 字段类型 * perf: signal event * perf: asset types automations (#9259) Co-authored-by: feng <1304903146@qq.com> * perf: 修改下载 rdp 文件时返回的 address 地址信息为空的问题 * perf: 修改 AssetSerializer.accounts.secret 为 write_only; 修改 DomainWithGatewaySerializer.gateways 返回 account 信息及 secret 字段; * perf: automation 干库 (#9260) Co-authored-by: feng <1304903146@qq.com> * perf: account push api * feat: 修改迁移文件 * feat: 删除无用代码 * feat: 优化部分资源无操作日志 * perf: 修改 account * perf: perm tree * perf: asset serializers retrieve * perf: 格式化代码 * perf: AutomationExecution (#9268) Co-authored-by: feng <1304903146@qq.com> * perf: AssetDetailSerializer 和 Asset Model 添加 specific_info 字段; * perf: 修改账号推送 * feat: handle ws heartbeat status * perf: k8s tree (#9269) Co-authored-by: feng <1304903146@qq.com> * perf: 修改账号推送 * perf: 修改 asset detail serializer * fix: 修复 windows 不能运行 powershell 命令的问题 * feat: 支持按照资源时间线查看操作活动 * feat: 翻译 * feat: 优化操作日志 * perf: asset clone * fix: 错误的修改改回去 * perf: create asset account * feat: 增加task 刷新续传功能 * fix: applet host deloypment filter host * perf: 修改了 common 结构,和 push accounts * perf: 整理 common 结构 * perf: 修改 const import * perf: 修改 allow bulk destroy * fix: applet host search fileds * perf: applet bulk delete * fix: applet list 404 * perf: 修改 common view * feat: 增加一些翻译, 修复 playbook 上传的错误 * fix: 修改错别字 * perf: 修改 applets status * perf: 修改网关 api * perf: automateion (#9281) Co-authored-by: feng <1304903146@qq.com> Co-authored-by: feng626 <57284900+feng626@users.noreply.github.com> * perf: 失效 connect methods 当 applet 删除 或者 host 删除 * perf: 网关账号的密码类型改成 LabelField * perf: chrome applet script * perf: verify code ttl (#9282) Co-authored-by: feng <1304903146@qq.com> * perf: database ping * perf: ws * perf: 修改网关创建 * perf: account task org (#9285) Co-authored-by: feng <1304903146@qq.com> * perf: asset test api * perf: port 添加 account * pref: 修改 db mapper permission * fix: db port mapper list api * perf: account change secret (#9286) Co-authored-by: feng <1304903146@qq.com> * perf: 修改 setup_eager_loading * perf: SecretStrategy * feat: 修改 ConnectionToken Create API 支持校验 ACL 逻辑 * feat: 修改 ConnectionToken Create API 支持校验 ACL 逻辑 * feat: 修改 ConnectionToken Create API 支持校验 ACL 逻辑 * pref: web database 信号转发 * perf: account push automation * perf: push filter account * perf: 修改 publish 版本 * perf: 修改网关 * fix: 修改资产 Specific 信息中 JSONField 字段返回 json.loads 对象 * feat: 远程应用内置Navicat Premium 16 * feat: 更新下载链接 * feat: 整理代码格式 * perf: 修改 terminal point * perf: update chrome applet script * fix: 资产 specific 获取 JSONField 时, 判断值的类型不为 list, dict * perf: domain (#9292) Co-authored-by: feng <1304903146@qq.com> * perf: 优化 endpoint 监听端口,仅 oracle 动态 * perf: 修改翻译 * perf: 修改文案 * perf: 修改缺失的翻译 * perf: 修改 endpoint help text * feat: 还原格式 * feat: 去掉基类 * feat: 增加特权账号字段 * perf: decode content * fix: check pid * perf: 修改 smart endpoint * perf: 修改 endpoint mysql default port * feat: 优化 * perf: 修改 endpoint mysql default port * perf: gateway test (#9295) Co-authored-by: feng <1304903146@qq.com> * perf: migrate * perf: 修改 endpoint mysql default port * fix: 修复获取任务执行结果死循环 * feat: 作业审计日志增加字段 * fix: add on_transaction_commit task post save * perf: gateway (#9297) Co-authored-by: feng <1304903146@qq.com> * feat: 过滤 jumpserver 自动产生的用户 * fix: 修复ops节点选择的问题 * fix: 修改 统一 connection-token 和 command 的 review API 返回数据 from_ticket_info * perf: change secret (#9298) Co-authored-by: feng <1304903146@qq.com> * perf: 修改 db port manager * perf: 修改 db port manager * perf: add celery log mark * perf: remove debug log data * fix: navicat use manual type * fix: remove navicate download url * perf: push_account_enabled (#9301) Co-authored-by: feng <1304903146@qq.com> * fix: 修改navicat启动程序MD5值 * perf: push account (#9303) Co-authored-by: feng <1304903146@qq.com> * feat: Redis/MongoDB 支持SSL * fix: 修改授权规则过滤字段 node_name,node_id; 修复获取授权节点下的资产为空的问题; * perf: push account button (#9305) Co-authored-by: feng <1304903146@qq.com> * perf: account push * fix: 修复获取 /user//assets/tree/ 返回用户授权的所有资产 * perf: asset ping (#9307) Co-authored-by: feng <1304903146@qq.com> * perf: asset enabled_info * perf: 优化activity记录都保存至operatelog中 * feat: 远程应用navicat支持试用版连接 * perf: 优化迁移文件 * perf: 修改资产列表 API category type 字段 choices 根据 category 进行返回 * fix * perf: 修改账号列表 API 解决根据 node_id asset_id 搜索账号列表无效的问题 * fix: navicat dba账号登录 * perf: 优化navicat连接 * perf: 修改账号列表 Model Manager 继承自 OrgManager,解决组织过滤问题 * perf: 修改账号列表 Filter 支持根据 platform,category,type 字段搜索 * perf: change secret email (#9312) Co-authored-by: feng <1304903146@qq.com> * feat: 保证认证信息一定清理 * perf: add mariadb * perf: 修改资产类型树数量统计资产或账号 * perf: applet chrome quit * perf: 优化关闭欢迎页面 * fix * perf: executed amount * perf: 修改 built-in applet installation * perf: 修改资产列表增加标签搜索 * perf: 修改资产列表增加标签搜索 * perf: account task automation (#9319) Co-authored-by: feng <1304903146@qq.com> * perf: account trigger * perf: 修改系统设置文案:批量命令执行 -> 作业中心 * perf: 优化migrate (#9320) Co-authored-by: feng <1304903146@qq.com> * perf: 修改资产节点树 API,支持搜索资产、节点 * perf: audit dashboard (#9321) Co-authored-by: feng <1304903146@qq.com> * fix: 修改 has_perm 权限判断兼容 list 和 str 类型 * perf: 修改一些换行 * perf: 修改 ansible config * fix: oracle依赖文件地址错误 (#9324) * perf: ansible mudules * perf: 修改 runner host cwd Co-authored-by: ibuler <ibuler@qq.com> Co-authored-by: Aaron3S <chenyang@fit2cloud.com> Co-authored-by: Bai <baijiangjie@gmail.com> Co-authored-by: feng <1304903146@qq.com> Co-authored-by: feng626 <57284900+feng626@users.noreply.github.com> Co-authored-by: Eric <xplzv@126.com> Co-authored-by: jiangweidong <weidong.jiang@fit2cloud.com> Co-authored-by: jiangweidong <80373698+Hi-JWD@users.noreply.github.com>
2023-01-16 11:02:09 +00:00
from .const import CELERY_LOG_MAGIC_MARK
from .models import CeleryTaskExecution
logger = get_logger(__name__)
2022-11-16 06:27:50 +00:00
class TaskLogWebsocket(AsyncJsonWebsocketConsumer):
disconnected = False
user_tasks = (
'ops.tasks.run_ops_job',
'ops.tasks.run_ops_job_execution',
)
log_types = {
'celery': get_celery_task_log_path,
'ansible': get_ansible_task_log_path
}
2022-11-16 06:27:50 +00:00
async def connect(self):
2021-01-14 02:10:10 +00:00
user = self.scope["user"]
if user.is_authenticated:
2022-11-16 06:27:50 +00:00
await self.accept()
2021-01-14 02:10:10 +00:00
else:
2022-11-16 06:27:50 +00:00
await self.close()
2022-11-16 06:27:50 +00:00
def get_log_path(self, task_id, log_type):
func = self.log_types.get(log_type)
if func:
return func(task_id)
@sync_to_async
def get_task(self, task_id):
task = CeleryTaskExecution.objects.filter(id=task_id).first()
# task.creator 是 foreign key, 会异步去查询的,在下面的 if task.creator 会报错, 所以这里先取出来
if task and task.creator != ' ':
return task
else:
return None
2022-11-16 06:27:50 +00:00
async def receive_json(self, content, **kwargs):
task_id = content.get('task')
task = await self.get_task(task_id)
if not task:
await self.send_json({'message': 'Task not found', 'task': task_id})
return
if task.name in self.user_tasks and task.creator != self.scope['user']:
await self.send_json({'message': 'No permission', 'task': task_id})
return
task_type = content.get('type', 'celery')
log_path = self.get_log_path(task_id, task_type)
2022-11-16 06:27:50 +00:00
await self.async_handle_task(task_id, log_path)
2019-10-10 04:47:21 +00:00
2022-11-16 06:27:50 +00:00
async def async_handle_task(self, task_id, log_path):
logger.info("Task id: {}".format(task_id))
2024-04-11 11:06:02 +00:00
timeout = 0
2019-10-23 02:14:07 +00:00
while not self.disconnected:
2024-04-11 11:06:02 +00:00
if timeout >= 60:
await self.send_json({'message': '\r\n', 'task': task_id})
await self.send_json({'message': 'Task log was not found, the directory may not be shared.',
'task': task_id})
break
2019-10-23 02:14:07 +00:00
if not os.path.exists(log_path):
2022-11-16 06:27:50 +00:00
await self.send_json({'message': '.', 'task': task_id})
2024-04-11 11:06:02 +00:00
timeout += 0.5
2022-11-16 06:27:50 +00:00
await asyncio.sleep(0.5)
else:
await self.send_task_log(task_id, log_path)
2019-10-23 02:14:07 +00:00
break
2022-11-16 06:27:50 +00:00
async def send_task_log(self, task_id, log_path):
await self.send_json({'message': '\r\n'})
try:
logger.debug('Task log path: {}'.format(log_path))
async with aiofiles.open(log_path, 'rb') as task_log_f:
while not self.disconnected:
data = await task_log_f.read(4096)
if data:
data = data.replace(b'\n', b'\r\n')
await self.send_json(
{'message': data.decode(errors='ignore'), 'task': task_id}
)
v3.0.0-rc1 (#9322) * perf:automation * pref: 修改账号推送 * perf: 修改 assets * perf: 修改 accounts * feat: 优化代码 * fix: 修复 ObjectRelatedField 获取 value attr 时先判断是否有 attr 属性 * perf: 增加翻译 * feat: 增加部分翻译 * feat: 去除无用列 * perf: ticket remove app * fix: 修复创建账号备份任务失败的问题 * perf: 添加 accounts app * perf: ticket type serializer (#9252) Co-authored-by: feng <1304903146@qq.com> * perf: ticket * perf: 修改 accounts api * perf: 优化 AssetPermissionSerializer fields 顺序 * perf: 修改 accounts * feat: 限制常用用户名api返回长度 * feat: 限制常用用户名api返回长度 * perf: 修改 LoginAssetACL 序列类,增加 users_username_group, accounts_username_group... 字段 * perf: 修改 CommandFilterACLSerializer 增加 command_groups_amount 字段 * perf: 修改rbac API啥的 (#9254) * perf: migrate * perf: 修改 AssetPermedSerializer domain 字段类型 * perf: 放开push account 权限位 * perf: 修改 accounts * perf: 修改 LoginACLSerializer 字段类型 * pref: 修改数据库 migrations * perf: filter asset systemuser * perf: 修改 SessionSerializer 字段类型 * pref: 修改 applet host * perf: 修改 SessionCommandSerializer 字段类型 * perf: 修改 accounts import * perf: 修改 celery datetime * perf: 修改 asset serializer * pref: 修改 labeled field * feat: 修改翻译 * perf: 修改 JobSerializer 字段类型 * feat: 支持使用 ws 发送终断任务 * perf: add AccessTokenAuthentication * perf: 修改 BaseStorageSerializer 字段类型 * perf: 修改 AppletHostSerializer 字段类型 * perf: signal event * perf: asset types automations (#9259) Co-authored-by: feng <1304903146@qq.com> * perf: 修改下载 rdp 文件时返回的 address 地址信息为空的问题 * perf: 修改 AssetSerializer.accounts.secret 为 write_only; 修改 DomainWithGatewaySerializer.gateways 返回 account 信息及 secret 字段; * perf: automation 干库 (#9260) Co-authored-by: feng <1304903146@qq.com> * perf: account push api * feat: 修改迁移文件 * feat: 删除无用代码 * feat: 优化部分资源无操作日志 * perf: 修改 account * perf: perm tree * perf: asset serializers retrieve * perf: 格式化代码 * perf: AutomationExecution (#9268) Co-authored-by: feng <1304903146@qq.com> * perf: AssetDetailSerializer 和 Asset Model 添加 specific_info 字段; * perf: 修改账号推送 * feat: handle ws heartbeat status * perf: k8s tree (#9269) Co-authored-by: feng <1304903146@qq.com> * perf: 修改账号推送 * perf: 修改 asset detail serializer * fix: 修复 windows 不能运行 powershell 命令的问题 * feat: 支持按照资源时间线查看操作活动 * feat: 翻译 * feat: 优化操作日志 * perf: asset clone * fix: 错误的修改改回去 * perf: create asset account * feat: 增加task 刷新续传功能 * fix: applet host deloypment filter host * perf: 修改了 common 结构,和 push accounts * perf: 整理 common 结构 * perf: 修改 const import * perf: 修改 allow bulk destroy * fix: applet host search fileds * perf: applet bulk delete * fix: applet list 404 * perf: 修改 common view * feat: 增加一些翻译, 修复 playbook 上传的错误 * fix: 修改错别字 * perf: 修改 applets status * perf: 修改网关 api * perf: automateion (#9281) Co-authored-by: feng <1304903146@qq.com> Co-authored-by: feng626 <57284900+feng626@users.noreply.github.com> * perf: 失效 connect methods 当 applet 删除 或者 host 删除 * perf: 网关账号的密码类型改成 LabelField * perf: chrome applet script * perf: verify code ttl (#9282) Co-authored-by: feng <1304903146@qq.com> * perf: database ping * perf: ws * perf: 修改网关创建 * perf: account task org (#9285) Co-authored-by: feng <1304903146@qq.com> * perf: asset test api * perf: port 添加 account * pref: 修改 db mapper permission * fix: db port mapper list api * perf: account change secret (#9286) Co-authored-by: feng <1304903146@qq.com> * perf: 修改 setup_eager_loading * perf: SecretStrategy * feat: 修改 ConnectionToken Create API 支持校验 ACL 逻辑 * feat: 修改 ConnectionToken Create API 支持校验 ACL 逻辑 * feat: 修改 ConnectionToken Create API 支持校验 ACL 逻辑 * pref: web database 信号转发 * perf: account push automation * perf: push filter account * perf: 修改 publish 版本 * perf: 修改网关 * fix: 修改资产 Specific 信息中 JSONField 字段返回 json.loads 对象 * feat: 远程应用内置Navicat Premium 16 * feat: 更新下载链接 * feat: 整理代码格式 * perf: 修改 terminal point * perf: update chrome applet script * fix: 资产 specific 获取 JSONField 时, 判断值的类型不为 list, dict * perf: domain (#9292) Co-authored-by: feng <1304903146@qq.com> * perf: 优化 endpoint 监听端口,仅 oracle 动态 * perf: 修改翻译 * perf: 修改文案 * perf: 修改缺失的翻译 * perf: 修改 endpoint help text * feat: 还原格式 * feat: 去掉基类 * feat: 增加特权账号字段 * perf: decode content * fix: check pid * perf: 修改 smart endpoint * perf: 修改 endpoint mysql default port * feat: 优化 * perf: 修改 endpoint mysql default port * perf: gateway test (#9295) Co-authored-by: feng <1304903146@qq.com> * perf: migrate * perf: 修改 endpoint mysql default port * fix: 修复获取任务执行结果死循环 * feat: 作业审计日志增加字段 * fix: add on_transaction_commit task post save * perf: gateway (#9297) Co-authored-by: feng <1304903146@qq.com> * feat: 过滤 jumpserver 自动产生的用户 * fix: 修复ops节点选择的问题 * fix: 修改 统一 connection-token 和 command 的 review API 返回数据 from_ticket_info * perf: change secret (#9298) Co-authored-by: feng <1304903146@qq.com> * perf: 修改 db port manager * perf: 修改 db port manager * perf: add celery log mark * perf: remove debug log data * fix: navicat use manual type * fix: remove navicate download url * perf: push_account_enabled (#9301) Co-authored-by: feng <1304903146@qq.com> * fix: 修改navicat启动程序MD5值 * perf: push account (#9303) Co-authored-by: feng <1304903146@qq.com> * feat: Redis/MongoDB 支持SSL * fix: 修改授权规则过滤字段 node_name,node_id; 修复获取授权节点下的资产为空的问题; * perf: push account button (#9305) Co-authored-by: feng <1304903146@qq.com> * perf: account push * fix: 修复获取 /user//assets/tree/ 返回用户授权的所有资产 * perf: asset ping (#9307) Co-authored-by: feng <1304903146@qq.com> * perf: asset enabled_info * perf: 优化activity记录都保存至operatelog中 * feat: 远程应用navicat支持试用版连接 * perf: 优化迁移文件 * perf: 修改资产列表 API category type 字段 choices 根据 category 进行返回 * fix * perf: 修改账号列表 API 解决根据 node_id asset_id 搜索账号列表无效的问题 * fix: navicat dba账号登录 * perf: 优化navicat连接 * perf: 修改账号列表 Model Manager 继承自 OrgManager,解决组织过滤问题 * perf: 修改账号列表 Filter 支持根据 platform,category,type 字段搜索 * perf: change secret email (#9312) Co-authored-by: feng <1304903146@qq.com> * feat: 保证认证信息一定清理 * perf: add mariadb * perf: 修改资产类型树数量统计资产或账号 * perf: applet chrome quit * perf: 优化关闭欢迎页面 * fix * perf: executed amount * perf: 修改 built-in applet installation * perf: 修改资产列表增加标签搜索 * perf: 修改资产列表增加标签搜索 * perf: account task automation (#9319) Co-authored-by: feng <1304903146@qq.com> * perf: account trigger * perf: 修改系统设置文案:批量命令执行 -> 作业中心 * perf: 优化migrate (#9320) Co-authored-by: feng <1304903146@qq.com> * perf: 修改资产节点树 API,支持搜索资产、节点 * perf: audit dashboard (#9321) Co-authored-by: feng <1304903146@qq.com> * fix: 修改 has_perm 权限判断兼容 list 和 str 类型 * perf: 修改一些换行 * perf: 修改 ansible config * fix: oracle依赖文件地址错误 (#9324) * perf: ansible mudules * perf: 修改 runner host cwd Co-authored-by: ibuler <ibuler@qq.com> Co-authored-by: Aaron3S <chenyang@fit2cloud.com> Co-authored-by: Bai <baijiangjie@gmail.com> Co-authored-by: feng <1304903146@qq.com> Co-authored-by: feng626 <57284900+feng626@users.noreply.github.com> Co-authored-by: Eric <xplzv@126.com> Co-authored-by: jiangweidong <weidong.jiang@fit2cloud.com> Co-authored-by: jiangweidong <80373698+Hi-JWD@users.noreply.github.com>
2023-01-16 11:02:09 +00:00
if data.find(CELERY_LOG_MAGIC_MARK) != -1:
await self.send_json(
{'event': 'end', 'task': task_id, 'message': ''}
)
logger.debug("Task log file magic mark found")
break
2022-11-16 06:27:50 +00:00
await asyncio.sleep(0.2)
except OSError as e:
logger.warn('Task log path open failed: {}'.format(e))
2019-10-10 04:47:21 +00:00
2022-11-16 06:27:50 +00:00
async def disconnect(self, close_code):
2019-10-10 04:47:21 +00:00
self.disconnected = True
close_old_connections()