4 Commits (53025178ef090087fe73aef94e636c42dd53fb3c)

Author	SHA1	Message	Date
Dakota Walsh	958a8e97e9	fix(migration): close the database before running backups EE-3627 (#7218) ... * fix(migration): close the database before running backups On certain filesystems, particuarly NTFS when a network mounted windows file server is used to store portainer's database, you are unable to copy the database while it is open. To fix this we simply close the database and then re-open it after a backup. * handle close and open errors * dont return error on nil	2 years ago
Prabhat Khera	e4241207cb	fix(database): db migration improvements EE-2688 (#6662)	3 years ago
Matt Hook	34cc8ea96a	feat(database): add encryption support EE-1983 (#6316) ... * bootstrap encryption key * secret key message change in cli and secret key file content trimmed * Migrate encryption code to latest version * pull in newer code * tidying up * working data encryption layer * fix tests * remove stray comment * fix a few minor issues and improve the comments * split out databasefilename with param to two methods to be more obvious * DB encryption integration (#6374) * json methods moved under DBConnection * store encryption fixed * cleaned * review comments addressed * newstore value fixed * backup test updated * logrus format config updated * Fix for newStore Co-authored-by: Matt Hook <hookenz@gmail.com> * Minor improvements * Improve the export code. Add missing webhook for import * rename HelmUserRepositorys to HelmUserRepositories * fix logging messages * when starting portainer with a key (first use) http is disabled by default. But when starting fresh without a key, http is enabled? * Fix bug for default settings on new installs Co-authored-by: Prabhat Khera <prabhat.khera@portainer.io> Co-authored-by: Prabhat Khera <91852476+prabhat-org@users.noreply.github.com>	3 years ago
Sven Dowideit	f99329eb7e	chore(store) EE-1981: Refactor/store/error checking, and other refactoring (#6173) ... * use the Store interface IsErrObjectNotFound() to avoid revealing internal errors Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * what happens when you extract the datastore interfaces into their own package Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * Start renaming Storage methods Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the boltdb specific code from the Portainer storage code (example, the others need the same) Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more extract bolt.Tx from datastore code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * minimise imports by putting moving the struct definition into the file that needs the Service imports Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more extraction of boltdb.Tx Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the use of bucket.SetSequence Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * almost done - just endpoint.Synchonise :/ Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * so, endpoint.Synchonize looks hard, but i can't find where we use it, so 'delete first refactoring' Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix test compile errors Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * test compile fixes after rebase Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix a mis-remembering I had wrt deserialisation - last time i used AnyData - jsoniter's bindTo looks interesting for the same reason Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * set us up to make the connection an interface Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * make the db connection a datastore interface, and separate out our datastore services from the bolt ones Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * rename methods to something less oltdb internals specific Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * these errors are not boltdb secific Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * start using the db-backend factory method too Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * export boltdb raw in case we can't export from the service layer Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add a raw export from boltdb to yaml for broken db's, and an export services to yaml in backup Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add the version info by hand for now Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * actually, the export from services can be fully typed - its the import that needs to do more work Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * redo raw export, and make import capable of using it Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add DockerHub Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * migration from anything older than v1.21.0 has been broken for quite a while, deleting the un-tested code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix go test ./... again Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * my goland wasn't setup to gofmt Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * move the two extremely dubious migration tests down into store, so they can use the test store code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * the migrator is now free of boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * reverse goland overzealous replcement of internal with boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more undo over-zealous goland internal->boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * yay, now bolt is only mentioned inside the api/database/ dir Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * and this might be the last of the boltdb references? Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add todo Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the store code into a separate module too Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * don't need the fileService in boltdb anymore Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * use IsErrObjectNotFound() Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * use a string to select what database backend we use Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * make isNew store an ephemeral bool that doesn't stay true after we've initialised it Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * move the import.json wip to a separate file so its more obvious - we'll be using it for testing, emergency fixups, and in the next part of the store work, when we improve migrations and data model lifecycles Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * undo vscode formatting html Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix app templates symbol (#6221) * feat(webhook) EE-2125 send registry auth haeder when update swarms service via webhook (#6220) * feat(webhook) EE-2125 add some helpers to registry utils * feat(webhook) EE-2125 persist registryID when creating a webhook * feat(webhook) EE-2125 send registry auth header when executing a webhook * feat(webhook) EE-2125 send registryID to backend when creating a service with webhook * feat(webhook) EE-2125 use the initial registry ID to create webhook on editing service screen * feat(webhook) EE-2125 update webhook when update registry * feat(webhook) EE-2125 add endpoint of update webhook * feat(webhook) EE-2125 code cleanup * feat(webhook) EE-2125 fix a typo * feat(webhook) EE-2125 fix circle import issue with unit test Co-authored-by: Simon Meng <simon.meng@portainer.io> * fix(kubeconfig): show kubeconfig download button for non admin users [EE-2123] (#6204) Co-authored-by: Simon Meng <simon.meng@portainer.io> * fix data-cy for k8s cluster menu (#6226) LGTM * feat(stack): make stack created from app template editable EE-1941 (#6104) feat(stack): make stack from app template editable * fix(container):disable Duplicate/Edit button when the container is portainer (#6223) * fix/ee-1909/show-pull-image-error (#6195) Co-authored-by: sunportainer <ericsun@SG1.local> * feat(cy): add data-cy to helm install button (#6241) * feat(cy): add data-cy to add registry button (#6242) * refactor(app): convert root folder files to es6 (#4159) * refactor(app): duplicate constants as es6 exports (#4158) * fix(docker): provide workaround to save network name variable (#6080) * fix/EE-1862/unable-to-stop-or-remove-stack workaround for var without default value in yaml file * fix/EE-1862/unable-to-stop-or-remove-stack check yaml file * fixed func and var names * wrapper error and used bool for stringset * UT case for createNetworkEnvFile * UT case for %s=%s * powerful StringSet * wrapper error for extract network name * wrapper all the return err * store more env * put to env file * make default value None * feat: gzip static resources (#6258) * fix(ssl)//handle --sslcert and --sslkey ee-2106 (#6203) * fix/ee-2106/handle-sslcert-sslkey Co-authored-by: sunportainer <ericsun@SG1.local> * fix(server):support disable https only ee-2068 (#6232) * fix/ee-2068/disable-forcely-https * feat(store): implement store tests EE-2112 (#6224) * add store tests * add some more tests * Update missing helm user repo methods * remove redundant comments * add webhook export * update webhooks * use the Store interface IsErrObjectNotFound() to avoid revealing internal errors Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * what happens when you extract the datastore interfaces into their own package Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * Start renaming Storage methods Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the boltdb specific code from the Portainer storage code (example, the others need the same) Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more extract bolt.Tx from datastore code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * minimise imports by putting moving the struct definition into the file that needs the Service imports Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more extraction of boltdb.Tx Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the use of bucket.SetSequence Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * almost done - just endpoint.Synchonise :/ Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * so, endpoint.Synchonize looks hard, but i can't find where we use it, so 'delete first refactoring' Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix test compile errors Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * test compile fixes after rebase Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix a mis-remembering I had wrt deserialisation - last time i used AnyData - jsoniter's bindTo looks interesting for the same reason Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * set us up to make the connection an interface Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * make the db connection a datastore interface, and separate out our datastore services from the bolt ones Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * rename methods to something less oltdb internals specific Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * these errors are not boltdb secific Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * start using the db-backend factory method too Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * export boltdb raw in case we can't export from the service layer Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add a raw export from boltdb to yaml for broken db's, and an export services to yaml in backup Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add the version info by hand for now Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * actually, the export from services can be fully typed - its the import that needs to do more work Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * redo raw export, and make import capable of using it Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add DockerHub Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * migration from anything older than v1.21.0 has been broken for quite a while, deleting the un-tested code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix go test ./... again Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * my goland wasn't setup to gofmt Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * move the two extremely dubious migration tests down into store, so they can use the test store code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * the migrator is now free of boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * reverse goland overzealous replcement of internal with boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more undo over-zealous goland internal->boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * yay, now bolt is only mentioned inside the api/database/ dir Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * and this might be the last of the boltdb references? Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add todo Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the store code into a separate module too Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * don't need the fileService in boltdb anymore Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * use IsErrObjectNotFound() Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * use a string to select what database backend we use Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * make isNew store an ephemeral bool that doesn't stay true after we've initialised it Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * move the import.json wip to a separate file so its more obvious - we'll be using it for testing, emergency fixups, and in the next part of the store work, when we improve migrations and data model lifecycles Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * undo vscode formatting html Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * Update missing helm user repo methods * feat(store): implement store tests EE-2112 (#6224) * add store tests * add some more tests * remove redundant comments * add webhook export * update webhooks * fix build issues after rebasing * move migratorparams * remove unneeded integer type conversions * disable the db import/export for now Co-authored-by: Richard Wei <54336863+WaysonWei@users.noreply.github.com> Co-authored-by: cong meng <mcpacino@gmail.com> Co-authored-by: Simon Meng <simon.meng@portainer.io> Co-authored-by: Marcelo Rydel <marcelorydel26@gmail.com> Co-authored-by: Hao Zhang <hao.zhang@portainer.io> Co-authored-by: sunportainer <93502624+sunportainer@users.noreply.github.com> Co-authored-by: sunportainer <ericsun@SG1.local> Co-authored-by: wheresolivia <78844659+wheresolivia@users.noreply.github.com> Co-authored-by: Chaim Lev-Ari <chiptus@users.noreply.github.com> Co-authored-by: Chao Geng <93526589+chaogeng77977@users.noreply.github.com> Co-authored-by: Dmitry Salakhov <to@dimasalakhov.com> Co-authored-by: Matt Hook <hookenz@gmail.com>	3 years ago