diff --git a/docs/en/api_reference/error.md b/docs/en/api_reference/error.md index 4ad74bd659c630a3b742d57c197f943ee22dc942..2e702e4d31c5a2f96047acc426e166bb13ea4801 100644 --- a/docs/en/api_reference/error.md +++ b/docs/en/api_reference/error.md @@ -1,26 +1,26 @@ # Public Errors -## openmind_hub.utils.OMValidationError +## openmind_hub.OMValidationError Parameter validation failed. -## openmind_hub.utils.EntryNotFoundError +## openmind_hub.EntryNotFoundError The file does not exist. -## openmind_hub.utils.RepositoryNotFoundError +## openmind_hub.RepositoryNotFoundError The repository does not exist. -## openmind_hub.utils.RevisionNotFoundError +## openmind_hub.RevisionNotFoundError The branch or committed version does not exist. -## openmind_hub.utils.GatedRepoError +## openmind_hub.GatedRepoError Access to the repository is restricted. -## openmind_hub.utils.LocalEntryNotFoundError +## openmind_hub.LocalEntryNotFoundError The local file does not exist. @@ -28,6 +28,6 @@ The local file does not exist. Access to the trustlist is restricted. -## openmind_hub.utils.OmHubHTTPError +## openmind_hub.OmHubHTTPError -Parent class of all HTTP exceptions in the openMind Hub Client. +Parent class of all HTTP exceptions in openMind Hub Client. diff --git a/docs/en/api_reference/upload_api.md b/docs/en/api_reference/upload_api.md index f8e14afdb87630e9f52b473896ec9975c74e0c84..3d53a5083968b429ca3ee55cf0a35da8ca44fa61 100644 --- a/docs/en/api_reference/upload_api.md +++ b/docs/en/api_reference/upload_api.md @@ -22,7 +22,7 @@ upload_file( + **path_in_repo** (`str`): path for uploading files to the repository. + **repo_id** (`str`): target repository, in the format of *username*/*repository name*. For the username and repository name, letters, digits, dots (.), underscores (_), and hyphens (-) are allowed. + **token** (`str`, *optional*): access token that has the write permission on the target repository. This parameter is required when this method is used separately. -+ **revision** (`str`, *optional*): branch to be uploaded. Letters, digits, underscores (_), and hyphens (-) are allowed. The default value is **main**. ++ **revision** (`str`, *optional*): branch to be uploaded. Letters, digits, underscores (_), and hyphens (-) are allowed. The default value is **None**. + **commit_message** (`str`, *optional*): commit message of the upload. The default value is **Upload {path_in_repo} with openMind hub**. + **commit_description** (`str`, *optional*): description of this commit. + **kwargs**: required only for compatibility with third-party components. @@ -54,7 +54,7 @@ upload_folder( + **commit_message** (`str`, *optional*): commit message of the upload. The default message is **Upload folder using openMind hub**. + **commit_description** (`str`, *optional*): description of this commit. + **token** (`str`, *optional*): access token that has the write permission on the target repository. This parameter is required when this method is used separately. -+ **revision** (`str`, *optional*): branch to be uploaded. Letters, digits, underscores (_), and hyphens (-) are allowed. The default value is **main**. ++ **revision** (`str`, *optional*): branch to be uploaded. Letters, digits, underscores (_), and hyphens (-) are allowed. The default value is **None**. + **allow_patterns** (`List[str]` or `str`, *optional*): only certain types of files can be uploaded. For example, `allow_patterns=["*.bin, *.py"]` indicates that only .bin and .py files can be uploaded. + **ignore_patterns** (`List[str]` or `str`, *optional*): Ignore the upload of a certain type of files. For example, `ignore_patterns="*.log"` indicates that all log files are ignored. + **num_threads** (`int`, *optional*): number of threads used for uploading. The default value is **5**. @@ -88,6 +88,6 @@ create_commit( + **commit_message** (`str`): commit message of the upload. + **commit_description** (`str`, *optional*): description of this commit. + **token** (`str`, *optional*): access token that has the write permission on the target repository. This parameter is required when this method is used separately. -+ **revision** (`str`, *optional*): branch to be uploaded. Letters, digits, underscores (_), and hyphens (-) are allowed. The default value is **main**. ++ **revision** (`str`, *optional*): branch to be uploaded. Letters, digits, underscores (_), and hyphens (-) are allowed. The default value is **None**. + **num_threads** (`int`, *optional*): number of threads used for uploading. The default value is **5**. + **kwargs**: required only for compatibility with third-party components. diff --git a/docs/en/basic_tutorial/third-party_platform.md b/docs/en/basic_tutorial/third-party_platform.md index 0a887e89f05297e97b4dc7a08b0da15f5ff6eee5..bf948b921910148a3d7ead20761bf42769548768 100644 --- a/docs/en/basic_tutorial/third-party_platform.md +++ b/docs/en/basic_tutorial/third-party_platform.md @@ -1,8 +1,8 @@ # Interconnection with Third-party Communities -The openMind Hub Client can connect to multiple communities so that you can upload and download files in different communities. Currently, the following communities are interconnected: +openMind Hub Client can connect to multiple communities so that you can upload and download files in different communities. Currently, the following communities are interconnected: -+ [Modelers](https://modelers.cn)): By default, this community is connected. All open interfaces are supported. ++ [Modelers](https://modelers.cn): This community is connected by default. All open interfaces are supported. + [Openl](https://openi.pcl.ac.cn): Only some interfaces are supported, covering repository creation and model upload and download. + [GitCode](https://gitcode.com): Only some interfaces are supported, covering file upload and download, and file acquisition in a repository. @@ -41,7 +41,7 @@ def http_get( #### om_hub_url -When obtaining the link for downloading private repository files, you need to add the **token** parameter. +When obtaining the link for downloading private repository files, you need to add the `token` parameter. ```python def om_hub_url( @@ -191,7 +191,7 @@ def try_to_load_from_cache( ```python import os - # Environment variables must be set before **openmind_hub** is imported. + # Environment variables must be set before openmind_hub is imported. os.environ["platform"] = "openi" from openmind_hub import * @@ -205,17 +205,17 @@ from openmind_hub import set_platform, create_repo, upload_folder, snapshot_down token = "token_in_openi" -# Download the PyTorch-NPU/t5_small model in the Modelers community to the **./t5_small** directory. +# Download the PyTorch-NPU/t5_small model in the Modelers community to the t5_small* directory. snapshot_download(repo_id="PyTorch-NPU/t5_small", local_dir="./t5_small") -# Download the FoundationModel/ChatGLM2-6B model in the specified community to the **./ChatGLM2-6B** directory. +# Download the FoundationModel/ChatGLM2-6B model in the specified community to the ./ChatGLM2-6B directory. # snapshot_download(repo_id="FoundationModel/ChatGLM2-6B", local_dir="./ChatGLM2-6B", platform="openi") # Set the default community. set_platform("openi") -# Create an **owner/cool-model** project and a **cool-model** in the specified community. (Use the actual user name and repository name.) +# Create an owner/cool-model project and a cool-model in the specified community. (Use the actual user name and repository name.) create_repo(repo_id="owner/cool-model", token=token) -# Upload files to the **cool-model** of the **owner/cool-model** project. +# Upload files to the cool-model of the owner/cool-model project. upload_folder(repo_id="owner/cool-model", folder_path="./t5_small", token=token) ``` @@ -229,7 +229,7 @@ The GitCode community adapts to the following methods. The parameters and behavi + upload_folder + create_commit + om_hub_download: `revision` supports only the branch name. An access token is required for downloading files from a public repository. -+ snapshot_download: revision supports only the branch name. An access token is required for downloading files from a public repository. ++ snapshot_download: `revision` supports only the branch name. An access token is required for downloading files from a public repository. + list_repo_tree: A maximum of 100 files can be listed at each level. ### Specifying the Community to Be Accessed diff --git a/docs/en/basic_tutorial/upload.md b/docs/en/basic_tutorial/upload.md index f136815d737d8bacea3b0f017eb0e1fe31a21284..c5b6476f7f1d7bfb33725da58b609341e9258ba4 100644 --- a/docs/en/basic_tutorial/upload.md +++ b/docs/en/basic_tutorial/upload.md @@ -1,6 +1,6 @@ # Uploading Files -[Access tokens](../quick_start.md# access token) are required to upload files to a repository. +[Access tokens](../quick_start.md#access-token) are required to upload files to a repository. The openMind Hub Client controls file upload paths through blocklist and trustlist. The absolute path of a file cannot start with a path in the blocklist (except the default cache path `~/.cache/openmind`) and must be in the trustlist. The blacklist includes `["/etc", "/var", "/bin", "/boot", "/lib", "~/."]`. The default trustlist is `["/tmp", "~/"]` in Linux and `["~/", "D:", "E:", "F:"]` in Windows. You can configure the environment variable `HUB_WHITE_LIST_PATHS` (separate each path by commas) to configure the trustlist. In addition, the priority of the blocklist permission is higher than that of the trustlist permission. @@ -21,7 +21,7 @@ upload_file( ) ``` -+ **path_in_repo** (`str`): path for uploading files to the repository. ++ `path_in_repo` (`str`): path for uploading files to the repository. + `token` (required): access token that has the write permission on the target repository. + `path_or_fileobj` (required): path of the local file to be uploaded. + `repo_id` (required): target repository. diff --git a/docs/en/developer_tutorial/figures/cacert_download.png b/docs/en/developer_tutorial/figures/cacert_download.png new file mode 100644 index 0000000000000000000000000000000000000000..c67a10f0f60680bbb5d19223647cd87901396068 Binary files /dev/null and b/docs/en/developer_tutorial/figures/cacert_download.png differ diff --git a/docs/en/developer_tutorial/figures/certificate download 1.png b/docs/en/developer_tutorial/figures/certificate download 1.png deleted file mode 100644 index 28f7d94357e2e3a41c051bb27cddeec676142dee..0000000000000000000000000000000000000000 Binary files a/docs/en/developer_tutorial/figures/certificate download 1.png and /dev/null differ diff --git a/docs/en/developer_tutorial/figures/certificate download 2.png b/docs/en/developer_tutorial/figures/certificate download 2.png deleted file mode 100644 index a608a2ec18f14c063394e0f1be5c95508a9a0fdf..0000000000000000000000000000000000000000 Binary files a/docs/en/developer_tutorial/figures/certificate download 2.png and /dev/null differ diff --git a/docs/en/developer_tutorial/figures/certificate download 3.png b/docs/en/developer_tutorial/figures/certificate download 3.png deleted file mode 100644 index 849e39819853f204bb28c85d5d78b32046007937..0000000000000000000000000000000000000000 Binary files a/docs/en/developer_tutorial/figures/certificate download 3.png and /dev/null differ diff --git a/docs/en/developer_tutorial/figures/certificate_download_1.png b/docs/en/developer_tutorial/figures/certificate_download_1.png index 28f7d94357e2e3a41c051bb27cddeec676142dee..854cff7de291d3eb00962e5bec6114d60b8ba810 100644 Binary files a/docs/en/developer_tutorial/figures/certificate_download_1.png and b/docs/en/developer_tutorial/figures/certificate_download_1.png differ diff --git a/docs/en/developer_tutorial/figures/certificate_download_2.png b/docs/en/developer_tutorial/figures/certificate_download_2.png index a608a2ec18f14c063394e0f1be5c95508a9a0fdf..d71bd54c9983345d258d32f6c9e03b445c6e9881 100644 Binary files a/docs/en/developer_tutorial/figures/certificate_download_2.png and b/docs/en/developer_tutorial/figures/certificate_download_2.png differ diff --git a/docs/en/developer_tutorial/trouble_shooting.md b/docs/en/developer_tutorial/trouble_shooting.md index 84438540651ad4081b1825f89e5c5d1dcc95ca6d..b533100f2fd8b42809176cdf4863d90dbea4ea08 100644 --- a/docs/en/developer_tutorial/trouble_shooting.md +++ b/docs/en/developer_tutorial/trouble_shooting.md @@ -1,10 +1,10 @@ # Troubleshooting -When using openMind Hub Client, you may encounter some problems. This document collects some common problems and their solutions. We will continue to update this document to help you troubleshoot faults. +When using openMind Hub Client, you may encounter some issues. This document collects some common issues and their solutions. We will continue to update this document to help you troubleshoot faults. ## Certificate Verification Failure: CERTIFICATE_VERIFY_FAILED -There are multiple causes for certificate verification failures. The following lists the solutions to different errors. +There are multiple causes for certificate verification failures. The following lists the solutions to different issues. Solution 1: If the Python root CA certificate does not match the requests library version, use either of the following methods to locate the fault. @@ -36,7 +36,7 @@ Solution 1: If the Python root CA certificate does not match the requests librar 2. Obtain the latest certificate from and download the cacert.pem file. - ![img.png](figures/cacert download.png) + ![img.png](figures/cacert_download.png) 3. Use the new cacert.pem certificate to replace the original one. Back up the certificate before the replacement. @@ -48,9 +48,9 @@ Solution 2: Manually configure the certificate of the Modelers community. 1. Download the certificate from Modelers Community. The following describes the procedure for downloading the certificate using Google Chrome. The procedure for downloading the certificate using other browsers is similar. In Windows, double-click the certificate to install it. In Linux, refer to the following procedure. - ![img.png](figures/certificate download 1.png) ![img_1.png](figures/certificate download 2.png) + ![img.png](figures/certificate_download_1.png) ![img_1.png](figures/certificate_download_2.png) - ![img_2.png](figures/certificate download 3.png) + ![img_2.png](figures/certificate_download_3.png) 2. Move the downloaded certificate to the `/usr/local/share/ca-certificates/` directory. 3. Run `update-ca-certificates`. diff --git a/docs/en/install.md b/docs/en/install.md index d3d199aced7cf1ace461a1f26923acde5c885a43..f3a66b65a387d2ca5a492cc7bc774e4896c54cb7 100644 --- a/docs/en/install.md +++ b/docs/en/install.md @@ -1,6 +1,6 @@ -# openMind Hub Client Installation +# Installing openMind Hub Client -Before the installation, you need to configure the local environment. The openMind Hub Client supports Python 3.8, 3.9, and 3.10. For details, see [Installation Description](https://docs.python.org/3/using/index.html). Compatibility with other versions is not guaranteed. +Before the installation, you need to configure the local environment. openMind Hub Client supports Python 3.8, 3.9, and 3.10. For details, see [Installation Description](https://docs.python.org/3/using/index.html). Compatibility with other versions is not guaranteed. ## Installation Guide @@ -11,22 +11,22 @@ To avoid compatibility issues between dependencies in different projects, Python - Create and activate a Python virtual environment. ```shell - # Create a Python virtual environment named **your_venv_name**. + # Create a Python virtual environment named your_venv_name. python3 -m venv /path/to/your_venv_name - # Activate the **your_venv_name** virtual environment. + # Activate the your_venv_name virtual environment. source /path/to/your_venv_name/bin/activate ``` - Create and activate a conda virtual environment. ```shell - # Create a Python 3.8 virtual environment named **your_venv_name**. + # Create a Python 3.8 virtual environment named your_venv_name. conda create -n your_venv_name python=3.8 - # Activate the **your_venv_name** virtual environment. + # Activate the your_venv_name virtual environment. conda activate your_venv_name ``` -2. Install the openMind Hub Client in pip mode as required. +2. Install openMind Hub Client in pip mode as required. - Use the default installation mode, which can interact with the Modelers community. @@ -56,9 +56,9 @@ python -c "from openmind_hub import repo_info; print(repo_info('PyTorch-NPU/t5_s The detailed information about the t5_small model will be displayed in the command output. -## Uninstallation +## Uninstalling openMind Hub Client -Uninstall openMind Hub Client: +Run the following command to uninstall openMind Hub Client: ```shell pip uninstall openmind_hub diff --git a/docs/en/overview.md b/docs/en/overview.md index 05c4d1881bc813a98ec5fa7ceacc7c5f19b87855..bc474106c9a94f7b9cb8eeb8a834d2d99e800857 100644 --- a/docs/en/overview.md +++ b/docs/en/overview.md @@ -1,3 +1,3 @@ # Overview -The openMind Hub Client helps you interact with the Modelers community without leaving the development environment, covering operations like file upload and download. For details, see [Quick Start](./quick_start.md). +openMind Hub Client allows you to interact with the Modelers community without leaving the development environment, covering operations like file upload and download. For details, see [Quick Start](./quick_start.md). diff --git a/docs/en/quick_start.md b/docs/en/quick_start.md index d8513041a28fe350262b66251106a9a9988539ac..7e4e5a19a1eb9c2b58f0736bc29b539b8370bc22 100644 --- a/docs/en/quick_start.md +++ b/docs/en/quick_start.md @@ -24,7 +24,7 @@ For details, see [Downloading Files](basic_tutorial/download.md). ## Access Token -Access tokens are required when you interact with Modelers, such as accessing private repository resources, creating repositories, and uploading files. Register or log in to Modelers, [create an access token](https://modelers.cn/my/tokens), and keep it secure. The token content is displayed only when it is created. +Access tokens are required when you interact with Modelers, such as accessing private repository resources, creating repositories, and uploading files. [Create an access token](https://modelers.cn/my/tokens) and keep it secure. The token content is displayed only when it is created. ## File Upload @@ -47,7 +47,7 @@ For details, see [Uploading Files](./basic_tutorial/upload.md). ## Advanced Use -The openMind Hub Client provides you with a simple way to interact with the Modelers community using Python. To learn more about how to manage files and model repositories in the community, refer to the following: +openMind Hub Client provides you with a simple way to interact with the Modelers community using Python. To learn more about how to manage files and model repositories in the community, refer to the following documents: + [Download](./basic_tutorial/download.md) files from a repository. + [Upload](./basic_tutorial/upload.md) files to a repository. diff --git a/docs/en/release_note.md b/docs/en/release_note.md index b76032b94f79f4a7634d955a4b1536c02eeb2f4f..81a4702d8cbcd9ae8de07120531e5482b1d943d9 100644 --- a/docs/en/release_note.md +++ b/docs/en/release_note.md @@ -1,3 +1,145 @@ # Release Notes -This document is being translated. We will upload it ad soon ad possible. +## Description of openMind Hub Client V0.8.0 + +### New Features + +#### New Functions + +- The **create_commit**, **upload_folder**, and **om_hub_download** APIs are added to upload and download a dataset in the Openl community. +- The **upload_folder**, **upload_file**, **create_commit**, **om_hub_download**, **snapshot_download**, and **list_repo_tree** APIs are supported by the GitCode community. + +#### Document Updates + +- Documents related to the OmFileSystem class and class methods are added. + +### Resolved Issues + +- The value of the environment variable **HUB_WHITE_LIST_PATHS** ends with a slash (/), and it does not take effect. + +### Known Issues + +None + +## Description of openMind Hub Client V0.7.1 + +## Version Updates + +### Update + +Changed the default domain name of the Modelers community to . + +### Resolved Issues + +Resolved the issue that `requests.exceptions.ChunkedEncodingError` occasionally occurs during download. + +## Description of openMind Hub Client V0.7.0 + +### New Features + +- openMind Hub Client supports interconnection with multiple communities and provides unified standard APIs. Currently, the function of uploading and downloading model files is verified based on third-party communities. + +### Previous Features + +### Repository Management + +You can manage repositories on the openMind platform, including creating and deleting repositories of various types, creating and deleting branches, and searching for and filtering repositories. + +- Universal APIs + + | API/Class | Description | + |----------------|------------------------| + | create_repo | Creates a repository. | + | delete_repo | Deletes a repository. | + | repo_info | Obtains information about a repository. | + | list_repo_tree | Obtains the list of files in a repository. | + | create_branch | Creates a branch. | + | delete_branch | Deletes a branch. | + | OmFileSystem | Accesses files in a remote repository as if they were local. | + | Repository | Encapsulates a series of Git commands and uses them to manage repositories.| + +- Model-related APIs + + | API/Class | Description | + |-----------------------|---------------| + | model_info | Obtains information about a model. | + | list_models | Searches for a model. | + | ModelFilter | Encapsulates filter criteria available for model search.| + +- Dataset-related APIs + + | API/Class | Description | + |-----------------|-----------| + | dataset_info | Obtains information about a dataset. | + | list_datasets | Searches for a dataset. | + | DatasetInfo | Encapsulates dataset information. | + | DatasetCard | Encapsulates dataset card information.| + | DatasetCardData | Encapsulates dataset card information.| + +- Space-related APIs + + | API | Description | + |---------------|-----------| + | space_info | Obtains information about a space. | + | list_spaces | Searches for a space. | + | restart_space | Restarts a space app.| + +### File Upload and Download + +You can upload and download files on the openMind platform. + +- Upload APIs + + | API/Class | Description | + |-----------------------|-------------| + | upload_file | Uploads a file. | + | upload_folder | Uploads files in a folder. | + | create_commit | Uploads or deletes a file. | + | CommitOperationAdd | Encapsulates information about the file to be uploaded.| + | CommitOperationDelete | Encapsulates information about the file to be deleted.| + +- Download APIs + + | API | Description | + |------------------------|-------------| + | om_hub_download | Downloads a single file. | + | snapshot_download | Downloads files from a repository. | + | get_om_file_metadata | Obtains file information. | + | om_hub_url | Concatenates the file download URL. | + | http_get | Bottom-layer API for downloading a file. | + | try_to_load_from_cache | Searches for the path of a file in the cache.| + +### Interconnection Between openMind Toolchain and openMind Platform + +Provides public APIs, classes, and constants to enable the openMind toolchain or other user software to connect to the openMind platform. In addition to the preceding repository management APIs and upload/download APIs, other APIs such as public errors and constants are also provided. + + | API/Class/Constant | Description | + |-------------------------|-----------------| + | OM_HOME | Default cache path of downloaded files. | + | OM_HUB_CACHE | Default cache path of downloaded files. | + | default_cache_path | Default cache path of downloaded files. | + | _CACHED_NO_EXIST | Default object when the file does not exist. | + | REGEX_COMMIT_HASH | Regular expression matching rule of the hash value. | + | list_metrics | Lists metrics. | + | whoami | Obtains user information. | + | get_full_repo_name | Obtains the complete repository name. | + | metadata_update | Updates a repository card. | + | om_raise_for_status | Throws an exception based on the status code of the response body.| + | build_om_headers | Constructs a request header. | + | OMValidationError | Verification error. | + | EntryNotFoundError | File not exist. | + | RepositoryNotFoundError | Repository not exist. | + | RevisionNotFoundError | Branch not exist. | + | GatedRepoError | Restricted access to a repository. | + | OmHubHTTPError | Network request error. | + | LocalEntryNotFoundError | Local file not exist. | + +### Document Development + +Includes quick start, basic tutorials, developer tutorials, and API reference. You can view details about the preceding APIs in the API document. + +## Resolved Issues + +- Resolved the issue that the resumable download function of `snapshot_download` is unavailable. +- Resolved the issue that `local_files_only` of `snapshot_download` is abnormal. +- Resolved the issue that a repository can be downloaded even if `repo_id` and `repo_type`are different from the actual repository name and type.