Comments

Comments

Comments

Article URL: https://preslav.me/2023/11/27/python-is-easy-golang-is-simple-simple-is-not-easy/

Comments URL: https://news.ycombinator.com/item?id=38433358

Points: 106

# Comments: 134

Comments

Comments

An introduction to database generated columns, using PostgreSQL and the new GeneratedField added in Django 5.0.

Comments

GeneFace: Generalized and High-Fidelity 3D Talking Face Synthesis; ICLR 2023; Official code

GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis | ICLR'23

Zhenhui Ye, Ziyue Jiang, Yi Ren, Jinglin Liu, Jinzheng He, Zhou Zhao | Zhejiang University, ByteDance

| | | | 中文文档

This repository is the official PyTorch implementation of our ICLR-2023 paper, in which we propose GeneFace for generalized and high-fidelity audio-driven talking face generation. The inference pipeline is as follows:

Our GeneFace achieves better lip synchronization and expressiveness to out-of-domain audios. Watch this video for a clear lip-sync comparison against previous NeRF-based methods. You can also visit our project page for more details.

🔥Update:

• 2023.3.16 We release a big update in this release, a video demo is here. including: 1) RAD-NeRF-based renderer, which could infer in real-time and be trained in 10 hours. 2) pytorch-based deep3d_reconstruction module, which is easier to install and is 8x faster than the previous TF-based version. 3) pitch-aware audio2motion module which could generate more lip-sync landmark. 4) fix some bugs that cause large memory usage. 5) We will upload the paper about this update soon.
• 2023.2.22 We release a 1 minute-long demo video, in which GeneFace is driven by a Chinese song generated by DiffSinger.
• 2023.2.20 We release a stable 3D landmark post-processing strategy in inference/nerfs/lm3d_nerf_infer.py, which improve the stability and quality of the final results by a large margin.

Quick Started!

We provide pre-trained models and processed datasets of GeneFace in this release to enable a quick start. In the following, we show how to infer the pre-trained models in 4 steps. If you want to train GeneFace on your own target person video, please reach to the following sections (Prepare Environments, Prepare Datasets, and Train Models).

• Step1. Create a new python env named geneface following the guide in docs/prepare_env/install_guide.md.

• Step2. Download the lrs3.zip and May.zip in the release and unzip it into the checkpoints directory.

• Step3. Process the dataset of May.mp4 following the guide in docs/process_data/process_target_person_video.md. Then you can see a output file named data/binary/videos/May/trainval_dataset.npy.

After the above steps, the structure of your checkpoints and data directory should look like this:

> checkpoints
    > lrs3
        > lm3d_vae_sync
        > syncnet
    > May
        > lm3d_postnet_sync
        > lm3d_radnerf
        > lm3d_radnerf_torso
> data
    > binary
        > videos
            > May
                trainval_dataset.npy

• Step4. Run the scripts below:

bash scripts/infer_postnet.sh
bash scripts/infer_lm3d_radnerf.sh
# bash scripts/infer_radnerf_gui.sh # you can also use GUI provided by RADNeRF

You can find a output video named infer_out/May/pred_video/zozo.mp4.

Prepare Environments

Please follow the steps in docs/prepare_env.

Prepare Datasets

Please follow the steps in docs/process_data.

Train Models

Please follow the steps in docs/train_models.

Train GeneFace on other target person videos

Apart from the May.mp4 provided in this repo, we also provide 8 target person videos that were used in our experiments. You can download them at this link. To train on a new video named <video_id>.mp4, you should place it into the data/raw/videos/ directory, then create a new folder at egs/datasets/videos/<video_id> and edit config files, according to the provided example folder egs/datasets/videos/May.

You can also record your own video and train a unique GeneFace model for yourself!

Citation

@article{ye2023geneface,
  title={GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis},
  author={Ye, Zhenhui and Jiang, Ziyue and Ren, Yi and Liu, Jinglin and He, Jinzheng and Zhao, Zhou},
  journal={arXiv preprint arXiv:2301.13430},
  year={2023}
}

Acknowledgements

Our codes are based on the following repos:

• NATSpeech (For the code template)
• AD-NeRF (For NeRF-related data preprocessing and vanilla NeRF implementation)
• RAD-NeRF (For RAD-NeRF implementation)
• Deep3DFaceRecon_pytorch (For 3DMM parameters extraction)

Single Image to 3D using Cross-Domain Diffusion

Wonder3D

Single Image to 3D using Cross-Domain Diffusion

Paper | Project page | Hugging Face Demo | Colab from @camenduru

Wonder3D reconstructs highly-detailed textured meshes from a single-view image in only 2 ∼ 3 minutes. Wonder3D first generates consistent multi-view normal maps with corresponding color images via a cross-domain diffusion model, and then leverages a novel normal fusion method to achieve fast and high-quality reconstruction.

Usage

import torch
import requests
from PIL import Image
import numpy as np
from torchvision.utils import make_grid, save_image
from diffusers import DiffusionPipeline  # only tested on diffusers[torch]==0.19.3, may have conflicts with newer versions of diffusers

def load_wonder3d_pipeline():

    pipeline = DiffusionPipeline.from_pretrained(
    'flamehaze1115/wonder3d-v1.0', # or use local checkpoint './ckpts'
    custom_pipeline='flamehaze1115/wonder3d-pipeline',
    torch_dtype=torch.float16
    )

    # enable xformers
    pipeline.unet.enable_xformers_memory_efficient_attention()

    if torch.cuda.is_available():
        pipeline.to('cuda:0')
    return pipeline

pipeline = load_wonder3d_pipeline()

# Download an example image.
cond = Image.open(requests.get("https://d.skis.ltd/nrp/sample-data/lysol.png", stream=True).raw)

# The object should be located in the center and resized to 80% of image height.
cond = Image.fromarray(np.array(cond)[:, :, :3])

# Run the pipeline!
images = pipeline(cond, num_inference_steps=20, output_type='pt', guidance_scale=1.0).images

result = make_grid(images, nrow=6, ncol=2, padding=0, value_range=(0, 1))

save_image(result, 'result.png')

Collaborations

Our overarching mission is to enhance the speed, affordability, and quality of 3D AIGC, making the creation of 3D content accessible to all. While significant progress has been achieved in the recent years, we acknowledge there is still a substantial journey ahead. We enthusiastically invite you to engage in discussions and explore potential collaborations in any capacity. If you're interested in connecting or partnering with us, please don't hesitate to reach out via email (xxlong@connect.hku.hk) .

More features

The repo is still being under construction, thanks for your patience.

• Local gradio demo.
• Detailed tutorial.
• GUI demo for mesh reconstruction
• Windows support
• Docker support

Schedule

• Inference code and pretrained models.
• Huggingface demo.
• New model with higher resolution.

Preparation for inference

Linux System Setup.

conda create -n wonder3d
conda activate wonder3d
pip install -r requirements.txt
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

Windows System Setup.

Please switch to branch main-windows to see details of windows setup.

Inference

1. Optional. If you have troubles to connect to huggingface. Make sure you have downloaded the following models. Download the checkpoints and into the root folder.

Wonder3D
|-- ckpts
    |-- unet
    |-- scheduler
    |-- vae
    ...

Then modify the file ./configs/mvdiffusion-joint-ortho-6views.yaml, set pretrained_model_name_or_path="./ckpts"

1. Download the SAM model. Put it to the sam_pt folder.

Wonder3D
|-- sam_pt
    |-- sam_vit_h_4b8939.pth

1. Predict foreground mask as the alpha channel. We use Clipdrop to segment the foreground object interactively. You may also use rembg to remove the backgrounds.

# !pip install rembg
import rembg
result = rembg.remove(result)
result.show()

1. Run Wonder3d to produce multiview-consistent normal maps and color images. Then you can check the results in the folder ./outputs. (we use rembg to remove backgrounds of the results, but the segmentations are not always perfect. May consider using Clipdrop to get masks for the generated normal maps and color images, since the quality of masks will significantly influence the reconstructed mesh quality.)

accelerate launch --config_file 1gpu.yaml test_mvdiffusion_seq.py \
            --config configs/mvdiffusion-joint-ortho-6views.yaml validation_dataset.root_dir={your_data_path} \
            validation_dataset.filepaths=['your_img_file'] save_dir={your_save_path}

see example:

accelerate launch --config_file 1gpu.yaml test_mvdiffusion_seq.py \
            --config configs/mvdiffusion-joint-ortho-6views.yaml validation_dataset.root_dir=./example_images \
            validation_dataset.filepaths=['owl.png'] save_dir=./outputs

Interactive inference: run your local gradio demo

python gradio_app_mv.py   # generate multi-view normals and colors

1. Mesh Extraction

Instant-NSR Mesh Extraction

cd ./instant-nsr-pl
python launch.py --config configs/neuralangelo-ortho-wmask.yaml --gpu 0 --train dataset.root_dir=../{your_save_path}/cropsize-{crop_size}-cfg{guidance_scale:.1f}/ dataset.scene={scene}

see example:

cd ./instant-nsr-pl
python launch.py --config configs/neuralangelo-ortho-wmask.yaml --gpu 0 --train dataset.root_dir=../outputs/cropsize-192-cfg1.0/ dataset.scene=owl

Our generated normals and color images are defined in orthographic views, so the reconstructed mesh is also in orthographic camera space. If you use MeshLab to view the meshes, you can click Toggle Orthographic Camera in View tab.

Interactive inference: run your local gradio demo

python gradio_app_recon.py   

NeuS-based Mesh Extraction

Since there are many complaints about the Windows setup of instant-nsr-pl, we provide the NeuS-based reconstruction, which may get rid of the requirement problems.

NeuS consumes less GPU memory and favors smooth surfaces without parameters tuning. However, NeuS consumes more times and its texture may be less sharp. If you are not sensitive to time, we recommend NeuS for optimization due to its robustness.

cd ./NeuS
bash run.sh output_folder_path scene_name 

Common questions

Q: Tips to get better results.

1. Wonder3D is sensitive the facing direciton of input images. By experiments, front-facing images always lead to good reconstruction.
2. Limited by resources, current implemetation only supports limited views (6 views) and low resolution (256x256). Any images will be first resized into 256x256 for generation, so images after such a downsample that still keep clear and sharp features will lead to good results.
3. Images with occlusions will cause worse reconstructions, since 6 views cannot cover the complete object. Images with less occlsuions lead to better results.
4. Increate optimization steps in instant-nsr-pl, modify trainer.max_steps: 3000 in instant-nsr-pl/configs/neuralangelo-ortho-wmask.yaml to more steps like trainer.max_steps: 10000. Longer optimization leads to better texture.

Q: The evelation and azimuth degrees of the generated views?

A: Unlike that the prior works such as Zero123, SyncDreamer and One2345 adopt object world system, our views are defined in the camera system of the input image. The six views are in the plane with 0 elevation degree in the camera system of the input image. Therefore we don't need to estimate an elevation degree for input image. The azimuth degrees of the six views are 0, 45, 90, 180, -90, -45 respectively.

Q: The focal length of the generated views?

A: We assume the input images are captured by orthographic camera, so the generated views are also in orthographic space. This design enables our model to keep strong generlaization on unreal images, but sometimes it may suffer from focal lens distortions on real-captured images.

Acknowledgement

We have intensively borrow codes from the following repositories. Many thanks to the authors for sharing their codes.

• stable diffusion
• zero123
• NeuS
• SyncDreamer
• instant-nsr-pl

License

Wonder3D is under AGPL-3.0, so any downstream solution and products (including cloud services) that include wonder3d code or a trained model (both pretrained or custom trained) inside it should be open-sourced to comply with the AGPL conditions. If you have any questions about the usage of Wonder3D, please contact us first.

Citation

If you find this repository useful in your project, please cite the following work. :)

@article{long2023wonder3d,
  title={Wonder3D: Single Image to 3D using Cross-Domain Diffusion},
  author={Long, Xiaoxiao and Guo, Yuan-Chen and Lin, Cheng and Liu, Yuan and Dou, Zhiyang and Liu, Lingjie and Ma, Yuexin and Zhang, Song-Hai and Habermann, Marc and Theobalt, Christian and others},
  journal={arXiv preprint arXiv:2310.15008},
  year={2023}
}

ChatGLM3 series: Open Bilingual Chat LLMs | 开源双语对话语言模型

ChatGLM3

🤗 HF Repo • 🤖 ModelScope • 🐦 Twitter • 📃 [GLM@ACL 22] [GitHub] • 📃 [GLM-130B@ICLR 23] [GitHub]

👋 加入我们的 Slack 和 微信

📍在 chatglm.cn 体验更大规模的 ChatGLM 模型。

Read this in English.

📔 更为详细的使用信息，可以参考：ChatGLM3技术文档

介绍

ChatGLM3 是智谱AI和清华大学 KEG 实验室联合发布的新一代对话预训练模型。ChatGLM3-6B 是 ChatGLM3 系列中的开源模型，在保留了前两代模型对话流畅、部署门槛低等众多优秀特性的基础上，ChatGLM3-6B 引入了如下特性：

1. 更强大的基础模型： ChatGLM3-6B 的基础模型 ChatGLM3-6B-Base 采用了更多样的训练数据、更充分的训练步数和更合理的训练策略。在语义、数学、推理、代码、知识等不同角度的数据集上测评显示，ChatGLM3-6B-Base 具有在 10B 以下的基础模型中最强的性能。
2. 更完整的功能支持： ChatGLM3-6B 采用了全新设计的 Prompt 格式，除正常的多轮对话外。同时原生支持工具调用（Function Call）、代码执行（Code Interpreter）和 Agent 任务等复杂场景。
3. 更全面的开源序列： 除了对话模型 ChatGLM3-6B 外，还开源了基础模型 ChatGLM3-6B-Base、长文本对话模型 ChatGLM3-6B-32K。以上所有权重对学术研究完全开放，在填写问卷进行登记后亦允许免费商业使用。

ChatGLM3 开源模型旨在与开源社区一起推动大模型技术发展，恳请开发者和大家遵守开源协议，勿将开源模型和代码及基于开源项目产生的衍生物用于任何可能给国家和社会带来危害的用途以及用于任何未经过安全评估和备案的服务。目前，本项目团队未基于 ChatGLM3 开源模型开发任何应用，包括网页端、安卓、苹果 iOS 及 Windows App 等应用。

尽管模型在训练的各个阶段都尽力确保数据的合规性和准确性，但由于 ChatGLM3-6B 模型规模较小，且模型受概率随机性因素影响，无法保证输出内容的准确。同时模型的输出容易被用户的输入误导。本项目不承担开源模型和代码导致的数据安全、舆情风险或发生任何模型被误导、滥用、传播、不当利用而产生的风险和责任。

模型列表

友情链接

对 ChatGLM3 进行加速的开源项目：

• chatglm.cpp: 类似 llama.cpp 的量化加速推理方案，实现笔记本上实时对话
• ChatGLM3-TPU: 采用TPU加速推理方案，在算能端侧芯片BM1684X（16T@FP16，内存16G）上实时运行约7.5 token/s

评测结果

典型任务

我们选取了 8 个中英文典型数据集，在 ChatGLM3-6B (base) 版本上进行了性能测试。

Best Baseline 指的是截止 2023年10月27日、模型参数在 10B 以下、在对应数据集上表现最好的预训练模型，不包括只针对某一项任务训练而未保持通用能力的模型。

对 ChatGLM3-6B-Base 的测试中，BBH 采用 3-shot 测试，需要推理的 GSM8K、MATH 采用 0-shot CoT 测试，MBPP 采用 0-shot 生成后运行测例计算 Pass@1 ，其他选择题类型数据集均采用 0-shot 测试。

我们在多个长文本应用场景下对 ChatGLM3-6B-32K 进行了人工评估测试。与二代模型相比，其效果平均提升了超过 50%。在论文阅读、文档摘要和财报分析等应用中，这种提升尤为显著。此外，我们还在 LongBench 评测集上对模型进行了测试，具体结果如下表所示

使用方式

环境安装

首先需要下载本仓库：

git clone https://github.com/THUDM/ChatGLM3
cd ChatGLM3

然后使用 pip 安装依赖：

pip install -r requirements.txt

• transformers 库版本应该 4.30.2 以及以上的版本 ，torch 库版本应为 2.0 及以上的版本，以获得最佳的推理性能。
• 为了保证 torch 的版本正确，请严格按照 官方文档 的说明安装。
• gradio 库版本应该为 3.x 的版本。

综合 Demo

我们提供了一个集成以下三种功能的综合 Demo，运行方法请参考 综合 Demo

• Chat: 对话模式，在此模式下可以与模型进行对话。
• Tool: 工具模式，模型除了对话外，还可以通过工具进行其他操作。
• Code Interpreter: 代码解释器模式，模型可以在一个 Jupyter 环境中执行代码并获取结果，以完成复杂任务。

代码调用

可以通过如下代码调用 ChatGLM 模型来生成对话：

>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True)
>>> model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True, device='cuda')
>>> model = model.eval()
>>> response, history = model.chat(tokenizer, "你好", history=[])
>>> print(response)
你好👋!我是人工智能助手 ChatGLM3-6B,很高兴见到你,欢迎问我任何问题。
>>> response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
>>> print(response)
晚上睡不着可能会让你感到焦虑或不舒服,但以下是一些可以帮助你入睡的方法:

1. 制定规律的睡眠时间表:保持规律的睡眠时间表可以帮助你建立健康的睡眠习惯,使你更容易入睡。尽量在每天的相同时间上床,并在同一时间起床。
2. 创造一个舒适的睡眠环境:确保睡眠环境舒适,安静,黑暗且温度适宜。可以使用舒适的床上用品,并保持房间通风。
3. 放松身心:在睡前做些放松的活动,例如泡个热水澡,听些轻柔的音乐,阅读一些有趣的书籍等,有助于缓解紧张和焦虑,使你更容易入睡。
4. 避免饮用含有咖啡因的饮料:咖啡因是一种刺激性物质,会影响你的睡眠质量。尽量避免在睡前饮用含有咖啡因的饮料,例如咖啡,茶和可乐。
5. 避免在床上做与睡眠无关的事情:在床上做些与睡眠无关的事情,例如看电影,玩游戏或工作等,可能会干扰你的睡眠。
6. 尝试呼吸技巧:深呼吸是一种放松技巧,可以帮助你缓解紧张和焦虑,使你更容易入睡。试着慢慢吸气,保持几秒钟,然后缓慢呼气。

如果这些方法无法帮助你入睡,你可以考虑咨询医生或睡眠专家,寻求进一步的建议。

从本地加载模型

以上代码会由 transformers 自动下载模型实现和参数。完整的模型实现在 Hugging Face Hub。如果你的网络环境较差，下载模型参数可能会花费较长时间甚至失败。此时可以先将模型下载到本地，然后从本地加载。

从 Hugging Face Hub 下载模型需要先安装Git LFS，然后运行

git clone https://huggingface.co/THUDM/chatglm3-6b

如果从你从 HuggingFace 下载比较慢，也可以从 ModelScope 中下载。

模型微调

请参考对话模型微调 ChatGLM3-6B 微调示例,或基座模型微调 ChatGLM3-6B-base 微调示例。 请注意，不同的微调脚本对应的模型并不相同，请根据需要选择对应的模型。

网页版对话 Demo

可以通过以下命令启动基于 Gradio 的网页版 demo：

python web_demo.py

可以通过以下命令启动基于 Streamlit 的网页版 demo：

streamlit run web_demo2.py

网页版 demo 会运行一个 Web Server，并输出地址。在浏览器中打开输出的地址即可使用。 经测试，基于 Streamlit 的网页版 Demo 会更流畅。

命令行对话 Demo

运行仓库中 cli_demo.py：

python cli_demo.py

程序会在命令行中进行交互式的对话，在命令行中输入指示并回车即可生成回复，输入 clear 可以清空对话历史，输入 stop 终止程序。

LangChain Demo

请参考 基于 LangChain 的工具调用 Demo。

工具调用

关于工具调用的方法请参考 工具调用。

API 部署

感谢 @xusenlinzy 实现了 OpenAI 格式的流式 API 部署，可以作为任意基于 ChatGPT 的应用的后端，比如 ChatGPT-Next-Web。可以通过运行仓库中的openai_api.py 进行部署：

cd openai_api_demo
python openai_api.py

同时，我们也书写了一个示例代码，用来测试API调用的性能。可以通过运行仓库中的openai_api_request.py 进行测试

• 使用Curl进行测试

curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"system\", \"content\": \"You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.\"}, {\"role\": \"user\", \"content\": \"你好，给我讲一个故事，大概100字\"}], \"stream\": false, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"

• 使用Python进行测试

cd openai_api_demo
python openai_api_request.py

如果测试成功，则模型应该返回一段故事。

低成本部署

模型量化

默认情况下，模型以 FP16 精度加载，运行上述代码需要大概 13GB 显存。如果你的 GPU 显存有限，可以尝试以量化方式加载模型，使用方法如下：

model = AutoModel.from_pretrained("THUDM/chatglm3-6b",trust_remote_code=True).quantize(4).cuda()

模型量化会带来一定的性能损失，经过测试，ChatGLM3-6B 在 4-bit 量化下仍然能够进行自然流畅的生成。

CPU 部署

如果你没有 GPU 硬件的话，也可以在 CPU 上进行推理，但是推理速度会更慢。使用方法如下（需要大概 32GB 内存）

model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).float()

Mac 部署

对于搭载了 Apple Silicon 或者 AMD GPU 的 Mac，可以使用 MPS 后端来在 GPU 上运行 ChatGLM3-6B。需要参考 Apple 的 官方说明 安装 PyTorch-Nightly（正确的版本号应该是2.x.x.dev2023xxxx，而不是 2.x.x）。

目前在 MacOS 上只支持从本地加载模型。将代码中的模型加载改为从本地加载，并使用 mps 后端：

model = AutoModel.from_pretrained("your local path", trust_remote_code=True).to('mps')

加载半精度的 ChatGLM3-6B 模型需要大概 13GB 内存。内存较小的机器（比如 16GB 内存的 MacBook Pro），在空余内存不足的情况下会使用硬盘上的虚拟内存，导致推理速度严重变慢。

多卡部署

如果你有多张 GPU，但是每张 GPU 的显存大小都不足以容纳完整的模型，那么可以将模型切分在多张GPU上。首先安装 accelerate: pip install accelerate，然后通过如下方法加载模型：

from utils import load_model_on_gpus

model = load_model_on_gpus("THUDM/chatglm3-6b", num_gpus=2)

即可将模型部署到两张 GPU 上进行推理。你可以将 num_gpus 改为你希望使用的 GPU 数。默认是均匀切分的，你也可以传入 device_map 参数来自己指定。

引用

如果你觉得我们的工作有帮助的话，请考虑引用下列论文。

@article{zeng2022glm,
  title={Glm-130b: An open bilingual pre-trained model},
  author={Zeng, Aohan and Liu, Xiao and Du, Zhengxiao and Wang, Zihan and Lai, Hanyu and Ding, Ming and Yang, Zhuoyi and Xu, Yifan and Zheng, Wendi and Xia, Xiao and others},
  journal={arXiv preprint arXiv:2210.02414},
  year={2022}
}

@inproceedings{du2022glm,
  title={GLM: General Language Model Pretraining with Autoregressive Blank Infilling},
  author={Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={320--335},
  year={2022}
}

Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference

Latent Consistency Models

Official Repository of the paper: Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference.

Official Repository of the paper: LCM-LoRA: A Universal Stable-Diffusion Acceleration Module.

Project Page: https://latent-consistency-models.github.io

Try our Demos:

🤗 Hugging Face Demo: 🔥🔥🔥

Replicate Demo:

OpenXLab Demo:

LCM Community: Join our LCM discord channels for discussions. Coders are welcome to contribute.

Breaking News 🔥🔥!!

• (❤️New) 2023/11/10 Training Scripts are released!! Check here.
• (🤯New) 2023/11/10 Training-free acceleration LCM-LoRA is born! See our technical report here and Hugging Face blog here.
• (⚡️New) 2023/11/10 LCM has a major update! We release 3 LCM-LoRA (SD-XL, SSD-1B, SD-V1.5), see here.
• (🚀New) 2023/11/10 LCM has a major update! We release 2 Full Param-tuned LCM (SD-XL, SSD-1B), see here.

News

• (🔥New) 2023/11/10 We support LCM Inference with C# and ONNX Runtime now! Thanks to @saddam213! Check the link here.
• (🔥New) 2023/11/01 Real-Time Latent Consistency Models is out!! Github link here. Thanks @radames for the really cool Huggingface🤗 demo Real-Time Image-to-Image, Real-Time Text-to-Image. Twitter/X Link.
• (🔥New) 2023/10/28 We support Img2Img for LCM! Please refer to "🔥 Image2Image Demos".
• (🔥New) 2023/10/25 We have official LCM Pipeline and LCM Scheduler in 🧨 Diffusers library now! Check the new "Usage".
• (🔥New) 2023/10/24 Simple Streamlit UI for local use: See the link Thanks for @akx.
• (🔥New) 2023/10/24 We support SD-Webui and ComfyUI now!! Thanks for @0xbitches. See the link: SD-Webui and ComfyUI.
• (🔥New) 2023/10/23 Running on Windows/Linux CPU is also supported! Thanks for @rupeshs See the link.
• (🔥New) 2023/10/22 Google Colab is supported now. Thanks for @camenduru See the link: Colab
• (🔥New) 2023/10/21 We support local gradio demo now. LCM can run locally!! Please refer to the "Local gradio Demos".
• (🔥New) 2023/10/19 We provide a demo of LCM in 🤗 Hugging Face Space. Try it here.
• (🔥New) 2023/10/19 We provide the LCM model (Dreamshaper_v7) in 🤗 Hugging Face. Download here.
• (🔥New) 2023/10/19 LCM is integrated in 🧨 Diffusers library. Please refer to the "Usage".

🔥 Image2Image Demos (Image-to-Image):

We support Img2Img now! Try the impressive img2img demos here: Replicate, SD-webui, ComfyUI, Colab

Local gradio for img2img is on the way!

🔥 Local gradio Demos (Text-to-Image):

To run the model locally, you can download the "local_gradio" folder:

1. Install Pytorch (CUDA). MacOS system can download the "MPS" version of Pytorch. Please refer to: https://pytorch.org. Install Intel Extension for Pytorch as well if you're using Intel GPUs.
2. Install the main library:

pip install diffusers transformers accelerate gradio==3.48.0 

1. Launch the gradio: (For MacOS users, need to set the device="mps" in app.py; For Intel GPU users, set device="xpu" in app.py)

python app.py

Demos & Models Released

Ours Hugging Face Demo and Model are released ! Latent Consistency Models are supported in 🧨 diffusers.

LCM Model Download: LCM_Dreamshaper_v7

For Chinese users, download LCM here: (中文用户可以在此下载LCM模型)

Hugging Face Demo:

Replicate Demo:

OpenXLab Demo:

Tungsten Demo:

Novita.AI Demo:

By distilling classifier-free guidance into the model's input, LCM can generate high-quality images in very short inference time. We compare the inference time at the setting of 768 x 768 resolution, CFG scale w=8, batchsize=4, using a A800 GPU.

Usage

We have official LCM Pipeline and LCM Scheduler in 🧨 Diffusers library now! The older usages will be deprecated.

You can try out Latency Consistency Models directly on:

To run the model yourself, you can leverage the 🧨 Diffusers library:

1. Install the library:

pip install --upgrade diffusers  # make sure to use at least diffusers >= 0.22
pip install transformers accelerate

1. Run the model:

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7")

# To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.
num_inference_steps = 4 

images = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=8.0, lcm_origin_steps=50, output_type="pil").images

For more information, please have a look at the official docs: 👉 https://huggingface.co/docs/diffusers/api/pipelines/latent_consistency_models#latent-consistency-models

Usage (Deprecated)

We have official LCM Pipeline and LCM Scheduler in 🧨 Diffusers library now! The older usages will be deprecated. But you can still use the older usages by adding revision="fb9c5d1" from from_pretrained(...)

To run the model yourself, you can leverage the 🧨 Diffusers library:

1. Install the library:

pip install diffusers transformers accelerate

1. Run the model:

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_txt2img", custom_revision="main", revision="fb9c5d")

# To save GPU memory, torch.float16 can be used, but it may compromise image quality.
pipe.to(torch_device="cuda", torch_dtype=torch.float32)

prompt = "Self-portrait oil painting, a beautiful cyborg with golden hair, 8k"

# Can be set to 1~50 steps. LCM support fast inference even <= 4 steps. Recommend: 1~8 steps.
num_inference_steps = 4 

images = pipe(prompt=prompt, num_inference_steps=num_inference_steps, guidance_scale=8.0, lcm_origin_steps=50, output_type="pil").images

Our Contributors :

BibTeX

LCM:
@misc{luo2023latent,
      title={Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference}, 
      author={Simian Luo and Yiqin Tan and Longbo Huang and Jian Li and Hang Zhao},
      year={2023},
      eprint={2310.04378},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

LCM-LoRA:
@article{luo2023lcm,
  title={LCM-LoRA: A Universal Stable-Diffusion Acceleration Module},
  author={Luo, Simian and Tan, Yiqin and Patil, Suraj and Gu, Daniel and von Platen, Patrick and Passos, Apolin{\'a}rio and Huang, Longbo and Li, Jian and Zhao, Hang},
  journal={arXiv preprint arXiv:2311.05556},
  year={2023}
}

A Solution Accelerator for the RAG pattern running in Azure, using Azure Cognitive Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences. This includes most common requirements and best practices.

"Chat with your data" Solution Accelerator

[!IMPORTANT]
As of November 15, 2023, Azure Cognitive Search has been renamed to Azure AI Search.

Welcome to the "Chat with your data" Solution Accelerator repository! The "Chat with your data" Solution Accelerator is a powerful tool that combines the capabilities of Azure AI Search and Large Language Models (LLMs) to create a conversational search experience. This solution accelerator uses an Azure OpenAI GPT model and an Azure AI Search index generated from your data, which is integrated into a web application to provide a natural language interface for search queries.

This repository provides a template for setting up the solution accelerator, along with detailed instructions on how to use and customize it to fit your specific needs.

About this repo

This repository provides a template for setting up the solution accelerator, along with detailed instructions on how to use and customize it to fit your specific needs. It provides the following features:

• Chat with an Azure OpenAI model using your own data
• Upload and process your documents
• Index public web pages
• Easy prompt configuration
• Multiple chunking strategies

When should you use this repo?

You should use this repo when your scenario customization needs exceed the out-of-the-box experience offered by Azure OpenAI on your data and you don't require to streamline the entire development cycle of your AI application, as you can with Azure Machine Learning prompt flow.

The accelerator presented here provides several options, for example:

• The ability to ground a model using both data and public web pages
• Advanced prompt engineering capabilities
• An admin site for ingesting/inspecting/configuring your dataset on the fly
• Running a Retrieval Augmented Generation (RAG) solution locally, as a Docker container

*Have you seen ChatGPT + Enterprise data with Azure OpenAI and AI Search demo? If you would like to experiment: Play with prompts, understanding RAG pattern different implementation approaches, see how different features interact with the RAG pattern and choose the best options for your RAG deployments, take a look at that repo.

Here is a comparison table with a few features offered by Azure, an available GitHub demo sample and this repo, that can provide guidance when you need to decide which one to use:

Supported file types

Out-of-the-box, you can upload the following file types:

• PDF
• JPEG
• JPG
• PNG
• TXT
• HTML
• MD (Markdown)
• DOCX

Prerequisites

• Azure subscription - Create one for free with contributor access.

• An Azure OpenAI resource and a deployment for one of the following Chat model and an embedding model:

  • Chat Models (select model version 0613 if you wish to use Azure OpenAI Functions as the orchestration stategy)
    • GPT-3.5
    • GPT-4
  • Embedding Model
    • text-embedding-ada-002

  NOTE: The deployment template defaults to gpt-35-turbo and text-embedding-ada-002. If your deployment names are different, update them in the deployment process.

Getting started

1. Click the following deployment button to create the required resources for this accelerator directly in your Azure Subscription.

   

2. Add the following fields:

   Field Description

   Resource group	The resource group that will contain the resources for this accelerator. You can select Create new to create a new group.
   Resource prefix	A text string that will be appended to each resource that gets created, and used as the website name for the web app. This name cannot contain spaces or special characters.
   Azure OpenAI resource	The name of your Azure OpenAI resource. This resource must have already been created previously.
   Azure OpenAI key	The access key associated with your Azure OpenAI resource.
   Orchestration strategy	Use Azure OpenAI Functions (openai_functions) or LangChain (langchain) for messages orchestration. If you are using a new model version 0613 select "openai_functions" (or "langchain"), if you are using a 0314 model version select "langchain"
   Azure Document Intelligence resource location	The location of the Azure Document Intelligence resource

   You can find the ARM template used, along with a Bicep file for deploying this accelerator in the /infrastructure directory.

   NOTE: By default, the deployment name in the application settings is equal to the model name (gpt-35-turbo and text-embedding-ada-002). If you named the deployment in a different way, you should update the application settings to match your deployment names.

3. Navigate to the admin site, where you can upload documents. It will be located at:

   https://{MY_RESOURCE_PREFIX}-website-admin.azurewebsites.net/

   Where {MY_RESOURCE_PREFIX} is replaced with the resource prefix you used during deployment. Then select Ingest Data and add your data. You can find sample data in the /data directory.

   

4. Navigate to the web app to start chatting on top of your data. The web app can be found at:

   https://{MY_RESOURCE_PREFIX}-website.azurewebsites.net/

   Where {MY_RESOURCE_PREFIX} is replaced with the resource prefix you used during deployment.

   

Troubleshooting

Unrecognized request argument supplied: functions

This error can occur when sending a message using the chat app. The error message "An error occurred. Please try again. If the problem persists, please contact the site administrator" is displayed on the UI and the exception with message "Unrecognized request argument supplied: functions" is logged to the developer console.

This is caused when the orchestration stategy is set to openai_functions but the OpenAI deployment model is not set to version 0613. To fix, please update your OpenAI deployment model to versoin 0613 and try again.

Development and run the accelerator locally

To customize the accelerator or run it locally, first, copy the .env.sample file to your development environment's .env file, and edit it according to environment variable values table below.

Running the full solution locally

You can run the full solution locally with the following commands - this will spin up 3 different Docker containers:

Run the following docker compose command.

cd docker
docker compose up

Develop & run the frontend locally

For faster development, you can run the frontend Typescript React UI app and the Python Flask app in development mode. This allows the app to "hot reload" meaning your changes will automatically be reflected in the app without having to refresh or restart the local servers.

To run the app locally with hot refresh, first follow the instructions to update your .env file with the needed values.

Starting the Flask app in dev mode

Open a terminal and enter the following commands

cd code
python -m pip install -r requirements.txt
cd app
python -m flask --app ./app.py --debug run

Starting the Typescript React app in dev mode

Open a new separate terminal and enter the following commands:

cd code
cd app
cd frontend
npm install
npm run dev

The local vite server will return a url that you can use to access the chat interface locally, such as http://localhost:5174/.

Building the user app Docker image

docker build -f docker\WebApp.Dockerfile -t YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE .
docker run --env-file .env -p 8080:80 YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE
docker push YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE

Develop & run the admin app

If you want to develop and run the backend container locally, use the following commands.

Running the admin app locally

cd code
cd admin
python -m pip install -r requirements.txt
streamlit run Admin.py

Then access http://localhost:8501/ for getting to the admin interface.

Building the backend Docker image

docker build -f docker\AdminWebApp.Dockerfile -t YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE .
docker run --env-file .env -p 8081:80 YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE
docker push YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE

NOTE: If you are using Linux, make sure to go to https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/blob/main/docker/docker-compose.yml#L9 and modify the docker-compose.yml to use forward slash /. The backslash version just works with Windows.

Develop & run the batch processing functions

If you want to develop and run the batch processing functions container locally, use the following commands.

Running the batch processing locally

First, install Azure Functions Core Tools.

cd code
cd batch
func start

Or use the Azure Functions VS Code extension.

Building the batch processing Docker image

docker build -f docker\Backend.Dockerfile -t YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE .
docker run --env-file .env -p 7071:80 YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE
docker push YOUR_DOCKER_REGISTRY/YOUR_DOCKER_IMAGE

Environment variables

Resources used in this solution

This solution accelerator deploys the following resources. It's crucial to comprehend the functionality of each. Below are the links to their respective documentation:

• Azure OpenAI Service Documentation
• Azure AI Search Documentation
• Azure Blob Storage Documentation
• Azure Functions Documentation
• Azure AI Document Intelligence Documentation
• Azure App Service Documentation

Pricing Considerations

This solution accelerator deploys multiple resources. Evaluate the cost of each component prior to deployment.

The following are links to the pricing details for some of the resources:

• Azure OpenAI service pricing. GPT and embedding models are charged separately.
• Azure AI Search pricing. AI Search core service and semantic search are charged separately.
• Azure Blob Storage pricing
• Azure Functions pricing
• Azure AI Document Intelligence pricing
• Azure Web App Pricing

Azure AI Search used as retriever in RAG

Azure AI Search, when used as a retriever in the Retrieval-Augmented Generation (RAG) pattern, plays a key role in fetching relevant information from a large corpus of data. The RAG pattern involves two key steps: retrieval of documents and generation of responses. Azure AI Search, in the retrieval phase, filters and ranks the most relevant documents from the dataset based on a given query.

The importance of optimizing data in the index for relevance lies in the fact that the quality of retrieved documents directly impacts the generation phase. The more relevant the retrieved documents are, the more accurate and pertinent the generated responses will be.

Azure AI Search allows for fine-tuning the relevance of search results through features such as scoring profiles, which assign weights to different fields, Lucene's powerful full-text search capabilities, vector search for similarity search, multi-modal search, recommendations, hybrid search and semantic search to use AI from Microsoft to rescore search results and moving results that have more semantic relevance to the top of the list. By leveraging these features, one can ensure that the most relevant documents are retrieved first, thereby improving the overall effectiveness of the RAG pattern.

Moreover, optimizing the data in the index also enhances the efficiency, the speed of the retrieval process and increases relevance which is an integral part of the RAG pattern.

Best practices specific to Azure AI Search

• Consider switching security keys and using RBAC instead for authentication.
• Consider setting up a firewall, private endpoints for inbound connections and shared private links for built-in pull indexers.
• For the best results, prepare your index data and consider analyzers.
• Analyze your resource capacity needs.

Best practices before deploying Azure RAG implementations to production

• Follow the best practices described in Azure Well-Architected-Framework.
• Understand the Retrieval Augmented Generation (RAG) in Azure AI Search.
• Understand the functionality and configuration that would adapt better to your solution and test with your own data for optimal retrieval.
• Experiment with different options, define the prompts that are optimal for your needs and find ways to implement functionality tailored to your business needs with this demo, so you can then adapt to the accelerator.
• Follow the Responsible AI best practices.
• Understand the levels of access of your users and application.

Chunking: Importance for RAG and strategies implemented as part of this repo

Chunking is essential for managing large data sets, optimizing relevance, preserving context, integrating workflows, and enhancing the user experience. See How to chunk documents for more information.

These are the chunking strategy options you can choose from:

• Layout: An AI approach to determine a good chunking strategy.
• Page: This strategy involves breaking down long documents into pages.
• Fixed-Size Overlap: This strategy involves defining a fixed size that’s sufficient for semantically meaningful paragraphs (for example, 250 words) and allows for some overlap (for example, 10-25% of the content). This usually helps creating good inputs for embedding vector models. Overlapping a small amount of text between chunks can help preserve the semantic context.
• Paragraph: This strategy allows breaking down a difficult text into more manageable pieces and rewrite these “chunks” with a summarization of all of them.

Licensing

This repository is licensed under the MIT License.

The data set under the /data folder is licensed under the CDLA-Permissive-2 License.

Data Set

The data set under the /data folder has been generated with Azure OpenAI GPT and DALL-E 2 models.

DISCLAIMER

This presentation, demonstration, and demonstration model are for informational purposes only and (1) are not subject to SOC 1 and SOC 2 compliance audits, and (2) are not designed, intended or made available as a medical device(s) or as a substitute for professional medical advice, diagnosis, treatment or judgment. Microsoft makes no warranties, express or implied, in this presentation, demonstration, and demonstration model. Nothing in this presentation, demonstration, or demonstration model modifies any of the terms and conditions of Microsoft’s written and signed agreements. This is not an offer and applicable terms and the information provided are subject to revision and may be changed at any time by Microsoft.

This presentation, demonstration, and demonstration model do not give you or your organization any license to any patents, trademarks, copyrights, or other intellectual property covering the subject matter in this presentation, demonstration, and demonstration model.

The information contained in this presentation, demonstration and demonstration model represents the current view of Microsoft on the issues discussed as of the date of presentation and/or demonstration, for the duration of your access to the demonstration model. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of presentation and/or demonstration and for the duration of your access to the demonstration model.

No Microsoft technology, nor any of its component technologies, including the demonstration model, is intended or made available as a substitute for the professional advice, opinion, or judgment of (1) a certified financial services professional, or (2) a certified medical professional. Partners or customers are responsible for ensuring the regulatory compliance of any solution they build using Microsoft technologies.

Create Reddit Videos with just✨ one command ✨

Reddit Video Maker Bot 🎥

All done WITHOUT video editing or asset compiling. Just pure ✨programming magic✨.

Created by Lewis Menelaws & TMRRW

Video Explainer

Motivation 🤔

These videos on TikTok, YouTube and Instagram get MILLIONS of views across all platforms and require very little effort. The only original thing being done is the editing and gathering of all materials...

... but what if we can automate that process? 🤔

Disclaimers 🚨

• At the moment, this repository won't attempt to upload this content through this bot. It will give you a file that you will then have to upload manually. This is for the sake of avoiding any sort of community guideline issues.

Requirements

• Python 3.10
• Playwright (this should install automatically in installation)

Installation 👩‍💻

1. Clone this repository
2. Run pip install -r requirements.txt
3. Run python -m playwright install and python -m playwright install-deps

EXPERIMENTAL!!!!

On macOS and Linux (debian, arch, fedora and centos, and based on those), you can run an install script that will automatically install steps 1 to 3. (requires bash)

bash <(curl -sL https://raw.githubusercontent.com/elebumm/RedditVideoMakerBot/master/install.sh)

This can also be used to update the installation

1. Run python main.py
2. Visit the Reddit Apps page., and set up an app that is a "script". Paste any URL in redirect URL. Ex:google.com
3. The bot will ask you to fill in your details to connect to the Reddit API, and configure the bot to your liking
4. Enjoy 😎
5. If you need to reconfigure the bot, simply open the config.toml file and delete the lines that need to be changed. On the next run of the bot, it will help you reconfigure those options.

(Note if you got an error installing or running the bot try first rerunning the command with a three after the name e.g. python3 or pip3)

If you want to read more detailed guide about the bot, please refer to the documentation

Video

https://user-images.githubusercontent.com/66544866/173453972-6526e4e6-c6ef-41c5-ab40-5d275e724e7c.mp4

Contributing & Ways to improve 📈

In its current state, this bot does exactly what it needs to do. However, improvements can always be made!

I have tried to simplify the code so anyone can read it and start contributing at any skill level. Don't be shy :) contribute!

• Creating better documentation and adding a command line interface.
• Allowing the user to choose background music for their videos.
• Allowing users to choose a reddit thread instead of being randomized.
• Allowing users to choose a background that is picked instead of the Minecraft one.
• Allowing users to choose between any subreddit.
• Allowing users to change voice.
• Checks if a video has already been created
• Light and Dark modes
• NSFW post filter

Please read our contributing guidelines for more detailed information.

For any questions or support join the Discord server

Developers and maintainers.

Elebumm (Lewis#6305) - https://github.com/elebumm (Founder)

Jason (JasonLovesDoggo#1904) - https://github.com/JasonLovesDoggo (Maintainer)

Simon (OpenSourceSimon) - https://github.com/OpenSourceSimon

CallumIO (c.#6837) - https://github.com/CallumIO

Verq (Verq#2338) - https://github.com/CordlessCoder

LukaHietala (Pix.#0001) - https://github.com/LukaHietala

Freebiell (Freebie#3263) - https://github.com/FreebieII

Aman Raza (electro199#8130) - https://github.com/electro199

LICENSE

Roboto Fonts are licensed under Apache License V2

A toolkit for developing and comparing reinforcement learning algorithms.

Important Notice

The team that has been maintaining Gym since 2021 has moved all future development to Gymnasium, a drop in replacement for Gym (import gymnasium as gym), and Gym will not be receiving any future updates. Please switch over to Gymnasium as soon as you're able to do so. If you'd like to read more about the story behind this switch, please check out this blog post.

Gym

Gym is an open source Python library for developing and comparing reinforcement learning algorithms by providing a standard API to communicate between learning algorithms and environments, as well as a standard set of environments compliant with that API. Since its release, Gym's API has become the field standard for doing this.

Gym documentation website is at https://www.gymlibrary.dev/, and you can propose fixes and changes to it here.

Gym also has a discord server for development purposes that you can join here: https://discord.gg/nHg2JRN489

Installation

To install the base Gym library, use pip install gym.

This does not include dependencies for all families of environments (there's a massive number, and some can be problematic to install on certain systems). You can install these dependencies for one family like pip install gym[atari] or use pip install gym[all] to install all dependencies.

We support Python 3.7, 3.8, 3.9 and 3.10 on Linux and macOS. We will accept PRs related to Windows, but do not officially support it.

API

The Gym API's API models environments as simple Python env classes. Creating environment instances and interacting with them is very simple- here's an example using the "CartPole-v1" environment:

import gym
env = gym.make("CartPole-v1")
observation, info = env.reset(seed=42)

for _ in range(1000):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)

    if terminated or truncated:
        observation, info = env.reset()
env.close()

Notable Related Libraries

Please note that this is an incomplete list, and just includes libraries that the maintainers most commonly point newcommers to when asked for recommendations.

• CleanRL is a learning library based on the Gym API. It is designed to cater to newer people in the field and provides very good reference implementations.
• Tianshou is a learning library that's geared towards very experienced users and is design to allow for ease in complex algorithm modifications.
• RLlib is a learning library that allows for distributed training and inferencing and supports an extraordinarily large number of features throughout the reinforcement learning space.
• PettingZoo is like Gym, but for environments with multiple agents.

Environment Versioning

Gym keeps strict versioning for reproducibility reasons. All environments end in a suffix like "_v0". When changes are made to environments that might impact learning results, the number is increased by one to prevent potential confusion.

MuJoCo Environments

The latest "_v4" and future versions of the MuJoCo environments will no longer depend on mujoco-py. Instead mujoco will be the required dependency for future gym MuJoCo environment versions. Old gym MuJoCo environment versions that depend on mujoco-py will still be kept but unmaintained. To install the dependencies for the latest gym MuJoCo environments use pip install gym[mujoco]. Dependencies for old MuJoCo environments can still be installed by pip install gym[mujoco_py].

Citation

A whitepaper from when Gym just came out is available https://arxiv.org/pdf/1606.01540, and can be cited with the following bibtex entry:

@misc{1606.01540,
  Author = {Greg Brockman and Vicki Cheung and Ludwig Pettersson and Jonas Schneider and John Schulman and Jie Tang and Wojciech Zaremba},
  Title = {OpenAI Gym},
  Year = {2016},
  Eprint = {arXiv:1606.01540},
}

Release Notes

There used to be release notes for all the new Gym versions here. New release notes are being moved to releases page on GitHub, like most other libraries do. Old notes can be viewed here.

OpenAgents: An Open Platform for Language Agents in the Wild

OpenAgents: An Open Platform for Language Agents in the Wild

English • 中文 • 日本語 • 한국어

Current language agent frameworks aim to facilitate the construction of proof-of-concept language agents while neglecting the non-expert user access to agents and paying little attention to application-level designs. We built OpenAgents, an open platform for using and hosting language agents in the wild of everyday life.

We have now implemented three agents in OpenAgents, and we host them on demo for free use!

1. Data Agent for data analysis with Python/SQL and data tools;
2. Plugins Agent with 200+ daily tools;
3. Web Agent for autonomous web browsing.

OpenAgents can analyze data, call plugins, control your browser as ChatGPT Plus, but with OPEN Code for

1. Easy deployment
2. Full stack
3. Chat Web UI
4. Agent methods
5. …

OpenAgents enables general users to interact with agent functionalities through a web UI optimized for swift responses and common failures, while offering developers and researchers a seamless deployment experience on local setups, providing a foundation for crafting innovative language agents and facilitating real-world evaluations. We elucidate both the challenges and promising opportunities, aspiring to set a foundation for future research and development of real-world language agents.

We welcome contributions from everyone. Before you start, please take a moment to read our CONTRIBUTING.md guidelines for issues and PRs. This will help ensure that your contribution process is smooth and consistent with the project’s standards.

🔫 Trouble Shooting

Join our Discord for help if you encounter any issues with our online demo or local deployment. Alternatively, create an issue if you have trouble with features or code.

🔥 News

• [2023, Oct 26] We've reached 3,000 users! 🚀 Heartfelt thanks to all our users and contributors! 🙏 Please bear with us as we navigate through unexpectedly high traffic on our servers. We appreciate your patience and ready to assist as soon as possible!
• [2023, Oct 18] Try out our Lemur, the SOTA and open-sourced foundation models for language agents, matching ChatGPT on 15 agent tasks!
• [2023, Oct 17] Check out the OpenAgents paper here!
• [2023, Oct 13] We've released OpenAgents platform code for all three agents, server backend and frontend! Feel free to setup your localhost one, and play with OpenAgents!
• [2023, Aug 17] Our platform has officially reached 500 users! 🚀
• [2023, Aug 8] We've released OpenAgents demos, including Data, Plugins, and Web agents! Check tutorials and use cases!

🥑 OpenAgents

We built three real-world agents with chat-based web UI as demonstration(check OpenAgents demos). Here is a brief overview of our OpenAgents platform. You can find more details about concepts & designs in our documentation.

Data Agent

Data Agent is a comprehensive toolkit designed for efficient data operations. It provides capabilities to:

• 🔍 Search: Quickly locate the data you need.
• 🛠️ Handle: Streamline data acquisition and processing.
• 🔄 Manipulate: Modify data to suit specific requirements.
• 📊 Visualize: Represent data in a clear and insightful manner.

With its proficiency in writing and executing code, Data Agent simplifies a wide range of data-centric tasks. Discover its potential through various use cases.

Plugins Agent

Plugins Agent seamlessly integrates with over 200 third-party plugins, each handpicked to enrich various facets of your daily life. With these plugins at its disposal, the agent empowers you to tackle a wide range of tasks and activities more efficiently.

🔌 Sample Plugins Include:

• 🛍️ Shopping: Klarna Shopping
• ☁️ Weather: XWeather
• 🔬 Scientific Exploration: Wolfram Alpha

Combined Plugin Usage

Harness the power of synergy! Plugins Agent supports the concurrent use of multiple plugins. Planning a trip? Seamlessly integrate functionalities from Klook, Currency converter, and WeatherViz.

Auto Plugin Selection

Simplify your choices with our Auto Plugin Selection feature. Let the agent intuitively search and suggest the best plugins tailored to your needs.

Dive into more use cases to see Plugins Agent in action.

Web Agent

Web Agent harnesses the power of a Chrome extension to navigate and explore websites automatically. This agent streamlines the web browsing experience, making it easier to find relevant information, access desired resources, and so on.

Examples of What Web Agent Can Do:

• 📍 Google Maps Navigation: Planning a journey? Simply relay your starting point and destination to Web Agent. It will navigate Google Maps for you and present the best routes.
• 🐦 Twitter Postings: Engage in a conversation with Web Agent and wish to share something on Twitter? Mention the content, and Web Agent will handle your tweet effortlessly.
• 📝 Google Form Assistance: Need to sign up for an event or activity? Share the Google Form link and the required details. Web Agent will populate the form for you.

Witness the full potential of Web Agent in these use cases.

💻 Localhost Deployment

We've released the OpenAgents platform code. Feel free to deploy on your own localhost!

Here is a brief system design of OpenAgents:

From Source Code

Please check the following folders and README files to set up & localhost:

1. Backend: the flask backend to host our three agents.
2. Frontend: the frontend UI and WeBot Chrome extension.

p.s.: We have renamed some arguments in code for better readability. If you have pulled the code before 10/26/2023, just a reminder that if you want to you pull the latest code, previous local chat history will be lost because of different key names.

Docker

Please follow the following steps to use the docker-compose to deploy the OpenAgents platform.

Note: the docker is under development, so there may be functions not working properly as expected and slower response. Please feel free to open an issue if you have any questions. If you want a more robust version, currently we recommend you to deploy from source code.

1. If you want to use kaggle's dataset, you must modify the information in the Dockerfile to your correct information.

ENV KAGGLE_USER="" \
    KAGGLE_KEY="" 

1. If you are not running locally, you need to modify the accessible IP to the backend service in frontend/Dockerfile

ENV NEXT_PUBLIC_BACKEND_ENDPOINT http://x.x.x.x:8000

1. Run the docker compose build command in the project root directory.
2. If you use openai unofficial services, such as FastChat, you need to modify OPENAI_API_BASE in docker-compose.yml;otherwise you only to put your OPENAI_API_KEY in docker-compose.yml
3. After completing the above steps, you can run docker compose up -d to start all services.

📜 Tutorial on Extending OpenAgents

Code Structure

Before we dive into how to extend OpenAgents, let's first take a glance at the code structure for better understanding. The code structure of OpenAgents is shown below:

├── backend  # backend code
│   ├── README.md  # backend README for setup
│   ├── api  # RESTful APIs, to be called by the frontend
│   ├── app.py  # main flask app
│   ├── display_streaming.py  # rendering the streaming response
│   ├── kernel_publisher.py  # queue for code execution
│   ├── main.py  # main entry for the backend
│   ├── memory.py  # memory(storage) for the backend
│   ├── schemas.py  # constant definitions
│   ├── setup_script.sh  # one-click setup script for the backend
│   ├── static  # static files, e.g., cache and figs
│   └── utils  # utilities
├── frontend  # frontend code
│   ├── README.md  # frontend README for setup
│   ├── components  # React components
│   ├── hooks  # custom React hooks
│   ├── icons  # icon assets
│   ├── next-env.d.ts  # TypeScript declarations for Next.js environment variables
│   ├── next-i18next.config.js  # configuration settings for internationalization
│   ├── next.config.js  # configuration settings for Next.js
│   ├── package-lock.json  # generated by npm that describes the exact dependency tree
│   ├── package.json  # manifest file that describes the dependencies
│   ├── pages  # Next.js pages
│   ├── postcss.config.js  # configuration settings for PostCSS
│   ├── prettier.config.js  # configuration settings for Prettier
│   ├── public  # static assets
│   ├── styles  # global styles
│   ├── tailwind.config.js  # configuration settings for Tailwind CSS
│   ├── tsconfig.json  # configuration settings for TypeScript
│   ├── types  # type declarations
│   ├── utils  # utilities or helper functions
│   ├── vitest.config.ts  # configuration settings for ViTest
│   └── webot_extension.zip  # Chrome extension for Web Agent
└── real_agents  # language agents
    ├── adapters  # shared components for the three agents to adapt to the backend
    ├── data_agent  # data agent implementation
    ├── plugins_agent  # plugins agent implementation
    └── web_agent  # web agent implementation

As shown, backend/ and frontend/ are self-contained and directly deployable (see here). It does not mean they cannot be modified. Instead, you can just follow the conventional client-server architecture to extend the backend and frontend as you wish. For real_agents/, we design it to be "one agent, one folder", so that it is easy to extend a new agent. It is worth noting that we name it "real agents" because not only the conceptual language agent part is included, but also the gaps between the language agent and the backend are filled here. For example, adapters/ contains the shared adapter components like stream parsing, data model, memory, callbacks, etc. We refer interested readers to our paper for concepts and implementation designs. And we thank LangChain as we base on their code to build real agents.

Extend A New Agent

If you want to build a new agent beyond the three agents we provide, you can follow the steps below:

• Refer to the real_agents/ folder to see how previous agents are implemented, and create a new folder for your agent.
• Implement the agent logic in the new folder. Use the components under adapters/ folder when needed.
• Add a chat_<new_agent>.py file under backend/api/ folder to define the chat API for the new agent, which will be called by the frontend.
• Register new constants in backend/schemas.py if needed.
• Add a new OpenAgentID in frontend/types/agent.ts and the corresponding API in frontend/utils/app/api.ts and frontend/utils/app/const.ts.
• Implement the agent UI in frontend/components/Chat/Chat.tsx and frontend/components/Chat/ChatMessage.tsx when needed.
• Run localhost script and test your new agent.

Note, if new data types, i.e., beyond text, image, table, and json, you may need to implement its parsing logic in backend/display_streaming.py and add new data models.

Extend A New LLM

Extending a new LLM as the agent backbone is simpler if the LLM is already hosted and can be called via API. Just register your new model in backend/api/language_model.py. Just refer to lemur-chat as a template.

If the LLM is not hosted yet, we have a tutorial on how to deploy a new LLM and expose it as an API here (LLM hosting to todo).

Extend A New Tool

If you want to extend a new tool in Plugins Agent, you can follow the steps below:

• Refer to the already built plugins in real_agents/plugins_agent/plugins/, and create a new folder for your tool.
• Implement the tool logic in the new folder. Note that ai-plugin.json and openapi.yaml are essential for the tool to be recognized(which can be generated by LLM following the others rather than manually written). And the paths/ are for the actual tool API call.
• Register the new tool name in real_agents/plugins_agent/plugins/plugin_names.py.

👏 Contributing

Thanks to open-sourced communities’ efforts, such as LangChain, ChatBot UI, Taxy.ai browser extension and others. We are able to build our interface prototype much more conveniently and efficiently.

We welcome contributions and suggestions, together we move further to make it better! Following the steps will be well-received:

• Step1: Post an issue if you want to add any additional features, enhancements, or encounter any problems during your experience. We would appreciate it if you follow the issue template. The issues will be discussed and assigned there.
• Step2: Whenever an issue is assigned, you can contribute by creating a Pull Request by following the PR template. You can also claim for any open issues. Together we can make OpenAgents better!
• Step3: PR will be merged or iterated after review and discussion. Thanks for your contribution!

Before you start, we highly recommend taking a moment to check here before contribution.

📖 Documentation

Please check here for full documentation, which will be updated to stay on pace with the demo changes and the code release.

🧙‍Participants

Tech Lead

Co-Lead Contributors

Key Contributors

Valuable Contributors

Acknowledgments (beyond code)

Heartfelt appreciation to Ziyi Huang, Roxy Rong, Haotian Li, Xingbo Wang, Jansen Wong, and Chen Henry Wu for their valuable contributions to the OpenAgents. Their expertise and insights were instrumental in bringing this project to fruition!

Open Source Contributors

Thanks to all the contributors!

Citation

If you find our work helpful, please cite us:

@misc{OpenAgents,
      title={OpenAgents: An Open Platform for Language Agents in the Wild}, 
      author={Tianbao Xie and Fan Zhou and Zhoujun Cheng and Peng Shi and Luoxuan Weng and Yitao Liu and Toh Jing Hua and Junning Zhao and Qian Liu and Che Liu and Leo Z. Liu and Yiheng Xu and Hongjin Su and Dongchan Shin and Caiming Xiong and Tao Yu},
      year={2023},
      eprint={2310.10634},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Acknowledgments

We would like to thank Google Research, Amazon AWS, and Salesforce Research for their research gift funds to this open-source effort!

⭐️ Star History

A ⭐️ to OpenAgents is to make it shine brighter and benefit more people.

A series of code large language models developed by PKU-KCL

🤗 Hugging Face • 🤖 ModelScope • ⭕️ WiseModel • 🌐 PKU-KCL

Introduction

CodeShell是北京大学知识计算实验室联合四川天府银行AI团队研发的多语言代码大模型基座。CodeShell具有70亿参数，在五千亿Tokens进行了训练，上下文窗口长度为8192。在权威的代码评估Benchmark（HumanEval与MBPP）上，CodeShell取得同等规模最好的性能。与此同时，我们提供了与CodeShell配套的部署方案与IDE插件，请参考代码库CodeShell。同时，为了方便中国用户下载，我们在Modelscope和Wisemodel中也上传了对应版本，国内用户可以访问。

本次开源的模型如下：

• CodeShell Base：CodelShell底座模型，具有强大的代码基础能力。
• CodeShell Chat：CodelShell对话模型，在代码问答、代码补全等下游任务重性能优异。
• CodeShell Chat 4bit：CodelShell对话模型4bit量化版本，在保证模型性能的前提下内存消耗更小，速度更快。
• CodeShell CPP：CodelShell对话模型CPP版本，支持开发者在没有GPU的个人电脑中使用。注意，CPP版本同样支持量化操作，用户可以在最小内存为8G的个人电脑中运行CodeShell。

Main Characteristics of CodeShell

• 强大的性能：CodelShell在HumanEval和MBPP上达到了7B代码基座大模型的最优性能
• 完整的体系：除了代码大模型，同时开源IDE（VS Code与JetBrains）插件，形成开源的全栈技术体系
• 轻量化部署：支持本地C++部署，提供轻量快速的本地化软件开发助手解决方案
• 全面的评测：提供支持完整项目上下文、覆盖代码生成、代码缺陷检测与修复、测试用例生成等常见软件开发活动的多任务评测体系（即将开源）
• 高效的训练：基于高效的数据治理体系，CodeShell在完全冷启动情况下，只训练了五千亿Token即获得了优异的性能

Performance

我们选取了目前最流行的两个代码评测数据集（HumanEval与MBPP）对模型进行评估，与目前最先进的两个7b代码大模型CodeLllama与Starcoder相比，Codeshell 取得了最优的成绩。具体评测结果如下。

Requirements

• python 3.8 and above
• pytorch 2.0 and above are recommended
• transformers 4.32 and above
• CUDA 11.8 and above are recommended (this is for GPU users, flash-attention users, etc.)

Quickstart

CodeShell系列模型已经上传至 Hugging Face，开发者可以通过Transformers快速调用CodeShell和CodeShell-Chat。

在开始之前，请确保已经正确设置了环境，并安装了必要的代码包，以及满足上一小节的环境要求。你可以通过下列代码快速安装相关依赖。

pip install -r requirements.txt

接下来你可以通过Transformers使用CodeShell。

Code Generation

开发者可以使用CodeShell快速生成代码，加速开发效率。

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = AutoTokenizer.from_pretrained("WisdomShell/CodeShell-7B")
model = AutoModelForCausalLM.from_pretrained("WisdomShell/CodeShell-7B", trust_remote_code=True, torch_dtype=torch.bfloat16).to(device)
inputs = tokenizer('def merge_sort():', return_tensors='pt').to(device)
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))

• Fill in the Moddle

CodeShell 支持Fill-in-the-Middle模式，从而更好的支持软件开发过程。

input_text = "<fim_prefix>def print_hello_world():\n    <fim_suffix>\n    print('Hello world!')<fim_middle>"
inputs = tokenizer(input_text, return_tensors='pt').to(device)
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))

• 代码问答

CodeShell同时开源了代码助手模型CodeShell-7B-Chat，开发者可以通过下列代码与模型进行交互。

model = AutoModelForCausalLM.from_pretrained('WisdomShell/CodeShell-7B-Chat', trust_remote_code=True, torch_dtype=torch.bfloat16).to(device)
tokenizer = AutoTokenizer.from_pretrained('WisdomShell/CodeShell-7B-Chat')

history = []
query = '你是谁?'
response = model.chat(query, history, tokenizer)
print(response)
history.append((query, response))

query = '用Python写一个HTTP server'
response = model.chat(query, history, tokenizer)
print(response)
history.append((query, response))

开发者也可以通过VS Code与JetBrains插件与CodeShell-7B-Chat交互，详情请参VSCode插件仓库与IntelliJ插件仓库。

• Model Quantization

CodeShell 支持4 bit/8 bit量化，4 bit量化后，占用显存大小约6G，用户可以在显存较小的GPU上使用CodeShell。

model = AutoModelForCausalLM.from_pretrained('WisdomShell/CodeShell-7B-Chat-int4', trust_remote_code=True).to(device)
tokenizer = AutoTokenizer.from_pretrained('WisdomShell/CodeShell-7B-Chat-int4')

• CodeShell in c/c++

由于大部分个人电脑没有GPU，CodeShell提供了C/C++版本的推理支持，开发者可以根据本地环境进行编译与使用，详见CodeShell C/C++本地化版。

Demo

我们提供了Web-UI、命令行、API、IDE四种形式的Demo。

Web UI

开发者通过下列命令启动Web服务，服务启动后，可以通过https://127.0.0.1:8000进行访问。

python demos/web_demo.py

CLI Demo

我们也提供了命令行交互的Demo版本，开发者可以通过下列命令运行。

python demos/cli_demo.py

API

CodeShell也提供了基于OpenAI API的部署方法。

python demos/openai_api.py

启动后即可通过HTTP请求与CodeShell交互。

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "CodeShell-7B-Chat",
    "messages": [
      {
        "role": "user",
        "content": "你好"
      }
    ]
  }'

IDE

CodeShell最后提供了线上IDE，开发者可以通过IDE进行代码补全、代码问答等操作。同时，IDE插件也同时发布，开发者可以自行在本地进行安装使用。插件相关问题欢迎在VSCode插件仓库与IntelliJ插件仓库中讨论。

Model Details

Code Shell使用GPT-2作为基础架构，采用Grouped-Query Attention、RoPE相对位置编码等技术。

Hyper-parameter

Data

CodeShell基于自己爬取的Github数据、Big Code开源的Stack和StarCoder数据集、以及少量高质量的中英文数据进行训练。在原始数据集的基础上，CodeShell采用基于Minihash对数据去重，基于KenLM以及高质量数据筛选模型对数据进行了过滤与筛选，最终得到高质量的预训练数据集。

Tokenizer

CodeShell基于Starcoder词表进行了优化，去除了使用频率较低的词语，并添加了部分中文词表，显著提升了中文的压缩率，为Chat版本的训练提供了基础。

License

社区使用CodeShell模型需要遵循《CodeShell模型许可协议》及Apache 2.0许可协议。CodeShell模型允许用于商业用途，但如果您计划将CodeShell模型或其派生产品用于商业用途，需要您确认主体符合以下条件：

1. 关联方的服务或产品的每日平均活跃用户数（DAU）不能超过100万。
2. 关联方不得是软件服务提供商或云服务提供商。
3. 关联方不存在将获得授予的商业许可，在未经许可的前提下将其再授权给其他第三方的可能性。

在满足上述条件的前提下，您需要通过向codeshell.opensource@gmail.com发送电子邮件，提交《CodeShell模型许可协议》要求的申请材料。经审核通过后，将授予您一个全球的、非排他的、不可转让的、不可再授权的商业版权许可。

Star History

An Autonomous LLM Agent for Complex Task Solving

English • 中文 • 日本語

Tutorial • Demo • Blog • Documentation • Citation

📖 Introduction

XAgent is an open-source experimental Large Language Model (LLM) driven autonomous agent that can automatically solve various tasks. It is designed to be a general-purpose agent that can be applied to a wide range of tasks. XAgent is still in its early stages, and we are working hard to improve it.

🏆 Our goal is to create a super-intelligent agent that can solve any given task!

We welcome diverse forms of collaborations, including full-time and part-time roles and more. If you are interested in the frontiers of agents and want to join us in realizing true autonomous agents, please contact us at xagentteam@gmail.com.

XAgent

XAgent is designed with the following features:

• Autonomy: XAgent can automatically solve various tasks without human participation.
• Safety: XAgent is designed to run safely. All actions are constrained inside a docker container. Run it anyway!
• Extensibility: XAgent is designed to be extensible. You can easily add new tools to enhance agent's abilities and even new agents！
• GUI: XAgent provides a friendly GUI for users to interact with the agent. You can also use the command line interface to interact with the agent.
• Cooperation with Human: XAgent can collaborate with you to tackle tasks. It not only has the capability to follow your guidance in solving complex tasks on the go but it can also seek your assistance when it encounters challenges.

XAgent is composed of three parts:

• 🤖 Dispatcher is responsible for dynamically instantiating and dispatching tasks to different agents. It allows us to add new agents and improve the agents' abilities.
• 🧐 Planner is responsible for generating and rectifying plans for tasks. It divides tasks into subtasks and generates milestones for them, allowing agents to solve tasks step by step.
• 🦾 Actor is responsible for conducting actions to achieve goals and finish subtasks. The actor utilizes various tools to solve subtasks, and it can also collaborate with humans to solve tasks.

🧰 ToolServer

ToolServer is the server that provides XAgent with powerful and safe tools to solve tasks. It is a docker container that provides a safe environment for XAgent to run. Currently, ToolServer provides the following tools:

• 📝 File Editor provides a text editing tool to write, read, and modify files.
• 📘 Python Notebook provides an interactive Python notebook that can run Python code to validate ideas, draw figures, etc.
• 🌏 Web Browser provides a web browser to search and visit webpages.
• 🖥️ Shell provides a bash shell tool that can execute any shell commands, even install programs and host services.
• 🧩 Rapid API provides a tool to retrieve APIs from Rapid API and call them, which offers a wide range of APIs for XAgent to use. See ToolBench to get more information about the Rapid API collections. You can also easily add new tools to ToolServer to enhance XAgent's abilities.

✨ Quickstart

🛠️ Build and Setup ToolServer

ToolServer is where XAgent's action takes place. It is a docker container that provides a safe environment for XAgent to run. So you should install docker and docker-compose first. After that, you should build the docker image for ToolServer and start the docker container.

docker-compose up --build

This will build the image for the ToolServer and start the ToolServer's container. If you want to run the container in the background, please use docker-compose up -d --build. Refer here for detailed information about our ToolServer.

If the ToolServer is updated, you have to rebuild the images:

docker compose build

🎮 Setup and Run XAgent

After setting up ToolServer, you can start to run XAgent.

• Install requirements (Require Python >= 3.10)

pip install -r requirements.txt

• Configure XAgent

1. You should configure XAgent in assets/config.yml before running it.
2. At least one OpenAI key is provided in assets/config.yml, which is used to access OpenAI API. We highly recommend using gpt-4-32k to run XAgent; gpt-4 is also OK for most simple tasks. In any case, at least one gpt-3.5-turbo-16k API key should be provided as a backup model. We do not test or recommend using gpt-3.5-turbo to run XAgent due to minimal context length; you should not try to run XAgent on that.
3. If you want to change the config_file path for XAgentServer, you should modify the CONFIG_FILE value in .env file and restart the docker container.

• Run XAgent

python run.py --task "put your task here" --model "gpt-4" --config_file "assets/config.yml"

1. You can use the argument --upload_files to select the initial files you want to submit to XAgent.

2. The local workspace for your XAgent is in local_workspace, where you can find all the files generated by XAgent throughout the running process.

3. After execution, the entire workspace in ToolServerNode will be copied to running_records for your convenience.

4. Besides, in running_records, you can find all the intermediate steps information, e.g., task statuses, LLM's input-output pairs, used tools, etc.

5. You can load from a record to reproduce a former run, just by setting record_dir in config(default to Null). The record is a system-level recording tied to the code version of XAgent. All running-config、query、code execution statuses (including errors)、server behavior will be documented.

6. We have removed all sensitive information (including API keys) from the record so you can safely share it with others. In the near future, we will introduce more granular sharing options highlighting the contributions of humans during execution.

• Run XAgent with GUI

## We ran the web ui docker when building the ToolServer network
## run nginx in docker
docker exec XAgent-Server systemctl start nginx

Build the docker image for XAgent-Server and start the docker container. You will see the XAgent Server listening on port 8090. You could visit http://localhost:5173 to interact with XAgent by using web UI. Refer here for the detailed information about our GUI Demo.

🎬 Demo

Here, we also show some cases of solving tasks by XAgent: You can check our live demo on XAgent Official Website. We also provide a video demo and showcases of using XAgent here:

Case 1. Data Analysis: Demonstrating the Effectiveness of Dual-Loop Mechanism

We start with a case of aiding users in intricate data analysis. Here, our user submitted an iris.zip file to XAgent, seeking assistance in data analysis. XAgent swiftly broke down the task into four sub-tasks: (1) data inspection and comprehension, (2) verification of the system's Python environment for relevant data analysis libraries, (3) crafting data analysis code for data processing and analysis, and (4) compiling an analytical report based on the Python code's execution results. Here is a figure drawn by XAgent.

Case 2. Recommendation: A New Paradigm of Human-Agent Interaction

Empowered with the unique capability to actively seek human assistance and collaborate in problem-solving, XAgent continues to redefine the boundaries of human-agent cooperation. As depicted in the screenshot below, a user sought XAgent's aid in recommending some great restaurants for a friendly gathering yet failed to provide specific details. Recognizing the insufficiency of the provided information, XAgent employed the AskForHumanHelp tool, prompting human intervention to elicit the user's preferred location, budget constraints, culinary preferences, and dietary restrictions. Armed with this valuable feedback, XAgent seamlessly generated tailored restaurant recommendations, ensuring a personalized and satisfying experience for the user and their friends.

Case 3. Training Model: A Sophisticated Tool User

XAgent not only tackles mundane tasks but also serves as an invaluable aid in complex tasks such as model training. Here, we show a scenario where a user desires to analyze movie reviews and evaluate the public sentiment surrounding particular films. In response, XAgent promptly initiates the process by downloading the IMDB dataset to train a cutting-edge BERT model (see screenshot below), harnessing the power of deep learning. Armed with this trained BERT model, XAgent seamlessly navigates the intricate nuances of movie reviews, offering insightful predictions regarding the public's perception of various films.

📊 Evaluation

We conduct human preference evaluation to evaluate XAgent's performance. We prepare over 50 real-world complex tasks for assessment, which can be categorized into 5 classes: Search and Report, Coding and Developing, Data Analysis, Math, and Life Assistant. We compare the results of XAgent with AutoGPT, which shows a total win of XAgent over AutoGPT. All running records can refer to here.

We report a significant improvement of XAgent over AutoGPT in terms of human preference.

We also evaluate XAgent on the following benchmarks:

🖌️ Blog

Our blog is available at here!

🌟 Our Contributors

A heartfelt thank you to all our contributors. Your efforts make this project grow and thrive. Every contribution, big or small, is invaluable.

🌟 Star History

Citation

If you find our repo useful, please kindly consider citing:

@misc{xagent2023,
      title={XAgent: An Autonomous Agent for Complex Task Solving}, 
      author={XAgent Team},
      year={2023},
}

Hello HN! I want to share something me and a few friends have been working on for a while now — Zeroshot, a web tool that builds image classifiers using text-image models and autolabeling. What does this mean in practice? You can put together an image classifier in about 30 seconds that’s faster and more accurate than CLIP, but that you can deploy yourself however you’d like. It’s open source, commercially licensed, and doesn’t require you to pay anyone per API call.

Here's a 2 minute video that shows it off: https://www.youtube.com/watch?v=S4R1gtmM-Lo

How/why does it work? We believe that with the rise of foundation vision models, computer vision will fundamentally change. These powerful models will let any devs “compile” a model ahead of time with a subset of the foundation model’s characteristics, using only text and a web-tool. The days of teams of MLEs building complex models and pipelines are ending.

Zeroshot works by using two powerful pre-trained models, CLIP and DINOv2 together. The web-app allows users to quickly create our training sets via text search. Using pre-cached DINOv2 features, we generate a simple linear model that can be trained and deployed without any fine-tuning. Since you can see what’s going into your training set, you can tune your prompts to get the type of performance or detail you want.

CLIP Small -- Size: 335 MB, Latency: 35ms

CLIP Large -- Size: 891 MB, Latency: 276ms

Zeroshot -- Size: 85 MB, Latency: 20ms

What’s next? We wanna see how people use or would use the tool before deciding what to do next. On the list: clients for iOS and NodeJS, speeding up GPU inference times via TensorRT, offering larger Zeroshot models for better accuracy, easier results refining, support for bringing your own data lake, model refinement using GPT-V, we’ve got plenty of ideas.

Comments URL: https://news.ycombinator.com/item?id=38450690

Points: 13

# Comments: 6

We are very excited to introduce JungleGym, our new open-source launch featuring datasets and tools for developing autonomous web agents, by Marco Mascorro and Matt Bornstein.

Addressing the challenge of benchmarking and testing AI agents, JungleGym aims to facilitate builders in this domain.

Project overview: https://junglegym.ai/

Live demo: https://junglegym.ai/TreeVoyager%20(LLM%20DOM%20Parser)

GitHub: https://github.com/a16z-infra/JungleGym

Datasets included: Mind2Web (Zhou et al.), WebArena (Shuyanzhxyc, Frankxu2004, _Hao_Zhu et al.), and AgentInstruct (Zeng et al.). These datasets allow for comprehensive testing of agents against established ground truths, accessible via the JungleGym API.

*TreeVoyager*: A new LLM-based DOM parser that simplifies task logic implementation for agent developers. Inspired by Tree of Thoughts (ShunyuYao12) and Minecraft Voyager (Guanzhi_Wang, DrJimFan).

This project builds on the * remarkable work * by the teams behind these datasets and incorporates feedback from agent developers like SigGravitas, DivGarg, Yohei Nakajima, and draws inspiration from projects like World of Bits (TimShi_ai).

Comments URL: https://news.ycombinator.com/item?id=38448995

Points: 11

# Comments: 1

Really excited to share this, with new features like v0, tldraw, magicpatterns, and other AI tools for UI generation, we figured why not do the same for APIs and backends?

Once you provide a prompt, image, or your code, ob1 generates the DDL for the SQL, the functions to power it, and even deploys it for you.

As far as next steps: our goal is to open-source this we're just dog fooding a lot of Outerbase right now to prove it out and we are peeling that back to make it so you can run it all locally.

Comments URL: https://news.ycombinator.com/item?id=38448646

Points: 10

# Comments: 1

Demo Video: https://www.youtube.com/watch?v=XpyVKUyt7k8

Hey all! A friend and I have been building projects with open-source LLMs for a while now (originally for other project ideas) and found that quickly iterating with different fine-tuning datasets is super hard. Training a model, setting up some inference code to try out the model and then going back and forth took 90% of our time.

That’s why we built Haven, a service to quickly try out different fine-tuning datasets and base-models. Going from uploading a dataset to chatting with the resulting model now takes less than 5 minutes (using a reasonably sized dataset).

We fine-tune the models using low-rank adapters, which not only means that the changes made to the model are very small (only 30mb for a 7b parameter LLM), it also allows us to host many fine-tuned models very efficiently by hot swapping adapters on demand. This helped us reduce cold-start times to below one second and makes it possible for us to host a single trained model for a few dollars per month. [Research has shown](https://arxiv.org/pdf/2305.14314.pdf) that low-rank fine-tuning performance stays almost on-par with full fine-tuning.

We charge between $0.004/1k training tokens, and after signing up, you get $5 in free credits. You can export all the models to Huggingface.

Right now we support Llama-2 and Zephyr (which is itself a fine-tune of Mistral) as base models. We’re gonna add some more soon. We hope you find this useful and we would love your feedback!

Comments URL: https://news.ycombinator.com/item?id=38448334

Points: 12

# Comments: 1

Hey HN,

I wanted a better load testing solution – so I built one with my team at Multiple. We just opened early access and would love to get your feedback.

We created Multiple to solve three challenges with existing tools:

1. Limited scripting capabilities. XML or GUI-based scripting can only test basic scenarios. Existing code-based tools struggle with auth, generating synthetic data, and testing anything other than HTTP requests. We went with JavaScript for ease of use, versatility, and integration with existing developer workflows.

2. Cannot use existing libraries or code. Instead of forcing you to learn a new system and rewrite code, Multiple leverages the JavaScript and NPM ecosystem so you can use packages you're already familiar with. By supporting NPM packages, Multiple can test nearly any API, service, or protocol.

3. Tedious infrastructure management. There's no reason to spend time spinning up and configuring machines, and then destroying them after a test. Multiple abstracts that away. You just enter the test size and duration, and press start.

My favorite feature we've built so far is the Debug Run. You can use Debug Run as you write your tests to execute a single run-through. It's helpful to verify correct behavior and capture logs, and it allows you to iterate quickly, without spinning up a full load test each time.

We have so much in store for developers: pass/fail conditions, CLI, and repo integration, to name a few. Thanks for reading, and let us know what you think.

Comments URL: https://news.ycombinator.com/item?id=38445762

Points: 13

# Comments: 7

Article URL: https://github.com/cap-so/cap

Comments URL: https://news.ycombinator.com/item?id=38445731

Points: 14

# Comments: 1

Hey, HN Community!

I'm a Brazilian software engineer, and my educational journey led to a significant discovery. During my master's degree, I often struggled to recall foundational concepts studied years earlier, which were crucial for my current studies. This challenge sparked a reevaluation of traditional note-taking and inspired the creation of cmaps.io.

Human thinking is inherently associative, not linear. We often draw connections between learned concepts unconsciously. To leverage this natural process, making these connections explicit is essential – and that's precisely what cmaps.io offers.

I can personally vouch for the impact of cmaps.io on my learning and retention. It's become an invaluable tool for me, and I'm confident it can be for you as well!

You can find more details on the ProductHunt post: https://www.producthunt.com/posts/cmaps-io

Discover cmaps.io and enjoy a revolutionary note-taking experience!

I eagerly await your feedback and thoughts

Comments URL: https://news.ycombinator.com/item?id=38444241

Points: 12

# Comments: 7

As an enthusiast of Indian classical music, I needed to write music notations in the traditional typeset format. When I didn't find any existing editors, I developed a Swara Notebook, a mobile focused web app to write North Indian Classical (Hindustani) music notations.

The notes (called Sargam, similar to Solfege) can be written in English, Devnagri and Bangla scripts. The transcribed song can be played back in 6 different rhythmic cycles (Taal) to the accompaniment of the Tabla(a type of drum) or a metronome. Here's an example of a transcribed song https://swaranotebook.com/view/vlB9hVgb5OdKcadbhOOYyzKwvpl2/...

Since North Indian classical music is oriented around Ragas (similar to modes in western classical music), the keyboard adapts to the notes of a Raga, making it contextually easier to key in notes.

A common question I get: can it handle microtones? I chose to not support microtones and other pitch ornamentation such as glissando (called Meend) since a mobile interface is not the easiest place to add such details.

I also find it a useful tool for ear training, by typing out notations to songs I know, and playing it back to know if I guessed the notes right.

It is an open source project written in Clojure/script https://github.com/Studio-kalavati/bandish-editor

Comments URL: https://news.ycombinator.com/item?id=38441584

Points: 17

# Comments: 3

Article URL: https://spaceflight-news.jerrynsh.com/

Comments URL: https://news.ycombinator.com/item?id=38440499

Points: 10

# Comments: 1

Hey all! My train the other day was delayed and I got curious where they all were at any given time, so I built a map and figured I'd share it.

Comments URL: https://news.ycombinator.com/item?id=38434574

Points: 103

# Comments: 71

Comments

Comments

Article URL: https://www.astralcodexten.com/p/god-help-us-lets-try-to-understand

Comments URL: https://news.ycombinator.com/item?id=38438261

Points: 102

# Comments: 40

Article URL: https://opencv.org/blog/where-is-opencv-5/

Comments URL: https://news.ycombinator.com/item?id=38430074

Points: 111

# Comments: 47

Comments

Comments

Comments

Article URL: https://arxiv.org/abs/2311.10770

Comments URL: https://news.ycombinator.com/item?id=38378242

Points: 109

# Comments: 26

Comments

Comments

Comments

What are you doing this week? Feel free to share!

Keep in mind it’s OK to do nothing at all, too.

Comments

Article URL: https://computer.rip/2023-11-25-the-curse-of-docker.html

Comments URL: https://news.ycombinator.com/item?id=38418359

Points: 111

# Comments: 54

Feel free to tell what you plan on doing this weekend and even ask for help or feedback.

Please keep in mind it’s more than OK to do nothing at all too!

Comments

Article URL: https://akkartik.itch.io/carousel

Comments URL: https://news.ycombinator.com/item?id=38397715

Points: 27

# Comments: 3

Comments

Hi everyone,

Sharing here the latest Materio Bootstrap 5 Django Admin Template. I previously shared the Sneat design version and this one is based on the Material design.

Like the Sneat version, Materio is also developer-friendly, and highly customizable Django Admin Dashboard. It is fast and easy to use as well as highly scalable. Furthermore, it even offers ultimate convenience and flexibility, too. You can easily build whatever application you want with very little hassle such as SaaS apps, eCommerce apps, Banking apps, Educational apps, CRM apps, etc.

Above all, it makes it app development process very smooth and easy. It saves tons of time as it is a pre-designed admin dashboard with most of the required features and components.

Features

• Built with Django 4
• Using CSS Framework Bootstrap 5.3.2 [Updated version]
• CRUD Example [New]
• User-based authentication
• Docker for Faster Development
• Vertical & Horizontal layouts
• Default, Bordered & Semi-dark themes
• Light, Dark & System mode support
• Internationalization/i18n & RTL Ready
• Python-Dotenv: Environment variables
• Theme Config: Customize our template without a sweat
• 5 Dashboard
• 10 Pre Built Apps
• 15+ Front Pages

Hope you all find this dashboard helpful.
Please share your valuable feedback.

1 post - 1 participant

Read full topic

In the past decade, we have seen a defined separation between the backend and the frontend with backend servers being used exclusively as API backends for a static frontend hosted on CDNs or a separate server like NextJs, Nuxt etc.

In a multi-tenant API backend, the server is hosted at a single endpoint and the tenant is mostly identified via a header or path-based routing. However, Django’s default sites framework assumes that the application will always serve multiple domains.

It would be great if we could have an AbstractSite model with a swappable Site model (like the user ), so that we could target the same API / functions while implementation details (like tenant lookup and additional fields) differ.

4 posts - 2 participants

Read full topic

On Postgres 16
If you create a collation:

CreateCollation(
  "case_insensitive",
  provider="icu",
  locale="und-u-ks-level2",
  deterministic=False,
),

And then a field with that collation

models.CharField(
  max_length=255,
  db_collation='case_insensitive',
)

And then try to do an icontains lookup (for example using the admin search), it will return this database error:

nondeterministic collations are not supported for LIKE

Which is surprising considering you expect it to use ILIKE instead of LIKE. Digging into the code pointed me to this bug created 15 years ago

https://code.djangoproject.com/ticket/3575

In which it was decided it was better to use UPPER('field') LIKE UPPER('%value%') instead of ILIKE for performance reasons, which I’m not sure are true anymore. Update: a quick and unscientific test I did on a million rows shows that ILIKE is 20% slower.

Since citext is deprecated in django and the “best” way to do case insensitive queries in postgres are collations, I’m not sure if the decision to not use ILIKE should be re-evaluated. Would it make sense for the documentation to warn of this?

In my case I just want the admin to do a case insensitive search on multiple fields, one of which has a nondeterministic collation, but I don’t see any easy way to do that. The simplest way I see is a custom field that when icontains is used, changes it use a custom different lookup that uses ILIKE

5 posts - 3 participants

Read full topic

Hi Dear Translators,

Django 5.0 RC1 was released on November 20 so this means this is now the last moment to check your Django translations are complete on Transifex , as strings shouldn’t change before the final 5.0 release on December 4. Thanks!

Best,
Mariusz

3 posts - 2 participants

Read full topic

On Django 4.2 and below, formsURLField adds the “http” scheme to scheme-less URLs. For example, it cleans example.com into http://example.com. Ticket 34380 makes this assumption https://example.com.

In the related PR, the implementation initially went for a hard cut, changing the field to assume “https” immediately. But this was changed to a deprecation after this comment by David Wobrock, stating it was backwards-incompatible.

I propose moving the implementation back to a hard cut approach before Django 5.0 has its impending final release.

I base this proposal on my experience fixing the warning in a project that I was testing on Django 5.0. I started drafting a blog post covering the multiple required techniques to set assume_scheme everywhere, covered at the end of this post. Whilst writing, I realized it’s a lot of work for such a minor change, and I think we can avoid hoisting this work on the community.

I would guess that most users entering a URL into a form copy-and-paste it from their browser, in which case it comes with a scheme. For the remaining cases where a URL comes with no scheme, it’s still overwhelmingly likely to work with HTTPS. According to Google Chrome’s statistics (“HTTPS Encryption by Chrome platform”), 93% of page loads on Windows use HTTPS.

Using the deprecation pathway to make this change adds warnings to projects for every URLField, including those generated by ModelForm, ModelAdmin, and third-party packages. It helps only a small number of projects that rely on users entering HTTP URLs without schemes (corporate intranets are my prime suspect).

I think we can make a hard cut without much problem. As a precedent, the now-removed ping_google command was hard cut to use HTTPS in Ticket 23829.

Sorry for the many words. Tagging people involved in the ticket: @felixxm.

Techniques from my drafted blog post:

Plain Forms with a URLField wrapper

For forms that you directly create, add the assume_scheme argument to their URLFields:

from django import forms


class CheckForm(forms.Form):
    url = forms.URLField(label="URL to check", assume_scheme="https")

If you have many URLFields, try functools.partial to reduce the repetition:

from functools import partial

from django import forms


HTTPSURLField = partial(forms.URLField, assume_scheme="https")


class CheckForm(forms.Form):
    url = HTTPSURLField(label="URL to check")

ModelForms with formfield_callback

ModelForms are a bit more tricky since their fields are automatically created “behind the scenes” from model fields. You can override these with explicit form fields, but that loses the automatic synchronization of other arguments like max_length.

Instead, you can use formfield_callback, added in Django 4.2. This callback hook allows you to wrap the creation of form fields to make adjustments as necessary. Use it to add assume_scheme=“https” to URLFields like so:

from django import forms
from django.db import models

from example.models import NicheMuseum


def urlfields_assume_https(db_field, **kwargs):
    """
    ModelForm.Meta.formfield_callback function to assume HTTPS for scheme-less
    domains in URLFields.
    """
    if isinstance(db_field, models.URLField):
        kwargs["assume_scheme"] = "https"
    return db_field.formfield(**kwargs)


class NicheMuseumForm(forms.ModelForm):
    class Meta:
        model = NicheMuseum
        fields = ["name", "website"]
        formfield_callback = urlfields_assume_https

For forms that already use formfield_callback, you’ll need to merge functions as appropriate.

ModelAdmin form field method override

If you’re using Django’s admin site, that’s another source of outdated URLFields. ModelAdmin classes automatically generate ModelForm classes, unless a form is explicitly declared. Because this form class is created per-request, the warning won’t appear at import time, but when you load the related “add” or “change” pages:

$ python -Xdev manage.py runserver
Watching for file changes with StatReloader
Performing system checks...

...
/.../django/db/models/fields/__init__.py:1140: RemovedInDjango60Warning: The default scheme will be changed from 'http' to 'https' in Django 6.0. Pass the forms.URLField.assume_scheme argument to silence this warning.
  return form_class(**defaults)
[25/Nov/2023 16:37:34] "GET /admin/example/nichemuseum/add/ HTTP/1.1" 200 8907
...

You can also expose the warnings in your test suite by testing all of your ModelAdmin classes. See my previous post on using parameterized tests to do this easily.

The easiest way to fix these warnings is with the undocumented ModelAdmin.formfield_for_dbfield() method. This is passed as ModelForm.formfield_callback, and applies some customizations for the admin site. Override it in custom admin classes that you then use as bases in your project:

from django.contrib import admin
from django.contrib.admin.options import BaseModelAdmin
from django.db import models

from example.models import NicheMuseum

# Custom admin classes for project-level modifications


class AdminMixin(BaseModelAdmin):
    def formfield_for_dbfield(self, db_field, request, **kwargs):
        """
        Assume HTTPS for scheme-less domains pasted into URLFields.
        """
        if isinstance(db_field, models.URLField):
            kwargs["assume_scheme"] = "https"
        return super().formfield_for_dbfield(db_field, request, **kwargs)


class ModelAdmin(AdminMixin, admin.ModelAdmin):
    pass


class StackedInline(AdminMixin, admin.StackedInline):
    pass


class TabularInline(AdminMixin, admin.TabularInline):
    pass


# Admin classes


@admin.register(NicheMuseum)
class NicheMuseumAdmin(ModelAdmin):
    pass

Third-party packages

URLFields in third-party packages may be hard to fix in your project. Some packages do allow you to replace their form classes, so you can use the above techniques. Others don’t, so you may need to resort to monkey-patching.

In either case, projects will hopefully accept pull requests to use assume_scheme=“https”, if appropriate. Packages can pass the argument on Django 5.0+ by using this pattern:

from functools import partial

import django
from django import forms

if django.VERSION >= (5, 0):
    URLField = partial(forms.URLField, assume_scheme="https")
else:
    URLField = forms.URLField


class CheckForm(forms.Form):
    url = URLField(label="URL to check")

1 post - 1 participant

Read full topic

please can you tell me more about this test, i’d like to understand why my pull request fails. on my local site, all the tests work.

2 posts - 2 participants

Read full topic

The exception flow for CSRF issues is a bit difficult. I think we can make it easier for developers to determine the root cause of a CSRF invalidation by doing the following:

1. Log a detailed explanation of the error. For example when a CSRF token is missing from the headers, the error message presented to the user is “CSRF token missing.” This doesn’t explain where it’s missing or how it could potentially be added in. For new Django devs this is a very cryptic message.
2. (less sure) Use a unique constant for each RejectRequest usage in django.middleware.csrf. This makes it easier to identify where exactly the error is coming from. It also creates consistency in how we create these errors.
3. Add a system check that confirms that all values in CSRF_TRUSTED_ORIGINS have a scheme specified. Not specifying a host doesn’t break things until a request is made and it fails to validate a request. So despite having a reasonable looking configuration, it doesn’t work until the user gets to a remote environment. If the setting requires a scheme to be specified (whereas ALLOWED_HOSTS doesn’t), Django should be validating those values. This would raise the issue sooner for the developer, allow for a more specific error message and be easier to solve.

Edit:
I’m opening this for discussion to get community buy-in before implementing any changes.

1 post - 1 participant

Read full topic

Greetings team,

I would love to join the Swahili team to help with translations. I have just sent my request and I would appreciate some feedback. Thank you

3 posts - 3 participants

Read full topic

Hi all!
While writing an article it occurred to me that it could be very useful to be able to use doctest to have unit tests in the method docstrings.

Here is an example:

class Person(models.Model):
    name = models.TextField()

    def __str__(self):
        """
        >>> Person(name="Jane Doe")
        <Person: Jane Doe>
        """
        return self.name

Obviously now when running doctest on a Django project this doesn’t work.

$ python3 -m doctest samples/models.py

I saw that in very old versions of Dajngo doctest it was supported and then it was removed.

I think that having tests in the docstrings of the methods of a class could greatly simplify development, provide an immediate example of how to use the methods, following the principle of concentrating the information in one point and could also help newcomers to get started to write tests in their Django projects.

What do you think?

2 posts - 2 participants

Read full topic

Problem

With every new feature release, Django increases the PBKDF2 iteration count (PBKDF2 is the default password hasher). For Django 4.2 it is 600,000, and for the upcoming Django 5.0 it will be 720,000.

I think that Django 5.0 iteration count is becoming too much. On commodity AWS instances, one password hash takes ~0.7s with 600,000, and it keeps increasing with every release. I don’t think server hardware gets faster at this rate.

It is quite easy to bring a set of servers to their knees with a simple DDOS on the login endpoint. Note that even if the username doesn’t exist, the password hasher still runs, as a mediation for timing attacks. It is difficult to protect against this attack, requiring captchas or annoying stuff like that. A very high iteration count makes this attack less expensive for the attacker.

Of course it’s possible to override the default (which I do), but it’s another thing to do and most people have no clue about it - the default matters. I also like that someone else worries about it for me.

Proposal

My suggestion is to change the policy to match the OWASP recommendation at the relevant time, which handily is currently 600,000, like Django 4.2. Django 5.0 will exceed it.

4 posts - 3 participants

Read full topic

My self-nomination statement for the 2024 Django Software Foundation (DSF) board of directors elections

News

Django 5.0 release candidate 1 released

Django 5.0 release candidate 1 is the final opportunity for you to try out a deluge of exciting new features before Django 5.0 is released.

djangoproject.com

Python 3.13.0 alpha 2 is now available

Python 3.13.0a2 is the second of seven planned alpha releases.

blogspot.com

Updates to Django

Last week we had 17 pull requests merged into Django by 11 different contributors - including 4 first time contributors! Congratulations to Andrew Cordery, jed212, William Hayes and Giannis Terzopoulos for having their first commits merged into Django - welcome on board!

Some key updates from last week:

• Python 3.12 support has been added to Django 4.2.8 (release scheduled for December 4th).
• From Django 5.1, the HTML attribute aria-describedby is available on form fieldsets - enabling screen readers to associate fieldsets with their help text.
• The Django accessibility team has publicly stated their targeted assistive technology support for the Django admin. If you are a user of any of these, we'd love for you to introduce yourself to the team in the Django Discord #accessibility channel 💚

The accessibility team is also looking for feedback on their guidelines for Django contributors. Both “general sentiment” comments and specific issues / possible improvements are very welcome!

Django Newsletter

Wagtail CMS

Wagtail CMS Accessibility Webinar

Join us on December 13th to explore accessibility within the world of Wagtail CMS. Whether you're a seasoned Wagtail editor, a developer, or on the hunt for an open-source CMS that prioritizes accessibility - this event is for you.

wagtail.org

Sponsored Ad

Lifetime access to SaaS Pegasus 50% off this week only

SaaS Pegasus is the configurable SaaS boilerplate trusted by more than a thousand Django developers. With built-in features like teams, billing, a modern front end, instant deployment and more, you can use it to launch your next application faster than you imagined possible. This week only, you can get a lifetime license for Pegasus for 50% off (a $500 savings).

saaspegasus.com

Articles

Django-related Deals for Black Friday and Cyber Monday 2023

Adam Johnson has collected a number of discounts including for his books, Michael Yin's, SaaS Pegasus, Tango with Django, testdriven.io, and Will Vincent's.

adamj.eu

How to find a ticket on Django's trac

A helpful guide to finding, filtering, assigning tickets to yourself on Django's trac.

.smithdc.uk

Querying every file in every release on the Python Package Index

How to query every PyPI file in every release to examine programming language usage trends and metadata adoption.

sethmlarson.dev

Maybe Ignore type=search — Adrian Roselli

If you have a valid, accessible search field (with a useful, sensible label) then you can probably ignore type="search" and use type="text" instead. Lots of examples are included for every web browser.

adrianroselli.com

Building a Bootstrap styled form in vanilla Django

A detailed look at rebuilding django-crispy-forms functionality using vanilla Django; no 3rd party packages allowed. Hint: it's hard!

.smithdc.uk

A Disorganized List of Maintainer Tasks

A lengthy list of maintainer tasks from the maintainer of Flask, David Lord. It should provide some appreciation for the work our Django Fellows and others do!

davidism.com

Tutorials

How to Render Markdown in a Django Application

A good tutorial demonstrating how to add markdown to a Django app, display it in the templates, and format code snippets using Pygments.

realpython.com

Sponsored Link

Wagtail CMS Developer Training - Introductory Offer

Build your knowledge and skills in essential Django and Wagtail CMS practices. A two-part training programme, led by Senior Wagtail Developers that extends way beyond typical tutorials and documentation. Only 10 places available per course. Apply here: https://bit.ly/wagtail-developer-training-course

bit.ly

Videos

Django Ninja 1.0 - What's new

Django Ninja v1.0 just released! It is a web framework for building APIs with Django and Python 3.6+ type hints.

youtube.com

DjangoCon US 2023: The First Talk about his First Contribution with Ryan Cheley

Jay Miller interviewed first-time-conference presenter Ryan Cheley at DjangoCon US 2023.

youtube.com

Django Day CPH 2023: Jon Gould -Unlocking the Django Job Market: Strategies for Developers and Employers

Jon Gould provides some expert tips on moving between jobs, obtaining your first job, and then switches focus to "How to hire a Django developer," focusing on writing a job spec, interviewing/testing, and making an offer.

youtu.be

Podcasts

Django Chat #151: Web Security with Mackenzie Jackson

Mackenzie is a developer advocate at GitGuardian. We discuss handling security reports in Django, the pros and cons of environment variables, why ChatGPT is a security concern and more.

djangochat.com

Django News Jobs

Full Stack Python Developer at Loginsoft

Senior Python Engineer at Loginsoft

Software Engineer - Ubuntu Systems Management at Canonical

Django Newsletter

Projects

radiac/django-flasky

Write a Django site which looks like Flask.

github.com

sampotts/plyr

A simple HTML5, YouTube and Vimeo player.

github.com

Django Forum

Django 5.0 string freeze is in effect, translations needed!

If you'd like to help translate the Django 5.0 docs, now is the time!

djangoproject.com

This RSS feed is published on https://django-news.com/. You can also subscribe via email.

An introduction to database generated columns, using PostgreSQL and the new GeneratedField added in Django 5.0.

A summary of the past year as I finish my third year working for Kraken Tech, an Octopus Energy Group subsidiary.

Note: As I write this I'm recovering from a sprained shoulder I received in a bicycle accident that makes it hard to type. I'm mostly using voice-to-text, so the text might not be in my normal "writing voice".

I changed roles

I transitioned from leading the tech team for Kraken Tech USA to being an individual contributor for the global team. I do this periodically, cycling between writing code to leading teams, and back again.

It is never easy to transition between roles but this time in particular it was particularly hard. The US Kraken Tech team is a group of talented, diverse, and passionate people. I'm so honored I had the chance to work with them. I'm constantly amazed by what they have accomplished and what they continue to do.

In my new role I'm on the "Developer Foundations" team, exploring ways to improve the coding experience at the company. I've enjoyed writing code full-time again, there's a clarity of purpose that's been fun to embrace. Mostly that's been working in Python with a bit of Django, with short detours into Rust and Golang.

One more thing: For the moment I still have the old management-sounding title, but expect to get something more accurate in the next few months.

I blogged more

Due to that I'm coding full-time again, I more than doubled the number of posts I wrote in 2022.

I worked in the London, UK Office

Audrey and Uma joined me on a move to London, which I documented here.

It has been a delight being able to work in-person with my peers, or at least close in timezone. I'm usually in the office about three out of every five days. I would make it five-out-of-five but I do better with deep thinking outside normal office commotion.

I attended large in-person tech events

After a break of four years I managed to attend the following in chronological order, with transit mode specified:

• Django London (walk or train)
• PyCon Italy (airplane)
• PyCon UK (train)
• Django Paris (train)

I very much enjoyed meeting old friends and making new ones. Plus, exploring the cities of London, Cardiff, and Paris were adventures of their own.

Helped scale up Kraken Tech to address climate change

As an individual concerned about global climate change I do my part:

• I compost about 80% of my family's food waste
• Do our best to recyle all the family's cardboard and paper products
• We live in a tiny flat with an electric stove
• Am experimenting with using a portable electric heater instead of the methane gas boiler
• Have reduced unnecessary air travel as much as possible (which is hard - I love airplanes and airports)

That's great, but I have managed to scale that up tens of thousands of times over by working for Kraken Tech.

As one employee of nine hundred and thirty seven (937) at Kraken Tech, I take payment from a collective of companies whose products are all designed with the purpose of saving the planet. Kraken Tech in particular is responsible for about 55 million people getting onto renewable energy or using non-renewable resources more efficiently.

Let's math that out, using conservative numbers:

58,697.97 = 55,000,000 / 937

Summary: Each employee of Kraken Tech is roughly responsible for 58,698 (rounding up) people contributing significantly less to global climate change.

So if you are in tech in any capacity (coding, management, etc), consider using your skills to help us in our mission of making a healthier planet.

Come and help me save the planet

I'll be upfront that if you want to make the biggest bucks possible in tech, travel a lot, and fly in first place class, go work for a Big Tech Company or a fintech. On the other hand, you risk more frequent layoffs. Also, we're making the planet better for our descendants. That to me is much more important than helping a billionaire score another billion.

I invite anyone who is reading this to join me on my mission of saving the planet. Our careers page is here.

Please understand that Kraken Tech only hires in countries in which we have a legal entity. So if a role isn't listed for the country in which you can legally work, we probably can't hire you.

If for whatever reason you can't get a job working with me, apply to work at somewhere else into decarbonization. Let me know where you go and I'll celebrate your joining our shared mission.

Django 5.0 release candidate 1 is the final opportunity for you to try out a deluge of exciting new features before Django 5.0 is released.

The release candidate stage marks the string freeze and the call for translators to submit translations. Provided no major bugs are discovered that can't be solved in the next two weeks, Django 5.0 final release is planned for December 4. Any potential delays will be communicated on the Django forum.

Please use this opportunity to help find and fix bugs (which should be reported to the issue tracker), you can grab a copy of the release candidate package from our downloads page or on PyPI.

The PGP key ID used for this release is Natalia Bidart: 2EE82A8D9470983E.

Here are some Django-related deals for this year’s Black Friday (24th Nov) and Cyber Monday (27th Nov), including my own.

I’ll keep updating this post as I learn about more deals. If you are also a creator, email me with details of your offer and I’ll add it here.

My books

My three books have a 50% discount, for both individual and team licenses, until the end of Cyber Monday (27th Nov). This deal stacks with the purchasing power parity discount for those in lower-income countries.

Buy now:

• Boost Your Django DX - $19.50 instead of $39
• Boost Your Git DX - $19.50 instead of $39 (released in October!)
• Speed Up Your Django Tests - $24.50 instead of $49

Aidas Bendoraitis’ GDPR Cookie Consent Package

Aidas Bendoraitis of djangotricks.com created this paid third-party app for Django. The package takes the pain out of setting up and customizing legally mandated GDPR Cookie Consent screens. Compared to commercial “one size fits all” solutions, it’s much simpler to use this third-party app to host and tweak your project’s cookie consent screen.

Use the discount code BLACKFRIDAY2023 for 20% off, from €150 to €120, until the end of November.

Buy it on Gumroad

Michael Yin’s Hotwire and Django book

Michael Yin has written a book on using Hotwire with Django. This is the “HTML-over-the-wire” suite of tools used heavily in Ruby on Rails.

Use this link for 20% off, from $39.99 to $27.80.

SaaS Pegasus

Corey Zue’s SaaS Pegasus, is a configurable Django project template with many preset niceties, including teams, Stripe subscriptions, a JavaScript pipeline, and multiple CSS themes. It can massively accelerate setting up a SaaS in Django.

The “unlimited lifetime license” is discounted 50%, from $999 to $499.50. This deal is available through 29 November.

Buy it on saaspegasus.com

Tango with Django

Tango with Django is an end-to-end beginner’s guide to Django, by Leif Azzopardi and David Maxwell.

The book’s “headline” price is discounted 50%, from $29.99 to $14.99.

Buy it on Leanpub

Will Vincent’s Three Book Bundle

Will Vincent is the author of three fantastic Django books:

• Django for Beginners
• Django for APIs
• Django for Professionals

He’s offering a 50% discount on the three-book bundle, from $97 to $48.50.

Buy it on Gumroad

Bonus: Django Itself

Okay, there’s no discount here, but there is a good deal! You can fund the framework that you love to ensure it continues to grow.

If you can spend some money on Django-related products this Black Friday, please consider sponsoring Django itself. You can support it by donating to the charity that runs the framework, the Django Software Foundation.

Your money will go towards:

• Paying the Django Fellows, Mariusz and Natalia, who merge respond to tickets, merge code, and make releases.
• Helping organize DjangoCons in Africa, America, and Europe, and other events.
• Hosting costs of the documentation, source code, ticketing system, and CI system.
• A long tail of activities for keeping the framework alive and thriving.

You can sponsor Django on:

• GitHub Sponsors - adds to your GitHub bill.
• Threadless - grab some custom-printed merchandise.
• The Fundraising Page - bills from a credit card with Stripe. This page also links to other schemes: Amazon Smile and Benevity Workplace Giving.

If you’re working with Django professionally, please consider sponsoring a few dollars a month. But better yet, get your organization to sponsor to pay back for all the advantages that Django offers you.

At the time of writing, Django is 59% towards its annual funding goal:

Let’s fill up that heart!

If you start runserver but your terminal session gets closed, the server can continue running in the background. For example, this can happen when your IDE crashes. This situation is annoying because you can’t stop the backgrounded runserver with Ctrl-C.

You can find and stop the server with your operating system’s process list tool, such as Windows Task Manager or macOS’ Activity Monitor. But this is manual and slow.

It’s faster to use this shell one-liner, for Bash or Zsh, on macOS or Linux:

lsof -i :8000 -t | xargs -t kill

Replace “8000” with the port the server is running on.

lsof is pre-installed on macOS but you may need to install it on Linux. xargs is pre-installed everywhere.

Here’s a quick breakdown of what’s going on:

1. lsof is a tool for listing processes and the files they have open. Since operating systems treat internet addresses as files, lsof has the -i option to select processes by the internet addresses they have open. -i :8000 selects processes bound to port 8000:

   $ lsof -i :8000
   COMMAND     PID        USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
   python3.1 43282 adamjohnson    9u  IPv4 0xe72dc161b280a8a9      0t0  TCP localhost:irdmi (LISTEN)
   

2. Adding -t makes lsof outputs only process IDs:

   $ lsof -i :8000 -t
   43282
   

3. xargs runs a named command for each line of its input. In this case the named command is kill, which signals processes to stop, and the input is the list of process IDs from lsof.

4. xargs -t prints the commands it runs, so you can see if any process IDs get stopped:

   $ lsof -i :8000 -t | xargs -t kill
   kill 43282
   

   If there are no matching processes, there will be no output:

   $ lsof -i :8000 -t | xargs -t kill
   

News

Paolo Melchiorre - My Django active developers Sprints proposal 🌅

The story of how the proposal to organize sprints on the Django active developers was born and how I proposed it during the past DjangoCon US 2022 in San Diego to various components of the Django community.

paulox.net

PyPI has completed its first security audit

The results of PyPI's very first external security audit. The first post in a three-part series.

pypi.org

Updates to Django

Last week we had 6 pull requests merged into Django by 6 different contributors - including 1 first time contributor! Congratulations to Prakhar Parashari for having their first commit merged into Django - welcome on board!

The main priority is resolving release blockers for Django 5.0, as the release candidate is planned for November 20th!

DjangoCon Africa was also last week and it looked AMAZING! Congratulations to the incredible folks who make such a wonderful conference.

"Django is not a framework, it's a community" 💯💜

Django Newsletter

Wagtail CMS

Wagtail 5.2.1

Wagtail 5.2.1 is a bugfix release that addresses over a dozen fixes.

github.com

Looking for sponsorship – Accessibility checks for site administrators

We’re building state-of-the-art accessibility checks within the CMS, replacing expensive monitoring tools. Help us make this happen faster!

wagtail.org

Wagtail Community Spotlight: LB

Meet LB, a Wagtail core team member, one of our best frontend experts, and a champion for new contributors.

wagtail.org

Sponsored Link

Wagtail CMS Developer Training - Introductory Offer

Build your knowledge and skills in essential Django and Wagtail CMS practices. A two-part training programme, led by Senior Wagtail Developers that extends way beyond typical tutorials and documentation. Only 10 places available per course. Apply here: https://bit.ly/wagtail-developer-training-course

bit.ly

Articles

HTML First

A set of principles that aims to make building web software easier, faster, more inclusive, and more maintainable.

html-first.com

Adding Full Text Search to Your Django App with django-watson

Learn how to supercharge your Django app with full-text search using Django-Watson. Dive deep into Postgres magic and boost search functionality.

idiomaticprogrammers.com

Why is Django's (Unofficial) Mascot a Pony?

A brief history of the Django pony mascot and its magical powers.

learndjango.com

Django: Maybe disable PostgreSQL’s JIT to speed up many-joined queries

A write-up of an optimization by disabling a PostgreSQL feature called the JIT (Just-In-Time) compiler which was taking a long time for little benefit.

adamj.eu

Error Culture

A list of some common characteristics of Error Culture.

ryancheley.com

Sponsored Ad

Sick of performance issues? Enter Scout's APM tool for Python apps. Easily pinpoint and fix slowdowns with intelligent tracing logic. Optimize performance hassle-free, delighting your users. Try us out for free!

ter.li

Videos

A simple pagination example in Django

Short and to-the-point demonstration of using the default paginator functionality in django.core.

youtube.com

Basic Search Engine Optimizations Tutorial

Learn how to do basic SEO such as correctly formatted HTML, image alt attributes, naming images, setting a robots file, and a sitemap.

youtube.com

Let's Code an AI Search Engine with LLM Embeddings, Django, and pgvector

How to build a prototype application that enables searching for job descriptions using an unstructured, English-language description of a job seeker.

youtube.com

Django News Jobs

Full Stack Python Developer at Loginsoft 🆕

Senior Python Engineer at Loginsoft

Software Engineer - Ubuntu Systems Management at Canonical

Senior Python/Django Engineer at Apexive

Senior Python/Django Software Engineer at Brivo Systems

Django Newsletter

Projects

nikolaik/docker-python-nodejs

Python with Node.js Docker image.

github.com

danihodovic/django-webhook

Outgoing Django webhooks triggered on model changes.

github.com

Django Forum

Change reusable apps naming recommendation

Adam Johnson raised the question on the Forum as well as on Mastodon and Twitter, proposing a chance to the docs default naming recommendation, such as django_prefix to prevent name clashes.

djangoproject.com

Checklists for upgrading packages and upgrading Python - Public Notes

Public notes on a step-by-step guide to upgrading packages and Python itself, especially in advance of Django 5.0's release.

djangoproject.com

This RSS feed is published on https://django-news.com/. You can also subscribe via email.

ChatGLM3 series: Open Bilingual Chat LLMs | 开源双语对话语言模型

ChatGLM3

🤗 HF Repo • 🤖 ModelScope • 🐦 Twitter • 📃 [GLM@ACL 22] [GitHub] • 📃 [GLM-130B@ICLR 23] [GitHub]

👋 加入我们的 Slack 和 微信

📍在 chatglm.cn 体验更大规模的 ChatGLM 模型。

Read this in English.

📔 更为详细的使用信息，可以参考：ChatGLM3技术文档

介绍

ChatGLM3 是智谱AI和清华大学 KEG 实验室联合发布的新一代对话预训练模型。ChatGLM3-6B 是 ChatGLM3 系列中的开源模型，在保留了前两代模型对话流畅、部署门槛低等众多优秀特性的基础上，ChatGLM3-6B 引入了如下特性：

1. 更强大的基础模型： ChatGLM3-6B 的基础模型 ChatGLM3-6B-Base 采用了更多样的训练数据、更充分的训练步数和更合理的训练策略。在语义、数学、推理、代码、知识等不同角度的数据集上测评显示，ChatGLM3-6B-Base 具有在 10B 以下的基础模型中最强的性能。
2. 更完整的功能支持： ChatGLM3-6B 采用了全新设计的 Prompt 格式，除正常的多轮对话外。同时原生支持工具调用（Function Call）、代码执行（Code Interpreter）和 Agent 任务等复杂场景。
3. 更全面的开源序列： 除了对话模型 ChatGLM3-6B 外，还开源了基础模型 ChatGLM3-6B-Base、长文本对话模型 ChatGLM3-6B-32K。以上所有权重对学术研究完全开放，在填写问卷进行登记后亦允许免费商业使用。

ChatGLM3 开源模型旨在与开源社区一起推动大模型技术发展，恳请开发者和大家遵守开源协议，勿将开源模型和代码及基于开源项目产生的衍生物用于任何可能给国家和社会带来危害的用途以及用于任何未经过安全评估和备案的服务。目前，本项目团队未基于 ChatGLM3 开源模型开发任何应用，包括网页端、安卓、苹果 iOS 及 Windows App 等应用。

尽管模型在训练的各个阶段都尽力确保数据的合规性和准确性，但由于 ChatGLM3-6B 模型规模较小，且模型受概率随机性因素影响，无法保证输出内容的准确。同时模型的输出容易被用户的输入误导。本项目不承担开源模型和代码导致的数据安全、舆情风险或发生任何模型被误导、滥用、传播、不当利用而产生的风险和责任。

模型列表

友情链接

对 ChatGLM3 进行加速的开源项目：

• chatglm.cpp: 类似 llama.cpp 的量化加速推理方案，实现笔记本上实时对话
• ChatGLM3-TPU: 采用TPU加速推理方案，在算能端侧芯片BM1684X（16T@FP16，内存16G）上实时运行约7.5 token/s

评测结果

典型任务

我们选取了 8 个中英文典型数据集，在 ChatGLM3-6B (base) 版本上进行了性能测试。

Best Baseline 指的是截止 2023年10月27日、模型参数在 10B 以下、在对应数据集上表现最好的预训练模型，不包括只针对某一项任务训练而未保持通用能力的模型。

对 ChatGLM3-6B-Base 的测试中，BBH 采用 3-shot 测试，需要推理的 GSM8K、MATH 采用 0-shot CoT 测试，MBPP 采用 0-shot 生成后运行测例计算 Pass@1 ，其他选择题类型数据集均采用 0-shot 测试。

我们在多个长文本应用场景下对 ChatGLM3-6B-32K 进行了人工评估测试。与二代模型相比，其效果平均提升了超过 50%。在论文阅读、文档摘要和财报分析等应用中，这种提升尤为显著。此外，我们还在 LongBench 评测集上对模型进行了测试，具体结果如下表所示

使用方式

环境安装

首先需要下载本仓库：

git clone https://github.com/THUDM/ChatGLM3
cd ChatGLM3

然后使用 pip 安装依赖：

pip install -r requirements.txt

• transformers 库版本应该 4.30.2 以及以上的版本 ，torch 库版本应为 2.0 及以上的版本，以获得最佳的推理性能。
• 为了保证 torch 的版本正确，请严格按照 官方文档 的说明安装。
• gradio 库版本应该为 3.x 的版本。

综合 Demo

我们提供了一个集成以下三种功能的综合 Demo，运行方法请参考 综合 Demo

• Chat: 对话模式，在此模式下可以与模型进行对话。
• Tool: 工具模式，模型除了对话外，还可以通过工具进行其他操作。
• Code Interpreter: 代码解释器模式，模型可以在一个 Jupyter 环境中执行代码并获取结果，以完成复杂任务。

代码调用

可以通过如下代码调用 ChatGLM 模型来生成对话：

>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True)
>>> model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True, device='cuda')
>>> model = model.eval()
>>> response, history = model.chat(tokenizer, "你好", history=[])
>>> print(response)
你好👋!我是人工智能助手 ChatGLM3-6B,很高兴见到你,欢迎问我任何问题。
>>> response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
>>> print(response)
晚上睡不着可能会让你感到焦虑或不舒服,但以下是一些可以帮助你入睡的方法:

1. 制定规律的睡眠时间表:保持规律的睡眠时间表可以帮助你建立健康的睡眠习惯,使你更容易入睡。尽量在每天的相同时间上床,并在同一时间起床。
2. 创造一个舒适的睡眠环境:确保睡眠环境舒适,安静,黑暗且温度适宜。可以使用舒适的床上用品,并保持房间通风。
3. 放松身心:在睡前做些放松的活动,例如泡个热水澡,听些轻柔的音乐,阅读一些有趣的书籍等,有助于缓解紧张和焦虑,使你更容易入睡。
4. 避免饮用含有咖啡因的饮料:咖啡因是一种刺激性物质,会影响你的睡眠质量。尽量避免在睡前饮用含有咖啡因的饮料,例如咖啡,茶和可乐。
5. 避免在床上做与睡眠无关的事情:在床上做些与睡眠无关的事情,例如看电影,玩游戏或工作等,可能会干扰你的睡眠。
6. 尝试呼吸技巧:深呼吸是一种放松技巧,可以帮助你缓解紧张和焦虑,使你更容易入睡。试着慢慢吸气,保持几秒钟,然后缓慢呼气。

如果这些方法无法帮助你入睡,你可以考虑咨询医生或睡眠专家,寻求进一步的建议。

从本地加载模型

以上代码会由 transformers 自动下载模型实现和参数。完整的模型实现在 Hugging Face Hub。如果你的网络环境较差，下载模型参数可能会花费较长时间甚至失败。此时可以先将模型下载到本地，然后从本地加载。

从 Hugging Face Hub 下载模型需要先安装Git LFS，然后运行

git clone https://huggingface.co/THUDM/chatglm3-6b

如果从你从 HuggingFace 下载比较慢，也可以从 ModelScope 中下载。

模型微调

请参考对话模型微调 ChatGLM3-6B 微调示例,或基座模型微调 ChatGLM3-6B-base 微调示例。 请注意，不同的微调脚本对应的模型并不相同，请根据需要选择对应的模型。

网页版对话 Demo

可以通过以下命令启动基于 Gradio 的网页版 demo：

python web_demo.py

可以通过以下命令启动基于 Streamlit 的网页版 demo：

streamlit run web_demo2.py

网页版 demo 会运行一个 Web Server，并输出地址。在浏览器中打开输出的地址即可使用。 经测试，基于 Streamlit 的网页版 Demo 会更流畅。

命令行对话 Demo

运行仓库中 cli_demo.py：

python cli_demo.py

程序会在命令行中进行交互式的对话，在命令行中输入指示并回车即可生成回复，输入 clear 可以清空对话历史，输入 stop 终止程序。

LangChain Demo

请参考 基于 LangChain 的工具调用 Demo。

工具调用

关于工具调用的方法请参考 工具调用。

API 部署

感谢 @xusenlinzy 实现了 OpenAI 格式的流式 API 部署，可以作为任意基于 ChatGPT 的应用的后端，比如 ChatGPT-Next-Web。可以通过运行仓库中的openai_api.py 进行部署：

cd openai_api_demo
python openai_api.py

同时，我们也书写了一个示例代码，用来测试API调用的性能。可以通过运行仓库中的openai_api_request.py 进行测试

• 使用Curl进行测试

curl -X POST "http://127.0.0.1:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d "{\"model\": \"chatglm3-6b\", \"messages\": [{\"role\": \"system\", \"content\": \"You are ChatGLM3, a large language model trained by Zhipu.AI. Follow the user's instructions carefully. Respond using markdown.\"}, {\"role\": \"user\", \"content\": \"你好，给我讲一个故事，大概100字\"}], \"stream\": false, \"max_tokens\": 100, \"temperature\": 0.8, \"top_p\": 0.8}"

• 使用Python进行测试

cd openai_api_demo
python openai_api_request.py

如果测试成功，则模型应该返回一段故事。

低成本部署

模型量化

默认情况下，模型以 FP16 精度加载，运行上述代码需要大概 13GB 显存。如果你的 GPU 显存有限，可以尝试以量化方式加载模型，使用方法如下：

model = AutoModel.from_pretrained("THUDM/chatglm3-6b",trust_remote_code=True).quantize(4).cuda()

模型量化会带来一定的性能损失，经过测试，ChatGLM3-6B 在 4-bit 量化下仍然能够进行自然流畅的生成。

CPU 部署

如果你没有 GPU 硬件的话，也可以在 CPU 上进行推理，但是推理速度会更慢。使用方法如下（需要大概 32GB 内存）

model = AutoModel.from_pretrained("THUDM/chatglm3-6b", trust_remote_code=True).float()

Mac 部署

对于搭载了 Apple Silicon 或者 AMD GPU 的 Mac，可以使用 MPS 后端来在 GPU 上运行 ChatGLM3-6B。需要参考 Apple 的 官方说明 安装 PyTorch-Nightly（正确的版本号应该是2.x.x.dev2023xxxx，而不是 2.x.x）。

目前在 MacOS 上只支持从本地加载模型。将代码中的模型加载改为从本地加载，并使用 mps 后端：

model = AutoModel.from_pretrained("your local path", trust_remote_code=True).to('mps')

加载半精度的 ChatGLM3-6B 模型需要大概 13GB 内存。内存较小的机器（比如 16GB 内存的 MacBook Pro），在空余内存不足的情况下会使用硬盘上的虚拟内存，导致推理速度严重变慢。

多卡部署

如果你有多张 GPU，但是每张 GPU 的显存大小都不足以容纳完整的模型，那么可以将模型切分在多张GPU上。首先安装 accelerate: pip install accelerate，然后通过如下方法加载模型：

from utils import load_model_on_gpus

model = load_model_on_gpus("THUDM/chatglm3-6b", num_gpus=2)

即可将模型部署到两张 GPU 上进行推理。你可以将 num_gpus 改为你希望使用的 GPU 数。默认是均匀切分的，你也可以传入 device_map 参数来自己指定。

引用

如果你觉得我们的工作有帮助的话，请考虑引用下列论文。

@article{zeng2022glm,
  title={Glm-130b: An open bilingual pre-trained model},
  author={Zeng, Aohan and Liu, Xiao and Du, Zhengxiao and Wang, Zihan and Lai, Hanyu and Ding, Ming and Yang, Zhuoyi and Xu, Yifan and Zheng, Wendi and Xia, Xiao and others},
  journal={arXiv preprint arXiv:2210.02414},
  year={2022}
}

@inproceedings{du2022glm,
  title={GLM: General Language Model Pretraining with Autoregressive Blank Infilling},
  author={Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie},
  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={320--335},
  year={2022}
}

Deep Learning System core principles introduction.

Deep Learning System & AI System

文字课程内容正在一节节补充更新，尽可能抽空继续更新正在 AISys ，希望您多多鼓励和参与进来！！！

文字课程开源在 AISys，系列视频托管B站和油管，PPT开源在github，欢迎取用！！！

非常希望您也参与到这个开源项目中，B站给ZOMI留言哦！

欢迎大家使用的过程中发现bug或者勘误直接提交代码PR到开源社区哦！

项目背景

这个开源项目英文名字叫做 Deep Learning System 或者 AI System(AISys)，中文名字叫做 深度学习系统 或者 AI系统。

本开源项目主要是跟大家一起探讨和学习人工智能、深度学习的系统设计，而整个系统是围绕着 ZOMI 在工作当中所积累、梳理、构建 AI 系统全栈的内容。希望跟所有关注 AI 开源项目的好朋友一起探讨研究，共同促进学习讨论。

课程内容大纲

课程主要包括以下六大模块：

第一部分，AI基础知识和AI系统的全栈概述的AI系统概述，以及深度学习系统的系统性设计和方法论，主要是整体了解AI训练和推理全栈的体系结构内容。

第二部分，硬核篇介绍AI芯片概况，这里就很硬核了，从芯片基础到AI芯片的范围都会涉及，芯片设计需要考虑上面AI框架的前端、后端编译，而不是停留在天天喊着吊打英伟达，被现实打趴。

第三部分，进阶篇介绍AI编译器原理，将站在系统设计的角度，思考在设计现代机器学习系统中需要考虑的编译器问题，特别是中间表达乃至后端优化。

第四部分，实际应用推理系统与引擎，讲了太多原理身体太虚容易消化不良，还是得回归到业务本质，让行业、企业能够真正应用起来，而推理系统涉及一些核心算法和注意的事情也分享下。

第五部分，介绍AI框架核心技术，首先介绍任何一个AI框架都离不开的自动微分，通过自动微分功能后就会产生表示神经网络的图和算子，然后介绍AI框架前端的优化，还有最近很火的大模型分布式训练在AI框架中的关键技术。

第六部分，汇总篇介绍大模型与AI系统，大模型是基于AI集群的全栈软硬件性能优化，通过最小的每一块AI芯片组成的AI集群，编译器使能到上层的AI框架，训练过程需要分布式并行、集群通信等算法支持，而且在大模型领域最近持续演进如智能体等新技术。

课程设立目的

本课程主要为本科生高年级、硕博研究生、AI系统从业者设计，帮助大家：

1. 完整了解AI的计算机系统架构，并通过实际问题和案例，来了解AI完整生命周期下的系统设计。

2. 介绍前沿系统架构和AI相结合的研究工作，了解主流框架、平台和工具来了解AI系统。

先修课程: C++/Python，计算机体系结构，人工智能基础

课程部分

一. AI系统概述

二. AI芯片体系结构

三. AI编译原理

四. AI推理系统

五. AI框架核心技术

六. 大模型训练

知识清单

贡献者

备注

这个仓已经到达疯狂的8G啦（ZOMI把所有制作过程、高清图片都原封不动提供），如果你要git clone会非常的慢，因此建议优先到 Releases · chenzomi12/DeepLearningSystem 来下载你需要的内容。

OpenAgents: An Open Platform for Language Agents in the Wild

OpenAgents: An Open Platform for Language Agents in the Wild

English • 中文 • 日本語 • 한국어

Current language agent frameworks aim to facilitate the construction of proof-of-concept language agents while neglecting the non-expert user access to agents and paying little attention to application-level designs. We built OpenAgents, an open platform for using and hosting language agents in the wild of everyday life.

We have now implemented three agents in OpenAgents, and we host them on demo for free use!

1. Data Agent for data analysis with Python/SQL and data tools;
2. Plugins Agent with 200+ daily tools;
3. Web Agent for autonomous web browsing.

OpenAgents can analyze data, call plugins, control your browser as ChatGPT Plus, but with OPEN Code for

1. Easy deployment
2. Full stack
3. Chat Web UI
4. Agent methods
5. …

OpenAgents enables general users to interact with agent functionalities through a web UI optimized for swift responses and common failures, while offering developers and researchers a seamless deployment experience on local setups, providing a foundation for crafting innovative language agents and facilitating real-world evaluations. We elucidate both the challenges and promising opportunities, aspiring to set a foundation for future research and development of real-world language agents.

We welcome contributions from everyone. Before you start, please take a moment to read our CONTRIBUTING.md guidelines for issues and PRs. This will help ensure that your contribution process is smooth and consistent with the project’s standards.

🔫 Trouble Shooting

Join our Discord for help if you encounter any issues with our online demo or local deployment. Alternatively, create an issue if you have trouble with features or code.

🔥 News

• [2023, Oct 26] We've reached 3,000 users! 🚀 Heartfelt thanks to all our users and contributors! 🙏 Please bear with us as we navigate through unexpectedly high traffic on our servers. We appreciate your patience and ready to assist as soon as possible!
• [2023, Oct 18] Try out our Lemur, the SOTA and open-sourced foundation models for language agents, matching ChatGPT on 15 agent tasks!
• [2023, Oct 17] Check out the OpenAgents paper here!
• [2023, Oct 13] We've released OpenAgents platform code for all three agents, server backend and frontend! Feel free to setup your localhost one, and play with OpenAgents!
• [2023, Aug 17] Our platform has officially reached 500 users! 🚀
• [2023, Aug 8] We've released OpenAgents demos, including Data, Plugins, and Web agents! Check tutorials and use cases!

🥑 OpenAgents

We built three real-world agents with chat-based web UI as demonstration(check OpenAgents demos). Here is a brief overview of our OpenAgents platform. You can find more details about concepts & designs in our documentation.

Data Agent

Data Agent is a comprehensive toolkit designed for efficient data operations. It provides capabilities to:

• 🔍 Search: Quickly locate the data you need.
• 🛠️ Handle: Streamline data acquisition and processing.
• 🔄 Manipulate: Modify data to suit specific requirements.
• 📊 Visualize: Represent data in a clear and insightful manner.

With its proficiency in writing and executing code, Data Agent simplifies a wide range of data-centric tasks. Discover its potential through various use cases.

Plugins Agent

Plugins Agent seamlessly integrates with over 200 third-party plugins, each handpicked to enrich various facets of your daily life. With these plugins at its disposal, the agent empowers you to tackle a wide range of tasks and activities more efficiently.

🔌 Sample Plugins Include:

• 🛍️ Shopping: Klarna Shopping
• ☁️ Weather: XWeather
• 🔬 Scientific Exploration: Wolfram Alpha

Combined Plugin Usage

Harness the power of synergy! Plugins Agent supports the concurrent use of multiple plugins. Planning a trip? Seamlessly integrate functionalities from Klook, Currency converter, and WeatherViz.

Auto Plugin Selection

Simplify your choices with our Auto Plugin Selection feature. Let the agent intuitively search and suggest the best plugins tailored to your needs.

Dive into more use cases to see Plugins Agent in action.

Web Agent

Web Agent harnesses the power of a Chrome extension to navigate and explore websites automatically. This agent streamlines the web browsing experience, making it easier to find relevant information, access desired resources, and so on.

Examples of What Web Agent Can Do:

• 📍 Google Maps Navigation: Planning a journey? Simply relay your starting point and destination to Web Agent. It will navigate Google Maps for you and present the best routes.
• 🐦 Twitter Postings: Engage in a conversation with Web Agent and wish to share something on Twitter? Mention the content, and Web Agent will handle your tweet effortlessly.
• 📝 Google Form Assistance: Need to sign up for an event or activity? Share the Google Form link and the required details. Web Agent will populate the form for you.

Witness the full potential of Web Agent in these use cases.

💻 Localhost Deployment

We've released the OpenAgents platform code. Feel free to deploy on your own localhost!

Here is a brief system design of OpenAgents:

From Source Code

Please check the following folders and README files to set up & localhost:

1. Backend: the flask backend to host our three agents.
2. Frontend: the frontend UI and WeBot Chrome extension.

p.s.: We have renamed some arguments in code for better readability. If you have pulled the code before 10/26/2023, just a reminder that if you want to you pull the latest code, previous local chat history will be lost because of different key names.

Docker

Please follow the following steps to use the docker-compose to deploy the OpenAgents platform.

Note: the docker is under development, so there may be functions not working properly as expected and slower response. Please feel free to open an issue if you have any questions. If you want a more robust version, currently we recommend you to deploy from source code.

1. If you want to use kaggle's dataset, you must modify the information in the Dockerfile to your correct information.

ENV KAGGLE_USER="" \
    KAGGLE_KEY="" 

1. If you are not running locally, you need to modify the accessible IP to the backend service in frontend/Dockerfile

ENV NEXT_PUBLIC_BACKEND_ENDPOINT http://x.x.x.x:8000

1. Run the docker compose build command in the project root directory.
2. If you use openai unofficial services, such as FastChat, you need to modify OPENAI_API_BASE in docker-compose.yml;otherwise you only to put your OPENAI_API_KEY in docker-compose.yml
3. After completing the above steps, you can run docker compose up -d to start all services.

📜 Tutorial on Extending OpenAgents

Code Structure

Before we dive into how to extend OpenAgents, let's first take a glance at the code structure for better understanding. The code structure of OpenAgents is shown below:

├── backend  # backend code
│   ├── README.md  # backend README for setup
│   ├── api  # RESTful APIs, to be called by the frontend
│   ├── app.py  # main flask app
│   ├── display_streaming.py  # rendering the streaming response
│   ├── kernel_publisher.py  # queue for code execution
│   ├── main.py  # main entry for the backend
│   ├── memory.py  # memory(storage) for the backend
│   ├── schemas.py  # constant definitions
│   ├── setup_script.sh  # one-click setup script for the backend
│   ├── static  # static files, e.g., cache and figs
│   └── utils  # utilities
├── frontend  # frontend code
│   ├── README.md  # frontend README for setup
│   ├── components  # React components
│   ├── hooks  # custom React hooks
│   ├── icons  # icon assets
│   ├── next-env.d.ts  # TypeScript declarations for Next.js environment variables
│   ├── next-i18next.config.js  # configuration settings for internationalization
│   ├── next.config.js  # configuration settings for Next.js
│   ├── package-lock.json  # generated by npm that describes the exact dependency tree
│   ├── package.json  # manifest file that describes the dependencies
│   ├── pages  # Next.js pages
│   ├── postcss.config.js  # configuration settings for PostCSS
│   ├── prettier.config.js  # configuration settings for Prettier
│   ├── public  # static assets
│   ├── styles  # global styles
│   ├── tailwind.config.js  # configuration settings for Tailwind CSS
│   ├── tsconfig.json  # configuration settings for TypeScript
│   ├── types  # type declarations
│   ├── utils  # utilities or helper functions
│   ├── vitest.config.ts  # configuration settings for ViTest
│   └── webot_extension.zip  # Chrome extension for Web Agent
└── real_agents  # language agents
    ├── adapters  # shared components for the three agents to adapt to the backend
    ├── data_agent  # data agent implementation
    ├── plugins_agent  # plugins agent implementation
    └── web_agent  # web agent implementation

As shown, backend/ and frontend/ are self-contained and directly deployable (see here). It does not mean they cannot be modified. Instead, you can just follow the conventional client-server architecture to extend the backend and frontend as you wish. For real_agents/, we design it to be "one agent, one folder", so that it is easy to extend a new agent. It is worth noting that we name it "real agents" because not only the conceptual language agent part is included, but also the gaps between the language agent and the backend are filled here. For example, adapters/ contains the shared adapter components like stream parsing, data model, memory, callbacks, etc. We refer interested readers to our paper for concepts and implementation designs. And we thank LangChain as we base on their code to build real agents.

Extend A New Agent

If you want to build a new agent beyond the three agents we provide, you can follow the steps below:

• Refer to the real_agents/ folder to see how previous agents are implemented, and create a new folder for your agent.
• Implement the agent logic in the new folder. Use the components under adapters/ folder when needed.
• Add a chat_<new_agent>.py file under backend/api/ folder to define the chat API for the new agent, which will be called by the frontend.
• Register new constants in backend/schemas.py if needed.
• Add a new OpenAgentID in frontend/types/agent.ts and the corresponding API in frontend/utils/app/api.ts and frontend/utils/app/const.ts.
• Implement the agent UI in frontend/components/Chat/Chat.tsx and frontend/components/Chat/ChatMessage.tsx when needed.
• Run localhost script and test your new agent.

Note, if new data types, i.e., beyond text, image, table, and json, you may need to implement its parsing logic in backend/display_streaming.py and add new data models.

Extend A New LLM

Extending a new LLM as the agent backbone is simpler if the LLM is already hosted and can be called via API. Just register your new model in backend/api/language_model.py. Just refer to lemur-chat as a template.

If the LLM is not hosted yet, we have a tutorial on how to deploy a new LLM and expose it as an API here (LLM hosting to todo).

Extend A New Tool

If you want to extend a new tool in Plugins Agent, you can follow the steps below:

• Refer to the already built plugins in real_agents/plugins_agent/plugins/, and create a new folder for your tool.
• Implement the tool logic in the new folder. Note that ai-plugin.json and openapi.yaml are essential for the tool to be recognized(which can be generated by LLM following the others rather than manually written). And the paths/ are for the actual tool API call.
• Register the new tool name in real_agents/plugins_agent/plugins/plugin_names.py.

👏 Contributing

Thanks to open-sourced communities’ efforts, such as LangChain, ChatBot UI, Taxy.ai browser extension and others. We are able to build our interface prototype much more conveniently and efficiently.

We welcome contributions and suggestions, together we move further to make it better! Following the steps will be well-received:

• Step1: Post an issue if you want to add any additional features, enhancements, or encounter any problems during your experience. We would appreciate it if you follow the issue template. The issues will be discussed and assigned there.
• Step2: Whenever an issue is assigned, you can contribute by creating a Pull Request by following the PR template. You can also claim for any open issues. Together we can make OpenAgents better!
• Step3: PR will be merged or iterated after review and discussion. Thanks for your contribution!

Before you start, we highly recommend taking a moment to check here before contribution.

📖 Documentation

Please check here for full documentation, which will be updated to stay on pace with the demo changes and the code release.

🧙‍Participants

Tech Lead

Co-Lead Contributors

Key Contributors

Valuable Contributors

Acknowledgments (beyond code)

Heartfelt appreciation to Ziyi Huang, Roxy Rong, Haotian Li, Xingbo Wang, Jansen Wong, and Chen Henry Wu for their valuable contributions to the OpenAgents. Their expertise and insights were instrumental in bringing this project to fruition!

Open Source Contributors

Thanks to all the contributors!

Citation

If you find our work helpful, please cite us:

@misc{OpenAgents,
      title={OpenAgents: An Open Platform for Language Agents in the Wild}, 
      author={Tianbao Xie and Fan Zhou and Zhoujun Cheng and Peng Shi and Luoxuan Weng and Yitao Liu and Toh Jing Hua and Junning Zhao and Qian Liu and Che Liu and Leo Z. Liu and Yiheng Xu and Hongjin Su and Dongchan Shin and Caiming Xiong and Tao Yu},
      year={2023},
      eprint={2310.10634},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Acknowledgments

We would like to thank Google Research, Amazon AWS, and Salesforce Research for their research gift funds to this open-source effort!

⭐️ Star History

A ⭐️ to OpenAgents is to make it shine brighter and benefit more people.

Deliver web apps with confidence 🚀

Angular - The modern web developer's platform

Angular is a development platform for building mobile and desktop web applications
using TypeScript/JavaScript and other languages.

angular.dev

Contributing Guidelines · Submit an Issue · Blog

Documentation

Get started with Angular, learn the fundamentals and explore advanced topics on our documentation website.

• Getting Started
• Architecture
• Components and Templates
• Forms
• API

Advanced

• Angular Elements
• Server Side Rendering
• Schematics
• Lazy Loading
• Animations

Development Setup

Prerequisites

• Install Node.js which includes Node Package Manager

Setting Up a Project

Install the Angular CLI globally:

npm install -g @angular/cli

Create workspace:

ng new [PROJECT NAME]

Run the application:

cd [PROJECT NAME]
ng serve

Angular is cross-platform, fast, scalable, has incredible tooling, and is loved by millions.

Quickstart

Get started in 5 minutes.

Ecosystem

• Angular Command Line (CLI)
• Angular Material

Changelog

Learn about the latest improvements.

Upgrading

Check out our upgrade guide to find out the best way to upgrade your project.

Contributing

Contributing Guidelines

Read through our contributing guidelines to learn about our submission process, coding rules, and more.

Want to Help?

Want to report a bug, contribute some code, or improve the documentation? Excellent! Read up on our guidelines for contributing and then check out one of our issues labeled as help wanted or good first issue.

Code of Conduct

Help us keep Angular open and inclusive. Please read and follow our Code of Conduct.

Community

Join the conversation and help the community.

• X (formerly Twitter)
• Discord
• Gitter
• YouTube
• StackOverflow
• Find a Local Meetup

Love Angular? Give our repo a star ⭐ ⬆.

The Startup CTO's Handbook, a book covering leadership, management and technical topics for leaders of software engineering teams

NOTE: As of October 2023 I'm still working on porting the book content into markdown. Everything is in there (via a .doc to .md auto-converter) but the formatting is all over the place and needs a lot of cleanup still, apologies for my mess in the interim!

The Book

You can view the latest content of the book in markdown here

You can buy the book on amazon

(Coming Soon) Link of the latest version of the markdown rendered to PDF

The original manuscript (now outdated) can be found as a google doc

Welcome

Hi, thanks for checking out the Startup CTO's Handbook! This repository has the latest version of the content of the book. You're welcome and encouraged to contribute issues or pull requests for additions / changes / suggestions / criticisms to be included in future editions. Please feel free to add your name to ACKNOWLEDGEMENTS if you do so.

The Author

Linkedin / Website / Email

Licensing

See the LICENSE file, but tl;dr - you're welcome to make copies, changes, redistribute etc. so long as you're not reselling, you keep my name/attribution attached, and you keep future versions open under a similar/the same license.

A Library for Advanced Deep Time Series Models.

Time Series Library (TSlib)

TSlib is an open-source library for deep learning researchers, especially for deep time series analysis.

We provide a neat code base to evaluate advanced deep time series models or develop your model, which covers five mainstream tasks: long- and short-term forecasting, imputation, anomaly detection, and classification.

🚩News (2023.10) We add an implementation to iTransformer, which is the state-of-the-art model for long-term forecasting. The official code and complete scripts of iTransformer can be found here.

🚩News (2023.09) We added a detailed tutorial for TimesNet and this library, which is quite friendly to beginners of deep time series analysis.

Leaderboard for Time Series Analysis

Till October 2023, the top three models for five different tasks are:

Note: We will keep updating this leaderboard. If you have proposed advanced and awesome models, you can send us your paper/code link or raise a pull request. We will add them to this repo and update the leaderboard as soon as possible.

Compared models of this leaderboard. ☑ means that their codes have already been included in this repo.

• iTransformer - iTransformer: Inverted Transformers Are Effective for Time Series Forecasting [arXiv 2023] [Code].
• PatchTST - A Time Series is Worth 64 Words: Long-term Forecasting with Transformers [ICLR 2023] [Code].
• TimesNet - TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis [ICLR 2023] [Code] [ICLR 2024] [Code].
• DLinear - Are Transformers Effective for Time Series Forecasting? [AAAI 2023] [Code].
• LightTS - Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures [arXiv 2022] [Code].
• ETSformer - ETSformer: Exponential Smoothing Transformers for Time-series Forecasting [arXiv 2022] [Code].
• Non-stationary Transformer - Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting [NeurIPS 2022] [Code].
• FEDformer - FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting [ICML 2022] [Code].
• Pyraformer - Pyraformer: Low-complexity Pyramidal Attention for Long-range Time Series Modeling and Forecasting [ICLR 2022] [Code].
• Autoformer - Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting [NeurIPS 2021] [Code].
• Informer - Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting [AAAI 2021] [Code].
• Reformer - Reformer: The Efficient Transformer [ICLR 2020] [Code].
• Transformer - Attention is All You Need [NeurIPS 2017] [Code].

See our latest paper [TimesNet] for the comprehensive benchmark. We will release a real-time updated online version soon.

Newly added baselines. We will add them to the leaderboard after a comprehensive evaluation.

• Koopa - Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors [NeurIPS 2023] [Code].
• FiLM - FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting [NeurIPS 2022][Code].
• MICN - MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting [ICLR 2023][Code].
• Crossformer - Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting [ICLR 2023][Code].

Usage

1. Install Python 3.8. For convenience, execute the following command.

pip install -r requirements.txt

1. Prepare Data. You can obtain the well pre-processed datasets from [Google Drive] or [Baidu Drive], Then place the downloaded data in the folder./dataset. Here is a summary of supported datasets.

1. Train and evaluate model. We provide the experiment scripts for all benchmarks under the folder ./scripts/. You can reproduce the experiment results as the following examples:

# long-term forecast
bash ./scripts/long_term_forecast/ETT_script/TimesNet_ETTh1.sh
# short-term forecast
bash ./scripts/short_term_forecast/TimesNet_M4.sh
# imputation
bash ./scripts/imputation/ETT_script/TimesNet_ETTh1.sh
# anomaly detection
bash ./scripts/anomaly_detection/PSM/TimesNet.sh
# classification
bash ./scripts/classification/TimesNet.sh

1. Develop your own model.

• Add the model file to the folder ./models. You can follow the ./models/Transformer.py.
• Include the newly added model in the Exp_Basic.model_dict of ./exp/exp_basic.py.
• Create the corresponding scripts under the folder ./scripts.

Citation

If you find this repo useful, please cite our paper.

@inproceedings{wu2023timesnet,
  title={TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis},
  author={Haixu Wu and Tengge Hu and Yong Liu and Hang Zhou and Jianmin Wang and Mingsheng Long},
  booktitle={International Conference on Learning Representations},
  year={2023},
}

Contact

If you have any questions or suggestions, feel free to contact:

• Haixu Wu (wuhx23@mails.tsinghua.edu.cn)
• Tengge Hu (htg21@mails.tsinghua.edu.cn)
• Haoran Zhang (z-hr20@mails.tsinghua.edu.cn)
• Jiawei Guo (guo-jw21@mails.tsinghua.edu.cn)

Or describe it in Issues.

Acknowledgement

This project is supported by the National Key R&D Program of China (2021YFB1715200).

This library is constructed based on the following repos:

• Forecasting: https://github.com/thuml/Autoformer.

• Anomaly Detection: https://github.com/thuml/Anomaly-Transformer.

• Classification: https://github.com/thuml/Flowformer.

All the experiment datasets are public, and we obtain them from the following links:

• Long-term Forecasting and Imputation: https://github.com/thuml/Autoformer.

• Short-term Forecasting: https://github.com/ServiceNow/N-BEATS.

• Anomaly Detection: https://github.com/thuml/Anomaly-Transformer.

• Classification: https://www.timeseriesclassification.com/.

All Thanks To Our Contributors

⚓ A collection of JavaScript tools written in Rust.

⚓ Oxc

The Oxidation Compiler is creating a suite of high-performance tools for JavaScript and TypeScript.

Oxc is building a parser, linter, formatter, transpiler, minifier, resolver ... all written in Rust.

💡 Philosophy

This project shares the same philosophies as Biome and Ruff.

1. JavaScript tooling could be rewritten in a more performant language.
2. An integrated toolchain can tap into efficiencies that are not available to a disparate set of tools.

⚡️ Quick Start

The linter is ready to catch mistakes for you. It comes with over 60 default rules and no configuration is required.

To start using, install oxlint or via npx:

npx oxlint@latest

To give you an idea of its capabilities, here is an example from the vscode repository, which finishes linting 4000+ files in 0.5 seconds.

⚡️ Performance

• The parser aim to be the fastest Rust-based ready-for-production parser.
• The linter is more than 50 times faster than ESLint, and scales with the number of CPU cores.

⌨️ Programming Usage

Rust

Individual crates are published, you may use them to build your own JavaScript tools.

• The umbrella crate oxc exports all public crates from this repository.
• The AST and parser crates oxc_ast and oxc_parser are production ready.
• See crates/*/examples for example usage.

While Rust has gained a reputation for its comparatively slower compilation speed, we have dedicated significant effort to fine-tune the Rust compilation speed. Our aim is to minimize any impact on your development workflow, ensuring that developing your own Oxc based tools remains a smooth and efficient experience.

This is demonstrated by our CI runs, where warm runs complete in 5 minutes.

Node.js

• You may use the parser via napi: oxc-parser

🎯 Tools

• AST and Parser
• Linter
• Resolver
• Minifier
• Formatter
• Transpiler
• Ezno Type Checker

🔸 AST and Parser

Oxc maintains its own AST and parser, which is by far the fastest and most conformant JavaScript and TypeScript (including JSX and TSX) parser written in Rust.

As the parser often represents a key performance bottleneck in JavaScript tooling, any minor improvements can have a cascading effect on our downstream tools. By developing our parser, we have the opportunity to explore and implement well-researched performance techniques.

While many existing JavaScript tools rely on estree as their AST specification, a notable drawback is its abundance of ambiguous nodes. This ambiguity often leads to confusion during development with estree.

The Oxc AST differs slightly from the estree AST by removing ambiguous nodes and introducing distinct types. For example, instead of using a generic estree Identifier, the Oxc AST provides specific types such as BindingIdentifier, IdentifierReference, and IdentifierName. This clear distinction greatly enhances the development experience by aligning more closely with the ECMAScript specification.

🏆 Parser Performance

Our benchmark reveals that the Oxc parser surpasses the speed of the swc parser by approximately 2 times and the Biome parser by 3 times.

• AST is allocated in a memory arena (bumpalo) for fast AST memory allocation and deallocation.
• Short strings are inlined by CompactString.
• No other heap allocations are done except the above two.
• Scope binding, symbol resolution and some syntax errors are not done in the parser, they are delegated to the semantic analyzer.

🔸 Linter

The linter embraces convention over configuration, eliminating the need for extensive configuration and plugin setup. Unlike other linters like ESLint, which often require intricate configurations and plugin installations (e.g. @typescript-eslint), our linter only requires a single command that you can immediately run on your codebase:

npx oxlint@latest

We also plan to port essential plugins such as eslint-plugin-import and eslint-plugin-jest.

🏆 Linter Performance

The linter is 50 - 100 times faster than ESLint depending on the number of rules and number of CPU cores used. It completes in less than a second for most codebases with a few hundred files and completes in a few seconds for larger monorepos. See bench-javascript-linter for details.

As an upside, the binary is approximately 3MB, whereas ESLint and its associated plugin dependencies can easily exceed 100.

You may also download the linter binary from the latest release tag as a standalone binary, this lets you run the linter without a Node.js installation in your CI.

• Oxc parser is used.
• AST visit is a fast operation due to linear memory scan from the memory arena.
• Files are linted in a multi-threaded environment, so scales with the total number of CPU cores.
• Every single lint rule is tuned for performance.

Linter Plugin

We are currently developing a DSL-based plugin system. The plugin system uses trustfall as its query engine and a subset of GraphQL as its query language.

You will not need to use JavaScript or Rust to write a plugin, this is useful for QAs and security researchers.

🔸 Resolver

Module resolution plays a crucial role in JavaScript tooling, especially for tasks like multi-file analysis or bundling. However, it can often become a performance bottleneck. To address this, we are actively working on porting enhanced-resolve.

eslint-plugin-import will be our first application for the resolver, since it is currently a performance and complexity blocker for a lot of projects.

🔸 Transpiler

A transpiler is responsible for turning higher versions of ECMAScript to a lower version that can be used in older browsers. We are currently focusing on an esnext to es2015 transpiler. See the umbrella issue for details.

🔸 Minifier

JavaScript minification plays a crucial role in optimizing website performance as it reduces the amount of data sent to users, resulting in faster page loads. This holds tremendous economic value, particularly for e-commerce websites, where every second can equate to millions of dollars.

However, existing minifiers typically require a trade-off between compression quality and speed. You have to choose between the slowest for the best compression or the fastest for less compression. But what if we could develop a faster minifier without compromising on compression?

We are actively working on a prototype that aims to achieve this goal, by porting all test cases from well-known minifiers such as google-closure-compiler, terser, esbuild, and tdewolff-minify.

Preliminary results indicate that we are on track to achieve our objectives. With the Oxc minifier, you can expect faster minification times without sacrificing compression quality.

🔸 Formatter

While prettier has established itself as the de facto code formatter for JavaScript, there is a significant demand in the developer community for a less opinionated alternative. Recognizing this need, our ambition is to undertake research and development to create a new JavaScript formatter that offers increased flexibility and customization options. Unfortunately we are currently lacking the resources to do so.

🔸 Ezno Type Checker

Developed by @kaleidawave, ezno is a TypeScript checker written in Rust with a focus on static analysis and runtime performance. You may read the announcement blog post for more information.

The type checker is available via npx oxidation-compiler@latest check path and the playground.

✍️ Contribute

See CONTRIBUTING.md for guidance.

Check out some of the good first issues or ask us on Discord.

If you are unable to contribute by code, you can still participate by:

• Add a GitHub Star to the project.
• Join us on Discord.
• Follow me on twitter and tweet about this project.

📚 Learning Resources

• My small tutorial on how to write a JavaScript Parser in Rust
• My small article Pursuit of Performance on Building a JavaScript Compiler
• Crafting Interpreters
• Andrew Kelley - Practical DOD

🧑‍💻 Maintenance

Oxc is being maintained by Boshen as the project lead with the help of contributors from all over the world.

Rules of thumb:

• APIs should be simple and well-documented.
• All performance issues (runtime and compilation speed) are considered as bugs in this project.
• Third-party dependencies should be minimal.
• Monitor code coverage for unused code. Aim for 99% code coverage.
• Embrace data-oriented design.
• Prefer smaller PRs. Try graphite.dev.

🤝 Credits

This project was incubated with the assistance of these exceptional mentors and their projects:

• Biome - @ematipico
• Ruff - @charliermarsh, @MichaReiser
• quick-lint-js - @strager
• elm-review - @jfmengels

Special thanks go to

• @domonji for contribution to the TypeScript parser
• @guan-wy for the project logo

And also

• @kaleidawave for Ezno Type Checker
• @zackradisic for tyvm
• @u9g for the work in progress oxc_linter_plugin

📖 License

Oxc is free and open-source software licensed under the MIT License.

Oxc partially copies code from the following projects, their licenses are listed in Third-party library licenses.

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

TensorRT-LLM

A TensorRT Toolbox for Optimized Large Language Model Inference

Architecture   |   Results   |   Examples   |   Documentation

---

Latest News

• [2023/11/13] H200 achieves nearly 12,000 tok/sec on Llama2-13B

H200 FP8 achieves 11,819 tok/s on Llama2-13B on a single GPU, and is up to 1.9x faster than H100.

• [2023/11/03] TensorRT-LLM is up to 4.6x faster on H100 than A100, achieving 10,000 tok/s at 100ms to first token.
• [2023/10/22] 🚀 RAG on Windows using TensorRT-LLM and LlamaIndex 🦙
• [2023/10/19] Getting Started Guide - Optimizing Inference on Large Language Models with NVIDIA TensorRT-LLM, Now Publicly Available
• [2023/10/17] Large Language Models up to 4x Faster on RTX With TensorRT-LLM for Windows
• [2023/9/9] NVIDIA TensorRT-LLM Supercharges Large Language Model Inference on NVIDIA H100 GPUs

2023/10/31 - Phind ; 2023/10/12 - Databricks (MosaicML) ; 2023/10/4 - Perplexity ; 2023/9/27 - CloudFlare;

Table of Contents

• TensorRT-LLM Overview
• Installation
• Quick Start
• Support Matrix
• Performance
• Advanced Topics
  • Quantization
  • In-flight Batching
  • Attention
  • Graph Rewriting
  • Benchmarking
• Troubleshooting
• Release Notes
  • Changelog
  • Known issues

TensorRT-LLM Overview

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. It also includes a backend for integration with the NVIDIA Triton Inference Server; a production-quality system to serve LLMs. Models built with TensorRT-LLM can be executed on a wide range of configurations going from a single GPU to multiple nodes with multiple GPUs (using Tensor Parallelism and/or Pipeline Parallelism).

The Python API of TensorRT-LLM is architectured to look similar to the PyTorch API. It provides users with a functional module containing functions like einsum, softmax, matmul or view. The layers module bundles useful building blocks to assemble LLMs; like an Attention block, a MLP or the entire Transformer layer. Model-specific components, like GPTAttention or BertAttention, can be found in the models module.

TensorRT-LLM comes with several popular models pre-defined. They can easily be modified and extended to fit custom needs. See below for a list of supported models.

To maximize performance and reduce memory footprint, TensorRT-LLM allows the models to be executed using different quantization modes (see examples/gpt for concrete examples). TensorRT-LLM supports INT4 or INT8 weights (and FP16 activations; a.k.a. INT4/INT8 weight-only) as well as a complete implementation of the SmoothQuant technique.

For a more detailed presentation of the software architecture and the key concepts used in TensorRT-LLM, we recommend you to read the following document.

Installation

For Windows installation, see Windows/.

TensorRT-LLM must be built from source, instructions can be found here. An image of a Docker container with TensorRT-LLM and its Triton Inference Server Backend will be made available soon.

The remaining commands in that document must be executed from the TensorRT-LLM container.

Quick Start

To create a TensorRT engine for an existing model, there are 3 steps:

1. Download pre-trained weights,
2. Build a fully-optimized engine of the model,
3. Deploy the engine.

The following sections show how to use TensorRT-LLM to run the BLOOM-560m model.

0. In the BLOOM folder

Inside the Docker container, you have to install the requirements:

pip install -r examples/bloom/requirements.txt
git lfs install

1. Download the model weights from HuggingFace

From the BLOOM example folder, you must download the weights of the model.

cd examples/bloom
rm -rf ./bloom/560M
mkdir -p ./bloom/560M && git clone https://huggingface.co/bigscience/bloom-560m ./bloom/560M

2. Build the engine

# Single GPU on BLOOM 560M
python build.py --model_dir ./bloom/560M/ \
                --dtype float16 \
                --use_gemm_plugin float16 \
                --use_gpt_attention_plugin float16 \
                --output_dir ./bloom/560M/trt_engines/fp16/1-gpu/

See the BLOOM example for more details and options regarding the build.py script.

3. Run

The summarize.py script can be used to perform the summarization of articles from the CNN Daily dataset:

python summarize.py --test_trt_llm \
                    --hf_model_location ./bloom/560M/ \
                    --data_type fp16 \
                    --engine_dir ./bloom/560M/trt_engines/fp16/1-gpu/

More details about the script and how to run the BLOOM model can be found in the example folder. Many more models than BLOOM are implemented in TensorRT-LLM. They can be found in the examples directory.

Support Matrix

TensorRT-LLM optimizes the performance of a range of well-known models on NVIDIA GPUs. The following sections provide a list of supported GPU architectures as well as important features implemented in TensorRT-LLM.

Devices

TensorRT-LLM is rigorously tested on the following GPUs:

• H100
• L40S
• A100/A30
• V100 (experimental)

If a GPU is not listed above, it is important to note that TensorRT-LLM is expected to work on GPUs based on the Volta, Turing, Ampere, Hopper and Ada Lovelace architectures. Certain limitations may, however, apply.

Precision

Various numerical precisions are supported in TensorRT-LLM. The support for some of those numerical features require specific architectures:

FP32 FP16 BF16 FP8 INT8 INT4

Volta (SM70)	Y	Y	N	N	Y	Y
Turing (SM75)	Y	Y	N	N	Y	Y
Ampere (SM80, SM86)	Y	Y	Y	N	Y	Y
Ada-Lovelace (SM89)	Y	Y	Y	Y	Y	Y
Hopper (SM90)	Y	Y	Y	Y	Y	Y

In this release of TensorRT-LLM, the support for FP8 and quantized data types (INT8 or INT4) is not implemented for all the models. See the precision document and the examples folder for additional details.

Key Features

TensorRT-LLM contains examples that implement the following features.

• Multi-head Attention(MHA)
• Multi-query Attention (MQA)
• Group-query Attention(GQA)
• In-flight Batching
• Paged KV Cache for the Attention
• Tensor Parallelism
• Pipeline Parallelism
• INT4/INT8 Weight-Only Quantization (W4A16 & W8A16)
• SmoothQuant
• GPTQ
• AWQ
• FP8
• Greedy-search
• Beam-search
• RoPE

In this release of TensorRT-LLM, some of the features are not enabled for all the models listed in the examples folder.

Models

The list of supported models is:

• Baichuan
• Bert
• Blip2
• BLOOM
• ChatGLM-6B
• ChatGLM2-6B
• Falcon
• GPT
• GPT-J
• GPT-Nemo
• GPT-NeoX
• LLaMA
• LLaMA-v2
• MPT
• OPT
• SantaCoder
• StarCoder

Performance

Please refer to the performance page for performance numbers. That page contains measured numbers for four variants of popular models (GPT-J, LLAMA-7B, LLAMA-70B, Falcon-180B), measured on the H100, L40S and A100 GPU(s).

Advanced Topics

Quantization

This document describes the different quantization methods implemented in TensorRT-LLM and contains a support matrix for the different models.

In-flight Batching

TensorRT-LLM supports in-flight batching of requests (also known as continuous batching or iteration-level batching). It's a technique that aims at reducing wait times in queues, eliminating the need for padding requests and allowing for higher GPU utilization.

Attention

TensorRT-LLM implements several variants of the Attention mechanism that appears in most the Large Language Models. This document summarizes those implementations and how they are optimized in TensorRT-LLM.

Graph Rewriting

TensorRT-LLM uses a declarative approach to define neural networks and contains techniques to optimize the underlying graph. For more details, please refer to doc

Benchmark

TensorRT-LLM provides C++ and Python tools to perform benchmarking. Note, however, that it is recommended to use the C++ version.

Troubleshooting

• It's recommended to add options –shm-size=1g –ulimit memlock=-1 to the docker or nvidia-docker run command. Otherwise you may see NCCL errors when running multiple GPU inferences. See https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#errors for details.

• When building models, memory-related issues such as

[09/23/2023-03:13:00] [TRT] [E] 9: GPTLMHeadModel/layers/0/attention/qkv/PLUGIN_V2_Gemm_0: could not find any supported formats consistent with input/output data types
[09/23/2023-03:13:00] [TRT] [E] 9: [pluginV2Builder.cpp::reportPluginError::24] Error Code 9: Internal Error (GPTLMHeadModel/layers/0/attention/qkv/PLUGIN_V2_Gemm_0: could not find any supported formats consistent with input/output data types)

may happen. One possible solution is to reduce the amount of memory needed by reducing the maximum batch size, input and output lengths. Another option is to enable plugins, for example: --use_gpt_attention_plugin.

Release notes

• TensorRT-LLM requires TensorRT 9.1.0.4 and 23.08 containers.

Change Log

• TensorRT-LLM v0.5.0 is the first public release.

Known Issues

Report Issues

You can use GitHub issues to report issues with TensorRT-LLM.

Atomicals CLI and Javascript Library

Atomicals Javascript Library

atomicals.xyz Documentation: https://docs.atomicals.xyz

Install, Build and Run Tests

Install

Download the github repo and then run:

npm install
npm run build

See all commands at:

npm run cli --help

Quick Start - Command Line (CLI)

First install packages and build, then follow the steps here to create your first Atomical and query the status. Use yarn clito get a list of all commands available.

0. Environment File (.env)

The environment file comes with defaults (.env.example), but it is highly recommend to install and operate your own ElectrumX server. Web browser communication is possible through the wss (secure websockets) interface of ElectrumX.

ELECTRUMX_WSS=wss://electrumx.atomicals.xyz:50012

// Optional (defaults to wallet.json)
WALLET_PATH=path-to-wallet.json

ELECTRUMX_WSS: URL of the ElectrumX with Atomicals support. Note that only wss endpoints are accessible from web browsers.

1. Wallet Setup

The purpose of the wallet is to create p2tr (pay-to-taproot) spend scripts and to receive change from the transactions made for the various operations. Do not put more funds than you can afford to lose, as this is still beta!

To initialize a new wallet.json file that will store your address for receiving change use the wallet-init command. Alternatively, you may populate the wallet.json manually, ensuring that the address at m/44'/0'/0'/0/0 is equal to the address and the derivePath is set correctly.

Configure the path in the environment .env file to point to your wallet file. defaults to ./wallet.json

Default:

WALLET_PATH=.
WALLET_FILE=wallet.json

Update to wallets/ directory:

WALLET_PATH=./wallets
WALLET_FILE=wallet.json

Create the wallet:

yarn cli wallet-init

>>>

Wallet created at wallet.json
phrase: maple maple maple maple maple maple maple maple maple maple maple maple
Legacy address (for change): 1FXL2CJ9nAC...u3e9Evdsa2pKrPhkag
Derive Path: m/44'/0'/0'/0/0
WIF: L5Sa65gNR6QsBjqK.....r6o4YzcqNRnJ1p4a6GPxqQQ
------------------------------------------------------

2. Explore the CLI

yarn cli --help

3. Quick Commands

Get all of the commands available:

npm run cli --help

Read the documentation at https://docs.atomicals.xyz

ElectrumX Server RPC Interface

See updated ElectrumX (https://github.com/atomicals/electrumx-atomicals)

Any questions or ideas?

https://atomicals.xyz

https://x.com/atomicalsxyz (X - Formerly Twitter)

Donate to Atomicals Development

We greatly appreciate any donation to help support Atomicals Protocol development. We worked out of passion and kindness for the world, we believe this technology must exist and be free for all to use. Bitcoin is our one hope for freedom and digital sovereignty and we intend to do our best to make it a reality.

BTC: bc1pljy9g0ugrgumpd5y6v9tv23rvz5y8dhaq980r9qfgyhd4dmgkwmqpdpr5q

An Autonomous LLM Agent for Complex Task Solving

English • 中文 • 日本語

Tutorial • Demo • Blog • Documentation • Citation

📖 Introduction

XAgent is an open-source experimental Large Language Model (LLM) driven autonomous agent that can automatically solve various tasks. It is designed to be a general-purpose agent that can be applied to a wide range of tasks. XAgent is still in its early stages, and we are working hard to improve it.

🏆 Our goal is to create a super-intelligent agent that can solve any given task!

We welcome diverse forms of collaborations, including full-time and part-time roles and more. If you are interested in the frontiers of agents and want to join us in realizing true autonomous agents, please contact us at xagentteam@gmail.com.

XAgent

XAgent is designed with the following features:

• Autonomy: XAgent can automatically solve various tasks without human participation.
• Safety: XAgent is designed to run safely. All actions are constrained inside a docker container. Run it anyway!
• Extensibility: XAgent is designed to be extensible. You can easily add new tools to enhance agent's abilities and even new agents！
• GUI: XAgent provides a friendly GUI for users to interact with the agent. You can also use the command line interface to interact with the agent.
• Cooperation with Human: XAgent can collaborate with you to tackle tasks. It not only has the capability to follow your guidance in solving complex tasks on the go but it can also seek your assistance when it encounters challenges.

XAgent is composed of three parts:

• 🤖 Dispatcher is responsible for dynamically instantiating and dispatching tasks to different agents. It allows us to add new agents and improve the agents' abilities.
• 🧐 Planner is responsible for generating and rectifying plans for tasks. It divides tasks into subtasks and generates milestones for them, allowing agents to solve tasks step by step.
• 🦾 Actor is responsible for conducting actions to achieve goals and finish subtasks. The actor utilizes various tools to solve subtasks, and it can also collaborate with humans to solve tasks.

🧰 ToolServer

ToolServer is the server that provides XAgent with powerful and safe tools to solve tasks. It is a docker container that provides a safe environment for XAgent to run. Currently, ToolServer provides the following tools:

• 📝 File Editor provides a text editing tool to write, read, and modify files.
• 📘 Python Notebook provides an interactive Python notebook that can run Python code to validate ideas, draw figures, etc.
• 🌏 Web Browser provides a web browser to search and visit webpages.
• 🖥️ Shell provides a bash shell tool that can execute any shell commands, even install programs and host services.
• 🧩 Rapid API provides a tool to retrieve APIs from Rapid API and call them, which offers a wide range of APIs for XAgent to use. See ToolBench to get more information about the Rapid API collections. You can also easily add new tools to ToolServer to enhance XAgent's abilities.

✨ Quickstart

🛠️ Build and Setup ToolServer

ToolServer is where XAgent's action takes place. It is a docker container that provides a safe environment for XAgent to run. So you should install docker and docker-compose first. After that, you should build the docker image for ToolServer and start the docker container.

docker-compose up --build

This will build the image for the ToolServer and start the ToolServer's container. If you want to run the container in the background, please use docker-compose up -d --build. Refer here for detailed information about our ToolServer.

If the ToolServer is updated, you have to rebuild the images:

docker compose build

🎮 Setup and Run XAgent

After setting up ToolServer, you can start to run XAgent.

• Install requirements (Require Python >= 3.10)

pip install -r requirements.txt

• Configure XAgent

1. You should configure XAgent in assets/config.yml before running it.
2. At least one OpenAI key is provided in assets/config.yml, which is used to access OpenAI API. We highly recommend using gpt-4-32k to run XAgent; gpt-4 is also OK for most simple tasks. In any case, at least one gpt-3.5-turbo-16k API key should be provided as a backup model. We do not test or recommend using gpt-3.5-turbo to run XAgent due to minimal context length; you should not try to run XAgent on that.
3. If you want to change the config_file path for XAgentServer, you should modify the CONFIG_FILE value in .env file and restart the docker container.

• Run XAgent

python run.py --task "put your task here" --model "gpt-4" --config_file "assets/config.yml"

1. You can use the argument --upload_files to select the initial files you want to submit to XAgent.

2. The local workspace for your XAgent is in local_workspace, where you can find all the files generated by XAgent throughout the running process.

3. After execution, the entire workspace in ToolServerNode will be copied to running_records for your convenience.

4. Besides, in running_records, you can find all the intermediate steps information, e.g., task statuses, LLM's input-output pairs, used tools, etc.

5. You can load from a record to reproduce a former run, just by setting record_dir in config(default to Null). The record is a system-level recording tied to the code version of XAgent. All running-config、query、code execution statuses (including errors)、server behavior will be documented.

6. We have removed all sensitive information (including API keys) from the record so you can safely share it with others. In the near future, we will introduce more granular sharing options highlighting the contributions of humans during execution.

• Run XAgent with GUI

## We ran the web ui docker when building the ToolServer network
## run nginx in docker
docker exec XAgent-Server systemctl start nginx

Build the docker image for XAgent-Server and start the docker container. You will see the XAgent Server listening on port 8090. You could visit http://localhost:5173 to interact with XAgent by using web UI. Refer here for the detailed information about our GUI Demo.

🎬 Demo

Here, we also show some cases of solving tasks by XAgent: You can check our live demo on XAgent Official Website. We also provide a video demo and showcases of using XAgent here:

Case 1. Data Analysis: Demonstrating the Effectiveness of Dual-Loop Mechanism

We start with a case of aiding users in intricate data analysis. Here, our user submitted an iris.zip file to XAgent, seeking assistance in data analysis. XAgent swiftly broke down the task into four sub-tasks: (1) data inspection and comprehension, (2) verification of the system's Python environment for relevant data analysis libraries, (3) crafting data analysis code for data processing and analysis, and (4) compiling an analytical report based on the Python code's execution results. Here is a figure drawn by XAgent.

Case 2. Recommendation: A New Paradigm of Human-Agent Interaction

Empowered with the unique capability to actively seek human assistance and collaborate in problem-solving, XAgent continues to redefine the boundaries of human-agent cooperation. As depicted in the screenshot below, a user sought XAgent's aid in recommending some great restaurants for a friendly gathering yet failed to provide specific details. Recognizing the insufficiency of the provided information, XAgent employed the AskForHumanHelp tool, prompting human intervention to elicit the user's preferred location, budget constraints, culinary preferences, and dietary restrictions. Armed with this valuable feedback, XAgent seamlessly generated tailored restaurant recommendations, ensuring a personalized and satisfying experience for the user and their friends.

Case 3. Training Model: A Sophisticated Tool User

XAgent not only tackles mundane tasks but also serves as an invaluable aid in complex tasks such as model training. Here, we show a scenario where a user desires to analyze movie reviews and evaluate the public sentiment surrounding particular films. In response, XAgent promptly initiates the process by downloading the IMDB dataset to train a cutting-edge BERT model (see screenshot below), harnessing the power of deep learning. Armed with this trained BERT model, XAgent seamlessly navigates the intricate nuances of movie reviews, offering insightful predictions regarding the public's perception of various films.

📊 Evaluation

We conduct human preference evaluation to evaluate XAgent's performance. We prepare over 50 real-world complex tasks for assessment, which can be categorized into 5 classes: Search and Report, Coding and Developing, Data Analysis, Math, and Life Assistant. We compare the results of XAgent with AutoGPT, which shows a total win of XAgent over AutoGPT. All running records can refer to here.

We report a significant improvement of XAgent over AutoGPT in terms of human preference.

We also evaluate XAgent on the following benchmarks:

🖌️ Blog

Our blog is available at here!

🌟 Our Contributors

A heartfelt thank you to all our contributors. Your efforts make this project grow and thrive. Every contribution, big or small, is invaluable.

🌟 Star History

Citation

If you find our repo useful, please kindly consider citing:

@misc{xagent2023,
      title={XAgent: An Autonomous Agent for Complex Task Solving}, 
      author={XAgent Team},
      year={2023},
}