Article URL: https://github.com/novuhq/

Comments URL: https://news.ycombinator.com/item?id=38419513

Points: 101

# Comments: 17

Article URL: https://sqids.org/

Comments URL: https://news.ycombinator.com/item?id=38414914

Points: 105

# Comments: 31

src: https://github.com/midzer/sdl-sopwith

via: https://fragglet.github.io/sdl-sopwith/

Comments URL: https://news.ycombinator.com/item?id=38414513

Points: 102

# Comments: 37

Hey HN, I'm Patrick a developer working on side projects beside a 9-5. I created a Site where you can create customised Meal Plans + Grocery Lists and get them automatically every Week into your Inbox.

Please Check it out, any feedback is welcome!

Pat

Comments URL: https://news.ycombinator.com/item?id=38413970

Points: 11

# Comments: 6

Two browser windows (acting as socket clients) communicate their:

- Screen dimensions - (screen.width, screen.height)

- Window dimensions - (window.innerWidth, window.innerHeight)

- Window X/Y position - (window.screenX, window.screenY)

...or whichever calculation works best for you.

The original work by Bjorn Staal https://twitter.com/_nonfigurativ_/status/172732259457002734 used localStorage, but I found sockets more fun, because if tweaked a bit, this can be shared with friends.

Here's a demo of how it works and the codebase: https://github.com/Momciloo/fun-with-sockets/

Comments URL: https://news.ycombinator.com/item?id=38413660

Points: 100

# Comments: 35

Hi HN community!

Proud to have recently become a founder myself and now to submit a "Show HN".

We are Dibulo, a young startup which is on a mission to redefine coloring sessions for kids! We have built our first worlds, sold our first coloring books and now we want to let the world know about us :) So far we have been praised by parents, and also got pretty good feedback from institutions like daycares and elementary school teachers.

What sets our coloring experience apart? Not only do we bring vibrant templates on paper to life on screens, but we also introduce a captivating group activity. Picture this: kids enthusiastically coloring away, their creations showcased prominently on a big screen for all to admire.

For us it is crucial that the user experience is smooth and easy. Our platform works in pretty much all modern browsers and on all major devices (PC, laptop, tablet, AndroidTV / FireTV). Setting up a coloring session on Dibulo takes less time than a kid can color our templates and uploading is as easy as snapping a picture. You can even share the coloring session with your family and friends and if they like they can participate and upload their drawings too (remotely).

Feel free to download and print our free templates and try them with your kids / nephews / at your daycare.

Thank you for checking us out! Looking forward to receiving feedback!

Comments URL: https://news.ycombinator.com/item?id=38413520

Points: 10

# Comments: 3

Article URL: https://github.com/AyaanZaveri/cognite

Comments URL: https://news.ycombinator.com/item?id=38411278

Points: 10

# Comments: 8

Hey all, wanted to share my recent weekend hack project. I wanted to improve upon the existing space for looking up information about IP addresses and ASNs.

The backend is written in Rust and pulls BGP data every 8 hours from the RIPE RIS project to build up a routing table and also adds geolocation information. The frontend is using React and Tailwind.

Would love any feedback or suggestions on what to improve.

Comments URL: https://news.ycombinator.com/item?id=38409164

Points: 108

# Comments: 38

Article URL: https://github.com/mostafasadeghi97/design2code

Comments URL: https://news.ycombinator.com/item?id=38408696

Points: 10

# Comments: 0

Comments

By William F. Godoy, Pedro Valero-Lara, Caira Anderson, Katrina W. Lee, Ana Gainaru, Rafael Ferreira da Silva, Jeffrey S. Vetter.

Abstract:

We evaluate Julia as a single language and ecosystem paradigm powered by LLVM to develop workflow components for high-performance computing. We run a Gray-Scott, 2-variable diffusion-reaction application using a memory-bound, 7-point stencil kernel on Frontier, the US Department of Energy’s first exascale supercomputer. We evaluate the performance, scaling, and trade-offs of (i) the computational kernel on AMD’s MI250x GPUs, (ii) weak scaling up to 4,096 MPI processes/GPUs or 512 nodes, (iii) parallel I/O writes using the ADIOS2 library bindings, and (iv) Jupyter Notebooks for interactive analysis. Results suggest that although Julia generates a reasonable LLVM-IR, a nearly 50% performance difference exists vs. native AMD HIP stencil codes when running on the GPUs. As expected, we observed near-zero overhead when using MPI and parallel I/O bindings for system-wide installed implementations. Consequently, Julia emerges as a compelling high-performance and high-productivity workflow composition language, as measured on the fastest supercomputer in the world.

Comments

Comments

Comments

Comments

Article URL: https://typst.app/

Comments URL: https://news.ycombinator.com/item?id=38354422

Points: 124

# Comments: 36

Comments

What are you doing this week? Feel free to share!

Keep in mind it’s OK to do nothing at all, too.

Comments

Article URL: https://rubjo.github.io/victor-mono/

Comments URL: https://news.ycombinator.com/item?id=38332635

Points: 101

# Comments: 73

Robust Speech Recognition via Large-Scale Weak Supervision

Whisper

[Blog] [Paper] [Model card] [Colab example]

Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification.

Approach

A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing a single model to replace many stages of a traditional speech-processing pipeline. The multitask training format uses a set of special tokens that serve as task specifiers or classification targets.

Setup

We used Python 3.9.9 and PyTorch 1.10.1 to train and test our models, but the codebase is expected to be compatible with Python 3.8-3.11 and recent PyTorch versions. The codebase also depends on a few Python packages, most notably OpenAI's tiktoken for their fast tokenizer implementation. You can download and install (or update to) the latest release of Whisper with the following command:

pip install -U openai-whisper

Alternatively, the following command will pull and install the latest commit from this repository, along with its Python dependencies:

pip install git+https://github.com/openai/whisper.git 

To update the package to the latest version of this repository, please run:

pip install --upgrade --no-deps --force-reinstall git+https://github.com/openai/whisper.git

It also requires the command-line tool ffmpeg to be installed on your system, which is available from most package managers:

# on Ubuntu or Debian
sudo apt update && sudo apt install ffmpeg

# on Arch Linux
sudo pacman -S ffmpeg

# on MacOS using Homebrew (https://brew.sh/)
brew install ffmpeg

# on Windows using Chocolatey (https://chocolatey.org/)
choco install ffmpeg

# on Windows using Scoop (https://scoop.sh/)
scoop install ffmpeg

You may need rust installed as well, in case tiktoken does not provide a pre-built wheel for your platform. If you see installation errors during the pip install command above, please follow the Getting started page to install Rust development environment. Additionally, you may need to configure the PATH environment variable, e.g. export PATH="$HOME/.cargo/bin:$PATH". If the installation fails with No module named 'setuptools_rust', you need to install setuptools_rust, e.g. by running:

pip install setuptools-rust

Available models and languages

There are five model sizes, four with English-only versions, offering speed and accuracy tradeoffs. Below are the names of the available models and their approximate memory requirements and inference speed relative to the large model; actual speed may vary depending on many factors including the available hardware.

The .en models for English-only applications tend to perform better, especially for the tiny.en and base.en models. We observed that the difference becomes less significant for the small.en and medium.en models.

Whisper's performance varies widely depending on the language. The figure below shows a performance breakdown of large-v3 and large-v2 models by language, using WERs (word error rates) or CER (character error rates, shown in Italic) evaluated on the Common Voice 15 and Fleurs datasets. Additional WER/CER metrics corresponding to the other models and datasets can be found in Appendix D.1, D.2, and D.4 of the paper, as well as the BLEU (Bilingual Evaluation Understudy) scores for translation in Appendix D.3.

Command-line usage

The following command will transcribe speech in audio files, using the medium model:

whisper audio.flac audio.mp3 audio.wav --model medium

The default setting (which selects the small model) works well for transcribing English. To transcribe an audio file containing non-English speech, you can specify the language using the --language option:

whisper japanese.wav --language Japanese

Adding --task translate will translate the speech into English:

whisper japanese.wav --language Japanese --task translate

Run the following to view all available options:

whisper --help

See tokenizer.py for the list of all available languages.

Python usage

Transcription can also be performed within Python:

import whisper

model = whisper.load_model("base")
result = model.transcribe("audio.mp3")
print(result["text"])

Internally, the transcribe() method reads the entire file and processes the audio with a sliding 30-second window, performing autoregressive sequence-to-sequence predictions on each window.

Below is an example usage of whisper.detect_language() and whisper.decode() which provide lower-level access to the model.

import whisper

model = whisper.load_model("base")

# load audio and pad/trim it to fit 30 seconds
audio = whisper.load_audio("audio.mp3")
audio = whisper.pad_or_trim(audio)

# make log-Mel spectrogram and move to the same device as the model
mel = whisper.log_mel_spectrogram(audio).to(model.device)

# detect the spoken language
_, probs = model.detect_language(mel)
print(f"Detected language: {max(probs, key=probs.get)}")

# decode the audio
options = whisper.DecodingOptions()
result = whisper.decode(model, mel, options)

# print the recognized text
print(result.text)

More examples

Please use the 🙌 Show and tell category in Discussions for sharing more example usages of Whisper and third-party extensions such as web demos, integrations with other tools, ports for different platforms, etc.

License

Whisper's code and model weights are released under the MIT License. See LICENSE for further details.

Comments

Article URL: https://www.theverge.com/2023/11/15/23960345/microsoft-cpu-gpu-ai-chips-azure-maia-cobalt-specifications-cloud-infrastructure

Comments URL: https://news.ycombinator.com/item?id=38278171

Points: 68

# Comments: 102

Article URL: https://simonwillison.net/2023/Nov/15/gpts/

Comments URL: https://news.ycombinator.com/item?id=38277926

Points: 112

# Comments: 80

Comments

Comments

Article URL: https://cloud.google.com/blog/products/compute/the-worlds-largest-distributed-llm-training-job-on-tpu-v5e

Comments URL: https://news.ycombinator.com/item?id=38222277

Points: 100

# Comments: 43

Comments

Comments

Comments

Article URL: https://blog.pypi.org/posts/2023-11-14-1-pypi-completes-first-security-audit/

Comments URL: https://news.ycombinator.com/item?id=38263786

Points: 101

# Comments: 19

Comments

Comments

Comments

Comments

Article URL: https://github.com/microsoft/ML-For-Beginners

Comments URL: https://news.ycombinator.com/item?id=38271959

Points: 103

# Comments: 18

🚀🧠💬 Supercharged Custom Instructions for ChatGPT (non-coding) and ChatGPT Advanced Data Analysis (coding).

ChatGPT AutoExpert

by Dustin Miller • Reddit • Substack

License: Attribution-NonCommercial-ShareAlike 4.0 International

Elevating Conversational AI to Expert Level

❇️ NEW ❇️

I've created a set of "Custom GPTs" with updated versions of these prompts:

• AutoExpert v6 (Chat)
• AutoExpert v6 (Dev)

Want to support these free prompts? My Substack offers paid subscriptions, that's the best way to show your appreciation.

Introduction

ChatGPT AutoExpert is a shockingly effective set of custom instructions aimed at enhancing the capabilities of GPT-4 and GPT-3.5-Turbo conversational models. These instructions maximize the depth and nuance in responses while minimizing general disclaimers and hand-holding. The ultimate objective is to provide users with accurate, context-rich information and an improved learning experience.

Getting Started

To get started with ChatGPT AutoExpert, choose which set of custom instructions you want to use:

• AutoExpert ("Standard Edition")
  (for non-coding tasks)
• AutoExpert ("Developer Edition")
  (requires GPT-4 with Advanced Data Analysis)

Features

"Standard Edition"

• ✳️ New to v5: Automatically Improves your Question
  Many of us still compose ambiguous questions when asking ChatGPT for help. The AutoExpert will automatically rewrite your question to be precise, and to elicit the best response the experts can provide.
• ✳️ New to v5: Slash Commands
  Slash commands offer an easy way to interact with the AutoExpert system. Get summaries, ideas for additional questions, alternative viewpoints…even ask ChatGPT to review its own answer and suggest improvements.
• ✳️ New to v5: Auto-selects Frameworks and Methodologies
  Designed to select a context-appropriate framework for formulating its best answers
• Maximized Depth and Nuance
  Receive high-quality, in-depth, and ✳️ New to v5: multi-turn responses (GPT-4 only) without compromising on the granularity of the information.
• Perfect for Everyday Use
  No need to switch these instructions on and off. They'll give you a greatly improved experience with ChatGPT, even if you're writing code. Although, if you are writing code, you should check the Developer Edition
• Automatically Identifies the Best Expert
  The AutoExpert custom instruction automatically finds the best expert roles to answer whatever question you ask, every time. You don't need a bunch of special prompts any more—this works with even the simplest of prompts!
• Minimized Hand-Holding
  Cut through the noise and get straight to the facts, reducing unnecessary disclaimers.
• Explicit Reasoning
  Your AutoExpert doesn't just provide answers; it offers an explanation, detailing the thought process behind each response.
• Resourceful Links
  Automatically generates inline links for related topics and "you may also like" topics, helpfully linked to Google search results to avoid hallucination (GPT-3.5 still hallucinates here, but not always. GPT-4 is rock-solid).

"Developer Edition"

[!IMPORTANT] This requires a ChatGPT professional subscription, as it needs both GPT-4 and Advanced Data Analysis!

• Verbosity Selection
  Easily choose the complexity of the generated code, from compact "code golf" type responses, up to complex, modular code samples
• Powered by Jupyter
  ChatGPT Advanced Data Analysis already runs a Jupyter kernel under the hood. AutoExpert (Developer Edition) comes with a companion Python script that you simply upload to your conversation. It will automatically take advantage of the sandbox Python environment for editing longer code samples, and activate a handful of extra "slash commands" to make your life even easier.
• Pick Up Where You Left Off
  You can start a new chat without worrying about ChatGPT forgetting what you were doing in the previous one. The /memory slash command will download all your files, and a history of everything that's been done during your session. Simply upload it (along with the companion script) in a new session, and pick up where you left off.
• Install Custom Wheels
  Yeah, you heard me. Wheels for Python packages can be uploaded and installed automatically.
  • Note that your ChatGPT sandbox uses Python 3.8, on a VM with x86_64 architecture (as of this writing).
• Save Your Work
  Among other /slash commands, AutoExpert (Developer Edition) will save all your code snippets, dehydrate its memory of your requirements and the work it's done—even back up the code cells themselves. Then it zips it up, and you can quickly download your coding conversation history.
• File and Symbol Tree
  By keeping a running history along with a file/symbol tree at the end of each response, ChatGPT will always remember what it just did, and you'll always see what files still need work. It's even smart enough to handle breaking down complex requirements in a way that allows it to write code over multiple turns.

ChatGPT AutoExpert (both standard and "Developer Edition")
by Dustin Miller is licensed under Attribution-NonCommercial-ShareAlike 4.0 International

uBlock Origin - An efficient blocker for Chromium and Firefox. Fast and lean.

uBlock Origin (uBO)

BEWARE! uBO is (and has always been) COMPLETELY UNRELATED to the website ublock.org.

uBlock Origin (uBO) is a CPU and memory-efficient wide-spectrum content blocker for Chromium and Firefox. It blocks ads, trackers, coin miners, popups, annoying anti-blockers, malware sites, etc., by default using EasyList, EasyPrivacy, Peter Lowe's Blocklist, Online Malicious URL Blocklist, and uBO filter lists. There are many other lists available to block even more. Hosts files are also supported. uBO uses the EasyList filter syntax and extends the syntax to work with custom rules and filters.

You may easily unselect any preselected filter lists if you think uBO blocks too much. For reference, Adblock Plus installs with only EasyList, ABP filters, and Acceptable Ads enabled by default.

It is important to note that using a blocker is NOT theft. Do not fall for this creepy idea. The ultimate logical consequence of blocking = theft is the criminalization of the inalienable right to privacy.

Ads, "unintrusive" or not, are just the visible portion of the privacy-invading means entering your browser when you visit most sites. uBO's primary goal is to help users neutralize these privacy-invading methods in a way that welcomes those users who do not wish to use more technical means.

• Documentation
• Installation
  • Firefox
  • Thunderbird
  • Chromium
  • All Programs
  • Enterprise Deployment
• Release History
• Translations
• About

Documentation

The simple popup user interface for an install-it-and-forget-it type of installation that is configured optimally by default.	The advanced popup user interface includes a point-and-click firewall that is configurable on a per-site basis.

Visit the Wiki for documentation.

For support, questions, or help, visit /r/uBlockOrigin.

Installation

Required Permissions

Firefox

Firefox Add-ons

Development Builds

uBO works best on Firefox and is available for desktop and Android versions.

Thunderbird

Thunderbird Add-ons

In Thunderbird, uBlock Origin does not affect emails, just feeds.

Chromium

Chrome Web Store

Microsoft Edge Add-ons (Published by: Nicole Rolls)

Opera Add-ons

Development Builds

uBO should be compatible with any Chromium-based browser.

All Programs

Do NOT use uBO with any other content blocker. uBO performs as well as or better than most popular blockers. Other blockers can prevent uBO's privacy or anti-blocker-defusing features from working correctly.

Manual Installation

Enterprise Deployment

Deploying uBO

Release History

Releases Page

Translations

Help translate uBO via Crowdin.

About

Manifesto

Privacy Policy

GPLv3 License

Free. Open-source. For users by users. No donations sought.

If you ever want to contribute something, think about the people working hard to maintain the filter lists you are using, which are available to use by all for free.

Playing Pokemon Red with Reinforcement Learning

Train RL agents to play Pokemon Red

Watch the Video on Youtube!

Join the discord server

Running the Pretrained Model Interactively 🎮

🐍 Python 3.10 is recommended. Other versions may work but have not been tested.
You also need to install ffmpeg and have it available in the command line.

1. Copy your legally obtained Pokemon Red ROM into the base directory. You can find this using google, it should be 1MB. Rename it to PokemonRed.gb if it is not already. The sha1 sum should be ea9bcae617fdf159b045185467ae58b2e4a48b9a, which you can verify by running shasum PokemonRed.gb.
2. Move into the baselines/ directory:
   cd baselines
3. Install dependencies:
   pip install -r requirements.txt
   It may be necessary in some cases to separately install the SDL libraries.
4. Run:
   python run_pretrained_interactive.py

Interact with the emulator using the arrow keys and the a and s keys (A and B buttons).
You can pause the AI's input during the game by editing agent_enabled.txt

Note: the Pokemon.gb file MUST be in the main directory and your current directory MUST be the baselines/ directory in order for this to work.

Training the Model 🏋️

10-21-23: Updated Version!

This version still needs some tuning, but it can clear the first gym in a small fraction of the time and compute resources. It can work with as few as 16 cores and ~20G of RAM. This is the place for active development and updates!

1. Previous steps 1-3
2. Run:
   python run_baseline_parallel_fast.py

Tracking Training Progress 📈

The current state of each game is rendered to images in the session directory.
You can track the progress in tensorboard by moving into the session directory and running:
tensorboard --logdir .
You can then navigate to localhost:6006 in your browser to view metrics.
To enable wandb integration, change use_wandb_logging in the training script to True.

Extra 🐜

Map visualization code can be found in visualization/ directory.

Supporting Libraries

Check out these awesome projects!

PyBoy

Stable Baselines 3

Article URL: https://github.com/KillianLucas/open-interpreter

Comments URL: https://news.ycombinator.com/item?id=38242343

Points: 100

# Comments: 49

Qt based cross-platform GUI proxy configuration manager (backend: v2ray / sing-box)

NekoRay / NekoBox For PC

Qt based cross-platform GUI proxy configuration manager (backend: v2ray / sing-box)

Support Windows / Linux out of the box now.

基于 Qt 的跨平台代理配置管理器 (后端 v2ray / sing-box)

目前支持 Windows / Linux 开箱即用

Readme Translations

中文 / English / fa_IR / 日本語

下载 / Download

GitHub Releases (Portable ZIP)

便携格式，无安装器。转到 Releases 下载预编译的二进制文件，解压后即可使用。

下载 / Download

安装包的说明，如果你不知道要下载哪一个

Package

AUR

• nekoray
• nekoray-git

archlinuxcn

• nekoray
• nekoray-git

Scoop Extras

scoop install nekoray

更改记录 & 发布频道 / Changelog & Telegram Channel

https://t.me/Matsuridayo

项目主页 & 文档 / Homepage & Documents

https://matsuridayo.github.io

代理 / Proxy

• SOCKS (4/4a/5)
• HTTP(S)
• Shadowsocks
• VMess
• VLESS
• Trojan
• TUIC ( sing-box )
• NaïveProxy ( Custom Core )
• Hysteria ( Custom Core or sing-box )
• Hysteria2 ( Custom Core or sing-box )
• Custom Outbound
• Custom Config
• Custom Core

订阅 / Subscription

• Raw: some widely used formats (like Shadowsocks, Clash and v2rayN)
• 原始格式: 一些广泛使用的格式 (如 Shadowsocks、Clash 和 v2rayN)

运行参数

运行参数

Windows 运行

若提示 DLL 缺失，无法运行，请下载 安装 微软 C++ 运行库

Linux 运行

Linux 运行教程

macOS

由于缺乏维护，不再提供 macOS 版本下载。

您还可以在下面的存储库中非正式地获取更新版本。非官方发布 / Unofficial releases

仍可以参照下方教程自行编译，常见问题请参考历史 Issue，欢迎 Pull Request 修复。

编译教程 / Compile Tutorial

请看 技术文档 / Technical documentation

捐助 / Donate

如果这个项目对您有帮助，可以通过捐赠的方式帮助我们维持这个项目。

捐赠满等额 50 USD 可以在「捐赠榜」显示头像，如果您未被添加到这里，欢迎联系我们补充。

Donations of 50 USD or more can display your avatar on the Donation List. If you are not added here, please contact us to add it.

USDT TRC20

TRhnA7SXE5Sap5gSG3ijxRmdYFiD4KRhPs

XMR

49bwESYQjoRL3xmvTcjZKHEKaiGywjLYVQJMUv79bXonGiyDCs8AzE3KiGW2ytTybBCpWJUvov8SjZZEGg66a4e59GXa6k5

• 您也可以通过 Google Play 购买捐赠，另有支付宝微信购买方式，见 TG 频道置顶。
• Donating via Google Play, using credit cards.

Credits

Core:

• v2fly/v2ray-core ( < 3.10 )
• MatsuriDayo/Matsuri ( < 3.10 )
• MatsuriDayo/v2ray-core ( < 3.10 )
• XTLS/Xray-core ( >= 3.10 )
• MatsuriDayo/Xray-core ( >= 3.10 )
• SagerNet/sing-box
• Matsuridayo/sing-box-extra

Gui:

• Qv2ray
• Qt
• protobuf
• yaml-cpp
• zxing-cpp
• QHotkey
• AppImageKit

A V2Ray client for Android, support Xray core and v2fly core

v2rayNG

A V2Ray client for Android, support Xray core and v2fly core

Telegram Channel

github_2dust

Usage

Geoip and Geosite

• geoip.dat and geosite.dat files are in Android/data/com.v2ray.ang/files/assets (path may differ on some Android device)
• download feature will get enhanced version in this repo (Note it need a working proxy)
• latest official domain list and ip list can be imported manually
• possible to use third party dat file in the same folder, like h2y

More in our wiki

Development guide

Android project under V2rayNG folder can be compiled directly in Android Studio, or using Gradle wrapper. But the v2ray core inside the aar is (probably) outdated.
The aar can be compiled from the Golang project AndroidLibV2rayLite or AndroidLibXrayLite. For a quick start, read guide for Go Mobile and Makefiles for Go Developers

v2rayNG can run on Android Emulators. For WSA, VPN permission need to be granted via appops set [package name] ACTIVATE_VPN allow

The cutest Discord client mod

Vencord

The cutest Discord client mod

Features

• Super easy to install (Download Installer, open, click install button, done)
• 100+ plugins built in: See a list
  • Some highlights: SpotifyControls, MessageLogger, Experiments, GameActivityToggle, Translate, NoTrack, QuickReply, Free Emotes/Stickers, PermissionsViewer, CustomCommands, ShowHiddenChannels, PronounDB
• Fairly lightweight despite the many inbuilt plugins
• Excellent Browser Support: Run Vencord in your Browser via extension or UserScript
• Works on any Discord branch: Stable, Canary or PTB all work (though for the best experience I recommend stable!)
• Custom CSS and Themes: Inbuilt css editor with support to import any css files (including BetterDiscord themes)
• Privacy friendly, blocks Discord analytics & crash reporting out of the box and has no telemetry
• Maintained very actively, broken plugins are usually fixed within 12 hours
• Settings sync: Keep your plugins and their settings synchronised between devices / apps (optional)

Installing / Uninstalling

Visit https://vencord.dev/download

Join our Support/Community Server

https://discord.gg/D9uwnFnqmd

Star History

Disclaimer

Discord is trademark of Discord Inc. and solely mentioned for the sake of descriptivity. Mention of it does not imply any affiliation with or endorsement by Discord Inc.

Client modifications are against Discord’s Terms of Service.

However, Discord is pretty indifferent about them and there are no known cases of users getting banned for using client mods! So you should generally be fine as long as you don’t use any plugins that implement abusive behaviour. But no worries, all inbuilt plugins are safe to use!

Regardless, if your account is very important to you and it getting disabled would be a disaster for you, you should probably not use any client mods (not exclusive to Vencord), just to be safe

Additionally, make sure not to post screenshots with Vencord in a server where you might get banned for it

Langchain + Docker + Neo4j + Ollama

GenAI Stack

The GenAI Stack will get you started building your own GenAI application in no time. The demo applications can serve as inspiration or as a starting point. Learn more about the details in the technical blog post.

Configure

Create a .env file from the environment template file env.example

Available variables:

LLM Configuration

MacOS and Linux users can use any LLM that's available via Ollama. Check the "tags" section under the model page you want to use on https://ollama.ai/library and write the tag for the value of the environment variable LLM= in th e.env file. All platforms can use GPT-3.5-turbo and GPT-4 (bring your own API keys for OpenAIs models).

MacOS Install Ollama on MacOS and start it before running docker compose up.

Linux No need to install Ollama manually, it will run in a container as part of the stack when running with the Linux profile: run docker compose --profile linux up. Make sure to set the OLLAMA_BASE_URL=http://llm:11434 in the .env file when using Ollama docker container.

To use the Linux-GPU profile: run docker compose --profile linux-gpu up. Also change OLLAMA_BASE_URL=http://llm-gpu:11434 in the .env file.

Windows Not supported by Ollama, so Windows users need to generate a OpenAI API key and configure the stack to use gpt-3.5 or gpt-4 in the .env file.

Develop

[!WARNING] There is a performance issue that impacts python applications in the 4.24.x releases of Docker Desktop. Please upgrade to the latest release before using this stack.

To start everything

docker compose up

If changes to build scripts has been made, rebuild.

docker compose up --build

To enter watch mode (auto rebuild on file changes). First start everything, then in new terminal:

docker compose watch

Shutdown If health check fails or containers don't start up as expected, shutdown completely to start up again.

docker compose down

Applications

Here's what's in this repo:

The database can be explored at http://localhost:7474.

App 1 - Support Agent Bot

UI: http://localhost:8501 DB client: http://localhost:7474

• answer support question based on recent entries
• provide summarized answers with sources
• demonstrate difference between
  • RAG Disabled (pure LLM response)
  • RAG Enabled (vector + knowledge graph context)
• allow to generate a high quality support ticket for the current conversation based on the style of highly rated questions in the database.

(Chat input + RAG mode selector)

(CTA to auto generate support ticket draft)	(UI of the auto generated support ticket draft)

App 2 - Loader

UI: http://localhost:8502 DB client: http://localhost:7474

• import recent Stack Overflow data for certain tags into a KG
• embed questions and answers and store in vector index
• UI: choose tags, run import, see progress, some stats of data in the database
• Load high ranked questions (regardless of tags) to support the ticket generation feature of App 1.

App 3 Question / Answer with a local PDF

UI: http://localhost:8503
DB client: http://localhost:7474

This application lets you load a local PDF into text chunks and embed it into Neo4j so you can ask questions about its contents and have the LLM answer them using vector similarity search.

App 4 Standalone HTTP API

Endpoints:

• http://localhost:8504/query?text=hello&rag=false (non streaming)
• http://localhost:8504/query-stream?text=hello&rag=false (SSE streaming)

Example cURL command:

curl http://localhost:8504/query-stream\?text\=minimal%20hello%20world%20in%20python\&rag\=false

Exposes the functionality to answer questions in the same way as App 1 above. Uses same code and prompts.

App 5 Static front-end

UI: http://localhost:8505

This application has the same features as App 1, but is built separate from the back-end code using modern best practices (Vite, Svelte, Tailwind).
The auto-reload on changes are instant using the Docker watch sync config.

Xray, Penetrates Everything. Also the best v2ray-core, with XTLS support. Fully compatible configuration.

Project X

Project X originates from XTLS protocol, providing a set of network tools such as Xray-core and REALITY.

README is open, so feel free to submit your project here.

License

Mozilla Public License Version 2.0

Documentation

Project X Official Website

Telegram

Project X

Project X Channel

Installation

• Linux Script
  • XTLS/Xray-install
• Docker
  • iamybj/docker-xray
  • teddysun/xray
• Web Panel
  • X-UI, X-UI-English, 3X-UI, X-UI, X-UI
  • Xray-UI, X-UI
  • Hiddify
  • Marzban
  • Libertea
• One Click
  • Xray-REALITY, xray-reality, reality-ezpz
  • Xray-script, Xray_bash_onekey, XTool
  • v2ray-agent, Xray_onekey, ProxySU
• Magisk
  • Xray4Magisk
  • Xray_For_Magisk
• Homebrew
  • brew install xray

Usage

• Example
  • VLESS-XTLS-uTLS-REALITY
  • VLESS-TCP-XTLS-Vision
  • All-in-One-fallbacks-Nginx
• Xray-examples
  • XTLS/Xray-examples
  • chika0801/Xray-examples
  • lxhao61/integrated-examples
• Tutorial
  • XTLS Vision
  • REALITY (English)
  • XTLS-Iran-Reality (English)
  • Xray REALITY with 'steal oneself' (English)

GUI Clients

• OpenWrt
  • PassWall, PassWall 2
  • ShadowSocksR Plus+
  • luci-app-xray (openwrt-xray)
• Windows
  • v2rayN
  • NekoRay
  • Furious
  • HiddifyN
  • Invisible Man - Xray
• Android
  • v2rayNG
  • HiddifyNG
  • X-flutter
• iOS & macOS arm64
  • Mango
  • FoXray
  • Streisand
• macOS arm64 & x64
  • V2rayU
  • V2RayXS
  • Furious
  • FoXray
• Linux
  • v2rayA
  • NekoRay
  • Furious

Others that support VLESS, XTLS, REALITY, XUDP, PLUX...

• iOS & macOS arm64
  • Shadowrocket
• Xray Tools
  • xray-knife
• Xray Wrapper
  • XTLS/libXray
  • xtlsapi
  • AndroidLibXrayLite
  • XrayKit
  • Xray-core-python
• XrayR
  • XrayR-release
  • XrayR-V2Board
• Clash.Meta
  • Clash Verge
  • clashN
  • Clash Meta for Android
  • meta_for_ios
• sing-box
  • installReality
  • sbox-reality
  • sing-box-for-ios

Contributing

Code of Conduct

Credits

• Xray-core v1.0.0 was forked from v2fly-core 9a03cc5, and we have made & accumulated a huge number of enhancements over time, check the release notes for each version.
• For third-party projects used in Xray-core, check your local or the latest go.mod.

Compilation

Windows

go build -o xray.exe -trimpath -ldflags "-s -w -buildid=" ./main

Linux / macOS

go build -o xray -trimpath -ldflags "-s -w -buildid=" ./main

Stargazers over time

I’d like to know if anyone already succeeded in joining two models that relate only on their geometry fields, without raw SQL. My use case is to annotate a parent model containing a polygon field with the number of points (PointField of another model) contained in each parent model instance.

I found this StackOverflow post (python - GeoDjango query: all point that are contained into a multi polygon - Stack Overflow) which is close to what I’m trying to achieve, but I’m unfortunately getting errors. I can develop my trials/errors, but I’d like first to know if someone found a solution for a similar use case.

Any success story here?

1 post - 1 participant

Read full topic

In my latest blog post, “Build a Reactive ToDo Application in Django with Django-Unicorn,” Idelve into the world of efficient web application development using the robust combination of Django and Django-Unicorn.

This comprehensive guide is designed for developers looking to create a ToDo application that is not only functional but also highly reactive, providing a smooth, real-time user experience.

With Django-Unicorn, I demonstrate how to bridge the gap between the simplicity of Django and the reactivity of modern JavaScript frameworks, enabling you to write less JavaScript and more Python.

Build a Reactive ToDo application in Django with Django-Unicorn

1 post - 1 participant

Read full topic

Hi,
I’d be interested in reopening ticket #21183 (Filter block on the right covers information in admin) – Django . This ticket was in limbo for very long time before going from “wontfix” → “new” → “closed” recently with a rationale that another ticket is implemented (#27471 (Make admin's list_filter choices collapsable) – Django) and “we dont need both of them”.

However the latter ticket is about collapsing individual items in the sidebar vertically, whereas the former one is about hiding the whole sidebar which provides more horizontal screen space, so the only similarity here is that these are related to the same sidebar (?)

The commenter reopening given ticket states that they’re affected by this, so implementation of #27471 havent apparently solved their issue.
I’m facing the same problem, so I’d like to challenge the decision for having this ticket closed, and I’m willing to contribute the code in case it could be reopened.

Thanks for consideration.

5 posts - 3 participants

Read full topic

Hey everyone. After spending some years using and promoting Postgres schemas for multi-tenancy in Django, I decided to give row level security a try.

GitHub - lorinkoz/django-pgrls: Django multi-tenancy through Postgres row...

Django multi-tenancy through Postgres row level security - GitHub - lorinkoz/django-pgrls: Django multi-tenancy through Postgres row level security

1 post - 1 participant

Read full topic

The docs page Advanced tutorial: How to write reusable apps covers packaging a Django app for PyPI. The “Choosing a name for your app” admonition recommends using an example module name polls with a package name django-polls.

I propose changing this page to recommend a module name of django_polls and making the admonition warn against not using the django_ prefix. The current approach causes name clashes which prevents using packages together. Here are the examples I know of:

• django-ratelimit had to rename after it collided with ratelimit: Conflict with PyPI ratelimit · Issue #214 · jsocol/django-ratelimit · GitHub
• @jeff recently reported a collision between django-markdownify and markdownify Jeff Triplett: "🤷 It's rare that Python namespaces stop me from u…" - Mastodon

I’m sure there are more.

3 posts - 3 participants

Read full topic

A year ago, during DjangoCon US 2022, I proposed the idea of organizing the “Django active developers Sprints”

Since then, I’ve talked about it with some community members and I updated the article.

I’ll share it again
https://www.paulox.net/2022/10/26/my-django-active-developers-sprints-proposal/

1 post - 1 participant

Read full topic

In the spirit of creating public notes (which I heard about from Ryan Cheley during his talk at DjangoCon US 2023), I’m posting on the chance it could be helpful for anyone else who is also learning.

Feedback welcome - as with everything, I’m sure there are iterations to be made as I learn more (just please keep in mind that I’m a beginner if responding).

Upgrading packages checklist [check ~monthly]

• Check for deprecation warnings by running the project’s tests
  python -Wa manage.py test >>> use if using Django Test module
  PYTHONWARNINGS=always pytest --capture=no >>> use if using pytest

  • For each deprecation warning, address what it’s warning about & then rerun the tests

• Check if there are any outdated dependencies / package versions in the project:
  pip list --outdated

• For each outdated package,

  • In requirements.in (or dev-requirements.in) file, update the version number of OutdatedPackageName to upgrade to the next incremental (e.g. from X.Y.Z to X.Y+1.Z to X.Y+2.X) release
    Note: OutdatedPackageName~=2.1.0 will install the latest version of OutdatedPackageName 2.1.X where X will update automatically when syncing your venv if a newer version is available (aka you won’t manually need to type the 2.1.1 to 2.1.2 to 2.1.3 updates in the requirements.in file each time)
    Note: Search for “OutdatedPackageName X.X.X Release Notes” to find info about changes in that version. Likely will also be in OutdatedPackageName GitHub Repo

• Save file

• Compile updated requirements.txt or (dev-requirements.txt)
  pip-compile requirements.in
  or
  pip-compile dev-requirements.in

• Sync your venv to make it match your dev-requirements.txt file
  pip-sync dev-requirements.txt
  Note: you always sync the dev-requirements.txt file (even if you only updated the requirements.in file) because the dev requirements inherit the requirements.txt too and you’re using the dev requirements locally (so the venv should always be updated to match any update)

• Double-check that the venv has the correct updated version number
  pip freeze

• Commit changes to Git
  git status
git add -A
git commit -m ‘upgraded OutdatedPackageName version to X.X.X; updated requirements files’

• Double-check everythings OK by running the project’s tests again
  python -Wa manage.py test >>> use if using Django Test module
  PYTHONWARNINGS=always pytest --capture=no >>> use if using pytest

Upgrading Homebrew / Python checklist (Mac) [check every ~3 months]

• Find the version of Python your project currently uses in venv
  python --version

• Find the release date of the version currently used and compare it to the release date of the newer versions of Python
  Download Python | Python.org

• Upgrade Python version (& other outdated stuff in Homebrew)

  • Ensure Homebrew is installed
    brew --version
  • Update Homebrew
    brew update
  • See outdated Homebrew packages
    brew outdated
  • Upgrade Homebrew packages
    brew upgrade
    Note: if desired, can do one at a time instead of all at once; more info

Some assumptions

• Mac
• pytest
• pip-tools

Some sources

• How to upgrade Django to a newer version | Django documentation | Django
• Django Dependency Management with pip-compile and pip-tools
• Using pip-compile to manage dependencies in your Python packages
• Managing your requirements.txt with pip-tools in python
• FAQ (Frequently Asked Questions) — Homebrew Documentation

1 post - 1 participant

Read full topic

Hi all,

With unique_together likely going away in the future in favour of UniqueConstraint, I noticed that Django does not raise a system check error or warning when you use both for the same set of fields on the same model. For example:

class TranslatableText(models.Model):
    translation_key = models.UUIDField(default=uuid.uuid4, editable=False)
    locale = models.CharField(max_length=8)
    content = models.TextField()

    class Meta:
        unique_together = [("translation_key", "locale")]
        constraints = [
            models.UniqueConstraint(
                fields=["translation_key", "locale"],
                name="unique_translation_key_locale",
            ),
        ]

It might be obvious that you shouldn’t do this. However, it’s not immediately obvious in the case of abstract models (e.g. from a third-party package that provides i18n) if you inherit the abstract model’s Meta as well. Without knowing that there’s a unique_together or a UniqueConstraint already in place, you might end up adding the other one and end up with two constraints for the same set of fields.

As a package maintainer, it’s pretty easy to add the system check yourself, but I’m wondering whether this should be included in Django itself. If so, I’d be happy to create a ticket and patch.

Although one might also argue it’s not worth to add the check into Django, as we’ll likely add another check to raise a deprecation warning about unique_together when it actually is going away anyway.

Another point is that Django also doesn’t raise an error if you have more than one UniqueConstraints with the same set of fields. This is understandable though, as UniqueConstraint has other options e.g. condition which means the constraints may have different purposes despite using the same fields. With unique_together, Django seems smart enough to consolidate duplicate entries into one when creating migrations (because they get converted to a set).

What do you think, shall we add such check(s)? Keen to hear your thoughts.

Thanks!

1 post - 1 participant

Read full topic

I found a number of people, myself included run into csrf validation problems. And unfortunately they aren’t the easiest thing to debug. This is my attempt at helping someone walk through the error and dice into the Django codebase to understand and fix the issue.

Debugging CSRF Failed / 403 Forbidden errors in Django

A guided deep dive into Django's source code to understand why your application is failing CSRF validation.

1 post - 1 participant

Read full topic

I published an introduction to database generated columns, using SQLite and the new GeneratedField added in Django 5.0

I’d be very happy to have your feedback on my article

https://www.paulox.net/2023/11/07/database-generated-columns-part-1-django-and-sqlite/

1 post - 1 participant

Read full topic

Here’s a write-up of an optimization I made in my client Silvr’s project. I ended up disabling a PostgreSQL feature called the JIT (Just-In-Time) compiler which was taking a long time for little benefit.

This behaviour was observed on PostgreSQL 14, so things may have improved since. Although the version 15 and 16 release notes don’t seem to contain any relevant-looking changes.

Okay, story time…

I was looking at the project’s slowest tests and found one that took five seconds despite containing few lines of code. Profiling the test revealed a single slow query, which was also slow in production. I grabbed the query plan and fed it to pgMustard using the technique I previously blogged about):

pgMustard reported that JIT compilation of the query was taking 99.7% of the 3217ms runtime. The five-star rating for this finding meant that it was ripe for optimization. But why was JIT compilation taking so long?

The query used InheritanceManager from django-model-utils, which extends Django’s multi-table inheritance to return heterogenous subclasses of a base model:

>>> Place.objects.select_subclasses()
[<Library: Downtown Books>, <BookShop: Giovanni’s>]

The generated query joins all subclasses’ tables. With many subclasses, that query has many joins. But each join only adds a few rows to the results since each object is of only one leaf subclass.

However, PostgreSQL estimated that the large number of joins would mean many results, making it decide to JIT compile the query. JIT compiling a query takes a while, which is typically offset by the time savings. But with few results, this query took three seconds to JIT compile but milliseconds to execute.

pgMustard recommended either improving PostgreSQL’s configuration to reduce cost estimates to prevent JIT compiling, or disabling the JIT altogether. I found the number of cost estimation options bamboozling, so I gave up on that and tried disabling the JIT. I at least found a PostgreSQL mailing list post reporting similar JIT misbehaviour with low result counts, which was some reassurance that disabling the JIT wasn’t such a bad idea.

To disable the JIT, I extended the DATABASES setting’s OPTIONS with psycopg’s options to disable PostgreSQL’s jit setting, with a hefty explanatory query:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        # ...
        "OPTIONS": {
            # Disable PostgreSQL JIT
            # https://www.postgresql.org/docs/current/jit.html
            # django-model-utils’ select_subclasses() method joins a bunch of
            # tables, which PostgreSQL sees as high cost and JIT’s. But each
            # table will only match zero or one row per base class row, so this
            # cost estimate is vastly wrong, and JIT compiling costs a lot more
            # than it’s worth (e.g. 3.5 seconds with JIT vs 10 milliseconds
            # without). We’re not the only ones to encounter such a problem:
            # https://www.postgresql.org/message-id/70ec49f7-e9c4-bea7-c689-a4dfbdd66f78%40jurr.org
            # Using libq 'options' param, via psycopg:
            # https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-OPTIONS
            "options": "-c jit=off",
        },
    },
}

This option could also be set in the server configuration, but putting it in Django’s settings is fast and guaranteed to apply to all environments.

The change sped up the individual test to take milliseconds. Overall, the test suite went from 106 seconds to 94 seconds, an 11% improvement.

There was a risk of performance regressions in production, so we monitored when deploying. Thankfully, things only improved :) It seems the JIT wasn’t providing much value for this project.

Comments

Let's say you're developing a Django project and you've lost track of the terminal window that ran your python manage.py runserver command. If you can't press Ctrl+C in the terminal, how do you stop the server?

ps aux

ps aux | grep manage.py

ps aux | grep 'manage.py runserver'

lance    12839  0.0  0.1  57068 49840 pts/25   S+   07:11   0:00 python manage.py runserver
lance    12840 23.0  0.2 435156 67752 pts/25   Sl+  07:11   4:40 /home/lance/django-workout/.venv/bin/python manage.py runserver
lance    23752  0.0  0.0   4040  1980 pts/30   S+   07:31   0:00 grep --color=auto manage.py runserver

ps aux | head --lines=1 && ps aux | grep 'manage.py runserver'

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
lance    12839  0.0  0.1  57068 49840 pts/25   S+   07:11   0:00 python manage.py runserver
lance    12840 23.0  0.2 435156 67752 pts/25   Sl+  07:11   4:40 /home/lance/django-workout/.venv/bin/python manage.py runserver
lance    23752  0.0  0.0   4040  1980 pts/30   S+   07:31   0:00 grep --color=auto manage.py runserver

kill 12839

ps aux | head --lines=1 && ps aux | grep 'manage.py runserver'

News

Django security releases issued: 4.2.7, 4.1.13, and 3.2.23

A new security release for a potential denial of service vulnerability in UsernameField on Windows. As ever, the best security practice is to keep your Django version up to date with the latest release.

djangoproject.com

Python Software Foundation News: Announcing our new Community Communications Manager!

The PSF's first Communications Manager will establish a PSF communications calendar, including annual messaging, newsletters, and blog posts. She will also partner with our Executive Director, Deb Nicholson, and other staffers to enhance our support for the Python community with various initiatives.

blogspot.com

Updates to Django

Last week we had another massive 18 pull requests merged into Django by 13 different contributors - including 4 first time contributors! Congratulations to ksg, Leo Suarez, Izzy Hyman and CheesyPhoenix for having their first commit merged into Django - welcome on board!

The System check framework docs now has a new section on writing integration tests: System check framework - Writing integration tests. I always love to see the documentation continue to improve to better serve the community!

Coming in Django 5.1, there will be a new template tag {% query_string %} that allows changing a django.http.QueryDict instance for use in links, for example, to generate a link to the next page while keeping any filtering options in place. This should be hugely useful! You can read more about this here: query_string docs

Django Newsletter

Wagtail CMS

Wagtail 5.2 (LTS) release notes

Wagtail 5.2 is designated a Long Term Support (LTS) release which includes a redesigned page listing view, OpenSearch support, responsive & multi-format images with the picture tag, and much more.

wagtail.org

Curate content better and make it greener in Wagtail 5.2

A nice overview of Wagtail 5.2's new features.

wagtail.org

Sponsored Link

Get Expert Developer Support from the creators of Wagtail CMS

Get support for your developers from our team of Wagtail experts: from reviewing your Wagtail set-up and planning complex features, to code reviews, pair programming and more. We offer a range of plans, special pricing for charities and universities, and profits are invested in Wagtail product development.

bit.ly

Articles

New goodies in Django 5.0

From Django Fellow Mariusz Felisiak, an exploration of the "deluge" of amazing new features added in Django 5.0

fly.io

Moving from Django Microframework to Django

A detailed look at recent efforts to create a single file Django "Hello, World" example and why analyzing those efforts demystifies some of the "magic" behind Django.

programmingmylife.com

Customize the Django admin to differentiate environments

An elegant way to configure the admin colors based on production/staging/local environment.

406.ch

Add infinite scroll with Django and HTMX in 60 seconds

Speed your initial load - get content only when the user wants it.

photondesigner.com

Toward sustainable development practices in open source

Mostly Python is a newsletter by Eric Matthes, the author of Python Crash Course and the django-simple-deploy package. This article talks about the new Djangonaut program and how it ties into sustainable open source.

substack.com

Guide to setting up GeoDjango on Mac M1

How to setup GeoDjango using conda on an M-series Mac.

avilpage.com

Not all open source work is equal

Python maintainer Hynek Schlaweck shares his thoughts on how being compensated for his work has helped him carve out more time to work on the open-source projects he maintains.

tidelift.com

Forum

Updating our DEP process / DEP 1 - Django Internals

There is a good discussion on the Django Enhancement Proposal (DEP) process and how it might be updated and streamlined nearly a decade later.

djangoproject.com

Events

DjangoCon Africa November 6-11

The first ever DjangoCon Africa is taking place in Zanzibar, Tanzania with six days of talks, workshops, education, and collaboration with the international Django community.

djangocon.africa

Python Pizza Hamburg (November 17th)

A microconference filled with pizza and 10-minute talks sharing lessons and experiences from real-world projects.

python.pizza

DjangoCon US 2023: Durham, NC Photos

DjangoCon US 2023 photos are live and were taken by Bartek Pawlik who has taken photos at many Django conferences and events all over the world going back a decade.

flickr.com

Sponsored Ad

Sick of performance issues?

Enter Scout's APM tool for Python apps. Easily pinpoint and fix slowdowns with intelligent tracing logic. Optimize performance hassle-free, delighting your users. Try us out for free!

ter.li

Videos

Django Day CPH 2023: Paris Kasidiaris - Taming rich text content in Django

A talk on the challenges and opportunities of working with rich text in Django, and the development of the django-prose third-party package.

youtu.be

Django Day CPH 2023: Wilhelm Klopp on How to spend less time writing tests

This talk shares a novel approach to deterministically generating working integration tests from traces recorded using kolo. This reduces the time it takes to write tests overall and the often painful experience of setting up fixtures, test data, and database state.

youtu.be

VIDEO + Live Demo: Build ChatGPT-like Apps with AI

If you're interested in the practical applications of AI and Large Language Models (LLMs), you'll find value in this talk and live demo. The presentation goes beyond theory to include real-world examples and best practices, including a GitHub repository packed with Python code and ChatGPT-like app examples that will help you spin up your own app.

sixfeetup.com

Python Community News: DjangoCon US 2023 Perspectives

A playlist with six and counting videos of DjangoCon US participants discussing the conference and their work in the Django community.

youtu.be

Django News Jobs

A few weeks ago, we launched Django News Jobs and there two new jobs this week.

Software Engineer - Ubuntu Systems Management at Canonical 🆕

Front End Web UI Django Developer (NC) at Ansys 🆕

Junior Web Developer at The Westervelt Company

Django Girls Communications Officer at Django Girls

Django Girls Awesomeness Ambassador at Django Girls

Senior Python/Django Engineer at Apexive

Senior Python/Django Software Engineer at Brivo Systems

Django Newsletter

Projects

ambient-innovation/django-pony-express

Class-based emails for Django shipping with a test suite.

github.com

django-cms/django-sekizai: Django Template Blocks with extra functionality

Django Template Blocks with extra functionality.

github.com

valberg/django-htmx-admin

What if we put HTMX into the Django Admin?

github.com

This RSS feed is published on https://django-news.com/. You can also subscribe via email.

Weeknotes (2023 week 44)

Unmaintained but maintained packages

There’s a discussion going on in the django-mptt issue tracker about the maintenance state of django-mptt. I have marked the project as unmaintained in March 2021 and haven’t regretted this decision at all. I haven’t had to fix inconsistencies in the tree structure once since switching to django-tree-queries. And if that wasn’t enough, I get little but only warm and thankful feedback for the latter, so that’s extra nice.

Despite marking django-mptt as unmaintained I seem to be doing a little bit of maintenance still. I’m still using it in old paid projects and so the things I do to make the package work for me is paid work. I’m not personally invested in the package anymore, so I’m able to tell people that there are absolutely no guarantees about the maintenance, and that feels good.

Read the Docs

I do understand why the .readthedocs.yaml file is now necessary. I wish that I wouldn’t have to do all the busywork of adding one to projects. I have just resubscribed to the Read the Docs Gold Membership which probably has expired at some point in the past. Read the Docs is excellent and everybody who can should support them.

Releases

• feincms3 4.5, feincms3-sites 0.20 and feincms3-language-sites 0.3: Fixed the check which only allows adding an application through the CMS to the page tree (yes, that’s right) once; feincms3 worked fine, feincms3-language-sites by accident but feincms3-sites didn’t.
• towel 0.31: Towel is one of my oldest packages which is still being used in real-world projects. Towel is a tool for building CRUD-type applications and is designed to keep you DRY while doing that. The project has been heavily inspired by a Django-based agency software I built many years back. The package even has docs! I’m still quite proud of the mostly transparent support for multitenancy, but apart from that I haven’t used it in many new projects.

Langchain + Docker + Neo4j + Ollama

GenAI Stack

The GenAI Stack will get you started building your own GenAI application in no time. The demo applications can serve as inspiration or as a starting point. Learn more about the details in the technical blog post.

Configure

Create a .env file from the environment template file env.example

Available variables:

LLM Configuration

MacOS and Linux users can use any LLM that's available via Ollama. Check the "tags" section under the model page you want to use on https://ollama.ai/library and write the tag for the value of the environment variable LLM= in th e.env file. All platforms can use GPT-3.5-turbo and GPT-4 (bring your own API keys for OpenAIs models).

MacOS Install Ollama on MacOS and start it before running docker compose up.

Linux No need to install Ollama manually, it will run in a container as part of the stack when running with the Linux profile: run docker compose --profile linux up. Make sure to set the OLLAMA_BASE_URL=http://llm:11434 in the .env file when using Ollama docker container.

To use the Linux-GPU profile: run docker compose --profile linux-gpu up. Also change OLLAMA_BASE_URL=http://llm-gpu:11434 in the .env file.

Windows Not supported by Ollama, so Windows users need to generate a OpenAI API key and configure the stack to use gpt-3.5 or gpt-4 in the .env file.

Develop

[!WARNING] There is a performance issue that impacts python applications in the 4.24.x releases of Docker Desktop. Please upgrade to the latest release before using this stack.

To start everything

docker compose up

If changes to build scripts has been made, rebuild.

docker compose up --build

To enter watch mode (auto rebuild on file changes). First start everything, then in new terminal:

docker compose watch

Shutdown If health check fails or containers don't start up as expected, shutdown completely to start up again.

docker compose down

Applications

Here's what's in this repo:

The database can be explored at http://localhost:7474.

App 1 - Support Agent Bot

UI: http://localhost:8501 DB client: http://localhost:7474

• answer support question based on recent entries
• provide summarized answers with sources
• demonstrate difference between
  • RAG Disabled (pure LLM response)
  • RAG Enabled (vector + knowledge graph context)
• allow to generate a high quality support ticket for the current conversation based on the style of highly rated questions in the database.

(Chat input + RAG mode selector)

(CTA to auto generate support ticket draft)	(UI of the auto generated support ticket draft)

App 2 - Loader

UI: http://localhost:8502 DB client: http://localhost:7474

• import recent Stack Overflow data for certain tags into a KG
• embed questions and answers and store in vector index
• UI: choose tags, run import, see progress, some stats of data in the database
• Load high ranked questions (regardless of tags) to support the ticket generation feature of App 1.

App 3 Question / Answer with a local PDF

UI: http://localhost:8503
DB client: http://localhost:7474

This application lets you load a local PDF into text chunks and embed it into Neo4j so you can ask questions about its contents and have the LLM answer them using vector similarity search.

App 4 Standalone HTTP API

Endpoints:

• http://localhost:8504/query?text=hello&rag=false (non streaming)
• http://localhost:8504/query-stream?text=hello&rag=false (SSE streaming)

Example cURL command:

curl http://localhost:8504/query-stream\?text\=minimal%20hello%20world%20in%20python\&rag\=false

Exposes the functionality to answer questions in the same way as App 1 above. Uses same code and prompts.

App 5 Static front-end

UI: http://localhost:8505

This application has the same features as App 1, but is built separate from the back-end code using modern best practices (Vite, Svelte, Tailwind).
The auto-reload on changes are instant using the Docker watch sync config.

PyTorch image models, scripts, pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNet-V3/V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

PyTorch Image Models

• What's New
• Introduction
• Models
• Features
• Results
• Getting Started (Documentation)
• Train, Validation, Inference Scripts
• Awesome PyTorch Resources
• Licenses
• Citing

What's New

❗Updates after Oct 10, 2022 are available in version >= 0.9❗

• Many changes since the last 0.6.x stable releases. They were previewed in 0.8.x dev releases but not everyone transitioned.
• timm.models.layers moved to timm.layers:
  • from timm.models.layers import name will still work via deprecation mapping (but please transition to timm.layers).
  • import timm.models.layers.module or from timm.models.layers.module import name needs to be changed now.
• Builder, helper, non-model modules in timm.models have a _ prefix added, ie timm.models.helpers -> timm.models._helpers, there are temporary deprecation mapping files but those will be removed.
• All models now support architecture.pretrained_tag naming (ex resnet50.rsb_a1).
  • The pretrained_tag is the specific weight variant (different head) for the architecture.
  • Using only architecture defaults to the first weights in the default_cfgs for that model architecture.
  • In adding pretrained tags, many model names that existed to differentiate were renamed to use the tag (ex: vit_base_patch16_224_in21k -> vit_base_patch16_224.augreg_in21k). There are deprecation mappings for these.
• A number of models had their checkpoints remaped to match architecture changes needed to better support features_only=True, there are checkpoint_filter_fn methods in any model module that was remapped. These can be passed to timm.models.load_checkpoint(..., filter_fn=timm.models.swin_transformer_v2.checkpoint_filter_fn) to remap your existing checkpoint.
• The Hugging Face Hub (https://huggingface.co/timm) is now the primary source for timm weights. Model cards include link to papers, original source, license.
• Previous 0.6.x can be cloned from 0.6.x branch or installed via pip with version.

Nov 3, 2023

• DFN (Data Filtering Networks) and MetaCLIP ViT weights added
• DINOv2 'register' ViT model weights added
• Add quickgelu ViT variants for OpenAI, DFN, MetaCLIP weights that use it (less efficient)
• Improved typing added to ResNet, MobileNet-v3 thanks to Aryan
• ImageNet-12k fine-tuned (from LAION-2B CLIP) convnext_xxlarge
• 0.9.9 release

Oct 20, 2023

• SigLIP image tower weights supported in vision_transformer.py.
  • Great potential for fine-tune and downstream feature use.
• Experimental 'register' support in vit models as per Vision Transformers Need Registers
• Updated RepViT with new weight release. Thanks wangao
• Add patch resizing support (on pretrained weight load) to Swin models
• 0.9.8 release pending

Sep 1, 2023

• TinyViT added by SeeFun
• Fix EfficientViT (MIT) to use torch.autocast so it works back to PT 1.10
• 0.9.7 release

Aug 28, 2023

• Add dynamic img size support to models in vision_transformer.py, vision_transformer_hybrid.py, deit.py, and eva.py w/o breaking backward compat.
  • Add dynamic_img_size=True to args at model creation time to allow changing the grid size (interpolate abs and/or ROPE pos embed each forward pass).
  • Add dynamic_img_pad=True to allow image sizes that aren't divisible by patch size (pad bottom right to patch size each forward pass).
  • Enabling either dynamic mode will break FX tracing unless PatchEmbed module added as leaf.
  • Existing method of resizing position embedding by passing different img_size (interpolate pretrained embed weights once) on creation still works.
  • Existing method of changing patch_size (resize pretrained patch_embed weights once) on creation still works.
  • Example validation cmd python validate.py /imagenet --model vit_base_patch16_224 --amp --amp-dtype bfloat16 --img-size 255 --crop-pct 1.0 --model-kwargs dynamic_img_size=True dyamic_img_pad=True

Aug 25, 2023

• Many new models since last release
  • FastViT - https://arxiv.org/abs/2303.14189
  • MobileOne - https://arxiv.org/abs/2206.04040
  • InceptionNeXt - https://arxiv.org/abs/2303.16900
  • RepGhostNet - https://arxiv.org/abs/2211.06088 (thanks https://github.com/ChengpengChen)
  • GhostNetV2 - https://arxiv.org/abs/2211.12905 (thanks https://github.com/yehuitang)
  • EfficientViT (MSRA) - https://arxiv.org/abs/2305.07027 (thanks https://github.com/seefun)
  • EfficientViT (MIT) - https://arxiv.org/abs/2205.14756 (thanks https://github.com/seefun)
• Add --reparam arg to benchmark.py, onnx_export.py, and validate.py to trigger layer reparameterization / fusion for models with any one of reparameterize(), switch_to_deploy() or fuse()
  • Including FastViT, MobileOne, RepGhostNet, EfficientViT (MSRA), RepViT, RepVGG, and LeViT
• Preparing 0.9.6 'back to school' release

Aug 11, 2023

• Swin, MaxViT, CoAtNet, and BEiT models support resizing of image/window size on creation with adaptation of pretrained weights
• Example validation cmd to test w/ non-square resize python validate.py /imagenet --model swin_base_patch4_window7_224.ms_in22k_ft_in1k --amp --amp-dtype bfloat16 --input-size 3 256 320 --model-kwargs window_size=8,10 img_size=256,320

Aug 3, 2023

• Add GluonCV weights for HRNet w18_small and w18_small_v2. Converted by SeeFun
• Fix selecsls* model naming regression
• Patch and position embedding for ViT/EVA works for bfloat16/float16 weights on load (or activations for on-the-fly resize)
• v0.9.5 release prep

July 27, 2023

• Added timm trained seresnextaa201d_32x8d.sw_in12k_ft_in1k_384 weights (and .sw_in12k pretrain) with 87.3% top-1 on ImageNet-1k, best ImageNet ResNet family model I'm aware of.
• RepViT model and weights (https://arxiv.org/abs/2307.09283) added by wangao
• I-JEPA ViT feature weights (no classifier) added by SeeFun
• SAM-ViT (segment anything) feature weights (no classifier) added by SeeFun
• Add support for alternative feat extraction methods and -ve indices to EfficientNet
• Add NAdamW optimizer
• Misc fixes

May 11, 2023

• timm 0.9 released, transition from 0.8.xdev releases

May 10, 2023

• Hugging Face Hub downloading is now default, 1132 models on https://huggingface.co/timm, 1163 weights in timm
• DINOv2 vit feature backbone weights added thanks to Leng Yue
• FB MAE vit feature backbone weights added
• OpenCLIP DataComp-XL L/14 feat backbone weights added
• MetaFormer (poolformer-v2, caformer, convformer, updated poolformer (v1)) w/ weights added by Fredo Guan
• Experimental get_intermediate_layers function on vit/deit models for grabbing hidden states (inspired by DINO impl). This is WIP and may change significantly... feedback welcome.
• Model creation throws error if pretrained=True and no weights exist (instead of continuing with random initialization)
• Fix regression with inception / nasnet TF sourced weights with 1001 classes in original classifiers
• bitsandbytes (https://github.com/TimDettmers/bitsandbytes) optimizers added to factory, use bnb prefix, ie bnbadam8bit
• Misc cleanup and fixes
• Final testing before switching to a 0.9 and bringing timm out of pre-release state

April 27, 2023

• 97% of timm models uploaded to HF Hub and almost all updated to support multi-weight pretrained configs
• Minor cleanup and refactoring of another batch of models as multi-weight added. More fused_attn (F.sdpa) and features_only support, and torchscript fixes.

April 21, 2023

• Gradient accumulation support added to train script and tested (--grad-accum-steps), thanks Taeksang Kim
• More weights on HF Hub (cspnet, cait, volo, xcit, tresnet, hardcorenas, densenet, dpn, vovnet, xception_aligned)
• Added --head-init-scale and --head-init-bias to train.py to scale classiifer head and set fixed bias for fine-tune
• Remove all InplaceABN (inplace_abn) use, replaced use in tresnet with standard BatchNorm (modified weights accordingly).

April 12, 2023

• Add ONNX export script, validate script, helpers that I've had kicking around for along time. Tweak 'same' padding for better export w/ recent ONNX + pytorch.
• Refactor dropout args for vit and vit-like models, separate drop_rate into drop_rate (classifier dropout), proj_drop_rate (block mlp / out projections), pos_drop_rate (position embedding drop), attn_drop_rate (attention dropout). Also add patch dropout (FLIP) to vit and eva models.
• fused F.scaled_dot_product_attention support to more vit models, add env var (TIMM_FUSED_ATTN) to control, and config interface to enable/disable
• Add EVA-CLIP backbones w/ image tower weights, all the way up to 4B param 'enormous' model, and 336x336 OpenAI ViT mode that was missed.

April 5, 2023

• ALL ResNet models pushed to Hugging Face Hub with multi-weight support
  • All past timm trained weights added with recipe based tags to differentiate
  • All ResNet strikes back A1/A2/A3 (seed 0) and R50 example B/C1/C2/D weights available
  • Add torchvision v2 recipe weights to existing torchvision originals
  • See comparison table in https://huggingface.co/timm/seresnextaa101d_32x8d.sw_in12k_ft_in1k_288#model-comparison
• New ImageNet-12k + ImageNet-1k fine-tunes available for a few anti-aliased ResNet models
  • resnetaa50d.sw_in12k_ft_in1k - 81.7 @ 224, 82.6 @ 288
  • resnetaa101d.sw_in12k_ft_in1k - 83.5 @ 224, 84.1 @ 288
  • seresnextaa101d_32x8d.sw_in12k_ft_in1k - 86.0 @ 224, 86.5 @ 288
  • seresnextaa101d_32x8d.sw_in12k_ft_in1k_288 - 86.5 @ 288, 86.7 @ 320

March 31, 2023

• Add first ConvNext-XXLarge CLIP -> IN-1k fine-tune and IN-12k intermediate fine-tunes for convnext-base/large CLIP models.

• Add EVA-02 MIM pretrained and fine-tuned weights, push to HF hub and update model cards for all EVA models. First model over 90% top-1 (99% top-5)! Check out the original code & weights at https://github.com/baaivision/EVA for more details on their work blending MIM, CLIP w/ many model, dataset, and train recipe tweaks.

• Multi-weight and HF hub for DeiT and MLP-Mixer based models

March 22, 2023

• More weights pushed to HF hub along with multi-weight support, including: regnet.py, rexnet.py, byobnet.py, resnetv2.py, swin_transformer.py, swin_transformer_v2.py, swin_transformer_v2_cr.py
• Swin Transformer models support feature extraction (NCHW feat maps for swinv2_cr_*, and NHWC for all others) and spatial embedding outputs.
• FocalNet (from https://github.com/microsoft/FocalNet) models and weights added with significant refactoring, feature extraction, no fixed resolution / sizing constraint
• RegNet weights increased with HF hub push, SWAG, SEER, and torchvision v2 weights. SEER is pretty poor wrt to performance for model size, but possibly useful.
• More ImageNet-12k pretrained and 1k fine-tuned timm weights:
  • rexnetr_200.sw_in12k_ft_in1k - 82.6 @ 224, 83.2 @ 288
  • rexnetr_300.sw_in12k_ft_in1k - 84.0 @ 224, 84.5 @ 288
  • regnety_120.sw_in12k_ft_in1k - 85.0 @ 224, 85.4 @ 288
  • regnety_160.lion_in12k_ft_in1k - 85.6 @ 224, 86.0 @ 288
  • regnety_160.sw_in12k_ft_in1k - 85.6 @ 224, 86.0 @ 288 (compare to SWAG PT + 1k FT this is same BUT much lower res, blows SEER FT away)
• Model name deprecation + remapping functionality added (a milestone for bringing 0.8.x out of pre-release). Mappings being added...
• Minor bug fixes and improvements.

Feb 26, 2023

• Add ConvNeXt-XXLarge CLIP pretrained image tower weights for fine-tune & features (fine-tuning TBD) -- see model card
• Update convnext_xxlarge default LayerNorm eps to 1e-5 (for CLIP weights, improved stability)
• 0.8.15dev0

Feb 20, 2023

• Add 320x320 convnext_large_mlp.clip_laion2b_ft_320 and convnext_lage_mlp.clip_laion2b_ft_soup_320 CLIP image tower weights for features & fine-tune
• 0.8.13dev0 pypi release for latest changes w/ move to huggingface org

Feb 16, 2023

• safetensor checkpoint support added
• Add ideas from 'Scaling Vision Transformers to 22 B. Params' (https://arxiv.org/abs/2302.05442) -- qk norm, RmsNorm, parallel block
• Add F.scaled_dot_product_attention support (PyTorch 2.0 only) to vit_*, vit_relpos*, coatnet / maxxvit (to start)
• Lion optimizer (w/ multi-tensor option) added (https://arxiv.org/abs/2302.06675)
• gradient checkpointing works with features_only=True

Feb 7, 2023

• New inference benchmark numbers added in results folder.
• Add convnext LAION CLIP trained weights and initial set of in1k fine-tunes
  • convnext_base.clip_laion2b_augreg_ft_in1k - 86.2% @ 256x256
  • convnext_base.clip_laiona_augreg_ft_in1k_384 - 86.5% @ 384x384
  • convnext_large_mlp.clip_laion2b_augreg_ft_in1k - 87.3% @ 256x256
  • convnext_large_mlp.clip_laion2b_augreg_ft_in1k_384 - 87.9% @ 384x384
• Add DaViT models. Supports features_only=True. Adapted from https://github.com/dingmyu/davit by Fredo.
• Use a common NormMlpClassifierHead across MaxViT, ConvNeXt, DaViT
• Add EfficientFormer-V2 model, update EfficientFormer, and refactor LeViT (closely related architectures). Weights on HF hub.
  • New EfficientFormer-V2 arch, significant refactor from original at (https://github.com/snap-research/EfficientFormer). Supports features_only=True.
  • Minor updates to EfficientFormer.
  • Refactor LeViT models to stages, add features_only=True support to new conv variants, weight remap required.
• Move ImageNet meta-data (synsets, indices) from /results to timm/data/_info.
• Add ImageNetInfo / DatasetInfo classes to provide labelling for various ImageNet classifier layouts in timm
  • Update inference.py to use, try: python inference.py /folder/to/images --model convnext_small.in12k --label-type detail --topk 5
• Ready for 0.8.10 pypi pre-release (final testing).

Jan 20, 2023

• Add two convnext 12k -> 1k fine-tunes at 384x384

  • convnext_tiny.in12k_ft_in1k_384 - 85.1 @ 384
  • convnext_small.in12k_ft_in1k_384 - 86.2 @ 384

• Push all MaxxViT weights to HF hub, and add new ImageNet-12k -> 1k fine-tunes for rw base MaxViT and CoAtNet 1/2 models

Jan 11, 2023

• Update ConvNeXt ImageNet-12k pretrain series w/ two new fine-tuned weights (and pre FT .in12k tags)
  • convnext_nano.in12k_ft_in1k - 82.3 @ 224, 82.9 @ 288 (previously released)
  • convnext_tiny.in12k_ft_in1k - 84.2 @ 224, 84.5 @ 288
  • convnext_small.in12k_ft_in1k - 85.2 @ 224, 85.3 @ 288

Jan 6, 2023

• Finally got around to adding --model-kwargs and --opt-kwargs to scripts to pass through rare args directly to model classes from cmd line
  • train.py /imagenet --model resnet50 --amp --model-kwargs output_stride=16 act_layer=silu
  • train.py /imagenet --model vit_base_patch16_clip_224 --img-size 240 --amp --model-kwargs img_size=240 patch_size=12
• Cleanup some popular models to better support arg passthrough / merge with model configs, more to go.

Jan 5, 2023

• ConvNeXt-V2 models and weights added to existing convnext.py
  • Paper: ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders
  • Reference impl: https://github.com/facebookresearch/ConvNeXt-V2 (NOTE: weights currently CC-BY-NC)

Dec 23, 2022 🎄☃

• Add FlexiViT models and weights from https://github.com/google-research/big_vision (check out paper at https://arxiv.org/abs/2212.08013)
  • NOTE currently resizing is static on model creation, on-the-fly dynamic / train patch size sampling is a WIP
• Many more models updated to multi-weight and downloadable via HF hub now (convnext, efficientnet, mobilenet, vision_transformer*, beit)
• More model pretrained tag and adjustments, some model names changed (working on deprecation translations, consider main branch DEV branch right now, use 0.6.x for stable use)
• More ImageNet-12k (subset of 22k) pretrain models popping up:
  • efficientnet_b5.in12k_ft_in1k - 85.9 @ 448x448
  • vit_medium_patch16_gap_384.in12k_ft_in1k - 85.5 @ 384x384
  • vit_medium_patch16_gap_256.in12k_ft_in1k - 84.5 @ 256x256
  • convnext_nano.in12k_ft_in1k - 82.9 @ 288x288

Dec 8, 2022

• Add 'EVA l' to vision_transformer.py, MAE style ViT-L/14 MIM pretrain w/ EVA-CLIP targets, FT on ImageNet-1k (w/ ImageNet-22k intermediate for some)
  • original source: https://github.com/baaivision/EVA

Dec 6, 2022

• Add 'EVA g', BEiT style ViT-g/14 model weights w/ both MIM pretrain and CLIP pretrain to beit.py.
  • original source: https://github.com/baaivision/EVA
  • paper: https://arxiv.org/abs/2211.07636

Dec 5, 2022

• Pre-release (0.8.0dev0) of multi-weight support (model_arch.pretrained_tag). Install with pip install --pre timm
  • vision_transformer, maxvit, convnext are the first three model impl w/ support
  • model names are changing with this (previous _21k, etc. fn will merge), still sorting out deprecation handling
  • bugs are likely, but I need feedback so please try it out
  • if stability is needed, please use 0.6.x pypi releases or clone from 0.6.x branch
• Support for PyTorch 2.0 compile is added in train/validate/inference/benchmark, use --torchcompile argument
• Inference script allows more control over output, select k for top-class index + prob json, csv or parquet output
• Add a full set of fine-tuned CLIP image tower weights from both LAION-2B and original OpenAI CLIP models

• Port of MaxViT Tensorflow Weights from official impl at https://github.com/google-research/maxvit
  • There was larger than expected drops for the upscaled 384/512 in21k fine-tune weights, possible detail missing, but the 21k FT did seem sensitive to small preprocessing

Oct 15, 2022

• Train and validation script enhancements
• Non-GPU (ie CPU) device support
• SLURM compatibility for train script
• HF datasets support (via ReaderHfds)
• TFDS/WDS dataloading improvements (sample padding/wrap for distributed use fixed wrt sample count estimate)
• in_chans !=3 support for scripts / loader
• Adan optimizer
• Can enable per-step LR scheduling via args
• Dataset 'parsers' renamed to 'readers', more descriptive of purpose
• AMP args changed, APEX via --amp-impl apex, bfloat16 supportedf via --amp-dtype bfloat16
• main branch switched to 0.7.x version, 0.6x forked for stable release of weight only adds
• master -> main branch rename

Oct 10, 2022

• More weights in maxxvit series, incl first ConvNeXt block based coatnext and maxxvit experiments:
  • coatnext_nano_rw_224 - 82.0 @ 224 (G) -- (uses ConvNeXt conv block, no BatchNorm)
  • maxxvit_rmlp_nano_rw_256 - 83.0 @ 256, 83.7 @ 320 (G) (uses ConvNeXt conv block, no BN)
  • maxvit_rmlp_small_rw_224 - 84.5 @ 224, 85.1 @ 320 (G)
  • maxxvit_rmlp_small_rw_256 - 84.6 @ 256, 84.9 @ 288 (G) -- could be trained better, hparams need tuning (uses ConvNeXt block, no BN)
  • coatnet_rmlp_2_rw_224 - 84.6 @ 224, 85 @ 320 (T)
  • NOTE: official MaxVit weights (in1k) have been released at https://github.com/google-research/maxvit -- some extra work is needed to port and adapt since my impl was created independently of theirs and has a few small differences + the whole TF same padding fun.

Sept 23, 2022

• LAION-2B CLIP image towers supported as pretrained backbones for fine-tune or features (no classifier)
  • vit_base_patch32_224_clip_laion2b
  • vit_large_patch14_224_clip_laion2b
  • vit_huge_patch14_224_clip_laion2b
  • vit_giant_patch14_224_clip_laion2b

Sept 7, 2022

• Hugging Face timm docs home now exists, look for more here in the future
• Add BEiT-v2 weights for base and large 224x224 models from https://github.com/microsoft/unilm/tree/master/beit2
• Add more weights in maxxvit series incl a pico (7.5M params, 1.9 GMACs), two tiny variants:
  • maxvit_rmlp_pico_rw_256 - 80.5 @ 256, 81.3 @ 320 (T)
  • maxvit_tiny_rw_224 - 83.5 @ 224 (G)
  • maxvit_rmlp_tiny_rw_256 - 84.2 @ 256, 84.8 @ 320 (T)

Introduction

PyTorch Image Models (timm) is a collection of image models, layers, utilities, optimizers, schedulers, data-loaders / augmentations, and reference training / validation scripts that aim to pull together a wide variety of SOTA models with ability to reproduce ImageNet training results.

The work of many others is present here. I've tried to make sure all source material is acknowledged via links to github, arxiv papers, etc in the README, documentation, and code docstrings. Please let me know if I missed anything.

Models

All model architecture families include variants with pretrained weights. There are specific model variants without any weights, it is NOT a bug. Help training new or better weights is always appreciated.

• Aggregating Nested Transformers - https://arxiv.org/abs/2105.12723
• BEiT - https://arxiv.org/abs/2106.08254
• Big Transfer ResNetV2 (BiT) - https://arxiv.org/abs/1912.11370
• Bottleneck Transformers - https://arxiv.org/abs/2101.11605
• CaiT (Class-Attention in Image Transformers) - https://arxiv.org/abs/2103.17239
• CoaT (Co-Scale Conv-Attentional Image Transformers) - https://arxiv.org/abs/2104.06399
• CoAtNet (Convolution and Attention) - https://arxiv.org/abs/2106.04803
• ConvNeXt - https://arxiv.org/abs/2201.03545
• ConvNeXt-V2 - http://arxiv.org/abs/2301.00808
• ConViT (Soft Convolutional Inductive Biases Vision Transformers)- https://arxiv.org/abs/2103.10697
• CspNet (Cross-Stage Partial Networks) - https://arxiv.org/abs/1911.11929
• DeiT - https://arxiv.org/abs/2012.12877
• DeiT-III - https://arxiv.org/pdf/2204.07118.pdf
• DenseNet - https://arxiv.org/abs/1608.06993
• DLA - https://arxiv.org/abs/1707.06484
• DPN (Dual-Path Network) - https://arxiv.org/abs/1707.01629
• EdgeNeXt - https://arxiv.org/abs/2206.10589
• EfficientFormer - https://arxiv.org/abs/2206.01191
• EfficientNet (MBConvNet Family)
  • EfficientNet NoisyStudent (B0-B7, L2) - https://arxiv.org/abs/1911.04252
  • EfficientNet AdvProp (B0-B8) - https://arxiv.org/abs/1911.09665
  • EfficientNet (B0-B7) - https://arxiv.org/abs/1905.11946
  • EfficientNet-EdgeTPU (S, M, L) - https://ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html
  • EfficientNet V2 - https://arxiv.org/abs/2104.00298
  • FBNet-C - https://arxiv.org/abs/1812.03443
  • MixNet - https://arxiv.org/abs/1907.09595
  • MNASNet B1, A1 (Squeeze-Excite), and Small - https://arxiv.org/abs/1807.11626
  • MobileNet-V2 - https://arxiv.org/abs/1801.04381
  • Single-Path NAS - https://arxiv.org/abs/1904.02877
  • TinyNet - https://arxiv.org/abs/2010.14819
• EfficientViT (MIT) - https://arxiv.org/abs/2205.14756
• EfficientViT (MSRA) - https://arxiv.org/abs/2305.07027
• EVA - https://arxiv.org/abs/2211.07636
• EVA-02 - https://arxiv.org/abs/2303.11331
• FastViT - https://arxiv.org/abs/2303.14189
• FlexiViT - https://arxiv.org/abs/2212.08013
• FocalNet (Focal Modulation Networks) - https://arxiv.org/abs/2203.11926
• GCViT (Global Context Vision Transformer) - https://arxiv.org/abs/2206.09959
• GhostNet - https://arxiv.org/abs/1911.11907
• GhostNet-V2 - https://arxiv.org/abs/2211.12905
• gMLP - https://arxiv.org/abs/2105.08050
• GPU-Efficient Networks - https://arxiv.org/abs/2006.14090
• Halo Nets - https://arxiv.org/abs/2103.12731
• HRNet - https://arxiv.org/abs/1908.07919
• InceptionNeXt - https://arxiv.org/abs/2303.16900
• Inception-V3 - https://arxiv.org/abs/1512.00567
• Inception-ResNet-V2 and Inception-V4 - https://arxiv.org/abs/1602.07261
• Lambda Networks - https://arxiv.org/abs/2102.08602
• LeViT (Vision Transformer in ConvNet's Clothing) - https://arxiv.org/abs/2104.01136
• MaxViT (Multi-Axis Vision Transformer) - https://arxiv.org/abs/2204.01697
• MetaFormer (PoolFormer-v2, ConvFormer, CAFormer) - https://arxiv.org/abs/2210.13452
• MLP-Mixer - https://arxiv.org/abs/2105.01601
• MobileNet-V3 (MBConvNet w/ Efficient Head) - https://arxiv.org/abs/1905.02244
  • FBNet-V3 - https://arxiv.org/abs/2006.02049
  • HardCoRe-NAS - https://arxiv.org/abs/2102.11646
  • LCNet - https://arxiv.org/abs/2109.15099
• MobileOne - https://arxiv.org/abs/2206.04040
• MobileViT - https://arxiv.org/abs/2110.02178
• MobileViT-V2 - https://arxiv.org/abs/2206.02680
• MViT-V2 (Improved Multiscale Vision Transformer) - https://arxiv.org/abs/2112.01526
• NASNet-A - https://arxiv.org/abs/1707.07012
• NesT - https://arxiv.org/abs/2105.12723
• NFNet-F - https://arxiv.org/abs/2102.06171
• NF-RegNet / NF-ResNet - https://arxiv.org/abs/2101.08692
• PNasNet - https://arxiv.org/abs/1712.00559
• PoolFormer (MetaFormer) - https://arxiv.org/abs/2111.11418
• Pooling-based Vision Transformer (PiT) - https://arxiv.org/abs/2103.16302
• PVT-V2 (Improved Pyramid Vision Transformer) - https://arxiv.org/abs/2106.13797
• RegNet - https://arxiv.org/abs/2003.13678
• RegNetZ - https://arxiv.org/abs/2103.06877
• RepVGG - https://arxiv.org/abs/2101.03697
• RepGhostNet - https://arxiv.org/abs/2211.06088
• RepViT - https://arxiv.org/abs/2307.09283
• ResMLP - https://arxiv.org/abs/2105.03404
• ResNet/ResNeXt
  • ResNet (v1b/v1.5) - https://arxiv.org/abs/1512.03385
  • ResNeXt - https://arxiv.org/abs/1611.05431
  • 'Bag of Tricks' / Gluon C, D, E, S variations - https://arxiv.org/abs/1812.01187
  • Weakly-supervised (WSL) Instagram pretrained / ImageNet tuned ResNeXt101 - https://arxiv.org/abs/1805.00932
  • Semi-supervised (SSL) / Semi-weakly Supervised (SWSL) ResNet/ResNeXts - https://arxiv.org/abs/1905.00546
  • ECA-Net (ECAResNet) - https://arxiv.org/abs/1910.03151v4
  • Squeeze-and-Excitation Networks (SEResNet) - https://arxiv.org/abs/1709.01507
  • ResNet-RS - https://arxiv.org/abs/2103.07579
• Res2Net - https://arxiv.org/abs/1904.01169
• ResNeSt - https://arxiv.org/abs/2004.08955
• ReXNet - https://arxiv.org/abs/2007.00992
• SelecSLS - https://arxiv.org/abs/1907.00837
• Selective Kernel Networks - https://arxiv.org/abs/1903.06586
• Sequencer2D - https://arxiv.org/abs/2205.01972
• Swin S3 (AutoFormerV2) - https://arxiv.org/abs/2111.14725
• Swin Transformer - https://arxiv.org/abs/2103.14030
• Swin Transformer V2 - https://arxiv.org/abs/2111.09883
• Transformer-iN-Transformer (TNT) - https://arxiv.org/abs/2103.00112
• TResNet - https://arxiv.org/abs/2003.13630
• Twins (Spatial Attention in Vision Transformers) - https://arxiv.org/pdf/2104.13840.pdf
• Visformer - https://arxiv.org/abs/2104.12533
• Vision Transformer - https://arxiv.org/abs/2010.11929
• VOLO (Vision Outlooker) - https://arxiv.org/abs/2106.13112
• VovNet V2 and V1 - https://arxiv.org/abs/1911.06667
• Xception - https://arxiv.org/abs/1610.02357
• Xception (Modified Aligned, Gluon) - https://arxiv.org/abs/1802.02611
• Xception (Modified Aligned, TF) - https://arxiv.org/abs/1802.02611
• XCiT (Cross-Covariance Image Transformers) - https://arxiv.org/abs/2106.09681

Features

Several (less common) features that I often utilize in my projects are included. Many of their additions are the reason why I maintain my own set of models, instead of using others' via PIP:

• All models have a common default configuration interface and API for
  • accessing/changing the classifier - get_classifier and reset_classifier
  • doing a forward pass on just the features - forward_features (see documentation)
  • these makes it easy to write consistent network wrappers that work with any of the models
• All models support multi-scale feature map extraction (feature pyramids) via create_model (see documentation)
  • create_model(name, features_only=True, out_indices=..., output_stride=...)
  • out_indices creation arg specifies which feature maps to return, these indices are 0 based and generally correspond to the C(i + 1) feature level.
  • output_stride creation arg controls output stride of the network by using dilated convolutions. Most networks are stride 32 by default. Not all networks support this.
  • feature map channel counts, reduction level (stride) can be queried AFTER model creation via the .feature_info member
• All models have a consistent pretrained weight loader that adapts last linear if necessary, and from 3 to 1 channel input if desired
• High performance reference training, validation, and inference scripts that work in several process/GPU modes:
  • NVIDIA DDP w/ a single GPU per process, multiple processes with APEX present (AMP mixed-precision optional)
  • PyTorch DistributedDataParallel w/ multi-gpu, single process (AMP disabled as it crashes when enabled)
  • PyTorch w/ single GPU single process (AMP optional)
• A dynamic global pool implementation that allows selecting from average pooling, max pooling, average + max, or concat([average, max]) at model creation. All global pooling is adaptive average by default and compatible with pretrained weights.
• A 'Test Time Pool' wrapper that can wrap any of the included models and usually provides improved performance doing inference with input images larger than the training size. Idea adapted from original DPN implementation when I ported (https://github.com/cypw/DPNs)
• Learning rate schedulers
  • Ideas adopted from
    • AllenNLP schedulers
    • FAIRseq lr_scheduler
    • SGDR: Stochastic Gradient Descent with Warm Restarts (https://arxiv.org/abs/1608.03983)
  • Schedulers include step, cosine w/ restarts, tanh w/ restarts, plateau
• Optimizers:
  • rmsprop_tf adapted from PyTorch RMSProp by myself. Reproduces much improved Tensorflow RMSProp behaviour.
  • radam by Liyuan Liu (https://arxiv.org/abs/1908.03265)
  • novograd by Masashi Kimura (https://arxiv.org/abs/1905.11286)
  • lookahead adapted from impl by Liam (https://arxiv.org/abs/1907.08610)
  • fused<name> optimizers by name with NVIDIA Apex installed
  • adamp and sgdp by Naver ClovAI (https://arxiv.org/abs/2006.08217)
  • adafactor adapted from FAIRSeq impl (https://arxiv.org/abs/1804.04235)
  • adahessian by David Samuel (https://arxiv.org/abs/2006.00719)
• Random Erasing from Zhun Zhong (https://arxiv.org/abs/1708.04896)
• Mixup (https://arxiv.org/abs/1710.09412)
• CutMix (https://arxiv.org/abs/1905.04899)
• AutoAugment (https://arxiv.org/abs/1805.09501) and RandAugment (https://arxiv.org/abs/1909.13719) ImageNet configurations modeled after impl for EfficientNet training (https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py)
• AugMix w/ JSD loss (https://arxiv.org/abs/1912.02781), JSD w/ clean + augmented mixing support works with AutoAugment and RandAugment as well
• SplitBachNorm - allows splitting batch norm layers between clean and augmented (auxiliary batch norm) data
• DropPath aka "Stochastic Depth" (https://arxiv.org/abs/1603.09382)
• DropBlock (https://arxiv.org/abs/1810.12890)
• Blur Pooling (https://arxiv.org/abs/1904.11486)
• Space-to-Depth by mrT23 (https://arxiv.org/abs/1801.04590) -- original paper?
• Adaptive Gradient Clipping (https://arxiv.org/abs/2102.06171, https://github.com/deepmind/deepmind-research/tree/master/nfnets)
• An extensive selection of channel and/or spatial attention modules:
  • Bottleneck Transformer - https://arxiv.org/abs/2101.11605
  • CBAM - https://arxiv.org/abs/1807.06521
  • Effective Squeeze-Excitation (ESE) - https://arxiv.org/abs/1911.06667
  • Efficient Channel Attention (ECA) - https://arxiv.org/abs/1910.03151
  • Gather-Excite (GE) - https://arxiv.org/abs/1810.12348
  • Global Context (GC) - https://arxiv.org/abs/1904.11492
  • Halo - https://arxiv.org/abs/2103.12731
  • Involution - https://arxiv.org/abs/2103.06255
  • Lambda Layer - https://arxiv.org/abs/2102.08602
  • Non-Local (NL) - https://arxiv.org/abs/1711.07971
  • Squeeze-and-Excitation (SE) - https://arxiv.org/abs/1709.01507
  • Selective Kernel (SK) - (https://arxiv.org/abs/1903.06586
  • Split (SPLAT) - https://arxiv.org/abs/2004.08955
  • Shifted Window (SWIN) - https://arxiv.org/abs/2103.14030

Results

Model validation results can be found in the results tables

Getting Started (Documentation)

The official documentation can be found at https://huggingface.co/docs/hub/timm. Documentation contributions are welcome.

Getting Started with PyTorch Image Models (timm): A Practitioner’s Guide by Chris Hughes is an extensive blog post covering many aspects of timm in detail.

timmdocs is an alternate set of documentation for timm. A big thanks to Aman Arora for his efforts creating timmdocs.

paperswithcode is a good resource for browsing the models within timm.

Train, Validation, Inference Scripts

The root folder of the repository contains reference train, validation, and inference scripts that work with the included models and other features of this repository. They are adaptable for other datasets and use cases with a little hacking. See documentation.

Awesome PyTorch Resources

One of the greatest assets of PyTorch is the community and their contributions. A few of my favourite resources that pair well with the models and components here are listed below.

Object Detection, Instance and Semantic Segmentation

• Detectron2 - https://github.com/facebookresearch/detectron2
• Segmentation Models (Semantic) - https://github.com/qubvel/segmentation_models.pytorch
• EfficientDet (Obj Det, Semantic soon) - https://github.com/rwightman/efficientdet-pytorch

Computer Vision / Image Augmentation

• Albumentations - https://github.com/albumentations-team/albumentations
• Kornia - https://github.com/kornia/kornia

Knowledge Distillation

• RepDistiller - https://github.com/HobbitLong/RepDistiller
• torchdistill - https://github.com/yoshitomo-matsubara/torchdistill

Metric Learning

• PyTorch Metric Learning - https://github.com/KevinMusgrave/pytorch-metric-learning

Training / Frameworks

• fastai - https://github.com/fastai/fastai

Licenses

Code

The code here is licensed Apache 2.0. I've taken care to make sure any third party code included or adapted has compatible (permissive) licenses such as MIT, BSD, etc. I've made an effort to avoid any GPL / LGPL conflicts. That said, it is your responsibility to ensure you comply with licenses here and conditions of any dependent licenses. Where applicable, I've linked the sources/references for various components in docstrings. If you think I've missed anything please create an issue.

Pretrained Weights

So far all of the pretrained weights available here are pretrained on ImageNet with a select few that have some additional pretraining (see extra note below). ImageNet was released for non-commercial research purposes only (https://image-net.org/download). It's not clear what the implications of that are for the use of pretrained weights from that dataset. Any models I have trained with ImageNet are done for research purposes and one should assume that the original dataset license applies to the weights. It's best to seek legal advice if you intend to use the pretrained weights in a commercial product.

Pretrained on more than ImageNet

Several weights included or references here were pretrained with proprietary datasets that I do not have access to. These include the Facebook WSL, SSL, SWSL ResNe(Xt) and the Google Noisy Student EfficientNet models. The Facebook models have an explicit non-commercial license (CC-BY-NC 4.0, https://github.com/facebookresearch/semi-supervised-ImageNet1K-models, https://github.com/facebookresearch/WSL-Images). The Google models do not appear to have any restriction beyond the Apache 2.0 license (and ImageNet concerns). In either case, you should contact Facebook or Google with any questions.

Citing

BibTeX

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
}

Latest DOI

Revolutionizing Database Interactions with Private LLM Technology

DB-GPT: Revolutionizing Database Interactions with Private LLM Technology

What is DB-GPT?

DB-GPT is an experimental open-source project that uses localized GPT large models to interact with your data and environment. With this solution, you can be assured that there is no risk of data leakage, and your data is 100% private and secure.

Contents

• Install
• Demo
• introduction
• features
• contribution
• roadmap
• contact

DB-GPT Youtube Video

Demo

Run on an RTX 4090 GPU.

Chat Excel

Chat Plugin

LLM Management

FastChat && vLLM

Trace

Chat Knowledge

Install

Usage Tutorial

• Install
  • Install Step by Step
  • Docker Install
  • Docker Compose
• How to Use
  • ChatData
  • ChatKnowledge
  • ChatExcel
  • Dashboard
  • LLM Management
  • Chat Agent
• How to Deploy LLM
  • Standalone
  • Cluster
  • vLLM
• How to Debug
• FAQ

Features

Currently, we have released multiple key features, which are listed below to demonstrate our current capabilities:

• Private KBQA & data processing

  The DB-GPT project offers a range of features to enhance knowledge base construction and enable efficient storage and retrieval of both structured and unstructured data. These include built-in support for uploading multiple file formats, the ability to integrate plug-ins for custom data extraction, and unified vector storage and retrieval capabilities for managing large volumes of information.

• Multiple data sources & visualization

  The DB-GPT project enables seamless natural language interaction with various data sources, including Excel, databases, and data warehouses. It facilitates effortless querying and retrieval of information from these sources, allowing users to engage in intuitive conversations and obtain insights. Additionally, DB-GPT supports the generation of analysis reports, providing users with valuable summaries and interpretations of the data.

• Multi-Agents&Plugins

  It supports custom plug-ins to perform tasks, natively supports the Auto-GPT plug-in model, and the Agents protocol adopts the Agent Protocol standard.

• Fine-tuning text2SQL

  An automated fine-tuning lightweight framework built around large language models, Text2SQL data sets, LoRA/QLoRA/Pturning, and other fine-tuning methods, making TextSQL fine-tuning as convenient as an assembly line. DB-GPT-Hub

• Multi LLMs Support, Supports multiple large language models, currently supporting

  Massive model support, including dozens of large language models such as open source and API agents. Such as LLaMA/LLaMA2, Baichuan, ChatGLM, Wenxin, Tongyi, Zhipu, etc.

  • Vicuna
  • vicuna-13b-v1.5
  • LLama2
  • baichuan2-13b
  • baichuan-7B
  • chatglm-6b
  • chatglm2-6b
  • falcon-40b
  • internlm-chat-7b
  • Qwen-7B-Chat/Qwen-14B-Chat
  • RWKV-4-Raven
  • CAMEL-13B-Combined-Data
  • dolly-v2-12b
  • h2ogpt-gm-oasst1-en-2048-open-llama-7b
  • fastchat-t5-3b-v1.0
  • mpt-7b-chat
  • gpt4all-13b-snoozy
  • Nous-Hermes-13b
  • codet5p-6b
  • guanaco-33b-merged
  • WizardLM-13B-V1.0
  • WizardLM/WizardCoder-15B-V1.0
  • Llama2-Chinese-13b-Chat
  • OpenLLaMa OpenInstruct

  Etc.

  • Support API Proxy LLMs
    • ChatGPT
    • Tongyi
    • Wenxin
    • ChatGLM

• Privacy and security

  The privacy and security of data are ensured through various technologies, such as privatized large models and proxy desensitization.

• Support Datasources

Introduction

The architecture of the entire DB-GPT is shown.

The core capabilities mainly consist of the following parts:

1. Multi-Models: Support multi-LLMs, such as LLaMA/LLaMA2、CodeLLaMA、ChatGLM, QWen、Vicuna and proxy model ChatGPT、Baichuan、tongyi、wenxin etc
2. Knowledge-Based QA: You can perform high-quality intelligent Q&A based on local documents such as PDF, word, excel, and other data.
3. Embedding: Unified data vector storage and indexing, Embed data as vectors and store them in vector databases, providing content similarity search.
4. Multi-Datasources: Used to connect different modules and data sources to achieve data flow and interaction.
5. Multi-Agents: Provides Agent and plugin mechanisms, allowing users to customize and enhance the system's behavior.
6. Privacy & Secure: You can be assured that there is no risk of data leakage, and your data is 100% private and secure.
7. Text2SQL: We enhance the Text-to-SQL performance by applying Supervised Fine-Tuning (SFT) on large language models

RAG-IN-Action

SubModule

• DB-GPT-Hub Text-to-SQL performance by applying Supervised Fine-Tuning (SFT) on large language models.
• DB-GPT-Plugins DB-GPT Plugins Can run autogpt plugin directly
• DB-GPT-Web ChatUI for DB-GPT

Image

🌐 AutoDL Image

Language Switching

In the .env configuration file, modify the LANGUAGE parameter to switch to different languages. The default is English (Chinese: zh, English: en, other languages to be added later).

Contribution

• Please run black . before submitting the code. Contributing guidelines, how to contribute

RoadMap

KBQA RAG optimization

• Multi Documents

  • PDF
  • Excel, CSV
  • Word
  • Text
  • MarkDown
  • Code
  • Images

• RAG

• Graph Database

  • Neo4j Graph
  • Nebula Graph

• Multi-Vector Database

  • Chroma
  • Milvus
  • Weaviate
  • PGVector
  • Elasticsearch
  • ClickHouse
  • Faiss

• Testing and Evaluation Capability Building

  • Knowledge QA datasets
  • Question collection [easy, medium, hard]:
  • Scoring mechanism
  • Testing and evaluation using Excel + DB datasets

Multi Datasource Support

• Multi Datasource Support
  • MySQL
  • PostgreSQL
  • Spark
  • DuckDB
  • Sqlite
  • MSSQL
  • ClickHouse
  • Oracle
  • Redis
  • MongoDB
  • HBase
  • Doris
  • DB2
  • Couchbase
  • Elasticsearch
  • OceanBase
  • TiDB
  • StarRocks

Multi-Models And vLLM

• Cluster Deployment
• Fastchat Support
• vLLM Support
• Cloud-native environment and support for Ray environment
• Service Registry(eg:nacos)
• Compatibility with OpenAI's interfaces
• Expansion and optimization of embedding models

Agents market and Plugins

• multi-agents framework
• custom plugin development
• plugin market
• Integration with CoT
• Enrich plugin sample library
• Support for AutoGPT protocol
• Integration of multi-agents and visualization capabilities, defining LLM+Vis new standards

Cost and Observability

• debugging
• Observability
• cost & budgets

Text2SQL Finetune

• support llms

  • LLaMA
  • LLaMA-2
  • BLOOM
  • BLOOMZ
  • Falcon
  • Baichuan
  • Baichuan2
  • InternLM
  • Qwen
  • XVERSE
  • ChatGLM2

• SFT Accuracy

As of October 10, 2023, by fine-tuning an open-source model of 13 billion parameters using this project, the execution accuracy on the Spider evaluation dataset has surpassed that of GPT-4!

More Information about Text2SQL finetune

Licence

The MIT License (MIT)

Contact Information

We are working on building a community, if you have any ideas for building the community, feel free to contact us.

Bisheng is an open LLM devops platform for next generation AI applications.

欢迎来到 Bisheng

Bisheng 是什么

Bisheng是一款领先的开源大模型应用开发平台，赋能和加速大模型应用开发落地，帮助用户以最佳体验进入下一代应用开发模式。

“毕昇”是活字印刷术的发明人，活字印刷术为人类知识的传递起到了巨大的推动作用。我们希望“毕昇”同样能够为智能应用的广泛落地提供有力的支撑。欢迎大家一道参与。

Bisheng 基于 Apache 2.0 License 协议发布，于 2023 年 8 月底正式开源。

产品亮点

• 便捷：即使是业务人员，基于我们预置的应用模板，通过简单直观的表单填写方式快速搭建以大模型为核心的智能应用。
• 灵活：对大模型技术有了解的人员，我们紧跟最前沿大模型技术生态提供数百种开发组件，基于可视化且自由的流程编排能力，可开发出任意类型的大模型应用，而不仅是简单的提示词工程。
• 可靠与企业级：当前许多同类的开源项目仅适用于实验测试场景，缺少真正生产使用的企业级特性，包括：高并发下的高可用、应用运营及效果持续迭代优化、贴合真实业务场景的实用功能等，这些都是毕昇平台的差异化能力；另外，更直观的是，企业内的数据质量参差不齐，想要真正把所有数据利用起来，首先需要有完备的非结构化数据治理能力，而这是过去几年我们团队所积累的核心能力，在毕昇的demo环境中您可以通过相关组件直接接入这些能力，并且这些能力免费不限量使用。

产品应用

使用毕昇平台，我们可以搭建各类丰富的大模型应用：

分析报告生成

• 📃 合同审核报告生成
• 🏦 信贷调查报告生成
• 📈 招股书分析报告生成
• 💼 智能投顾报告生成
• 👀 文档摘要生成

知识库问答

• 👩‍💻 用户手册问答
• 👩🏻‍🔬 研报知识库问答
• 🗄 规章制度问答
• 💊 《中华药典》知识问答
• 📊 股价数据库问答

对话

• 🎭 扮演面试官对话
• 📍 小红书文案助手
• 👩‍🎤 扮演外教对话
• 👨‍🏫 简历优化助手

要素提取

• 📄 合同关键要素提取
• 🏗️ 工程报告要素提取
• 🗂️ 通用元数据提取
• 🎫 卡证票据要素提取

各类应用构建方法详见：应用案例

我们认为在企业真实场景中，“对话”仅是众多交互形式中的一种，未来我们还将新增流程自动化、搜索等更多应用形态的支持。

快速开始

启动 Bisheng

• 安装 Bisheng

源码编译 Bisheng

Todo: 待补充

获取更多内容，请阅读 开发者文档。

贡献代码

欢迎向 Bisheng 社区贡献你的代码。代码贡献流程或提交补丁等相关信息详见 代码贡献准则。 参考 社区仓库 了解社区管理准则并获取更多社区资源。

All Thanks To Our Contributors:

Bisheng 文档

获取更多有关安装、开发、部署和管理的指南，请查看 Bisheng 文档.

社区

• 欢迎加入 Slack 频道分享你的建议与问题。
• 你也可以通过 FAQ 页面，查看常见问题及解答。
• 你也可以加入 讨论组 发起问题和讨论。

关注 Bisheng 社交媒体：

• Bisheng 技术交流微信群

加入我们

DataElem Inc. 是 Bisheng 项目的幕后公司。我们正在 招聘 算法、开发和全栈工程师。欢迎加入我们，让我们携手构建下一代的智能应用开发平台。

特别感谢

Bisheng 采用了以下依赖库:

• 感谢开源模型预估框架 Triton 。
• 感谢开源LLM应用开发库 langchain。
• 感谢开源非结构化数据解析引擎 unstructured。
• 感谢开源langchain可视化工具 langflow。

Star History

Main Sigma Rule Repository

Sigma - Generic Signature Format for SIEM Systems

Welcome to the Sigma main rule repository. The place where detection engineers, threat hunters and all defensive security practitioners collaborate on detection rules. The repository offers more than 3000 detection rules of different type and aims to make reliable detections accessible to all at no cost.

Currently the repository offers three types of rules:

• Generic Detection Rules - Are threat agnostic, their aim is to detect a behavior or an implementation of a technique or procedure that was, can or will be used by a potential threat actor.
• Threat Hunting Rules - Are broader in scope and are meant to give the analyst a starting point to hunt for potential suspicious or malicious activity
• Emerging Threat Rules - Are rules that cover specific threats, that are timely and relevant for certain periods of time. These threats include specific APT campaigns, exploitation of Zero-Day vulnerabilities, specific malware used during an attack,...etc.

Explore Sigma

To start exploring the Sigma ecosystem, please visit the official website sigmahq.io

What is Sigma

Sigma is a generic and open signature format that allows you to describe relevant log events in a straightforward manner. The rule format is very flexible, easy to write and applicable to any type of log file.

The main purpose of this project is to provide a structured form in which researchers or analysts can describe their once developed detection methods and make them shareable with others.

Sigma is for log files what Snort is for network traffic and YARA is for files.

Why Sigma

Today, everyone collects log data for analysis. People start working on their own, processing numerous white papers, blog posts and log analysis guidelines, extracting the necessary information and build their own searches and dashboard. Some of their searches and correlations are great and very useful but they lack a standardized format in which they can share their work with others.

Others provide excellent analyses, include IOCs and YARA rules to detect the malicious files and network connections, but have no way to describe a specific or generic detection method in log events. Sigma is meant to be an open standard in which such detection mechanisms can be defined, shared and collected in order to improve the detection capabilities for everyone.

🌟 Key Features

• A continuously growing list of detection and hunting rules, peer reviewed by a community of professional Detection Engineers.
• Vendor agnostic detection rules.
• Easily shareable across communities and reports

🏗️ Rule Creation

To start writing Sigma rules please check the following guides:

• Rule Creation Guide
• How to Write Sigma Rules - Nextron Systems

🔎 Contributing & Making PRs

Please refer to the CONTRIBUTING guide for detailed instructions on how you can start contributing new rules.

📦 Rule Packages

You can download the latest rule packages from the release page and start leveraging Sigma rules today.

🧬 Rule Usage and Conversion

• You can start converting Sigma rules today using Sigma CLI or sigconverter.io the GUI interface

• To integrate Sigma rules in your own toolchain or products use pySigma.

🚨 Reporting False Positives or New Rule Ideas

If you find a false positive or would like to propose a new detection rule idea but do not have the time to create one, please create a new issue on the GitHub repository by selecting one of the available templates.

📚 Resources & Further Reading

• Hack.lu 2017 Sigma - Generic Signatures for Log Events by Thomas Patzke
• MITRE ATT&CK® and Sigma Alerting SANS Webcast Recording
• Sigma - Generic Signatures for SIEM Systems by Florian Roth

Projects or Products that use or integrate Sigma rules

• Atomic Threat Coverage (Since December 2018)
• Confluent Sigma
• IBM QRadar
• Joe Sandbox
• MISP (Since Version 2.4.70, March 2017)
• Nextron's Aurora Agent
• Nextron's THOR Scanner - Scan with Sigma rules on endpoints
• RANK VASA
• Sekoia.io XDR - XDR supporting Sigma and Sigma Correlation rules languages
• SIΣGMA - SIEM consumable generator that utilizes Sigma for query conversion
• SOC Prime
• TA-Sigma-Searches (Splunk App)
• TimeSketch
• ypsilon - Automated Use Case Testing

📜 Maintainers

• Nasreddine Bencherchali (@nas_bench)
• Florian Roth (@cyb3rops)
• Christian Burkard (@phantinuss)
• François Hubaut (@frack113)
• Thomas Patzke (@blubbfiction)

Credits

This project would've never reached this hight without the help of the hundreds of contributors. Thanks to all past and present contributors for their help.

Licenses

The content of this repository is released under the Detection Rule License (DRL) 1.1.

Easy-to-use LLM fine-tuning framework (LLaMA-2, BLOOM, Falcon, Baichuan, Qwen, ChatGLM3)

LLaMA Factory: Training and Evaluating Large Language Models with Minimal Effort

👋 Join our WeChat.

[ English | 中文 ]

LLaMA Board: A One-stop Web UI for Getting Started with LLaMA Factory

Launch LLaMA Board via CUDA_VISIBLE_DEVICES=0 python src/train_web.py. (multiple GPUs are not supported yet)

Here is an example of altering the self-cognition of an instruction-tuned language model within 10 minutes on a single GPU.

https://github.com/hiyouga/LLaMA-Factory/assets/16256802/6ba60acc-e2e2-4bec-b846-2d88920d5ba1

Changelog

[23/10/21] We supported NEFTune trick for fine-tuning. Try --neft_alpha argument to activate NEFTune, e.g., --neft_alpha 5.

[23/09/27] We supported $S^2$-Attn proposed by LongLoRA for the LLaMA models. Try --shift_attn argument to enable shift short attention.

[23/09/23] We integrated MMLU, C-Eval and CMMLU benchmarks in this repo. See this example to evaluate your models.

[23/09/10] We supported using FlashAttention-2 for the LLaMA models. Try --flash_attn argument to enable FlashAttention-2 if you are using RTX4090, A100 or H100 GPUs.

[23/08/12] We supported RoPE scaling to extend the context length of the LLaMA models. Try --rope_scaling linear argument in training and --rope_scaling dynamic argument at inference to extrapolate the position embeddings.

[23/08/11] We supported DPO training for instruction-tuned models. See this example to train your models.

[23/07/31] We supported dataset streaming. Try --streaming and --max_steps 10000 arguments to load your dataset in streaming mode.

[23/07/29] We released two instruction-tuned 13B models at Hugging Face. See these Hugging Face Repos (LLaMA-2 / Baichuan) for details.

[23/07/18] We developed an all-in-one Web UI for training, evaluation and inference. Try train_web.py to fine-tune models in your Web browser. Thank @KanadeSiina and @codemayq for their efforts in the development.

[23/07/09] We released FastEdit ⚡🩹, an easy-to-use package for editing the factual knowledge of large language models efficiently. Please follow FastEdit if you are interested.

[23/06/29] We provided a reproducible example of training a chat model using instruction-following datasets, see Baichuan-7B-sft for details.

[23/06/22] We aligned the demo API with the OpenAI's format where you can insert the fine-tuned model in arbitrary ChatGPT-based applications.

[23/06/03] We supported quantized training and inference (aka QLoRA). Try --quantization_bit 4/8 argument to work with quantized models.

Supported Models

[!NOTE] Default module is used for the --lora_target argument, you can use --lora_target all to specify all the available modules.

For the "base" models, the --template argument can be chosen from default, alpaca, vicuna etc. But make sure to use the corresponding template for the "chat" models.

Please refer to template.py for a full list of models we supported.

Supported Training Approaches

[!NOTE] Use --quantization_bit 4/8 argument to enable QLoRA.

Provided Datasets

• Wiki Demo (en)
• RefinedWeb (en)
• RedPajama V2 (en)
• Wikipedia (en)
• Wikipedia (zh)
• Pile (en)
• SkyPile (zh)
• The Stack (en)
• StarCoder (en)

• Stanford Alpaca (en)
• Stanford Alpaca (zh)
• GPT-4 Generated Data (en&zh)
• Self-cognition (zh)
• Open Assistant (multilingual)
• ShareGPT (zh)
• Guanaco Dataset (multilingual)
• BELLE 2M (zh)
• BELLE 1M (zh)
• BELLE 0.5M (zh)
• BELLE Dialogue 0.4M (zh)
• BELLE School Math 0.25M (zh)
• BELLE Multiturn Chat 0.8M (zh)
• UltraChat (en)
• LIMA (en)
• OpenPlatypus (en)
• CodeAlpaca 20k (en)
• Alpaca CoT (multilingual)
• MathInstruct (en)
• Firefly 1.1M (zh)
• Web QA (zh)
• WebNovel (zh)
• Ad Gen (zh)
• ShareGPT Hyperfiltered (en)
• ShareGPT4 (en&zh)
• UltraChat 200k (en)
• AgentInstruct (en)
• LMSYS Chat 1M (en)
• Evol Instruct V2 (en)

• HH-RLHF (en)
• Open Assistant (multilingual)
• GPT-4 Generated Data (en&zh)

Please refer to data/README.md for details.

Some datasets require confirmation before using them, so we recommend logging in with your Hugging Face account using these commands.

pip install --upgrade huggingface_hub
huggingface-cli login

Requirement

• Python 3.8+ and PyTorch 1.13.1+
• 🤗Transformers, Datasets, Accelerate, PEFT and TRL
• sentencepiece, protobuf and tiktoken
• fire, jieba, rouge-chinese and nltk (used at evaluation and predict)
• gradio and matplotlib (used in web UI)
• uvicorn, fastapi and sse-starlette (used in API)

And powerful GPUs!

Getting Started

Data Preparation (optional)

Please refer to data/README.md for checking the details about the format of dataset files. You can either use a single .json file or a dataset loading script with multiple files to create a custom dataset.

[!NOTE] Please update data/dataset_info.json to use your custom dataset. About the format of this file, please refer to data/README.md.

Dependence Installation (optional)

git clone https://github.com/hiyouga/LLaMA-Factory.git
conda create -n llama_factory python=3.10
conda activate llama_factory
cd LLaMA-Factory
pip install -r requirements.txt

If you want to enable the quantized LoRA (QLoRA) on the Windows platform, you will be required to install a pre-built version of bitsandbytes library, which supports CUDA 11.1 to 12.1.

pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.39.1-py3-none-win_amd64.whl

Train on a single GPU

[!IMPORTANT] If you want to train models on multiple GPUs, please refer to Distributed Training.

Pre-Training

CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
    --stage pt \
    --model_name_or_path path_to_llama_model \
    --do_train \
    --dataset wiki_demo \
    --finetuning_type lora \
    --lora_target q_proj,v_proj \
    --output_dir path_to_pt_checkpoint \
    --overwrite_cache \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --logging_steps 10 \
    --save_steps 1000 \
    --learning_rate 5e-5 \
    --num_train_epochs 3.0 \
    --plot_loss \
    --fp16

Supervised Fine-Tuning

CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
    --stage sft \
    --model_name_or_path path_to_llama_model \
    --do_train \
    --dataset alpaca_gpt4_en \
    --template default \
    --finetuning_type lora \
    --lora_target q_proj,v_proj \
    --output_dir path_to_sft_checkpoint \
    --overwrite_cache \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --logging_steps 10 \
    --save_steps 1000 \
    --learning_rate 5e-5 \
    --num_train_epochs 3.0 \
    --plot_loss \
    --fp16

Reward Modeling

CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
    --stage rm \
    --model_name_or_path path_to_llama_model \
    --do_train \
    --dataset comparison_gpt4_en \
    --template default \
    --finetuning_type lora \
    --lora_target q_proj,v_proj \
    --resume_lora_training False \
    --checkpoint_dir path_to_sft_checkpoint \
    --output_dir path_to_rm_checkpoint \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --logging_steps 10 \
    --save_steps 1000 \
    --learning_rate 1e-6 \
    --num_train_epochs 1.0 \
    --plot_loss \
    --fp16

PPO Training

CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
    --stage ppo \
    --model_name_or_path path_to_llama_model \
    --do_train \
    --dataset alpaca_gpt4_en \
    --template default \
    --finetuning_type lora \
    --lora_target q_proj,v_proj \
    --resume_lora_training False \
    --checkpoint_dir path_to_sft_checkpoint \
    --reward_model path_to_rm_checkpoint \
    --output_dir path_to_ppo_checkpoint \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --logging_steps 10 \
    --save_steps 1000 \
    --learning_rate 1e-5 \
    --num_train_epochs 1.0 \
    --plot_loss \
    --fp16

DPO Training

CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
    --stage dpo \
    --model_name_or_path path_to_llama_model \
    --do_train \
    --dataset comparison_gpt4_en \
    --template default \
    --finetuning_type lora \
    --lora_target q_proj,v_proj \
    --resume_lora_training False \
    --checkpoint_dir path_to_sft_checkpoint \
    --output_dir path_to_dpo_checkpoint \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --logging_steps 10 \
    --save_steps 1000 \
    --learning_rate 1e-5 \
    --num_train_epochs 1.0 \
    --plot_loss \
    --fp16

Distributed Training

Use Huggingface Accelerate

accelerate config # configure the environment
accelerate launch src/train_bash.py # arguments (same as above)

compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
downcast_bf16: 'no'
gpu_ids: all
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

Use DeepSpeed

deepspeed --num_gpus 8 --master_port=9901 src/train_bash.py \
    --deepspeed ds_config.json \
    ... # arguments (same as above)

{
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "zero_allow_untested_optimizer": true,
  "fp16": {
    "enabled": "auto",
    "loss_scale": 0,
    "initial_scale_power": 16,
    "loss_scale_window": 1000,
    "hysteresis": 2,
    "min_loss_scale": 1
  },  
  "zero_optimization": {
    "stage": 2,
    "allgather_partitions": true,
    "allgather_bucket_size": 5e8,
    "reduce_scatter": true,
    "reduce_bucket_size": 5e8,
    "overlap_comm": false,
    "contiguous_gradients": true
  }
}

Export model

python src/export_model.py \
    --model_name_or_path path_to_llama_model \
    --template default \
    --finetuning_type lora \
    --checkpoint_dir path_to_checkpoint \
    --export_dir path_to_export

API Demo

python src/api_demo.py \
    --model_name_or_path path_to_llama_model \
    --template default \
    --finetuning_type lora \
    --checkpoint_dir path_to_checkpoint

[!NOTE] Visit http://localhost:8000/docs for API documentation.

CLI Demo

python src/cli_demo.py \
    --model_name_or_path path_to_llama_model \
    --template default \
    --finetuning_type lora \
    --checkpoint_dir path_to_checkpoint

Web Demo

python src/web_demo.py \
    --model_name_or_path path_to_llama_model \
    --template default \
    --finetuning_type lora \
    --checkpoint_dir path_to_checkpoint

Evaluation

CUDA_VISIBLE_DEVICES=0 python src/evaluate.py \
    --model_name_or_path path_to_llama_model \
    --finetuning_type lora \
    --checkpoint_dir path_to_checkpoint \
    --template vanilla \
    --task mmlu \
    --split test \
    --lang en \
    --n_shot 5 \
    --batch_size 4

Predict

CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
    --stage sft \
    --model_name_or_path path_to_llama_model \
    --do_predict \
    --dataset alpaca_gpt4_en \
    --template default \
    --finetuning_type lora \
    --checkpoint_dir path_to_checkpoint \
    --output_dir path_to_predict_result \
    --per_device_eval_batch_size 8 \
    --max_samples 100 \
    --predict_with_generate

[!NOTE] We recommend using --per_device_eval_batch_size=1 and --max_target_length 128 at 4/8-bit predict.

Projects using LLaMA Factory

• StarWhisper: A large language model for Astronomy, based on ChatGLM2-6B and Qwen-14B.
• DISC-LawLLM: A large language model specialized in Chinese legal domain, based on Baichuan-13B, is capable of retrieving and reasoning on legal knowledge.
• Sunsimiao: A large language model specialized in Chinese medical domain, based on Baichuan-7B and ChatGLM-6B.
• CareGPT: A series of large language models for Chinese medical domain, based on LLaMA2-7B and Baichuan-13B.

License

This repository is licensed under the Apache-2.0 License.

Please follow the model licenses to use the corresponding model weights: LLaMA / LLaMA-2 / BLOOM / Falcon / Baichuan / Baichuan2 / InternLM / Qwen / ChatGLM3 / Phi-1.5

Citation

If this work is helpful, please kindly cite as:

@Misc{llama-factory,
  title = {LLaMA Factory},
  author = {hiyouga},
  howpublished = {\url{https://github.com/hiyouga/LLaMA-Factory}},
  year = {2023}
}

Acknowledgement

This repo benefits from PEFT, QLoRA and FastChat. Thanks for their wonderful works.

Star History

localpilot

Use GitHub Copilot locally on your Macbook with one-click!

Demo Video

https://github.com/danielgross/localpilot/assets/279531/3259981b-39f7-4bfa-8a45-84bde6d4ba4c

This video is not sped up or slowed down.

Installation

1. First, open VS Code Settings and add the following to your settings.json file:

"github.copilot.advanced": {
    "debug.testOverrideProxyUrl": "http://localhost:5001",
    "debug.overrideProxyUrl": "http://localhost:5001"
}

1. Create a virtualenv to run this Python process, install the requirements, and download the models.

virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
# First setup run. This will download several models to your ~/models folder.
python app.py --setup 

1. Run it!

python app.py

Enjoy your on-device Copilot!

Caveat FAQ

Is the code as good as GitHub Copilot?

For simple line completions yes. For simple function completions, mostly. For complex functions... maybe.

Is it as fast as GitHub Copilot?

On my Macbook Pro with an Apple M2 Max, the 7b models are roughly as fast. The 34b models are not. Please consider this repo a demonstration of a very inefficient implementation. I'm sure we can make it faster; please do submit a pull request if you'd like to help. For example, I think we need debouncer because sometimes llama.cpp/GGML isn't fast at interrupting itself when a newer request comes in.

Can this be packaged as a simple Mac app?

Yes!, I'm sure it can be, I just haven't had the time. Please do submit a pull request if you're into that sort of thing!

Should there be a meta-model that routes to a 1b for autocomplete, 7b for more complex autocomplete, and a 34b for program completion?

Hmm, that seems like an interesting idea.

OK, but in summary, is it good?

Only if your network is bad. I don't think it's competitive if you have fast Internet. But it sure is awesome on airplanes and while tethering!

3D printer / slicing GUI built on top of the Uranium framework

       

       

       

Ultimaker Cura

State-of-the-art slicer app to prepare
your 3D models for your 3D printer.

With hundreds of settings & community-managed print profiles,
Ultimaker Cura is sure to lead your next project to a success.

       

       

Efficient Streaming Language Models with Attention Sinks

Efficient Streaming Language Models with Attention Sinks

[paper] [slides][video]

https://github.com/mit-han-lab/streaming-llm/assets/40906949/2bd1cda4-a0bd-47d1-a023-fbf7779b8358

TL;DR

We deploy LLMs for infinite-length inputs without sacrificing efficiency and performance.

News

• [2023/10] StreamingLLM is integrated into Intel Extension for Transformers.
• [2023/10] Check out Attention Sinks, a third-party implementation to enable StreamingLLM on more Huggingface LLMs.

Abstract

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach --- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a ``sink'' even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup.

Usage

Environment Setup

conda create -yn streaming python=3.8
conda activate streaming

pip install torch torchvision torchaudio
pip install transformers==4.33.0 accelerate datasets evaluate wandb scikit-learn scipy sentencepiece

python setup.py develop

Run Streaming Llama Chatbot

CUDA_VISIBLE_DEVICES=0 python examples/run_streaming_llama.py  --enable_streaming

FAQ

1. What does "working on infinite-length inputs" imply for LLMs?

   Handling infinite-length text with LLMs presents challenges. Notably, storing all previous Key and Value (KV) states demands significant memory, and models might struggle to generate text beyond their training sequence length. StreamingLLM addresses this by retaining only the most recent tokens and attention sinks, discarding intermediate tokens. This enables the model to generate coherent text from recent tokens without a cache reset — a capability not seen in earlier methods.

2. Is the context window of LLMs expanded?

   No. The context window remains unchanged. Only the most recent tokens and attention sinks are retained, discarding middle tokens. This means the model can only process the latest tokens. The context window remains constrained by its initial pre-training. For instance, if Llama-2 is pre-trained with a context window of 4096 tokens, then the maximum cache size for StreamingLLM on Llama-2 remains 4096.

3. Can I input an extensive text, like a book, into StreamingLLM for summarization?

   While you can input a lengthy text, the model will only recognize the latest tokens. Thus, if a book is an input, StreamingLLM might only summarize the concluding paragraphs, which might not be very insightful. As emphasized earlier, we neither expand the LLMs' context window nor enhance their long-term memory. StreamingLLM's strength lies in generating fluent text from recent tokens without needing a cache refresh.

4. What is the ideal use case for StreamingLLM?

   StreamingLLM is optimized for streaming applications, such as multi-round dialogues. It's ideal for scenarios where a model needs to operate continually without requiring extensive memory or dependency on past data. An example is a daily assistant based on LLMs. StreamingLLM would let the model function continuously, basing its responses on recent conversations without needing to refresh its cache. Earlier methods would either need a cache reset when the conversation length exceeded the training length (losing recent context) or recompute KV states from recent text history, which can be time-consuming.

5. How does StreamingLLM relate to recent works on context extension?

   StreamingLLM is orthogonal to recent context extension methods and can be integrated with them. In StreamingLLM's context, "context extension" refers to the possibility of using a larger cache size to store more recent tokens. For a practical demonstration, refer to Figure 9 in our paper, where we implement StreamingLLM with models like LongChat-7B-v1.5-32K and Llama-2-7B-32K-Instruct.

TODOs

We will release the code and data in the following order, please stay tuned!

• Release core code of StreamingLLM, including Llama-2, MPT, Falcon, and Pythia.
• Release perplexity evaluation code
• Release Streaming Llama Chatbot demo.
• Release StreamEval dataset and evaluation code.

Citation

If you find StreamingLLM useful or relevant to your project and research, please kindly cite our paper:

@article{xiao2023streamingllm,
        title={Efficient Streaming Language Models with Attention Sinks},
        author={Xiao, Guangxuan and Tian, Yuandong and Chen, Beidi and Han, Song and Lewis, Mike},
        journal={arXiv},
        year={2023}
        }

A collective list of free APIs

Status

The Project
Contributing Guide • API for this project • Issues • Pull Requests • License

Alternative sites for the project (unofficials)
Free APIs • Dev Resources • Public APIs Site • Apihouse • Collective APIs

---

[ Become a sponsor and support Public APIs and their maintainers ]

Special thanks to:

The fastest way to integrate APIs into any product

Explore, discover and consume public APIs as simpler programmable building blocks all on one platform for a 10x developer experience.

The fastest way to integrate APIs into any product

Explore, discover and consume public APIs as simpler programmable building blocks all on one platform for a 10x developer experience.

---

Index

• Animals
• Anime
• Anti-Malware
• Art & Design
• Authentication & Authorization
• Blockchain
• Books
• Business
• Calendar
• Cloud Storage & File Sharing
• Continuous Integration
• Cryptocurrency
• Currency Exchange
• Data Validation
• Development
• Dictionaries
• Documents & Productivity
• Email
• Entertainment
• Environment
• Events
• Finance
• Food & Drink
• Games & Comics
• Geocoding
• Government
• Health
• Jobs
• Machine Learning
• Music
• News
• Open Data
• Open Source Projects
• Patent
• Personality
• Phone
• Photography
• Programming
• Science & Math
• Security
• Shopping
• Social
• Sports & Fitness
• Test Data
• Text Analysis
• Tracking
• Transportation
• URL Shorteners
• Vehicle
• Video
• Weather

Animals

API Description Auth HTTPS CORS

AdoptAPet	Resource to help get pets adopted	apiKey	Yes	Yes
Axolotl	Collection of axolotl pictures and facts	No	Yes	No
Cat Facts	Daily cat facts	No	Yes	No
Cataas	Cat as a service (cats pictures and gifs)	No	Yes	No
Cats	Pictures of cats from Tumblr	apiKey	Yes	No
Dog Facts	Random dog facts	No	Yes	Yes
Dog Facts	Random facts of Dogs	No	Yes	Yes
Dogs	Based on the Stanford Dogs Dataset	No	Yes	Yes
eBird	Retrieve recent or notable birding observations within a region	apiKey	Yes	No
FishWatch	Information and pictures about individual fish species	No	Yes	Yes
HTTP Cat	Cat for every HTTP Status	No	Yes	Yes
HTTP Dog	Dogs for every HTTP response status code	No	Yes	Yes
IUCN	IUCN Red List of Threatened Species	apiKey	No	No
MeowFacts	Get random cat facts	No	Yes	No
Movebank	Movement and Migration data of animals	No	Yes	Yes
Petfinder	Petfinder is dedicated to helping pets find homes, another resource to get pets adopted	apiKey	Yes	Yes
PlaceBear	Placeholder bear pictures	No	Yes	Yes
PlaceDog	Placeholder Dog pictures	No	Yes	Yes
PlaceKitten	Placeholder Kitten pictures	No	Yes	Yes
RandomDog	Random pictures of dogs	No	Yes	Yes
RandomDuck	Random pictures of ducks	No	Yes	No
RandomFox	Random pictures of foxes	No	Yes	No
RescueGroups	Adoption	No	Yes	Unknown
Shibe.Online	Random pictures of Shiba Inu, cats or birds	No	Yes	Yes
The Dog	A public service all about Dogs, free to use when making your fancy new App, Website or Service	apiKey	Yes	No
xeno-canto	Bird recordings	No	Yes	Unknown
Zoo Animals	Facts and pictures of zoo animals	No	Yes	Yes

⬆ Back to Index

Anime

API Description Auth HTTPS CORS

AniAPI	Anime discovery, streaming & syncing with trackers	OAuth	Yes	Yes
AniDB	Anime Database	apiKey	No	Unknown
AniList	Anime discovery & tracking	OAuth	Yes	Unknown
AnimeChan	Anime quotes (over 10k+)	No	Yes	No
AnimeFacts	Anime Facts (over 100+)	No	Yes	Yes
AnimeNewsNetwork	Anime industry news	No	Yes	Yes
Catboy	Neko images, funny GIFs & more	No	Yes	Yes
Danbooru Anime	Thousands of anime artist database to find good anime art	apiKey	Yes	Yes
Jikan	Unofficial MyAnimeList API	No	Yes	Yes
Kitsu	Anime discovery platform	OAuth	Yes	Yes
MangaDex	Manga Database and Community	apiKey	Yes	Unknown
Mangapi	Translate manga pages from one language to another	apiKey	Yes	Unknown
MyAnimeList	Anime and Manga Database and Community	OAuth	Yes	Unknown
NekosBest	Neko Images & Anime roleplaying GIFs	No	Yes	Yes
Shikimori	Anime discovery, tracking, forum, rates	OAuth	Yes	Unknown
Studio Ghibli	Resources from Studio Ghibli films	No	Yes	Yes
Trace Moe	A useful tool to get the exact scene of an anime from a screenshot	No	Yes	No
Waifu.im	Get waifu pictures from an archive of over 4000 images and multiple tags	No	Yes	Yes
Waifu.pics	Image sharing platform for anime images	No	Yes	No

⬆ Back to Index

Anti-Malware

API Description Auth HTTPS CORS

AbuseIPDB	IP/domain/URL reputation	apiKey	Yes	Unknown
AlienVault Open Threat Exchange (OTX)	IP/domain/URL reputation	apiKey	Yes	Unknown
CAPEsandbox	Malware execution and analysis	apiKey	Yes	Unknown
Google Safe Browsing	Google Link/Domain Flagging	apiKey	Yes	Unknown
MalDatabase	Provide malware datasets and threat intelligence feeds	apiKey	Yes	Unknown
MalShare	Malware Archive / file sourcing	apiKey	Yes	No
MalwareBazaar	Collect and share malware samples	apiKey	Yes	Unknown
Metacert	Metacert Link Flagging	apiKey	Yes	Unknown
NoPhishy	Check links to see if they're known phishing attempts	apiKey	Yes	Yes
Phisherman	IP/domain/URL reputation	apiKey	Yes	Unknown
Scanii	Simple REST API that can scan submitted documents/files for the presence of threats	apiKey	Yes	Yes
URLhaus	Bulk queries and Download Malware Samples	No	Yes	Yes
URLScan.io	Scan and Analyse URLs	apiKey	Yes	Unknown
VirusTotal	VirusTotal File/URL Analysis	apiKey	Yes	Unknown
Web of Trust	IP/domain/URL reputation	apiKey	Yes	Unknown

⬆ Back to Index

Art & Design

API Description Auth HTTPS CORS

Améthyste	Generate images for Discord users	apiKey	Yes	Unknown
Art Institute of Chicago	Art	No	Yes	Yes
Colormind	Color scheme generator	No	No	Unknown
ColourLovers	Get various patterns, palettes and images	No	No	Unknown
Cooper Hewitt	Smithsonian Design Museum	apiKey	Yes	Unknown
Dribbble	Discover the world’s top designers & creatives	OAuth	Yes	Unknown
EmojiHub	Get emojis by categories and groups	No	Yes	Yes
Europeana	European Museum and Galleries content	apiKey	Yes	Unknown
Harvard Art Museums	Art	apiKey	No	Unknown
Icon Horse	Favicons for any website, with fallbacks	No	Yes	Yes
Iconfinder	Icons	apiKey	Yes	Unknown
Icons8	Icons (find "search icon" hyperlink in page)	No	Yes	Unknown
Lordicon	Icons with predone Animations	No	Yes	Yes
Metropolitan Museum of Art	Met Museum of Art	No	Yes	No
Noun Project	Icons	OAuth	No	Unknown
PHP-Noise	Noise Background Image Generator	No	Yes	Yes
Pixel Encounter	SVG Icon Generator	No	Yes	No
Rijksmuseum	RijksMuseum Data	apiKey	Yes	Unknown
Word Cloud	Easily create word clouds	apiKey	Yes	Unknown
xColors	Generate & convert colors	No	Yes	Yes

⬆ Back to Index

Authentication & Authorization

API Description Auth HTTPS CORS

Auth0	Easy to implement, adaptable authentication and authorization platform	apiKey	Yes	Yes
GetOTP	Implement OTP flow quickly	apiKey	Yes	No
Micro User Service	User management and authentication	apiKey	Yes	No
MojoAuth	Secure and modern passwordless authentication platform	apiKey	Yes	Yes
SAWO Labs	Simplify login and improve user experience by integrating passwordless authentication in your app	apiKey	Yes	Yes
Stytch	User infrastructure for modern applications	apiKey	Yes	No
Warrant	APIs for authorization and access control	apiKey	Yes	Yes

⬆ Back to Index

Blockchain

API Description Auth HTTPS CORS

Bitquery	Onchain GraphQL APIs & DEX APIs	apiKey	Yes	Yes
Chainlink	Build hybrid smart contracts with Chainlink	No	Yes	Unknown
Chainpoint	Chainpoint is a global network for anchoring data to the Bitcoin blockchain	No	Yes	Unknown
Covalent	Multi-blockchain data aggregator platform	apiKey	Yes	Unknown
Etherscan	Ethereum explorer API	apiKey	Yes	Yes
Helium	Helium is a global, distributed network of Hotspots that create public, long-range wireless coverage	No	Yes	Unknown
Nownodes	Blockchain-as-a-service solution that provides high-quality connection via API	apiKey	Yes	Unknown
Steem	Blockchain-based blogging and social media website	No	No	No
The Graph	Indexing protocol for querying networks like Ethereum with GraphQL	apiKey	Yes	Unknown
Walltime	To retrieve Walltime's market info	No	Yes	Unknown
Watchdata	Provide simple and reliable API access to Ethereum blockchain	apiKey	Yes	Unknown

⬆ Back to Index

Books

API Description Auth HTTPS CORS

A Bíblia Digital	Do not worry about managing the multiple versions of the Bible	apiKey	Yes	No
Bhagavad Gita	Open Source Shrimad Bhagavad Gita API including 21+ authors translation in Sanskrit/English/Hindi	apiKey	Yes	Yes
Bhagavad Gita	Bhagavad Gita text	OAuth	Yes	Yes
Bhagavad Gita telugu	Bhagavad Gita API in telugu and odia languages	No	Yes	Yes
Bible-api	Free Bible API with multiple languages	No	Yes	Yes
British National Bibliography	Books	No	No	Unknown
Crossref Metadata Search	Books & Articles Metadata	No	Yes	Unknown
Ganjoor	Classic Persian poetry works including access to related manuscripts, recitations and music tracks	OAuth	Yes	Yes
Google Books	Books	OAuth	Yes	Unknown
GurbaniNow	Fast and Accurate Gurbani RESTful API	No	Yes	Unknown
Gutendex	Web-API for fetching data from Project Gutenberg Books Library	No	Yes	Unknown
Open Library	Books, book covers and related data	No	Yes	No
Penguin Publishing	Books, book covers and related data	No	Yes	Yes
PoetryDB	Enables you to get instant data from our vast poetry collection	No	Yes	Yes
Quran	RESTful Quran API with multiple languages	No	Yes	Yes
Quran Cloud	A RESTful Quran API to retrieve an Ayah, Surah, Juz or the entire Holy Quran	No	Yes	Yes
Quran-api	Free Quran API Service with 90+ different languages and 400+ translations	No	Yes	Yes
Rig Veda	Gods and poets, their categories, and the verse meters, with the mandal and sukta number	No	Yes	Unknown
The Bible	Everything you need from the Bible in one discoverable place	apiKey	Yes	Unknown
Thirukkural	1330 Thirukkural poems and explanation in Tamil and English	No	Yes	Yes
Vedic Society	Descriptions of all nouns (names, places, animals, things) from vedic literature	No	Yes	Unknown
Wizard World	Get information from the Harry Potter universe	No	Yes	Yes
Wolne Lektury	API for obtaining information about e-books available on the WolneLektury.pl website	No	Yes	Unknown

⬆ Back to Index

Business

API Description Auth HTTPS CORS

Apache Superset	API to manage your BI dashboards and data sources on Superset	apiKey	Yes	Yes
Charity Search	Non-profit charity data	apiKey	No	Unknown
Clearbit Logo	Search for company logos and embed them in your projects	apiKey	Yes	Unknown
Domainsdb.info	Registered Domain Names Search	No	Yes	No
Freelancer	Hire freelancers to get work done	OAuth	Yes	Unknown
Gmail	Flexible, RESTful access to the user's inbox	OAuth	Yes	Unknown
Google Analytics	Collect, configure and analyze your data to reach the right audience	OAuth	Yes	Unknown
Instatus	Post to and update maintenance and incidents on your status page through an HTTP REST API	apiKey	Yes	Unknown
Mailchimp	Send marketing campaigns and transactional mails	apiKey	Yes	Unknown
mailjet	Marketing email can be sent and mail templates made in MJML or HTML can be sent using API	apiKey	Yes	Unknown
markerapi	Trademark Search	No	No	Unknown
ORB Intelligence	Company lookup	apiKey	Yes	Unknown
Redash	Access your queries and dashboards on Redash	apiKey	Yes	Yes
Smartsheet	Allows you to programmatically access and Smartsheet data and account information	OAuth	Yes	No
Square	Easy way to take payments, manage refunds, and help customers checkout online	OAuth	Yes	Unknown
SwiftKanban	Kanban software, Visualize Work, Increase Organizations Lead Time, Throughput & Productivity	apiKey	Yes	Unknown
Tenders in Hungary	Get data for procurements in Hungary in JSON format	No	Yes	Unknown
Tenders in Poland	Get data for procurements in Poland in JSON format	No	Yes	Unknown
Tenders in Romania	Get data for procurements in Romania in JSON format	No	Yes	Unknown
Tenders in Spain	Get data for procurements in Spain in JSON format	No	Yes	Unknown
Tenders in Ukraine	Get data for procurements in Ukraine in JSON format	No	Yes	Unknown
Tomba email finder	Email Finder for B2B sales and email marketing and email verifier	apiKey	Yes	Yes
Trello	Boards, lists and cards to help you organize and prioritize your projects	OAuth	Yes	Unknown

⬆ Back to Index

Calendar

API Description Auth HTTPS CORS

Abstract Public Holidays	Data on national, regional, and religious holidays via API	apiKey	Yes	Yes
Calendarific	Worldwide Holidays	apiKey	Yes	Unknown
Checkiday - National Holiday API	Industry-leading Holiday API. Over 5,000 holidays and thousands of descriptions. Trusted by the World’s leading companies	apiKey	Yes	Unknown
Church Calendar	Catholic liturgical calendar	No	No	Unknown
Czech Namedays Calendar	Lookup for a name and returns nameday date	No	No	Unknown
Festivo Public Holidays	Fastest and most advanced public holiday and observance service on the market	apiKey	Yes	Yes
Google Calendar	Display, create and modify Google calendar events	OAuth	Yes	Unknown
Hebrew Calendar	Convert between Gregorian and Hebrew, fetch Shabbat and Holiday times, etc	No	No	Unknown
Holidays	Historical data regarding holidays	apiKey	Yes	Unknown
LectServe	Protestant liturgical calendar	No	No	Unknown
Nager.Date	Public holidays for more than 90 countries	No	Yes	No
Namedays Calendar	Provides namedays for multiple countries	No	Yes	Yes
Non-Working Days	Database of ICS files for non working days	No	Yes	Unknown
Non-Working Days	Simple REST API for checking working, non-working or short days for Russia, CIS, USA and other	No	Yes	Yes
Russian Calendar	Check if a date is a Russian holiday or not	No	Yes	No
UK Bank Holidays	Bank holidays in England and Wales, Scotland and Northern Ireland	No	Yes	Unknown

⬆ Back to Index

Cloud Storage & File Sharing

API Description Auth HTTPS CORS

AnonFiles	Upload and share your files anonymously	No	Yes	Unknown
BayFiles	Upload and share your files	No	Yes	Unknown
Box	File Sharing and Storage	OAuth	Yes	Unknown
ddownload	File Sharing and Storage	apiKey	Yes	Unknown
Dropbox	File Sharing and Storage	OAuth	Yes	Unknown
File.io	Super simple file sharing, convenient, anonymous and secure	No	Yes	Unknown
Filestack	Filestack File Uploader & File Upload API	apiKey	Yes	Unknown
GoFile	Unlimited size file uploads for free	apiKey	Yes	Unknown
Google Drive	File Sharing and Storage	OAuth	Yes	Unknown
Gyazo	Save & Share screen captures instantly	apiKey	Yes	Unknown
Imgbb	Simple and quick private image sharing	apiKey	Yes	Unknown
OneDrive	File Sharing and Storage	OAuth	Yes	Unknown
Pantry	Free JSON storage for small projects	No	Yes	Yes
Pastebin	Plain Text Storage	apiKey	Yes	Unknown
Pinata	IPFS Pinning Services API	apiKey	Yes	Unknown
Quip	File Sharing and Storage for groups	apiKey	Yes	Yes
Storj	Decentralized Open-Source Cloud Storage	apiKey	Yes	Unknown
The Null Pointer	No-bullshit file hosting and URL shortening service	No	Yes	Unknown
Web3 Storage	File Sharing and Storage for Free with 1TB Space	apiKey	Yes	Yes

⬆ Back to Index

Continuous Integration

API Description Auth HTTPS CORS

Azure DevOps Health	Resource health helps you diagnose and get support when an Azure issue impacts your resources	apiKey	No	No
Bitrise	Build tool and processes integrations to create efficient development pipelines	apiKey	Yes	Unknown
Buddy	The fastest continuous integration and continuous delivery platform	OAuth	Yes	Unknown
CircleCI	Automate the software development process using continuous integration and continuous delivery	apiKey	Yes	Unknown
Codeship	Codeship is a Continuous Integration Platform in the cloud	apiKey	Yes	Unknown
Travis CI	Sync your GitHub projects with Travis CI to test your code in minutes	apiKey	Yes	Unknown

⬆ Back to Index

Cryptocurrency

API Description Auth HTTPS CORS

0x	API for querying token and pool stats across various liquidity pools	No	Yes	Yes
1inch	API for querying decentralize exchange	No	Yes	Unknown
Alchemy Ethereum	Ethereum Node-as-a-Service Provider	apiKey	Yes	Yes
apilayer coinlayer	Real-time Crypto Currency Exchange Rates	apiKey	Yes	Unknown
Binance	Exchange for Trading Cryptocurrencies based in China	apiKey	Yes	Unknown
Bitcambio	Get the list of all traded assets in the exchange	No	Yes	Unknown
BitcoinAverage	Digital Asset Price Data for the blockchain industry	apiKey	Yes	Unknown
BitcoinCharts	Financial and Technical Data related to the Bitcoin Network	No	Yes	Unknown
Bitfinex	Cryptocurrency Trading Platform	apiKey	Yes	Unknown
Bitmex	Real-Time Cryptocurrency derivatives trading platform based in Hong Kong	apiKey	Yes	Unknown
Bittrex	Next Generation Crypto Trading Platform	apiKey	Yes	Unknown
Block	Bitcoin Payment, Wallet & Transaction Data	apiKey	Yes	Unknown
Blockchain	Bitcoin Payment, Wallet & Transaction Data	apiKey	Yes	Unknown
blockfrost Cardano	Interaction with the Cardano mainnet and several testnets	apiKey	Yes	Unknown
Brave NewCoin	Real-time and historic crypto data from more than 200+ exchanges	apiKey	Yes	Unknown
BtcTurk	Real-time cryptocurrency data, graphs and API that allows buy&sell	apiKey	Yes	Yes
Bybit	Cryptocurrency data feed and algorithmic trading	apiKey	Yes	Unknown
CoinAPI	All Currency Exchanges integrate under a single api	apiKey	Yes	No
Coinbase	Bitcoin, Bitcoin Cash, Litecoin and Ethereum Prices	apiKey	Yes	Unknown
Coinbase Pro	Cryptocurrency Trading Platform	apiKey	Yes	Unknown
CoinCap	Real time Cryptocurrency prices through a RESTful API	No	Yes	Unknown
CoinDCX	Cryptocurrency Trading Platform	apiKey	Yes	Unknown
CoinDesk	CoinDesk's Bitcoin Price Index (BPI) in multiple currencies	No	Yes	Unknown
CoinGecko	Cryptocurrency Price, Market, and Developer/Social Data	No	Yes	Yes
Coinigy	Interacting with Coinigy Accounts and Exchange Directly	apiKey	Yes	Unknown
Coinlib	Crypto Currency Prices	apiKey	Yes	Unknown
Coinlore	Cryptocurrencies prices, volume and more	No	Yes	Unknown
CoinMarketCap	Cryptocurrencies Prices	apiKey	Yes	Unknown
Coinpaprika	Cryptocurrencies prices, volume and more	No	Yes	Yes
CoinRanking	Live Cryptocurrency data	apiKey	Yes	Unknown
Coinremitter	Cryptocurrencies Payment & Prices	apiKey	Yes	Unknown
CoinStats	Crypto Tracker	No	Yes	Unknown
CryptAPI	Cryptocurrency Payment Processor	No	Yes	Unknown
CryptingUp	Cryptocurrency data	No	Yes	Unknown
CryptoCompare	Cryptocurrencies Comparison	No	Yes	Unknown
CryptoMarket	Cryptocurrencies Trading platform	apiKey	Yes	Yes
Cryptonator	Cryptocurrencies Exchange Rates	No	Yes	Unknown
dYdX	Decentralized cryptocurrency exchange	apiKey	Yes	Unknown
Ethplorer	Ethereum tokens, balances, addresses, history of transactions, contracts, and custom structures	apiKey	Yes	Unknown
EXMO	Cryptocurrencies exchange based in UK	apiKey	Yes	Unknown
FTX	Complete REST, websocket, and FTX APIs to suit your algorithmic trading needs	apiKey	Yes	Yes
Gateio	API provides spot, margin and futures trading operations	apiKey	Yes	Unknown
Gemini	Cryptocurrencies Exchange	No	Yes	Unknown
Hirak Exchange Rates	Exchange rates between 162 currency & 300 crypto currency update each 5 min, accurate, no limits	apiKey	Yes	Unknown
Huobi	Seychelles based cryptocurrency exchange	apiKey	Yes	Unknown
icy.tools	GraphQL based NFT API	apiKey	Yes	Unknown
Indodax	Trade your Bitcoin and other assets with rupiah	apiKey	Yes	Unknown
INFURA Ethereum	Interaction with the Ethereum mainnet and several testnets	apiKey	Yes	Yes
Kraken	Cryptocurrencies Exchange	apiKey	Yes	Unknown
KuCoin	Cryptocurrency Trading Platform	apiKey	Yes	Unknown
Localbitcoins	P2P platform to buy and sell Bitcoins	No	Yes	Unknown
Mempool	Bitcoin API Service focusing on the transaction fee	No	Yes	No
MercadoBitcoin	Brazilian Cryptocurrency Information	No	Yes	Unknown
Messari	Provides API endpoints for thousands of crypto assets	No	Yes	Unknown
Nexchange	Automated cryptocurrency exchange service	No	No	Yes
Nomics	Historical and realtime cryptocurrency prices and market data	apiKey	Yes	Yes
NovaDax	NovaDAX API to access all market data, trading management endpoints	apiKey	Yes	Unknown
OKEx	Cryptocurrency exchange based in Seychelles	apiKey	Yes	Unknown
Poloniex	US based digital asset exchange	apiKey	Yes	Unknown
Solana JSON RPC	Provides various endpoints to interact with the Solana Blockchain	No	Yes	Unknown
Technical Analysis	Cryptocurrency prices and technical analysis	apiKey	Yes	No
VALR	Cryptocurrency Exchange based in South Africa	apiKey	Yes	Unknown
WorldCoinIndex	Cryptocurrencies Prices	apiKey	Yes	Unknown
ZMOK	Ethereum JSON RPC API and Web3 provider	No	Yes	Unknown

⬆ Back to Index

Currency Exchange

API Description Auth HTTPS CORS

1Forge	Forex currency market data	apiKey	Yes	Unknown
Amdoren	Free currency API with over 150 currencies	apiKey	Yes	Unknown
apilayer fixer.io	Exchange rates and currency conversion	apiKey	No	Unknown
Bank of Russia	Exchange rates and currency conversion	No	Yes	Unknown
Currency-api	Free Currency Exchange Rates API with 150+ Currencies & No Rate Limits	No	Yes	Yes
CurrencyFreaks	Provides current and historical currency exchange rates with free plan 1K requests/month	apiKey	Yes	Yes
Currencylayer	Exchange rates and currency conversion	apiKey	Yes	Unknown
CurrencyScoop	Real-time and historical currency rates JSON API	apiKey	Yes	Yes
Czech National Bank	A collection of exchange rates	No	Yes	Unknown
Economia.Awesome	Portuguese free currency prices and conversion with no rate limits	No	Yes	Unknown
ExchangeRate-API	Free currency conversion	apiKey	Yes	Yes
Exchangerate.host	Free foreign exchange & crypto rates API	No	Yes	Unknown
Exchangeratesapi.io	Exchange rates with currency conversion	apiKey	Yes	Yes
Frankfurter	Exchange rates, currency conversion and time series	No	Yes	Yes
FreeForexAPI	Real-time foreign exchange rates for major currency pairs	No	Yes	No
National Bank of Poland	A collection of currency exchange rates (data in XML and JSON)	No	Yes	Yes
VATComply.com	Exchange rates, geolocation and VAT number validation	No	Yes	Yes

⬆ Back to Index

Data Validation

API Description Auth HTTPS CORS

Lob.com	US Address Verification	apiKey	Yes	Unknown
Postman Echo	Test api server to receive and return value from HTTP method	No	Yes	Unknown
PurgoMalum	Content validator against profanity & obscenity	No	No	Unknown
US Autocomplete	Enter address data quickly with real-time address suggestions	apiKey	Yes	Yes
US Extract	Extract postal addresses from any text including emails	apiKey	Yes	Yes
US Street Address	Validate and append data for any US postal address	apiKey	Yes	Yes
vatlayer	VAT number validation	apiKey	Yes	Unknown

⬆ Back to Index

Development

API Description Auth HTTPS CORS

24 Pull Requests	Project to promote open source collaboration during December	No	Yes	Yes
Abstract Screenshot	Take programmatic screenshots of web pages from any website	apiKey	Yes	Yes
Agify.io	Estimates the age from a first name	No	Yes	Yes
API Grátis	Multiples services and public APIs	No	Yes	Unknown
ApicAgent	Extract device details from user-agent string	No	Yes	Yes
ApiFlash	Chrome based screenshot API for developers	apiKey	Yes	Unknown
apilayer userstack	Secure User-Agent String Lookup JSON API	OAuth	Yes	Unknown
APIs.guru	Wikipedia for Web APIs, OpenAPI/Swagger specs for public APIs	No	Yes	Unknown
Azure DevOps	The Azure DevOps basic components of a REST API request/response pair	apiKey	Yes	Unknown
Base	Building quick backends	apiKey	Yes	Yes
Beeceptor	Build a mock Rest API endpoint in seconds	No	Yes	Yes
Bitbucket	Bitbucket API	OAuth	Yes	Unknown
Blague.xyz	La plus grande API de Blagues FR/The biggest FR jokes API	apiKey	Yes	Yes
Blitapp	Schedule screenshots of web pages and sync them to your cloud	apiKey	Yes	Unknown
Blynk-Cloud	Control IoT Devices from Blynk IoT Cloud	apiKey	No	Unknown
Bored	Find random activities to fight boredom	No	Yes	Unknown
Brainshop.ai	Make A Free A.I Brain	apiKey	Yes	Yes
Browshot	Easily make screenshots of web pages in any screen size, as any device	apiKey	Yes	Yes
CDNJS	Library info on CDNJS	No	Yes	Unknown
Changelogs.md	Structured changelog metadata from open source projects	No	Yes	Unknown
Ciprand	Secure random string generator	No	Yes	No
Cloudflare Trace	Get IP Address, Timestamp, User Agent, Country Code, IATA, HTTP Version, TLS/SSL Version & More	No	Yes	Yes
Codex	Online Compiler for Various Languages	No	Yes	Unknown
Contentful Images	Used to retrieve and apply transformations to images	apiKey	Yes	Yes
CORS Proxy	Get around the dreaded CORS error by using this proxy as a middle man	No	Yes	Yes
CountAPI	Free and simple counting service. You can use it to track page hits and specific events	No	Yes	Yes
Databricks	Service to manage your databricks account,clusters, notebooks, jobs and workspaces	apiKey	Yes	Yes
DigitalOcean Status	Status of all DigitalOcean services	No	Yes	Unknown
Docker Hub	Interact with Docker Hub	apiKey	Yes	Yes
DomainDb Info	Domain name search to find all domains containing particular words/phrases/etc	No	Yes	Unknown
ExtendsClass JSON Storage	A simple JSON store API	No	Yes	Yes
GeekFlare	Provide numerous capabilities for important testing and monitoring methods for websites	apiKey	Yes	Unknown
Genderize.io	Estimates a gender from a first name	No	Yes	Yes
GETPing	Trigger an email notification with a simple GET request	apiKey	Yes	Unknown
Ghost	Get Published content into your Website, App or other embedded media	apiKey	Yes	Yes
GitHub	Make use of GitHub repositories, code and user info programmatically	OAuth	Yes	Yes
Gitlab	Automate GitLab interaction programmatically	OAuth	Yes	Unknown
Gitter	Chat for Developers	OAuth	Yes	Unknown
Glitterly	Image generation API	apiKey	Yes	Yes
Google Docs	API to read, write, and format Google Docs documents	OAuth	Yes	Unknown
Google Firebase	Google's mobile application development platform that helps build, improve, and grow app	apiKey	Yes	Yes
Google Fonts	Metadata for all families served by Google Fonts	apiKey	Yes	Unknown
Google Keep	API to read, write, and format Google Keep notes	OAuth	Yes	Unknown
Google Sheets	API to read, write, and format Google Sheets data	OAuth	Yes	Unknown
Google Slides	API to read, write, and format Google Slides presentations	OAuth	Yes	Unknown
Gorest	Online REST API for Testing and Prototyping	OAuth	Yes	Unknown
Hasura	GraphQL and REST API Engine with built in Authorization	apiKey	Yes	Yes
Heroku	REST API to programmatically create apps, provision add-ons and perform other task on Heroku	OAuth	Yes	Yes
host-t.com	Basic DNS query via HTTP GET request	No	Yes	No
Host.io	Domains Data API for Developers	apiKey	Yes	Yes
HTTP2.Pro	Test endpoints for client and server HTTP/2 protocol support	No	Yes	Unknown
Httpbin	A Simple HTTP Request & Response Service	No	Yes	Yes
Httpbin Cloudflare	A Simple HTTP Request & Response Service with HTTP/3 Support by Cloudflare	No	Yes	Yes
Hunter	API for domain search, professional email finder, author finder and email verifier	apiKey	Yes	Unknown
IBM Text to Speech	Convert text to speech	apiKey	Yes	Yes
Icanhazepoch	Get Epoch time	No	Yes	Yes
Icanhazip	IP Address API	No	Yes	Yes
IFTTT	IFTTT Connect API	No	Yes	Unknown
Image-Charts	Generate charts, QR codes and graph images	No	Yes	Yes
import.io	Retrieve structured data from a website or RSS feed	apiKey	Yes	Unknown
ip-fast.com	IP address, country and city	No	Yes	Yes
IP2WHOIS Information Lookup	WHOIS domain name lookup	apiKey	Yes	Unknown
ipfind.io	Geographic location of an IP address or any domain name along with some other useful information	apiKey	Yes	Yes
IPify	A simple IP Address API	No	Yes	Unknown
IPinfo	Another simple IP Address API	No	Yes	Unknown
jsDelivr	Package info and download stats on jsDelivr CDN	No	Yes	Yes
JSON 2 JSONP	Convert JSON to JSONP (on-the-fly) for easy cross-domain data requests using client-side JavaScript	No	Yes	Unknown
JSONbin.io	Free JSON storage service. Ideal for small scale Web apps, Websites and Mobile apps	apiKey	Yes	Yes
Kroki	Creates diagrams from textual descriptions	No	Yes	Yes
License-API	Unofficial REST API for choosealicense.com	No	Yes	No
Logs.to	Generate logs	apiKey	Yes	Unknown
Lua Decompiler	Online Lua 5.1 Decompiler	No	Yes	Yes
MAC address vendor lookup	Retrieve vendor details and other information regarding a given MAC address or an OUI	apiKey	Yes	Yes
Micro DB	Simple database service	apiKey	Yes	Unknown
MicroENV	Fake Rest API for developers	No	Yes	Unknown
Mocky	Mock user defined test JSON for REST API endpoints	No	Yes	Yes
MY IP	Get IP address information	No	Yes	Unknown
Nationalize.io	Estimate the nationality of a first name	No	Yes	Yes
Netlify	Netlify is a hosting service for the programmable web	OAuth	Yes	Unknown
NetworkCalc	Network calculators, including subnets, DNS, binary, and security tools	No	Yes	Yes
npm Registry	Query information about your favorite Node.js libraries programatically	No	Yes	Unknown
OneSignal	Self-serve customer engagement solution for Push Notifications, Email, SMS & In-App	apiKey	Yes	Unknown
Open Page Rank	API for calculating and comparing metrics of different websites using Page Rank algorithm	apiKey	Yes	Unknown
OpenAPIHub	The All-in-one API Platform	X-Mashape-Key	Yes	Unknown
OpenGraphr	Really simple API to retrieve Open Graph data from an URL	apiKey	Yes	Unknown
oyyi	API for Fake Data, image/video conversion, optimization, pdf optimization and thumbnail generation	No	Yes	Yes
PageCDN	Public API for javascript, css and font libraries on PageCDN	apiKey	Yes	Yes
Postman	Tool for testing APIs	apiKey	Yes	Unknown
ProxyCrawl	Scraping and crawling anticaptcha service	apiKey	Yes	Unknown
ProxyKingdom	Rotating Proxy API that produces a working proxy on every request	apiKey	Yes	Yes
Pusher Beams	Push notifications for Android & iOS	apiKey	Yes	Unknown
QR code	Create an easy to read QR code and URL shortener	No	Yes	Yes
QR code	Generate and decode / read QR code graphics	No	Yes	Unknown
Qrcode Monkey	Integrate custom and unique looking QR codes into your system or workflow	No	Yes	Unknown
QuickChart	Generate chart and graph images	No	Yes	Yes
Random Stuff	Can be used to get AI Response, jokes, memes, and much more at lightning-fast speed	apiKey	Yes	Yes
Rejax	Reverse AJAX service to notify clients	apiKey	Yes	No
ReqRes	A hosted REST-API ready to respond to your AJAX requests	No	Yes	Unknown
RSS feed to JSON	Returns RSS feed in JSON format using feed URL	No	Yes	Yes
SavePage.io	A free, RESTful API used to screenshot any desktop, or mobile website	apiKey	Yes	Yes
ScrapeNinja	Scraping API with Chrome fingerprint and residential proxies	apiKey	Yes	Unknown
ScraperApi	Easily build scalable web scrapers	apiKey	Yes	Unknown
scraperBox	Undetectable web scraping API	apiKey	Yes	Yes
scrapestack	Real-time, Scalable Proxy & Web Scraping REST API	apiKey	Yes	Unknown
ScrapingAnt	Headless Chrome scraping with a simple API	apiKey	Yes	Unknown
ScrapingDog	Proxy API for Web scraping	apiKey	Yes	Unknown
ScreenshotAPI.net	Create pixel-perfect website screenshots	apiKey	Yes	Yes
Serialif Color	Color conversion, complementary, grayscale and contrasted text	No	Yes	No
serpstack	Real-Time & Accurate Google Search Results API	apiKey	Yes	Yes
Sheetsu	Easy google sheets integration	apiKey	Yes	Unknown
SHOUTCLOUD	ALL-CAPS AS A SERVICE	No	No	Unknown
Sonar	Project Sonar DNS Enumeration API	No	Yes	Yes
SonarQube	SonarQube REST APIs to detect bugs, code smells & security vulnerabilities	OAuth	Yes	Unknown
StackExchange	Q&A forum for developers	OAuth	Yes	Unknown
Statically	A free CDN for developers	No	Yes	Yes
Supportivekoala	Autogenerate images with template	apiKey	Yes	Yes
Tyk	Api and service management platform	apiKey	Yes	Yes
Wandbox	Code compiler supporting 35+ languages mentioned at wandbox.org	No	Yes	Unknown
WebScraping.AI	Web Scraping API with built-in proxies and JS rendering	apiKey	Yes	Yes
ZenRows	Web Scraping API that bypasses anti-bot solutions while offering JS rendering, and rotating proxies	apiKey	Yes	Unknown

⬆ Back to Index

Dictionaries

API Description Auth HTTPS CORS

Chinese Character Web	Chinese character definitions and pronunciations	No	No	No
Chinese Text Project	Online open-access digital library for pre-modern Chinese texts	No	Yes	Unknown
Collins	Bilingual Dictionary and Thesaurus Data	apiKey	Yes	Unknown
Free Dictionary	Definitions, phonetics, pronounciations, parts of speech, examples, synonyms	No	Yes	Unknown
Indonesia Dictionary	Indonesia dictionary many words	No	Yes	Unknown
Lingua Robot	Word definitions, pronunciations, synonyms, antonyms and others	apiKey	Yes	Yes
Merriam-Webster	Dictionary and Thesaurus Data	apiKey	Yes	Unknown
OwlBot	Definitions with example sentence and photo if available	apiKey	Yes	Yes
Oxford	Dictionary Data	apiKey	Yes	No
Synonyms	Synonyms, thesaurus and antonyms information for any given word	apiKey	Yes	Unknown
Wiktionary	Collaborative dictionary data	No	Yes	Yes
Wordnik	Dictionary Data	apiKey	Yes	Unknown
Words	Definitions and synonyms for more than 150,000 words	apiKey	Yes	Unknown

⬆ Back to Index

Documents & Productivity

API Description Auth HTTPS CORS

Airtable	Integrate with Airtable	apiKey	Yes	Unknown
Api2Convert	Online File Conversion API	apiKey	Yes	Unknown
apilayer pdflayer	HTML/URL to PDF	apiKey	Yes	Unknown
Asana	Programmatic access to all data in your asana system	apiKey	Yes	Yes
ClickUp	ClickUp is a robust, cloud-based project management tool for boosting productivity	OAuth	Yes	Unknown
Clockify	Clockify's REST-based API can be used to push/pull data to/from it & integrate it with other systems	apiKey	Yes	Unknown
CloudConvert	Online file converter for audio, video, document, ebook, archive, image, spreadsheet, presentation	apiKey	Yes	Unknown
Cloudmersive Document and Data Conversion	HTML/URL to PDF/PNG, Office documents to PDF, image conversion	apiKey	Yes	Yes
Code::Stats	Automatic time tracking for programmers	apiKey	Yes	No
CraftMyPDF	Generate PDF documents from templates with a drop-and-drop editor and a simple API	apiKey	Yes	No
Flowdash	Automate business workflows	apiKey	Yes	Unknown
Html2PDF	HTML/URL to PDF	apiKey	Yes	Unknown
iLovePDF	Convert, merge, split, extract text and add page numbers for PDFs. Free for 250 documents/month	apiKey	Yes	Yes
JIRA	JIRA is a proprietary issue tracking product that allows bug tracking and agile project management	OAuth	Yes	Unknown
Mattermost	An open source platform for developer collaboration	OAuth	Yes	Unknown
Mercury	Web parser	apiKey	Yes	Unknown
Monday	Programmatically access and update data inside a monday.com account	apiKey	Yes	Unknown
Notion	Integrate with Notion	OAuth	Yes	Unknown
PandaDoc	DocGen and eSignatures API	apiKey	Yes	No
Pocket	Bookmarking service	OAuth	Yes	Unknown
Podio	File sharing and productivity	OAuth	Yes	Unknown
PrexView	Data from XML or JSON to PDF, HTML or Image	apiKey	Yes	Unknown
Restpack	Provides screenshot, HTML to PDF and content extraction APIs	apiKey	Yes	Unknown
Todoist	Todo Lists	OAuth	Yes	Unknown
Smart Image Enhancement API	Performs image upscaling by adding detail to images through multiple super-resolution algorithms	apiKey	Yes	Unknown
Vector Express v2.0	Free vector file converting API	No	Yes	No
WakaTime	Automated time tracking leaderboards for programmers	No	Yes	Unknown
Zube	Full stack project management	OAuth	Yes	Unknown

⬆ Back to Index

Email

API Description Auth HTTPS CORS

Abstract Email Validation	Validate email addresses for deliverability and spam	apiKey	Yes	Yes
apilayer mailboxlayer	Email address validation	apiKey	Yes	Unknown
Cloudmersive Validate	Validate email addresses, phone numbers, VAT numbers and domain names	apiKey	Yes	Yes
Disify	Validate and detect disposable and temporary email addresses	No	Yes	Yes
DropMail	GraphQL API for creating and managing ephemeral e-mail inboxes	No	Yes	Unknown
EVA	Validate email addresses	No	Yes	Yes
Guerrilla Mail	Disposable temporary Email addresses	No	Yes	Unknown
ImprovMX	API for free email forwarding service	apiKey	Yes	Unknown
Kickbox	Email verification API	No	Yes	Yes
mail.gw	10 Minute Mail	No	Yes	Yes
mail.tm	Temporary Email Service	No	Yes	Yes
MailboxValidator	Validate email address to improve deliverability	apiKey	Yes	Unknown
MailCheck.ai	Prevent users to sign up with temporary email addresses	No	Yes	Unknown
Mailtrap	A service for the safe testing of emails sent from the development and staging environments	apiKey	Yes	Unknown
Sendgrid	A cloud-based SMTP provider that allows you to send emails without having to maintain email servers	apiKey	Yes	Unknown
Sendinblue	A service that provides solutions relating to marketing and/or transactional email and/or SMS	apiKey	Yes	Unknown
Verifier	Verifies that a given email is real	apiKey	Yes	Yes

⬆ Back to Index

Entertainment

API Description Auth HTTPS CORS

chucknorris.io	JSON API for hand curated Chuck Norris jokes	No	Yes	Unknown
Corporate Buzz Words	REST API for Corporate Buzz Words	No	Yes	Yes
Excuser	Get random excuses for various situations	No	Yes	Unknown
Fun Fact	A simple HTTPS api that can randomly select and return a fact from the FFA database	No	Yes	Yes
Imgflip	Gets an array of popular memes	No	Yes	Unknown
Meme Maker	REST API for create your own meme	No	Yes	Unknown
NaMoMemes	Memes on Narendra Modi	No	Yes	Unknown
Random Useless Facts	Get useless, but true facts	No	Yes	Unknown
Techy	JSON and Plaintext API for tech-savvy sounding phrases	No	Yes	Unknown
Yo Momma Jokes	REST API for Yo Momma Jokes	No	Yes	Unknown

⬆ Back to Index

Environment

API Description Auth HTTPS CORS

BreezoMeter Pollen	Daily Forecast pollen conditions data for a specific location	apiKey	Yes	Unknown
Carbon Interface	API to calculate carbon (C02) emissions estimates for common C02 emitting activities	apiKey	Yes	Yes
Climatiq	Calculate the environmental footprint created by a broad range of emission-generating activities	apiKey	Yes	Yes
Cloverly	API calculates the impact of common carbon-intensive activities in real time	apiKey	Yes	Unknown
CO2 Offset	API calculates and validates the carbon footprint	No	Yes	Unknown
Danish data service Energi	Open energy data from Energinet to society	No	Yes	Unknown
GrünstromIndex	Green Power Index for Germany (Grünstromindex/GSI)	No	No	Yes
IQAir	Air quality and weather data	apiKey	Yes	Unknown
Luchtmeetnet	Predicted and actual air quality components for The Netherlands (RIVM)	No	Yes	Unknown
National Grid ESO	Open data from Great Britain’s Electricity System Operator	No	Yes	Unknown
OpenAQ	Open air quality data	apiKey	Yes	Unknown
PM2.5 Open Data Portal	Open low-cost PM2.5 sensor data	No	Yes	Unknown
PM25.in	Air quality of China	apiKey	No	Unknown
PVWatts	Energy production photovoltaic (PV) energy systems	apiKey	Yes	Unknown
Srp Energy	Hourly usage energy report for Srp customers	apiKey	Yes	No
UK Carbon Intensity	The Official Carbon Intensity API for Great Britain developed by National Grid	No	Yes	Unknown
Website Carbon	API to estimate the carbon footprint of loading web pages	No	Yes	Unknown

⬆ Back to Index

Events

API Description Auth HTTPS CORS

Eventbrite	Find events	OAuth	Yes	Unknown
SeatGeek	Search events, venues and performers	apiKey	Yes	Unknown
Ticketmaster	Search events, attractions, or venues	apiKey	Yes	Unknown

⬆ Back to Index

Finance

API Description Auth HTTPS CORS

Abstract VAT Validation	Validate VAT numbers and calculate VAT rates	apiKey	Yes	Yes
Aletheia	Insider trading data, earnings call analysis, financial statements, and more	apiKey	Yes	Yes
Alpaca	Realtime and historical market data on all US equities and ETFs	apiKey	Yes	Yes
Alpha Vantage	Realtime and historical stock data	apiKey	Yes	Unknown
apilayer marketstack	Real-Time, Intraday & Historical Market Data API	apiKey	Yes	Unknown
Banco do Brasil	All Banco do Brasil financial transaction APIs	OAuth	Yes	Yes
Bank Data API	Instant IBAN and SWIFT number validation across the globe	apiKey	Yes	Unknown
Billplz	Payment platform	apiKey	Yes	Unknown
Binlist	Public access to a database of IIN/BIN information	No	Yes	Unknown
Boleto.Cloud	A api to generate boletos in Brazil	apiKey	Yes	Unknown
Citi	All Citigroup account and statement data APIs	apiKey	Yes	Unknown
Econdb	Global macroeconomic data	No	Yes	Yes
Fed Treasury	U.S. Department of the Treasury Data	No	Yes	Unknown
Finage	Finage is a stock, currency, cryptocurrency, indices, and ETFs real-time & historical data provider	apiKey	Yes	Unknown
Financial Modeling Prep	Realtime and historical stock data	apiKey	Yes	Unknown
Finnhub	Real-Time RESTful APIs and Websocket for Stocks, Currencies, and Crypto	apiKey	Yes	Unknown
FRED	Economic data from the Federal Reserve Bank of St. Louis	apiKey	Yes	Yes
Front Accounting APIs	Front accounting is multilingual and multicurrency software for small businesses	OAuth	Yes	Yes
Hotstoks	Stock market data powered by SQL	apiKey	Yes	Yes
IEX Cloud	Realtime & Historical Stock and Market Data	apiKey	Yes	Yes
IG	Spreadbetting and CFD Market Data	apiKey	Yes	Unknown
Indian Mutual Fund	Get complete history of India Mutual Funds Data	No	Yes	Unknown
Intrinio	A wide selection of financial data feeds	apiKey	Yes	Unknown
Klarna	Klarna payment and shopping service	apiKey	Yes	Unknown
MercadoPago	Mercado Pago API reference - all the information you need to develop your integrations	apiKey	Yes	Unknown
Mono	Connect with users’ bank accounts and access transaction data in Africa	apiKey	Yes	Unknown
Moov	The Moov API makes it simple for platforms to send, receive, and store money	apiKey	Yes	Unknown
Nordigen	Connect to bank accounts using official bank APIs and get raw transaction data	apiKey	Yes	Unknown
OpenFIGI	Equity, index, futures, options symbology from Bloomberg LP	apiKey	Yes	Yes
Plaid	Connect with user's bank accounts and access transaction data	apiKey	Yes	Unknown
Polygon	Historical stock market data	apiKey	Yes	Unknown
Portfolio Optimizer	Portfolio analysis and optimization	No	Yes	Yes
Razorpay IFSC	Indian Financial Systems Code (Bank Branch Codes)	No	Yes	Unknown
Real Time Finance	Websocket API to access realtime stock data	apiKey	No	Unknown
SEC EDGAR Data	API to access annual reports of public US companies	No	Yes	Yes
SmartAPI	Gain access to set of and create end-to-end broking services	apiKey	Yes	Unknown
StockData	Real-Time, Intraday & Historical Market Data, News and Sentiment API	apiKey	Yes	Yes
Styvio	Realtime and historical stock data and current stock sentiment	apiKey	Yes	Unknown
Tax Data API	Instant VAT number and tax validation across the globe	apiKey	Yes	Unkown
Tradier	US equity/option market data (delayed, intraday, historical)	OAuth	Yes	Yes
Twelve Data	Stock market data (real-time & historical)	apiKey	Yes	Unknown
WallstreetBets	WallstreetBets Stock Comments Sentiment Analysis	No	Yes	Unknown
Yahoo Finance	Real time low latency Yahoo Finance API for stock market, crypto currencies, and currency exchange	apiKey	Yes	Yes
YNAB	Budgeting & Planning	OAuth	Yes	Yes
Zoho Books	Online accounting software, built for your business	OAuth	Yes	Unknown

⬆ Back to Index

Food & Drink

API Description Auth HTTPS CORS

BaconMockup	Resizable bacon placeholder images	No	Yes	Yes
Chomp	Data about various grocery products and foods	apiKey	Yes	Unknown
Coffee	Random pictures of coffee	No	Yes	Unknown
Edamam nutrition	Nutrition Analysis	apiKey	Yes	Unknown
Edamam recipes	Recipe Search	apiKey	Yes	Unknown
Foodish	Random pictures of food dishes	No	Yes	Yes
Fruityvice	Data about all kinds of fruit	No	Yes	Unknown
Kroger	Supermarket Data	apiKey	Yes	Unknown
LCBO	Alcohol	apiKey	Yes	Unknown
Open Brewery DB	Breweries, Cideries and Craft Beer Bottle Shops	No	Yes	Yes
Open Food Facts	Food Products Database	No	Yes	Unknown
PunkAPI	Brewdog Beer Recipes	No	Yes	Unknown
Rustybeer	Beer brewing tools	No	Yes	No
Spoonacular	Recipes, Food Products, and Meal Planning	apiKey	Yes	Unknown
Systembolaget	Govornment owned liqour store in Sweden	apiKey	Yes	Unknown
TacoFancy	Community-driven taco database	No	No	Unknown
Tasty	API to query data about recipe, plan, ingredients	apiKey	Yes	Unknown
The Report of the Week	Food & Drink Reviews	No	Yes	Unknown
TheCocktailDB	Cocktail Recipes	apiKey	Yes	Yes
TheMealDB	Meal Recipes	apiKey	Yes	Yes
Untappd	Social beer sharing	OAuth	Yes	Unknown
What's on the menu?	NYPL human-transcribed historical menu collection	apiKey	No	Unknown
WhiskyHunter	Past online whisky auctions statistical data	No	Yes	Unknown
Zestful	Parse recipe ingredients	apiKey	Yes	Yes

⬆ Back to Index

Games & Comics

API Description Auth HTTPS CORS

Age of Empires II	Get information about Age of Empires II resources	No	Yes	No
AmiiboAPI	Nintendo Amiibo Information	No	Yes	Yes
Animal Crossing: New Horizons	API for critters, fossils, art, music, furniture and villagers	No	Yes	Unknown
Autochess VNG	Rest Api for Autochess VNG	No	Yes	Yes
Barter.VG	Provides information about Game, DLC, Bundles, Giveaways, Trading	No	Yes	Yes
Battle.net	Diablo III, Hearthstone, StarCraft II and World of Warcraft game data APIs	OAuth	Yes	Yes
Board Game Geek	Board games, RPG and videogames	No	Yes	No
Brawl Stars	Brawl Stars Game Information	apiKey	Yes	Unknown
Bugsnax	Get information about Bugsnax	No	Yes	Yes
CheapShark	Steam/PC Game Prices and Deals	No	Yes	Yes
Chess.com	Chess.com read-only REST API	No	Yes	Unknown
Chuck Norris Database	Jokes	No	No	Unknown
Clash of Clans	Clash of Clans Game Information	apiKey	Yes	Unknown
Clash Royale	Clash Royale Game Information	apiKey	Yes	Unknown
Comic Vine	Comics	No	Yes	Unknown
Crafatar	API for Minecraft skins and faces	No	Yes	Yes
Cross Universe	Cross Universe Card Data	No	Yes	Yes
Deck of Cards	Deck of Cards	No	No	Unknown
Destiny The Game	Bungie Platform API	apiKey	Yes	Unknown
Digimon Information	Provides information about digimon creatures	No	Yes	Unknown
Digimon TCG	Search for Digimon cards in digimoncard.io	No	Yes	Unknown
Disney	Information of Disney characters	No	Yes	Yes
Dota 2	Provides information about Player stats , Match stats, Rankings for Dota 2	apiKey	Yes	Unknown
Dungeons and Dragons	Reference for 5th edition spells, classes, monsters, and more	No	No	No
Dungeons and Dragons (Alternate)	Includes all monsters and spells from the SRD (System Reference Document) as well as a search API	No	Yes	Yes
Eve Online	Third-Party Developer Documentation	OAuth	Yes	Unknown
FFXIV Collect	Final Fantasy XIV data on collectables	No	Yes	Yes
FIFA Ultimate Team	FIFA Ultimate Team items API	No	Yes	Unknown
Final Fantasy XIV	Final Fantasy XIV Game data API	No	Yes	Yes
Fortnite	Fortnite Stats	apiKey	Yes	Unknown
Forza	Show random image of car from Forza	No	Yes	Unknown
FreeToGame	Free-To-Play Games Database	No	Yes	Yes
Fun Facts	Random Fun Facts	No	Yes	Yes
FunTranslations	Translate Text into funny languages	No	Yes	Yes
GamerPower	Game Giveaways Tracker	No	Yes	Yes
GDBrowser	Easy way to use the Geometry Dash Servers	No	Yes	Unknown
Geek-Jokes	Fetch a random geeky/programming related joke for use in all sorts of applications	No	Yes	Yes
Genshin Impact	Genshin Impact game data	No	Yes	Yes
Giant Bomb	Video Games	apiKey	Yes	Unknown
GraphQL Pokemon	GraphQL powered Pokemon API. Supports generations 1 through 8	No	Yes	Yes
Guild Wars 2	Guild Wars 2 Game Information	apiKey	Yes	Unknown
GW2Spidy	GW2Spidy API, Items data on the Guild Wars 2 Trade Market	No	Yes	Unknown
Halo	Halo 5 and Halo Wars 2 Information	apiKey	Yes	Unknown
Hearthstone	Hearthstone Cards Information	X-Mashape-Key	Yes	Unknown
Humble Bundle	Humble Bundle's current bundles	apiKey	Yes	Unknown
Humor	Humor, Jokes, and Memes	apiKey	Yes	Unknown
Hypixel	Hypixel player stats	apiKey	Yes	Unknown
Hyrule Compendium	Data on all interactive items from The Legend of Zelda: BOTW	No	Yes	Unknown
Hytale	Hytale blog posts and jobs	No	Yes	Unknown
IGDB.com	Video Game Database	apiKey	Yes	Unknown
JokeAPI	Programming, Miscellaneous and Dark Jokes	No	Yes	Yes
Jokes One	Joke of the day and large category of jokes accessible via REST API	apiKey	Yes	Yes
Jservice	Jeopardy Question Database	No	No	Unknown
Lichess	Access to all data of users, games, puzzles and etc on Lichess	OAuth	Yes	Unknown
Magic The Gathering	Magic The Gathering Game Information	No	No	Unknown
Mario Kart Tour	API for Drivers, Karts, Gliders and Courses	OAuth	Yes	Unknown
Marvel	Marvel Comics	apiKey	Yes	Unknown
Minecraft Server Status	API to get Information about a Minecraft Server	No	Yes	No
MMO Games	MMO Games Database, News and Giveaways	No	Yes	No
mod.io	Cross Platform Mod API	apiKey	Yes	Unknown
Mojang	Mojang / Minecraft API	apiKey	Yes	Unknown
Monster Hunter World	Monster Hunter World data	No	Yes	Yes
Open Trivia	Trivia Questions	No	Yes	Unknown
PandaScore	E-sports games and results	apiKey	Yes	Unknown
Path of Exile	Path of Exile Game Information	OAuth	Yes	Unknown
PlayerDB	Query Minecraft, Steam and XBox Accounts	No	Yes	Unknown
Pokéapi	Pokémon Information	No	Yes	Unknown
PokéAPI (GraphQL)	The Unofficial GraphQL for PokeAPI	No	Yes	Yes
Pokémon TCG	Pokémon TCG Information	No	Yes	Unknown
Psychonauts	Psychonauts World Characters Information and PSI Powers	No	Yes	Yes
PUBG	Access in-game PUBG data	apiKey	Yes	Yes
Puyo Nexus	Puyo Puyo information from Puyo Nexus Wiki	No	Yes	Yes
quizapi.io	Access to various kind of quiz questions	apiKey	Yes	Yes
Raider	Provides detailed character and guild rankings for Raiding and Mythic+ content in World of Warcraft	No	Yes	Unknown
RAWG.io	500,000+ games for 50 platforms including mobiles	apiKey	Yes	Unknown
Rick and Morty	All the Rick and Morty information, including images	No	Yes	Yes
Riot Games	League of Legends Game Information	apiKey	Yes	Unknown
RPS 101	Rock, Paper, Scissors with 101 objects	No	Yes	Yes
RuneScape	RuneScape and OSRS RPGs information	No	Yes	No
Sakura CardCaptor	Sakura CardCaptor Cards Information	No	Yes	Unknown
Scryfall	Magic: The Gathering database	No	Yes	Yes
SpaceTradersAPI	A playable inter-galactic space trading MMOAPI	OAuth	Yes	Yes
Steam	Steam Web API documentation	apiKey	Yes	No
Steam	Internal Steam Web API documentation	No	Yes	No
SuperHeroes	All SuperHeroes and Villains data from all universes under a single API	apiKey	Yes	Unknown
TCGdex	Multi languages Pokémon TCG Information	No	Yes	Yes
Tebex	Tebex API for information about game purchases	X-Mashape-Key	Yes	No
TETR.IO	TETR.IO Tetra Channel API	No	Yes	Unknown
Tronald Dump	The dumbest things Donald Trump has ever said	No	Yes	Unknown
Universalis	Final Fantasy XIV market board data	No	Yes	Yes
Valorant (non-official)	An extensive API containing data of most Valorant in-game items, assets and more	No	Yes	Unknown
Warface (non-official)	Official API proxy with better data structure and more features	No	Yes	No
Wargaming.net	Wargaming.net info and stats	apiKey	Yes	No
When is next MCU film	Upcoming MCU film information	No	Yes	Unknown
xkcd	Retrieve xkcd comics as JSON	No	Yes	No
Yu-Gi-Oh!	Yu-Gi-Oh! TCG Information	No	Yes	Unknown

⬆ Back to Index

Geocoding

API Description Auth HTTPS CORS

Abstract IP Geolocation	Geolocate website visitors from their IPs	apiKey	Yes	Yes
Actinia Grass GIS	Actinia is an open source REST API for geographical data that uses GRASS GIS	apiKey	Yes	Unknown
administrative-divisons-db	Get all administrative divisions of a country	No	Yes	Yes
adresse.data.gouv.fr	Address database of France, geocoding and reverse	No	Yes	Unknown
Airtel IP	IP Geolocation API. Collecting data from multiple sources	No	Yes	Unknown
Apiip	Get location information by IP address	apiKey	Yes	Yes
apilayer ipstack	Locate and identify website visitors by IP address	apiKey	Yes	Unknown
Battuta	A (country/region/city) in-cascade location API	apiKey	No	Unknown
BigDataCloud	Provides fast and accurate IP geolocation APIs along with security checks and confidence area	apiKey	Yes	Unknown
Bing Maps	Create/customize digital maps based on Bing Maps data	apiKey	Yes	Unknown
bng2latlong	Convert British OSGB36 easting and northing (British National Grid) to WGS84 latitude and longitude	No	Yes	Yes
Cartes.io	Create maps and markers for anything	No	Yes	Unknown
Cep.la	Brazil RESTful API to find information about streets, zip codes, neighborhoods, cities and states	No	No	Unknown
CitySDK	Open APIs for select European cities	No	Yes	Unknown
Country	Get your visitor's country from their IP	No	Yes	Yes
CountryStateCity	World countries, states, regions, provinces, cities & towns in JSON, SQL, XML, YAML, & CSV format	apiKey	Yes	Yes
Ducks Unlimited	API explorer that gives a query URL with a JSON response of locations and cities	No	Yes	No
FreeGeoIP	Free geo ip information, no registration required. 15k/hour rate limit	No	Yes	Yes
GeoApi	French geographical data	No	Yes	Unknown
Geoapify	Forward and reverse geocoding, address autocomplete	apiKey	Yes	Yes
Geocod.io	Address geocoding / reverse geocoding in bulk	apiKey	Yes	Unknown
Geocode.xyz	Provides worldwide forward/reverse geocoding, batch geocoding and geoparsing	No	Yes	Unknown
Geocodify.com	Worldwide geocoding, geoparsing and autocomplete for addresses	apiKey	Yes	Yes
Geodata.gov.gr	Open geospatial data and API service for Greece	No	Yes	Unknown
GeoDataSource	Geocoding of city name by using latitude and longitude coordinates	apiKey	Yes	Unknown
GeoDB Cities	Get global city, region, and country data	apiKey	Yes	Unknown
GeographQL	A Country, State, and City GraphQL API	No	Yes	Yes
GeoJS	IP geolocation with ChatOps integration	No	Yes	Yes
Geokeo	Geokeo geocoding service- with 2500 free api requests daily	No	Yes	Yes
GeoNames	Place names and other geographical data	No	No	Unknown
geoPlugin	IP geolocation and currency conversion	No	Yes	Yes
Google Earth Engine	A cloud-based platform for planetary-scale environmental data analysis	apiKey	Yes	Unknown
Google Maps	Create/customize digital maps based on Google Maps data	apiKey	Yes	Unknown
Graph Countries	Country-related data like currencies, languages, flags, regions+subregions and bordering countries	No	Yes	Unknown
HelloSalut	Get hello translation following user language	No	Yes	Unknown
HERE Maps	Create/customize digital maps based on HERE Maps data	apiKey	Yes	Unknown
Hirak IP to Country	Ip to location with country code, currency code & currency name, fast response, unlimited requests	apiKey	Yes	Unknown
Hong Kong GeoData Store	API for accessing geo-data of Hong Kong	No	Yes	Unknown
IBGE	Aggregate services of IBGE (Brazilian Institute of Geography and Statistics)	No	Yes	Unknown
IP 2 Country	Map an IP to a country	No	Yes	Unknown
IP Address Details	Find geolocation with ip address	No	Yes	Unknown
IP Vigilante	Free IP Geolocation API	No	Yes	Unknown
ip-api	Find location with IP address or domain	No	No	Unknown
IP2Location	IP geolocation web service to get more than 55 parameters	apiKey	Yes	Unknown
IP2Proxy	Detect proxy and VPN using IP address	apiKey	Yes	Unknown
ipapi.co	Find IP address location information	No	Yes	Yes
ipapi.com	Real-time Geolocation & Reverse IP Lookup REST API	apiKey	Yes	Unknown
IPGEO	Unlimited free IP Address API with useful information	No	Yes	Unknown
ipgeolocation	IP Geolocation AP with free plan 30k requests per month	apiKey	Yes	Yes
IPInfoDB	Free Geolocation tools and APIs for country, region, city and time zone lookup by IP address	apiKey	Yes	Unknown
ipstack	Locate and identify website visitors by IP address	apiKey	Yes	Unknown
Kakao Maps	Kakao Maps provide multiple APIs for Korean maps	apiKey	Yes	Unknown
keycdn IP Location Finder	Get the IP geolocation data through the simple REST API. All the responses are JSON encoded	apiKey	Yes	Unknown
LocationIQ	Provides forward/reverse geocoding and batch geocoding	apiKey	Yes	Yes
Longdo Map	Interactive map with detailed places and information portal in Thailand	apiKey	Yes	Yes
Mapbox	Create/customize beautiful digital maps	apiKey	Yes	Unknown
MapQuest	To access tools and resources to map the world	apiKey	Yes	No
Mexico	Mexico RESTful zip codes API	No	Yes	Unknown
Nominatim	Provides worldwide forward / reverse geocoding	No	Yes	Yes
One Map, Singapore	Singapore Land Authority REST API services for Singapore addresses	apiKey	Yes	Unknown
OnWater	Determine if a lat/lon is on water or land	No	Yes	Unknown
Open Topo Data	Elevation and ocean depth for a latitude and longitude	No	Yes	No
OpenCage	Forward and reverse geocoding using open data	apiKey	Yes	Yes
openrouteservice.org	Directions, POIs, isochrones, geocoding (+reverse), elevation, and more	apiKey	Yes	Unknown
OpenStreetMap	Navigation, geolocation and geographical data	OAuth	No	Unknown
Pinball Map	A crowdsourced map of public pinball machines	No	Yes	Yes
positionstack	Forward & Reverse Batch Geocoding REST API	apiKey	Yes	Unknown
Postali	Mexico Zip Codes API	No	Yes	Yes
PostcodeData.nl	Provide geolocation data based on postcode for Dutch addresses	No	No	Unknown
Postcodes.io	Postcode lookup & Geolocation for the UK	No	Yes	Yes
Queimadas INPE	Access to heat focus data (probable wildfire)	No	Yes	Unknown
REST Countries	Get information about countries via a RESTful API	No	Yes	Yes
RoadGoat Cities	Cities content & photos API	apiKey	Yes	No
Rwanda Locations	Rwanda Provences, Districts, Cities, Capital City, Sector, cells, villages and streets	No	Yes	Unknown
SLF	German city, country, river, database	No	Yes	Yes
SpotSense	Add location based interactions to your mobile app	apiKey	Yes	Unknown
Telize	Telize offers location information from any IP address	apiKey	Yes	Yes
TomTom	Maps, Directions, Places and Traffic APIs	apiKey	Yes	Yes
Uebermaps	Discover and share maps with friends	apiKey	Yes	Unknown
US ZipCode	Validate and append data for any US ZipCode	apiKey	Yes	Yes
Utah AGRC	Utah Web API for geocoding Utah addresses	apiKey	Yes	Unknown
ViaCep	Brazil RESTful zip codes API	No	Yes	Unknown
What3Words	Three words as rememberable and unique coordinates worldwide	apiKey	Yes	Unknown
Yandex.Maps Geocoder	Use geocoding to get an object's coordinates from its address	apiKey	Yes	Unknown
ZipCodeAPI	US zip code distance, radius and location API	apiKey	Yes	Unknown
Zippopotam.us	Get information about place such as country, city, state, etc	No	No	Unknown
Ziptastic	Get the country, state, and city of any US zip-code	No	Yes	Unknown

⬆ Back to Index

Government

API Description Auth HTTPS CORS

Bank Negara Malaysia Open Data	Malaysia Central Bank Open Data	No	Yes	Unknown
BCLaws	Access to the laws of British Columbia	No	No	Unknown
Brazil	Community driven API for Brazil Public Data	No	Yes	Yes
Brazil Central Bank Open Data	Brazil Central Bank Open Data	No	Yes	Unknown
Brazil Receita WS	Consult companies by CNPJ for Brazilian companies	No	Yes	Unknown
Brazilian Chamber of Deputies Open Data	Provides legislative information in Apis XML and JSON, as well as files in various formats	No	Yes	No
Census.gov	The US Census Bureau provides various APIs and data sets on demographics and businesses	No	Yes	Unknown
City, Berlin	Berlin(DE) City Open Data	No	Yes	Unknown
City, Gdańsk	Gdańsk (PL) City Open Data	No	Yes	Unknown
City, Gdynia	Gdynia (PL) City Open Data	No	No	Unknown
City, Helsinki	Helsinki(FI) City Open Data	No	Yes	Unknown
City, Lviv	Lviv(UA) City Open Data	No	Yes	Unknown
City, Nantes Open Data	Nantes(FR) City Open Data	apiKey	Yes	Unknown
City, New York Open Data	New York (US) City Open Data	No	Yes	Unknown
City, Prague Open Data	Prague(CZ) City Open Data	No	No	Unknown
City, Toronto Open Data	Toronto (CA) City Open Data	No	Yes	Yes
Code.gov	The primary platform for Open Source and code sharing for the U.S. Federal Government	apiKey	Yes	Unknown
Colorado Information Marketplace	Colorado State Government Open Data	No	Yes	Unknown
Data USA	US Public Data	No	Yes	Unknown
Data.gov	US Government Data	apiKey	Yes	Unknown
Data.parliament.uk	Contains live datasets including information about petitions, bills, MP votes, attendance and more	No	No	Unknown
Deutscher Bundestag DIP	This API provides read access to DIP entities (e.g. activities, persons, printed material)	apiKey	Yes	Unknown
District of Columbia Open Data	Contains D.C. government public datasets, including crime, GIS, financial data, and so on	No	Yes	Unknown
EPA	Web services and data sets from the US Environmental Protection Agency	No	Yes	Unknown
FBI Wanted	Access information on the FBI Wanted program	No	Yes	Unknown
FEC	Information on campaign donations in federal elections	apiKey	Yes	Unknown
Federal Register	The Daily Journal of the United States Government	No	Yes	Unknown
Food Standards Agency	UK food hygiene rating data API	No	No	Unknown
Gazette Data, UK	UK official public record API	OAuth	Yes	Unknown
Gun Policy	International firearm injury prevention and policy	apiKey	Yes	Unknown
INEI	Peruvian Statistical Government Open Data	No	No	Unknown
Interpol Red Notices	Access and search Interpol Red Notices	No	Yes	Unknown
Istanbul (İBB) Open Data	Data sets from the İstanbul Metropolitan Municipality (İBB)	No	Yes	Unknown
National Park Service, US	Data from the US National Park Service	apiKey	Yes	Yes
Open Government, ACT	Australian Capital Territory Open Data	No	Yes	Unknown
Open Government, Argentina	Argentina Government Open Data	No	Yes	Unknown
Open Government, Australia	Australian Government Open Data	No	Yes	Unknown
Open Government, Austria	Austria Government Open Data	No	Yes	Unknown
Open Government, Belgium	Belgium Government Open Data	No	Yes	Unknown
Open Government, Canada	Canadian Government Open Data	No	No	Unknown
Open Government, Colombia	Colombia Government Open Data	No	No	Unknown
Open Government, Cyprus	Cyprus Government Open Data	No	Yes	Unknown
Open Government, Czech Republic	Czech Republic Government Open Data	No	Yes	Unknown
Open Government, Denmark	Denmark Government Open Data	No	Yes	Unknown
Open Government, Estonia	Estonia Government Open Data	apiKey	Yes	Unknown
Open Government, Finland	Finland Government Open Data	No	Yes	Unknown
Open Government, France	French Government Open Data	apiKey	Yes	Unknown
Open Government, Germany	Germany Government Open Data	No	Yes	Unknown
Open Government, Greece	Greece Government Open Data	OAuth	Yes	Unknown
Open Government, India	Indian Government Open Data	apiKey	Yes	Unknown
Open Government, Ireland	Ireland Government Open Data	No	Yes	Unknown
Open Government, Italy	Italy Government Open Data	No	Yes	Unknown
Open Government, Korea	Korea Government Open Data	apiKey	Yes	Unknown
Open Government, Lithuania	Lithuania Government Open Data	No	Yes	Unknown
Open Government, Luxembourg	Luxembourgish Government Open Data	apiKey	Yes	Unknown
Open Government, Mexico	Mexican Statistical Government Open Data	No	Yes	Unknown
Open Government, Mexico	Mexico Government Open Data	No	Yes	Unknown
Open Government, Netherlands	Netherlands Government Open Data	No	Yes	Unknown
Open Government, New South Wales	New South Wales Government Open Data	apiKey	Yes	Unknown
Open Government, New Zealand	New Zealand Government Open Data	No	Yes	Unknown
Open Government, Norway	Norwegian Government Open Data	No	Yes	Yes
Open Government, Peru	Peru Government Open Data	No	Yes	Unknown
Open Government, Poland	Poland Government Open Data	No	Yes	Yes
Open Government, Portugal	Portugal Government Open Data	No	Yes	Yes
Open Government, Queensland Government	Queensland Government Open Data	No	Yes	Unknown
Open Government, Romania	Romania Government Open Data	No	No	Unknown
Open Government, Saudi Arabia	Saudi Arabia Government Open Data	No	Yes	Unknown
Open Government, Singapore	Singapore Government Open Data	No	Yes	Unknown
Open Government, Slovakia	Slovakia Government Open Data	No	Yes	Unknown
Open Government, Slovenia	Slovenia Government Open Data	No	Yes	No
Open Government, South Australian Government	South Australian Government Open Data	No	Yes	Unknown
Open Government, Spain	Spain Government Open Data	No	Yes	Unknown
Open Government, Sweden	Sweden Government Open Data	No	Yes	Unknown
Open Government, Switzerland	Switzerland Government Open Data	No	Yes	Unknown
Open Government, Taiwan	Taiwan Government Open Data	No	Yes	Unknown
Open Government, Thailand	Thailand Government Open Data	apiKey	Yes	Unknown
Open Government, UK	UK Government Open Data	No	Yes	Unknown
Open Government, USA	United States Government Open Data	No	Yes	Unknown
Open Government, Victoria State Government	Victoria State Government Open Data	No	Yes	Unknown
Open Government, West Australia	West Australia Open Data	No	Yes	Unknown
PRC Exam Schedule	Unofficial Philippine Professional Regulation Commission's examination schedule	No	Yes	Yes
Represent by Open North	Find Canadian Government Representatives	No	Yes	Unknown
UK Companies House	UK Companies House Data from the UK government	OAuth	Yes	Unknown
US Presidential Election Data by TogaTech	Basic candidate data and live electoral vote counts for top two parties in US presidential election	No	Yes	No
USA.gov	Authoritative information on U.S. programs, events, services and more	apiKey	Yes	Unknown
USAspending.gov	US federal spending data	No	Yes	Unknown

⬆ Back to Index

Health

API Description Auth HTTPS CORS

CMS.gov	Access to the data from the CMS - medicare.gov	apiKey	Yes	Unknown
Coronavirus	HTTP API for Latest Covid-19 Data	No	Yes	Unknown
Coronavirus in the UK	UK Government coronavirus data, including deaths and cases by region	No	Yes	Unknown
Covid Tracking Project	Covid-19 data for the US	No	Yes	No
Covid-19	Covid 19 spread, infection and recovery	No	Yes	Yes
Covid-19	Covid 19 cases, deaths and recovery per country	No	Yes	Yes
Covid-19 Datenhub	Maps, datasets, applications and more in the context of COVID-19	No	Yes	Unknown
Covid-19 Government Response	Government measures tracker to fight against the Covid-19 pandemic	No	Yes	Yes
Covid-19 India	Covid 19 statistics state and district wise about cases, vaccinations, recovery within India	No	Yes	Unknown
Covid-19 JHU CSSE	Open-source API for exploring Covid19 cases based on JHU CSSE	No	Yes	Yes
Covid-19 Live Data	Global and countrywise data of Covid 19 daily Summary, confirmed cases, recovered and deaths	No	Yes	Yes
Covid-19 Philippines	Unofficial Covid-19 Web API for Philippines from data collected by DOH	No	Yes	Yes
COVID-19 Tracker Canada	Details on Covid-19 cases across Canada	No	Yes	Unknown
COVID-19 Tracker Sri Lanka	Provides situation of the COVID-19 patients reported in Sri Lanka	No	Yes	Unknown
COVID-ID	Indonesian government Covid data per province	No	Yes	Yes
Dataflow Kit COVID-19	COVID-19 live statistics into sites per hour	No	Yes	Unknown
FoodData Central	National Nutrient Database for Standard Reference	apiKey	Yes	Unknown
Healthcare.gov	Educational content about the US Health Insurance Marketplace	No	Yes	Unknown
Humanitarian Data Exchange	Humanitarian Data Exchange (HDX) is open platform for sharing data across crises and organisations	No	Yes	Unknown
Infermedica	NLP based symptom checker and patient triage API for health diagnosis from text	apiKey	Yes	Yes
LAPIS	SARS-CoV-2 genomic sequences from public sources	No	Yes	Yes
Lexigram	NLP that extracts mentions of clinical concepts from text, gives access to clinical ontology	apiKey	Yes	Unknown
Makeup	Makeup Information	No	No	Unknown
MyVaccination	Vaccination data for Malaysia	No	Yes	Unknown
NPPES	National Plan & Provider Enumeration System, info on healthcare providers registered in US	No	Yes	Unknown
Nutritionix	Worlds largest verified nutrition database	apiKey	Yes	Unknown
Open Data NHS Scotland	Medical reference data and statistics by Public Health Scotland	No	Yes	Unknown
Open Disease	API for Current cases and more stuff about COVID-19 and Influenza	No	Yes	Yes
openFDA	Public FDA data about drugs, devices and foods	apiKey	Yes	Unknown
Orion Health	Medical platform which allows the development of applications for different healthcare scenarios	OAuth	Yes	Unknown
Quarantine	Coronavirus API with free COVID-19 live updates	No	Yes	Yes

⬆ Back to Index

Jobs

API Description Auth HTTPS CORS

Adzuna	Job board aggregator	apiKey	Yes	Unknown
Arbeitnow	API for Job board aggregator in Europe / Remote	No	Yes	Yes
Arbeitsamt	API for the "Arbeitsamt", which is a german Job board aggregator	OAuth	Yes	Unknown
Careerjet	Job search engine	apiKey	No	Unknown
DevITjobs UK	Jobs with GraphQL	No	Yes	Yes
Findwork	Job board	apiKey	Yes	Unknown
GraphQL Jobs	Jobs with GraphQL	No	Yes	Yes
Jobs2Careers	Job aggregator	apiKey	Yes	Unknown
Jooble	Job search engine	apiKey	Yes	Unknown
Juju	Job search engine	apiKey	No	Unknown
Open Skills	Job titles, skills and related jobs data	No	No	Unknown
Reed	Job board aggregator	apiKey	Yes	Unknown
The Muse	Job board and company profiles	apiKey	Yes	Unknown
Upwork	Freelance job board and management system	OAuth	Yes	Unknown
USAJOBS	US government job board	apiKey	Yes	Unknown
WhatJobs	Job search engine	apiKey	Yes	Unknown
ZipRecruiter	Job search app and website	apiKey	Yes	Unknown

⬆ Back to Index

Machine Learning

API Description Auth HTTPS CORS

AI For Thai	Free Various Thai AI API	apiKey	Yes	Yes
Clarifai	Computer Vision	OAuth	Yes	Unknown
Cloudmersive	Image captioning, face recognition, NSFW classification	apiKey	Yes	Yes
Deepcode	AI for code review	No	Yes	Unknown
Dialogflow	Natural Language Processing	apiKey	Yes	Unknown
EXUDE-API	Used for the primary ways for filtering the stopping, stemming words from the text data	No	Yes	Yes
Hirak FaceAPI	Face detection, face recognition with age estimation/gender estimation, accurate, no quota limits	apiKey	Yes	Unknown
Imagga	Image Recognition Solutions like Tagging, Visual Search, NSFW moderation	apiKey	Yes	Unknown
Inferdo	Computer Vision services like Facial detection, Image labeling, NSFW classification	apiKey	Yes	Unknown
IPS Online	Face and License Plate Anonymization	apiKey	Yes	Unknown
Irisnet	Realtime content moderation API that blocks or blurs unwanted images in real-time	apiKey	Yes	Yes
Keen IO	Data Analytics	apiKey	Yes	Unknown
Machinetutors	AI Solutions: Video/Image Classification & Tagging, NSFW, Icon/Image/Audio Search, NLP	apiKey	Yes	Yes
MessengerX.io	A FREE API for developers to build and monetize personalized ML based chat apps	apiKey	Yes	Yes
NLP Cloud	NLP API using spaCy and transformers for NER, sentiments, classification, summarization, and more	apiKey	Yes	Unknown
OpenVisionAPI	Open source computer vision API based on open source models	No	Yes	Yes
Perspective	NLP API to return probability that if text is toxic, obscene, insulting or threatening	apiKey	Yes	Unknown
Roboflow Universe	Pre-trained computer vision models	apiKey	Yes	Yes
SkyBiometry	Face Detection, Face Recognition and Face Grouping	apiKey	Yes	Unknown
Time Door	A time series analysis API	apiKey	Yes	Yes
Unplugg	Forecasting API for timeseries data	apiKey	Yes	Unknown
WolframAlpha	Provides specific answers to questions using data and algorithms	apiKey	Yes	Unknown

⬆ Back to Index

Music

API Description Auth HTTPS CORS

7digital	Api of Music store 7digital	OAuth	Yes	Unknown
AI Mastering	Automated Music Mastering	apiKey	Yes	Yes
Audiomack	Api of the streaming music hub Audiomack	OAuth	Yes	Unknown
Bandcamp	API of Music store Bandcamp	OAuth	Yes	Unknown
Bandsintown	Music Events	No	Yes	Unknown
Deezer	Music	OAuth	Yes	Unknown
Discogs	Music	OAuth	Yes	Unknown
Freesound	Music Samples	apiKey	Yes	Unknown
Gaana	API to retrieve song information from Gaana	No	Yes	Unknown
Genius	Crowdsourced lyrics and music knowledge	OAuth	Yes	Unknown
Genrenator	Music genre generator	No	Yes	Unknown
iTunes Search	Software products	No	Yes	Unknown
Jamendo	Music	OAuth	Yes	Unknown
JioSaavn	API to retrieve song information, album meta data and many more from JioSaavn	No	Yes	Unknown
KKBOX	Get music libraries, playlists, charts, and perform out of KKBOX's platform	OAuth	Yes	Unknown
KSoft.Si Lyrics	API to get lyrics for songs	apiKey	Yes	Unknown
LastFm	Music	apiKey	Yes	Unknown
Lyrics.ovh	Simple API to retrieve the lyrics of a song	No	Yes	Unknown
Mixcloud	Music	OAuth	Yes	Yes
MusicBrainz	Music	No	Yes	Unknown
Musixmatch	Music	apiKey	Yes	Unknown
Napster	Music	apiKey	Yes	Yes
Openwhyd	Download curated playlists of streaming tracks (YouTube, SoundCloud, etc...)	No	Yes	No
Phishin	A web-based archive of legal live audio recordings of the improvisational rock band Phish	apiKey	Yes	No
Radio Browser	List of internet radio stations	No	Yes	Yes
Songkick	Music Events	apiKey	Yes	Unknown
Songlink / Odesli	Get all the services on which a song is available	apiKey	Yes	Yes
Songsterr	Provides guitar, bass and drums tabs and chords	No	Yes	Unknown
SoundCloud	With SoundCloud API you can build applications that will give more power to control your content	OAuth	Yes	Unknown
Spotify	View Spotify music catalog, manage users' libraries, get recommendations and more	OAuth	Yes	Unknown
TasteDive	Similar artist API (also works for movies and TV shows)	apiKey	Yes	Unknown
TheAudioDB	Music	apiKey	Yes	Unknown
Vagalume	Crowdsourced lyrics and music knowledge	apiKey	Yes	Unknown

⬆ Back to Index

News

API Description Auth HTTPS CORS

apilayer mediastack	Free, Simple REST API for Live News & Blog Articles	apiKey	Yes	Unknown
Associated Press	Search for news and metadata from Associated Press	apiKey	Yes	Unknown
Chronicling America	Provides access to millions of pages of historic US newspapers from the Library of Congress	No	No	Unknown
Currents	Latest news published in various news sources, blogs and forums	apiKey	Yes	Yes
Feedbin	RSS reader	OAuth	Yes	Unknown
GNews	Search for news from various sources	apiKey	Yes	Yes
Graphs for Coronavirus	Each Country separately and Worldwide Graphs for Coronavirus. Daily updates	No	Yes	Yes
Inshorts News	Provides news from inshorts	No	Yes	Unknown
MarketAux	Live stock market news with tagged tickers + sentiment and stats JSON API	apiKey	Yes	Yes
New York Times	The New York Times Developer Network	apiKey	Yes	Unknown
News	Headlines currently published on a range of news sources and blogs	apiKey	Yes	Unknown
NewsData	News data API for live-breaking news and headlines from reputed news sources	apiKey	Yes	Unknown
NewsX	Get or Search Latest Breaking News with ML Powered Summaries 🤖	apiKey	Yes	Unknown
NPR One	Personalized news listening experience from NPR	OAuth	Yes	Unknown
Spaceflight News	Spaceflight related news 🚀	No	Yes	Yes
The Guardian	Access all the content the Guardian creates, categorised by tags and section	apiKey	Yes	Unknown
The Old Reader	RSS reader	apiKey	Yes	Unknown
TheNews	Aggregated headlines, top story and live news JSON API	apiKey	Yes	Yes
Trove	Search through the National Library of Australia collection of 1000s of digitised newspapers	apiKey	Yes	Unknown

⬆ Back to Index

Open Data

API Description Auth HTTPS CORS

18F	Unofficial US Federal Government API Development	No	No	Unknown
API Setu	An Indian Government platform that provides a lot of APIS for KYC, business, education & employment	No	Yes	Yes
Archive.org	The Internet Archive	No	Yes	No
Black History Facts	Contribute or search one of the largest black history fact databases on the web	apiKey	Yes	Yes
BotsArchive	JSON formatted details about Telegram Bots available in database	No	Yes	Unknown
Callook.info	United States ham radio callsigns	No	Yes	Unknown
CARTO	Location Information Prediction	apiKey	Yes	Unknown
CollegeScoreCard.ed.gov	Data on higher education institutions in the United States	No	Yes	Unknown
Enigma Public	Broadest collection of public data	apiKey	Yes	Yes
French Address Search	Address search via the French Government	No	Yes	Unknown
GENESIS	Federal Statistical Office Germany	OAuth	Yes	Unknown
Joshua Project	People groups of the world with the fewest followers of Christ	apiKey	Yes	Unknown
Kaggle	Create and interact with Datasets, Notebooks, and connect with Kaggle	apiKey	Yes	Unknown
LinkPreview	Get JSON formatted summary with title, description and preview image for any requested URL	apiKey	Yes	Yes
Lowy Asia Power Index	Get measure resources and influence to rank the relative power of states in Asia	No	Yes	Unknown
Microlink.io	Extract structured data from any website	No	Yes	Yes
Nasdaq Data Link	Stock market data	apiKey	Yes	Unknown
Nobel Prize	Open data about nobel prizes and events	No	Yes	Yes
Open Data Minneapolis	Spatial (GIS) and non-spatial city data for Minneapolis	No	Yes	No
openAFRICA	Large datasets repository of African open data	No	Yes	Unknown
OpenCorporates	Data on corporate entities and directors in many countries	apiKey	Yes	Unknown
OpenSanctions	Data on international sanctions, crime and politically exposed persons	No	Yes	Yes
PeakMetrics	News articles and public datasets	apiKey	Yes	Unknown
Recreation Information Database	Recreational areas, federal lands, historic sites, museums, and other attractions/resources(US)	apiKey	Yes	Unknown
Scoop.it	Content Curation Service	apiKey	No	Unknown
Socrata	Access to Open Data from Governments, Non-profits and NGOs around the world	OAuth	Yes	Yes
Teleport	Quality of Life Data	No	Yes	Unknown
Umeå Open Data	Open data of the city Umeå in northen Sweden	No	Yes	Yes
Universities List	University names, countries and domains	No	Yes	Unknown
University of Oslo	Courses, lecture videos, detailed information for courses etc. for the University of Oslo (Norway)	No	Yes	Unknown
UPC database	More than 1.5 million barcode numbers from all around the world	apiKey	Yes	Unknown
Urban Observatory	The largest set of publicly available real time urban data in the UK	No	No	No
Wikidata	Collaboratively edited knowledge base operated by the Wikimedia Foundation	OAuth	Yes	Unknown
Wikipedia	Mediawiki Encyclopedia	No	Yes	Unknown
Yelp	Find Local Business	OAuth	Yes	Unknown

⬆ Back to Index

Open Source Projects

API Description Auth HTTPS CORS

Countly	Countly web analytics	No	No	Unknown
Creative Commons Catalog	Search among openly licensed and public domain works	OAuth	Yes	Yes
Datamuse	Word-finding query engine	No	Yes	Unknown
Drupal.org	Drupal.org	No	Yes	Unknown
Evil Insult Generator	Evil Insults	No	Yes	Yes
GitHub Contribution Chart Generator	Create an image of your GitHub contributions	No	Yes	Yes
GitHub ReadMe Stats	Add dynamically generated statistics to your GitHub profile ReadMe	No	Yes	Yes
Metabase	An open source Business Intelligence server to share data and analytics inside your company	No	Yes	Yes
Shields	Concise, consistent, and legible badges in SVG and raster format	No	Yes	Unknown

⬆ Back to Index

Patent

API Description Auth HTTPS CORS

EPO	European patent search system api	OAuth	Yes	Unknown
PatentsView	API is intended to explore and visualize trends/patterns across the US innovation landscape	No	Yes	Unknown
TIPO	Taiwan patent search system api	apiKey	Yes	Unknown
USPTO	USA patent api services	No	Yes	Unknown

⬆ Back to Index

Personality

API Description Auth HTTPS CORS

Advice Slip	Generate random advice slips	No	Yes	Unknown
Biriyani As A Service	Biriyani images placeholder	No	Yes	No
Dev.to	Access Forem articles, users and other resources via API	apiKey	Yes	Unknown
Dictum	API to get access to the collection of the most inspiring expressions of mankind	No	Yes	Unknown
FavQs.com	FavQs allows you to collect, discover and share your favorite quotes	apiKey	Yes	Unknown
FOAAS	Fuck Off As A Service	No	No	Unknown
Forismatic	Inspirational Quotes	No	No	Unknown
icanhazdadjoke	The largest selection of dad jokes on the internet	No	Yes	Unknown
Inspiration	Motivational and Inspirational quotes	No	Yes	Yes
kanye.rest	REST API for random Kanye West quotes	No	Yes	Yes
kimiquotes	Team radio and interview quotes by Finnish F1 legend Kimi Räikkönen	No	Yes	Yes
Medium	Community of readers and writers offering unique perspectives on ideas	OAuth	Yes	Unknown
Programming Quotes	Programming Quotes API for open source projects	No	Yes	Unknown
Quotable Quotes	Quotable is a free, open source quotations API	No	Yes	Unknown
Quote Garden	REST API for more than 5000 famous quotes	No	Yes	Unknown
quoteclear	Ever-growing list of James Clear quotes from the 3-2-1 Newsletter	No	Yes	Yes
Quotes on Design	Inspirational Quotes	No	Yes	Unknown
Stoicism Quote	Quotes about Stoicism	No	Yes	Unknown
They Said So Quotes	Quotes Trusted by many fortune brands around the world	No	Yes	Unknown
Traitify	Assess, collect and analyze Personality	No	Yes	Unknown
Udemy(instructor)	API for instructors on Udemy	apiKey	Yes	Unknown
Vadivelu HTTP Codes	On demand HTTP Codes with images	No	Yes	No
Zen Quotes	Large collection of Zen quotes for inspiration	No	Yes	Yes

⬆ Back to Index

Phone

API Description Auth HTTPS CORS

Abstract Phone Validation	Validate phone numbers globally	apiKey	Yes	Yes
apilayer numverify	Phone number validation	apiKey	Yes	Unknown
Cloudmersive Validate	Validate international phone numbers	apiKey	Yes	Yes
Phone Specification	Rest Api for Phone specifications	No	Yes	Yes
Veriphone	Phone number validation & carrier lookup	apiKey	Yes	Yes

⬆ Back to Index

Photography

API Description Auth HTTPS CORS

apilayer screenshotlayer	URL 2 Image	No	Yes	Unknown
APITemplate.io	Dynamically generate images and PDFs from templates with a simple API	apiKey	Yes	Yes
Bruzu	Image generation with query string	apiKey	Yes	Yes
CheetahO	Photo optimization and resize	apiKey	Yes	Unknown
Dagpi	Image manipulation and processing	apiKey	Yes	Unknown
Duply	Generate, Edit, Scale and Manage Images and Videos Smarter & Faster	apiKey	Yes	Yes
DynaPictures	Generate Hundreds of Personalized Images in Minutes	apiKey	Yes	Yes
Flickr	Flickr Services	OAuth	Yes	Unknown
Getty Images	Build applications using the world's most powerful imagery	OAuth	Yes	Unknown
Gfycat	Jiffier GIFs	OAuth	Yes	Unknown
Giphy	Get all your gifs	apiKey	Yes	Unknown
Google Photos	Integrate Google Photos with your apps or devices	OAuth	Yes	Unknown
Image Upload	Image Optimization	apiKey	Yes	Unknown
Imgur	Images	OAuth	Yes	Unknown
Imsea	Free image search	No	Yes	Unknown
Lorem Picsum	Images from Unsplash	No	Yes	Unknown
ObjectCut	Image Background removal	apiKey	Yes	Yes
Pexels	Free Stock Photos and Videos	apiKey	Yes	Yes
PhotoRoom	Remove background from images	apiKey	Yes	Unknown
Pixabay	Photography	apiKey	Yes	Unknown
PlaceKeanu	Resizable Keanu Reeves placeholder images with grayscale and young Keanu options	No	Yes	Unknown
Readme typing SVG	Customizable typing and deleting text SVG	No	Yes	Unknown
Remove.bg	Image Background removal	apiKey	Yes	Unknown
ReSmush.it	Photo optimization	No	No	Unknown
shutterstock	Stock Photos and Videos	OAuth	Yes	Unknown
Sirv	Image management solutions like optimization, manipulation, hosting	apiKey	Yes	Unknown
Unsplash	Photography	OAuth	Yes	Unknown
Wallhaven	Wallpapers	apiKey	Yes	Unknown
Webdam	Images	OAuth	Yes	Unknown

⬆ Back to Index

Programming

API Description Auth HTTPS CORS

Codeforces	Get access to Codeforces data	apiKey	Yes	Unknown
Hackerearth	For compiling and running code in several languages	apiKey	Yes	Unknown
Judge0 CE	Online code execution system	apiKey	Yes	Unknown
KONTESTS	For upcoming and ongoing competitive coding contests	No	Yes	Unknown
Mintlify	For programmatically generating documentation for code	apiKey	Yes	Yes

⬆ Back to Index

Science & Math

API Description Auth HTTPS CORS

arcsecond.io	Multiple astronomy data sources	No	Yes	Unknown
arXiv	Curated research-sharing platform: physics, mathematics, quantitative finance, and economics	No	Yes	Unknown
CORE	Access the world's Open Access research papers	apiKey	Yes	Unknown
GBIF	Global Biodiversity Information Facility	No	Yes	Yes
iDigBio	Access millions of museum specimens from organizations around the world	No	Yes	Unknown
inspirehep.net	High Energy Physics info. system	No	Yes	Unknown
isEven (humor)	Check if a number is even	No	Yes	Unknown
ISRO	ISRO Space Crafts Information	No	Yes	No
ITIS	Integrated Taxonomic Information System	No	Yes	Unknown
Launch Library 2	Spaceflight launches and events database	No	Yes	Yes
Materials Platform for Data Science	Curated experimental data for materials science	apiKey	Yes	No
Minor Planet Center	Asterank.com Information	No	No	Unknown
NASA	NASA data, including imagery	No	Yes	No
NASA ADS	NASA Astrophysics Data System	OAuth	Yes	Yes
Newton	Symbolic and Arithmetic Math Calculator	No	Yes	No
Noctua	REST API used to access NoctuaSky features	No	Yes	Unknown
Numbers	Number of the day, random number, number facts and anything else you want to do with numbers	apiKey	Yes	No
Numbers	Facts about numbers	No	No	No
Ocean Facts	Facts pertaining to the physical science of Oceanography	No	Yes	Unknown
Open Notify	ISS astronauts, current location, etc	No	No	No
Open Science Framework	Repository and archive for study designs, research materials, data, manuscripts, etc	No	Yes	Unknown
Purple Air	Real Time Air Quality Monitoring	No	Yes	Unknown
Remote Calc	Decodes base64 encoding and parses it to return a solution to the calculation in JSON	No	Yes	Yes
SHARE	A free, open, dataset about research and scholarly activities	No	Yes	No
SpaceX	Company, vehicle, launchpad and launch data	No	Yes	No
SpaceX	GraphQL, Company, Ships, launchpad and launch data	No	Yes	Unknown
Sunrise and Sunset	Sunset and sunrise times for a given latitude and longitude	No	Yes	No
Times Adder	With this API you can add each of the times introduced in the array sended	No	Yes	No
TLE	Satellite information	No	Yes	No
USGS Earthquake Hazards Program	Earthquakes data real-time	No	Yes	No
USGS Water Services	Water quality and level info for rivers and lakes	No	Yes	No
World Bank	World Data	No	Yes	No
xMath	Random mathematical expressions	No	Yes	Yes

⬆ Back to Index

Security

API Description Auth HTTPS CORS

Application Environment Verification	Android library and API to verify the safety of user devices, detect rooted devices and other risks	apiKey	Yes	Yes
BinaryEdge	Provide access to BinaryEdge 40fy scanning platform	apiKey	Yes	Yes
BitWarden	Best open-source password manager	OAuth	Yes	Unknown
Botd	Botd is a browser library for JavaScript bot detection	apiKey	Yes	Yes
Bugcrowd	Bugcrowd API for interacting and tracking the reported issues programmatically	apiKey	Yes	Unknown
Censys	Search engine for Internet connected host and devices	apiKey	Yes	No
Classify	Encrypting & decrypting text messages	No	Yes	Yes
Complete Criminal Checks	Provides data of offenders from all U.S. States and Pureto Rico	apiKey	Yes	Yes
CRXcavator	Chrome extension risk scoring	apiKey	Yes	Unknown
Dehash.lt	Hash decryption MD5, SHA1, SHA3, SHA256, SHA384, SHA512	No	Yes	Unknown
EmailRep	Email address threat and risk prediction	No	Yes	Unknown
Escape	An API for escaping different kind of queries	No	Yes	No
FilterLists	Lists of filters for adblockers and firewalls	No	Yes	Unknown
FingerprintJS Pro	Fraud detection API offering highly accurate browser fingerprinting	apiKey	Yes	Yes
FraudLabs Pro	Screen order information using AI to detect frauds	apiKey	Yes	Unknown
FullHunt	Searchable attack surface database of the entire internet	apiKey	Yes	Unknown
GitGuardian	Scan files for secrets (API Keys, database credentials)	apiKey	Yes	No
GreyNoise	Query IPs in the GreyNoise dataset and retrieve a subset of the full IP context data	apiKey	Yes	Unknown
HackerOne	The industry’s first hacker API that helps increase productivity towards creative bug bounty hunting	apiKey	Yes	Unknown
Hashable	A REST API to access high level cryptographic functions and methods	No	Yes	Yes
HaveIBeenPwned	Passwords which have previously been exposed in data breaches	apiKey	Yes	Unknown
Intelligence X	Perform OSINT via Intelligence X	apiKey	Yes	Unknown
LoginRadius	Managed User Authentication Service	apiKey	Yes	Yes
Microsoft Security Response Center (MSRC)	Programmatic interfaces to engage with the Microsoft Security Response Center (MSRC)	No	Yes	Unknown
Mozilla http scanner	Mozilla observatory http scanner	No	Yes	Unknown
Mozilla tls scanner	Mozilla observatory tls scanner	No	Yes	Unknown
National Vulnerability Database	U.S. National Vulnerability Database	No	Yes	Unknown
Passwordinator	Generate random passwords of varying complexities	No	Yes	Yes
PhishStats	Phishing database	No	Yes	Unknown
Privacy.com	Generate merchant-specific and one-time use credit card numbers that link back to your bank	apiKey	Yes	Unknown
Pulsedive	Scan, search and collect threat intelligence data in real-time	apiKey	Yes	Unknown
SecurityTrails	Domain and IP related information such as current and historical WHOIS and DNS records	apiKey	Yes	Unknown
Shodan	Search engine for Internet connected devices	apiKey	Yes	Unknown
Spyse	Access data on all Internet assets and build powerful attack surface management applications	apiKey	Yes	Unknown
Threat Jammer	Risk scoring service from curated threat intelligence data	apiKey	Yes	Unknown
UK Police	UK Police data	No	Yes	Unknown
Virushee	Virushee file/data scanning	No	Yes	Yes
VulDB	VulDB API allows to initiate queries for one or more items along with transactional bots	apiKey	Yes	Unknown

⬆ Back to Index

Shopping

API Description Auth HTTPS CORS

Best Buy	Products, Buying Options, Categories, Recommendations, Stores and Commerce	apiKey	Yes	Unknown
Digi-Key	Retrieve price and inventory of electronic components as well as place orders	OAuth	Yes	Unknown
Dummy Products	An api to fetch dummy e-commerce products JSON data with placeholder images	apiKey	Yes	Yes
eBay	Sell and Buy on eBay	OAuth	Yes	Unknown
Etsy	Manage shop and interact with listings	OAuth	Yes	Unknown
Flipkart Marketplace	Product listing management, Order Fulfilment in the Flipkart Marketplace	OAuth	Yes	Yes
Lazada	Retrieve product ratings and seller performance metrics	apiKey	Yes	Unknown
Mercadolibre	Manage sales, ads, products, services and Shops	apiKey	Yes	Unknown
Octopart	Electronic part data for manufacturing, design, and sourcing	apiKey	Yes	Unknown
OLX Poland	Integrate with local sites by posting, managing adverts and communicating with OLX users	apiKey	Yes	Unknown
Rappi	Manage orders from Rappi's app	OAuth	Yes	Unknown
Shopee	Shopee's official API for integration of various services from Shopee	apiKey	Yes	Unknown
Tokopedia	Tokopedia's Official API for integration of various services from Tokopedia	OAuth	Yes	Unknown
WooCommerce	WooCommerce REST APIS to create, read, update, and delete data on wordpress website in JSON format	apiKey	Yes	Yes

⬆ Back to Index

Social

API Description Auth HTTPS CORS

4chan	Simple image-based bulletin board dedicated to a variety of topics	No	Yes	Yes
Ayrshare	Social media APIs to post, get analytics, and manage multiple users social media accounts	apiKey	Yes	Yes
aztro	Daily horoscope info for yesterday, today, and tomorrow	No	Yes	Unknown
Blogger	The Blogger APIs allows client applications to view and update Blogger content	OAuth	Yes	Unknown
Cisco Spark	Team Collaboration Software	OAuth	Yes	Unknown
Dangerous Discord Database	Database of malicious Discord accounts	apiKey	Yes	Unknown
Discord	Make bots for Discord, integrate Discord onto an external platform	OAuth	Yes	Unknown
Disqus	Communicate with Disqus data	OAuth	Yes	Unknown
Doge-Meme	Top meme posts from r/dogecoin which include 'Meme' flair	No	Yes	Yes
Facebook	Facebook Login, Share on FB, Social Plugins, Analytics and more	OAuth	Yes	Unknown
Foursquare	Interact with Foursquare users and places (geolocation-based checkins, photos, tips, events, etc)	OAuth	Yes	Unknown
Fuck Off as a Service	Asks someone to fuck off	No	Yes	Unknown
Full Contact	Get Social Media profiles and contact Information	OAuth	Yes	Unknown
HackerNews	Social news for CS and entrepreneurship	No	Yes	Unknown
Hashnode	A blogging platform built for developers	No	Yes	Unknown
Instagram	Instagram Login, Share on Instagram, Social Plugins and more	OAuth	Yes	Unknown
Kakao	Kakao Login, Share on KakaoTalk, Social Plugins and more	OAuth	Yes	Unknown
Lanyard	Retrieve your presence on Discord through an HTTP REST API or WebSocket	No	Yes	Yes
Line	Line Login, Share on Line, Social Plugins and more	OAuth	Yes	Unknown
LinkedIn	The foundation of all digital integrations with LinkedIn	OAuth	Yes	Unknown
Meetup.com	Data about Meetups from Meetup.com	apiKey	Yes	Unknown
Microsoft Graph	Access the data and intelligence in Microsoft 365, Windows 10, and Enterprise Mobility	OAuth	Yes	Unknown
NAVER	NAVER Login, Share on NAVER, Social Plugins and more	OAuth	Yes	Unknown
Open Collective	Get Open Collective data	No	Yes	Unknown
Pinterest	The world's catalog of ideas	OAuth	Yes	Unknown
Product Hunt	The best new products in tech	OAuth	Yes	Unknown
Reddit	Homepage of the internet	OAuth	Yes	Unknown
Revolt	Revolt open source Discord alternative	apiKey	Yes	Unknown
Saidit	Open Source Reddit Clone	OAuth	Yes	Unknown
Slack	Team Instant Messaging	OAuth	Yes	Unknown
TamTam	Bot API to interact with TamTam	apiKey	Yes	Unknown
Telegram Bot	Simplified HTTP version of the MTProto API for bots	apiKey	Yes	Unknown
Telegram MTProto	Read and write Telegram data	OAuth	Yes	Unknown
Telegraph	Create attractive blogs easily, to share	apiKey	Yes	Unknown
TikTok	Fetches user info and user's video posts on TikTok platform	OAuth	Yes	Unknown
Trash Nothing	A freecycling community with thousands of free items posted every day	OAuth	Yes	Yes
Tumblr	Read and write Tumblr Data	OAuth	Yes	Unknown
Twitch	Game Streaming API	OAuth	Yes	Unknown
Twitter	Read and write Twitter data	OAuth	Yes	No
vk	Read and write vk data	OAuth	Yes	Unknown

⬆ Back to Index

Sports & Fitness

API Description Auth HTTPS CORS

API-FOOTBALL	Get information about Football Leagues & Cups	apiKey	Yes	Yes
ApiMedic	ApiMedic offers a medical symptom checker API primarily for patients	apiKey	Yes	Unknown
balldontlie	Balldontlie provides access to stats data from the NBA	No	Yes	Yes
Canadian Football League (CFL)	Official JSON API providing real-time league, team and player statistics about the CFL	apiKey	Yes	No
City Bikes	City Bikes around the world	No	Yes	Unknown
Cloudbet	Official Cloudbet API provides real-time sports odds and betting API to place bets programmatically	apiKey	Yes	Yes
CollegeFootballData.com	Unofficial detailed American college football statistics, records, and results API	apiKey	Yes	Unknown
Ergast F1	F1 data from the beginning of the world championships in 1950	No	Yes	Unknown
Fitbit	Fitbit Information	OAuth	Yes	Unknown
Football	A simple Open Source Football API to get squads’ stats, best scorers and more	X-Mashape-Key	Yes	Unknown
Football (Soccer) Videos	Embed codes for goals and highlights from Premier League, Bundesliga, Serie A and many more	No	Yes	Yes
Football Standings	Display football standings e.g epl, la liga, serie a etc. The data is based on espn site	No	Yes	Yes
Football-Data	Football data with matches info, players, teams, and competitions	X-Mashape-Key	Yes	Unknown
JCDecaux Bike	JCDecaux's self-service bicycles	apiKey	Yes	Unknown
MLB Records and Stats	Current and historical MLB statistics	No	No	Unknown
NBA Data	All NBA Stats DATA, Games, Livescore, Standings, Statistics	apiKey	Yes	Unknown
NBA Stats	Current and historical NBA Statistics	No	Yes	Unknown
NHL Records and Stats	NHL historical data and statistics	No	Yes	Unknown
Oddsmagnet	Odds history from multiple UK bookmakers	No	Yes	Yes
OpenLigaDB	Crowd sourced sports league results	No	Yes	Yes
Premier League Standings	All Current Premier League Standings and Statistics	apiKey	Yes	Unknown
Sport Data	Get sports data from all over the world	apiKey	Yes	Unknown
Sport List & Data	List of and resources related to sports	No	Yes	Yes
Sport Places	Crowd-source sports places around the world	No	Yes	No
Sport Vision	Identify sport, brands and gear in an image. Also does image sports captioning	apiKey	Yes	Yes
Sportmonks Cricket	Live cricket score, player statistics and fantasy API	apiKey	Yes	Unknown
Sportmonks Football	Football score/schedule, news api, tv channels, stats, history, display standing e.g. epl, la liga	apiKey	Yes	Unknown
Squiggle	Fixtures, results and predictions for Australian Football League matches	No	Yes	Yes
Strava	Connect with athletes, activities and more	OAuth	Yes	Unknown
SuredBits	Query sports data, including teams, players, games, scores and statistics	No	No	No
TheSportsDB	Crowd-Sourced Sports Data and Artwork	apiKey	Yes	Yes
Tredict	Get and set activities, health data and more	OAuth	Yes	Unknown
Wger	Workout manager data as exercises, muscles or equipment	apiKey	Yes	Unknown

⬆ Back to Index

Test Data

API Description Auth HTTPS CORS

Bacon Ipsum	A Meatier Lorem Ipsum Generator	No	Yes	Unknown
Dicebear Avatars	Generate random pixel-art avatars	No	Yes	No
English Random Words	Generate English Random Words with Pronunciation	No	Yes	No
FakeJSON	Service to generate test and fake data	apiKey	Yes	Yes
FakerAPI	APIs collection to get fake data	No	Yes	Yes
FakeStoreAPI	Fake store rest API for your e-commerce or shopping website prototype	No	Yes	Unknown
GeneradorDNI	Data generator API. Profiles, vehicles, banks and cards, etc	apiKey	Yes	Unknown
ItsThisForThat	Generate Random startup ideas	No	Yes	No
JSONPlaceholder	Fake data for testing and prototyping	No	No	Unknown
Loripsum	The "lorem ipsum" generator that doesn't suck	No	No	Unknown
Mailsac	Disposable Email	apiKey	Yes	Unknown
Metaphorsum	Generate demo paragraphs giving number of words and sentences	No	No	Unknown
Mockaroo	Generate fake data to JSON, CSV, TXT, SQL and XML	apiKey	Yes	Unknown
QuickMocker	API mocking tool to generate contextual, fake or random data	No	Yes	Yes
Random Data	Random data generator	No	Yes	Unknown
Randommer	Random data generator	apiKey	Yes	Yes
RandomUser	Generates and list user data	No	Yes	Unknown
RoboHash	Generate random robot/alien avatars	No	Yes	Unknown
Spanish random names	Generate spanish names (with gender) randomly	No	Yes	Unknown
Spanish random words	Generate spanish words randomly	No	Yes	Unknown
This Person Does not Exist	Generates real-life faces of people who do not exist	No	Yes	Unknown
Toolcarton	Generate random testimonial data	No	Yes	Unknown
UUID Generator	Generate UUIDs	No	Yes	No
What The Commit	Random commit message generator	No	No	Yes
Yes No	Generate yes or no randomly	No	Yes	Unknown

⬆ Back to Index

Text Analysis

API Description Auth HTTPS CORS

Code Detection API	Detect, label, format and enrich the code in your app or in your data pipeline	OAuth	Yes	Unknown
apilayer languagelayer	Language Detection JSON API supporting 173 languages	OAuth	Yes	Unknown
Aylien Text Analysis	A collection of information retrieval and natural language APIs	apiKey	Yes	Unknown
Cloudmersive Natural Language Processing	Natural language processing and text analysis	apiKey	Yes	Yes
Detect Language	Detects text language	apiKey	Yes	Unknown
ELI	Natural Language Processing Tools for Thai Language	apiKey	Yes	Unknown
Google Cloud Natural	Natural language understanding technology, including sentiment, entity and syntax analysis	apiKey	Yes	Unknown
Hirak OCR	Image to text -text recognition- from image more than 100 language, accurate, unlimited requests	apiKey	Yes	Unknown
Hirak Translation	Translate between 21 of most used languages, accurate, unlimited requests	apiKey	Yes	Unknown
Lecto Translation	Translation API with free tier and reasonable prices	apiKey	Yes	Yes
LibreTranslate	Translation tool with 17 available languages	No	Yes	Unknown
Semantria	Text Analytics with sentiment analysis, categorization & named entity extraction	OAuth	Yes	Unknown
Sentiment Analysis	Multilingual sentiment analysis of texts from different sources	apiKey	Yes	Yes
Tisane	Text Analytics with focus on detection of abusive content and law enforcement applications	OAuth	Yes	Yes
Watson Natural Language Understanding	Natural language processing for advanced text analysis	OAuth	Yes	Unknown

⬆ Back to Index

Tracking

API Description Auth HTTPS CORS

Aftership	API to update, manage and track shipment efficiently	apiKey	Yes	Yes
Correios	Integration to provide information and prepare shipments using Correio's services	apiKey	Yes	Unknown
Pixela	API for recording and tracking habits or effort, routines	X-Mashape-Key	Yes	Yes
PostalPinCode	API for getting Pincode details in India	No	Yes	Unknown
Postmon	An API to query Brazilian ZIP codes and orders easily, quickly and free	No	No	Unknown
PostNord	Provides information about parcels in transport for Sweden and Denmark	apiKey	No	Unknown
UPS	Shipment and Address information	apiKey	Yes	Unknown
WeCanTrack	Automatically place subids in affiliate links to attribute affiliate conversions to click data	apiKey	Yes	Yes
WhatPulse	Small application that measures your keyboard/mouse usage	No	Yes	Unknown

⬆ Back to Index

Transportation

API Description Auth HTTPS CORS

ADS-B Exchange	Access real-time and historical data of any and all airborne aircraft	No	Yes	Unknown
airportsapi	Get name and website-URL for airports by ICAO code	No	Yes	Unknown
AIS Hub	Real-time data of any marine and inland vessel equipped with AIS tracking system	apiKey	No	Unknown
Amadeus for Developers	Travel Search - Limited usage	OAuth	Yes	Unknown
apilayer aviationstack	Real-time Flight Status & Global Aviation Data API	OAuth	Yes	Unknown
AviationAPI	FAA Aeronautical Charts and Publications, Airport Information, and Airport Weather	No	Yes	No
AZ511	Access traffic data from the ADOT API	apiKey	Yes	Unknown
Bay Area Rapid Transit	Stations and predicted arrivals for BART	apiKey	No	Unknown
BC Ferries	Sailing times and capacities for BC Ferries	No	Yes	Yes
BIC-Boxtech	Container technical detail for the global container fleet	OAuth	Yes	Unknown
BlaBlaCar	Search car sharing trips	apiKey	Yes	Unknown
Boston MBTA Transit	Stations and predicted arrivals for MBTA	apiKey	Yes	Unknown
Community Transit	Transitland API	No	Yes	Unknown
Compare Flight Prices	API for comparing flight prices across platforms	apiKey	Yes	Unknown
CTS	CTS Realtime API	apiKey	Yes	Yes
Grab	Track deliveries, ride fares, payments and loyalty points	OAuth	Yes	Unknown
GraphHopper	A-to-B routing with turn-by-turn instructions	apiKey	Yes	Unknown
Icelandic APIs	Open APIs that deliver services in or regarding Iceland	No	Yes	Unknown
Impala Hotel Bookings	Hotel content, rates and room bookings	apiKey	Yes	No
Izi	Audio guide for travellers	apiKey	Yes	Unknown
Land Transport Authority DataMall, Singapore	Singapore transport information	apiKey	No	Unknown
Metro Lisboa	Delays in subway lines	No	No	No
Navitia	The open API for building cool stuff with transport data	apiKey	Yes	Unknown
Open Charge Map	Global public registry of electric vehicle charging locations	apiKey	Yes	Yes
OpenSky Network	Free real-time ADS-B aviation data	No	Yes	Unknown
Railway Transport for France	SNCF public API	apiKey	Yes	Unknown
REFUGE Restrooms	Provides safe restroom access for transgender, intersex and gender nonconforming individuals	No	Yes	Unknown
Sabre for Developers	Travel Search - Limited usage	apiKey	Yes	Unknown
Schiphol Airport	Schiphol	apiKey	Yes	Unknown
Tankerkoenig	German realtime gas/diesel prices	apiKey	Yes	Yes
TransitLand	Transit Aggregation	No	Yes	Unknown
Transport for Atlanta, US	Marta	No	No	Unknown
Transport for Auckland, New Zealand	Auckland Transport	No	Yes	Unknown
Transport for Belgium	The iRail API is a third-party API for Belgian public transport by train	No	Yes	Yes
Transport for Berlin, Germany	Third-party VBB API	No	Yes	Unknown
Transport for Bordeaux, France	Bordeaux Métropole public transport and more (France)	apiKey	Yes	Unknown
Transport for Budapest, Hungary	Budapest public transport API	No	Yes	Unknown
Transport for Chicago, US	Chicago Transit Authority (CTA)	apiKey	No	Unknown
Transport for Czech Republic	Czech transport API	No	Yes	Unknown
Transport for Denver, US	RTD	No	No	Unknown
Transport for Finland	Finnish transport API	No	Yes	Unknown
Transport for Germany	Deutsche Bahn (DB) API	apiKey	No	Unknown
Transport for Grenoble, France	Grenoble public transport	No	No	No
Transport for Hessen, Germany	RMV API (Public Transport in Hessen)	No	Yes	Unknown
Transport for Honolulu, US	Honolulu Transportation Information	apiKey	No	Unknown
Transport for Lisbon, Portugal	Data about buses routes, parking and traffic	apiKey	Yes	Unknown
Transport for London, England	TfL API	apiKey	Yes	Unknown
Transport for Los Angeles, US	Data about positions of Metro vehicles in real time and travel their routes	No	Yes	Unknown
Transport for Manchester, England	TfGM transport network data	apiKey	Yes	No
Transport for Norway	Transport APIs and dataset for Norway	No	Yes	Unknown
Transport for Ottawa, Canada	OC Transpo API	apiKey	Yes	Unknown
Transport for Paris, France	RATP Open Data API	No	No	Unknown
Transport for Philadelphia, US	SEPTA APIs	No	No	Unknown
Transport for Sao Paulo, Brazil	SPTrans	OAuth	No	Unknown
Transport for Spain	Public trains of Spain	No	Yes	Unknown
Transport for Sweden	Public Transport consumer	OAuth	Yes	Unknown
Transport for Switzerland	Official Swiss Public Transport Open Data	apiKey	Yes	Unknown
Transport for Switzerland	Swiss public transport API	No	Yes	Unknown
Transport for The Netherlands	NS, only trains	apiKey	No	Unknown
Transport for The Netherlands	OVAPI, country-wide public transport	No	Yes	Unknown
Transport for Toronto, Canada	TTC	No	Yes	Unknown
Transport for UK	Transport API and dataset for UK	apiKey	Yes	Unknown
Transport for United States	NextBus API	No	No	Unknown
Transport for Vancouver, Canada	TransLink	OAuth	Yes	Unknown
Transport for Washington, US	Washington Metro transport API	OAuth	Yes	Unknown
transport.rest	Community maintained, developer-friendly public transport API	No	Yes	Yes
Tripadvisor	Rating content for a hotel, restaurant, attraction or destination	apiKey	Yes	Unknown
Uber	Uber ride requests and price estimation	OAuth	Yes	Yes
Velib metropolis, Paris, France	Velib Open Data API	No	Yes	No

⬆ Back to Index

URL Shorteners

API Description Auth HTTPS CORS

1pt	A simple URL shortener	No	Yes	Yes
Bitly	URL shortener and link management	OAuth	Yes	Unknown
CleanURI	URL shortener service	No	Yes	Yes
ClickMeter	Monitor, compare and optimize your marketing links	apiKey	Yes	Unknown
Clico	URL shortener service	apiKey	Yes	Unknown
Cutt.ly	URL shortener service	apiKey	Yes	Unknown
Drivet URL Shortener	Shorten a long URL easily and fast	No	Yes	Unknown
Free Url Shortener	Free URL Shortener offers a powerful API to interact with other sites	No	Yes	Unknown
Git.io	Git.io URL shortener	No	Yes	Unknown
GoTiny	A lightweight URL shortener, focused on ease-of-use for the developer and end-user	No	Yes	Yes
Kutt	Free Modern URL Shortener	apiKey	Yes	Yes
Mgnet.me	Torrent URL shorten API	No	Yes	No
owo	A simple link obfuscator/shortener	No	Yes	Unknown
Rebrandly	Custom URL shortener for sharing branded links	apiKey	Yes	Unknown
Short Link	Short URLs support so many domains	No	Yes	Unknown
Shrtcode	URl Shortener with multiple Domains	No	Yes	Yes
Shrtlnk	Simple and efficient short link creation	apiKey	Yes	Yes
TinyURL	Shorten long URLs	apiKey	Yes	No
UrlBae	Simple and efficient short link creation	apiKey	Yes	Yes

⬆ Back to Index

Vehicle

API Description Auth HTTPS CORS

Brazilian Vehicles and Prices	Vehicles information from Fundação Instituto de Pesquisas Econômicas - Fipe	No	Yes	No
Helipaddy sites	Helicopter and passenger drone landing site directory, Helipaddy data and much more	apiKey	Yes	Unknown
Kelley Blue Book	Vehicle info, pricing, configuration, plus much more	apiKey	Yes	No
Mercedes-Benz	Telematics data, remotely access vehicle functions, car configurator, locate service dealers	apiKey	Yes	No
NHTSA	NHTSA Product Information Catalog and Vehicle Listing	No	Yes	Unknown
Smartcar	Lock and unlock vehicles and get data like odometer reading and location. Works on most new cars	OAuth	Yes	Yes

⬆ Back to Index

Video

API Description Auth HTTPS CORS

An API of Ice And Fire	Game Of Thrones API	No	Yes	Unknown
Bob's Burgers	Bob's Burgers API	No	Yes	Yes
Breaking Bad	Breaking Bad API	No	Yes	Unknown
Breaking Bad Quotes	Some Breaking Bad quotes	No	Yes	Unknown
Catalogopolis	Doctor Who API	No	Yes	Unknown
Catch The Show	REST API for next-episode.net	No	Yes	Unknown
Czech Television	TV programme of Czech TV	No	No	Unknown
Dailymotion	Dailymotion Developer API	OAuth	Yes	Unknown
Dune	A simple API which provides you with book, character, movie and quotes JSON data	No	Yes	Yes
Final Space	Final Space API	No	Yes	Yes
Game of Thrones Quotes	Some Game of Thrones quotes	No	Yes	Unknown
Harry Potter Charactes	Harry Potter Characters Data with with imagery	No	Yes	Unknown
IMDb-API	API for receiving movie, serial and cast information	apiKey	Yes	Unknown
IMDbOT	Unofficial IMDb Movie / Series Information	No	Yes	Yes
JSON2Video	Create and edit videos programmatically: watermarks,resizing,slideshows,voice-over,text animations	apiKey	Yes	No
Lucifer Quotes	Returns Lucifer quotes	No	Yes	Unknown
MCU Countdown	A Countdown to the next MCU Film	No	Yes	Yes
Motivational Quotes	Random Motivational Quotes	No	Yes	Unknown
Movie Quote	Random Movie and Series Quotes	No	Yes	Yes
Open Movie Database	Movie information	apiKey	Yes	Unknown
Owen Wilson Wow	API for actor Owen Wilson's "wow" exclamations in movies	No	Yes	Yes
Ron Swanson Quotes	Television	No	Yes	Unknown
Simkl	Movie, TV and Anime data	apiKey	Yes	Unknown
STAPI	Information on all things Star Trek	No	No	No
Stranger Things Quotes	Returns Stranger Things quotes	No	Yes	Unknown
Stream	Czech internet television, films, series and online videos for free	No	Yes	No
Stromberg Quotes	Returns Stromberg quotes and more	No	Yes	Unknown
SWAPI	All the Star Wars data you've ever wanted	No	Yes	Yes
SWAPI	All things Star Wars	No	Yes	Yes
SWAPI GraphQL	Star Wars GraphQL API	No	Yes	Unknown
The Lord of the Rings	The Lord of the Rings API	apiKey	Yes	Unknown
The Vampire Diaries	TV Show Data	apiKey	Yes	Yes
ThronesApi	Game Of Thrones Characters Data with imagery	No	Yes	Unknown
TMDb	Community-based movie data	apiKey	Yes	Unknown
TrailerAddict	Easily embed trailers from TrailerAddict	apiKey	No	Unknown
Trakt	Movie and TV Data	apiKey	Yes	Yes
TVDB	Television data	apiKey	Yes	Unknown
TVMaze	TV Show Data	No	No	Unknown
uNoGS	Unofficial Netflix Online Global Search, Search all netflix regions in one place	apiKey	Yes	Yes
Vimeo	Vimeo Developer API	OAuth	Yes	Unknown
Watchmode	API for finding out the streaming availability of movies & shows	apiKey	Yes	Unknown
Web Series Quotes Generator	API generates various Web Series Quote Images	No	Yes	Yes
YouTube	Add YouTube functionality to your sites and apps	OAuth	Yes	Unknown

⬆ Back to Index

Weather

API Description Auth HTTPS CORS

7Timer!	Weather, especially for Astroweather	No	No	Unknown
AccuWeather	Weather and forecast data	apiKey	No	Unknown
Aemet	Weather and forecast data from Spain	apiKey	Yes	Unknown
apilayer weatherstack	Real-Time & Historical World Weather Data API	apiKey	Yes	Unknown
APIXU	Weather	apiKey	Yes	Unknown
AQICN	Air Quality Index Data for over 1000 cities	apiKey	Yes	Unknown
AviationWeather	NOAA aviation weather forecasts and observations	No	Yes	Unknown
ColorfulClouds	Weather	apiKey	Yes	Yes
Euskalmet	Meteorological data of the Basque Country	apiKey	Yes	Unknown
Foreca	Weather	OAuth	Yes	Unknown
HG Weather	Provides weather forecast data for cities in Brazil	apiKey	Yes	Yes
Hong Kong Obervatory	Provide weather information, earthquake information, and climate data	No	Yes	Unknown
MetaWeather	Weather	No	Yes	No
Meteorologisk Institutt	Weather and climate data	User-Agent	Yes	Unknown
Micro Weather	Real time weather forecasts and historic data	apiKey	Yes	Unknown
ODWeather	Weather and weather webcams	No	No	Unknown
Oikolab	70+ years of global, hourly historical and forecast weather data from NOAA and ECMWF	apiKey	Yes	Yes
Open-Meteo	Global weather forecast API for non-commercial use	No	Yes	Yes
openSenseMap	Data from Personal Weather Stations called senseBoxes	No	Yes	Yes
OpenUV	Real-time UV Index Forecast	apiKey	Yes	Unknown
OpenWeatherMap	Weather	apiKey	Yes	Unknown
QWeather	Location-based weather data	apiKey	Yes	Yes
RainViewer	Radar data collected from different websites across the Internet	No	Yes	Unknown
Storm Glass	Global marine weather from multiple sources	apiKey	Yes	Yes
Tomorrow	Weather API Powered by Proprietary Technology	apiKey	Yes	Unknown
US Weather	US National Weather Service	No	Yes	Yes
Visual Crossing	Global historical and weather forecast data	apiKey	Yes	Yes
weather-api	A RESTful free API to check the weather	No	Yes	No
WeatherAPI	Weather API with other stuff like Astronomy and Geolocation API	apiKey	Yes	Yes
Weatherbit	Weather	apiKey	Yes	Unknown
Yandex.Weather	Assesses weather condition in specific locations	apiKey	Yes	No

⬆ Back to Index

License

MIT (c) 2022 public-apis