MAGIC Agent Skills is now open source! Star on GitHub
MAGIC Agent SkillsMAGIC Agent Skills
Getting Started

Supported Tools

The Computational Linguistics suite integrates with a wide range of computational linguistics and ML tooling. Each skill references the tools relevant to its domain.

Language Identification

ToolCoverageUsed By
GlotLID (2024)1,600+ languages, paragraph-levelmagic-linguistic-corpus
FastText LID (176 languages)High-resource speedmagic-linguistic-corpus
CLD3Broad coveragemagic-linguistic-corpus (fallback)

Unicode & Script Tools

ToolPurposeUsed By
Python unicodedataNFC/NFKC normalizationmagic-linguistic-scripts
Unicode TR39 confusable dataMixed-script deduplicationmagic-linguistic-scripts
detect_confusables.pyFold/detect confusables + joinersmagic-linguistic-scripts

Tokenization & Vocab Extension

ToolPurposeUsed By
SentencePiece 0.1.96+Unigram/BPE tokenizer trainingmagic-linguistic-tokenize
FOCUSVocab extension (close script pairs)magic-linguistic-tokenize
OFAVocab extension (with parallel data)magic-linguistic-tokenize
HyperOfaVocab extension (minimal data)magic-linguistic-tokenize

Data Sources & Corpora

ResourceLanguagesUsed By
CulturaX167 languagesmagic-linguistic-corpus
MADLAD-400400+ languagesmagic-linguistic-corpus
Glot500500+ languagesmagic-linguistic-corpus
OLDI (Open Language Data Initiative)100+ languagesmagic-linguistic-corpus
Wikipedia dumps300+ languagesmagic-linguistic-corpus
OPUSParallel data, many languagesmagic-linguistic-bitext
FLORES-200200 languages, evalmagic-linguistic-eval
NTREX-128128 languages, evalmagic-linguistic-eval
Belebele122 languages, reading compmagic-linguistic-eval

Bitext Mining & Alignment

ToolPurposeUsed By
LASER3Sentence embeddings for alignmentmagic-linguistic-bitext
SONAR (Meta 2024)Embeddings — stronger for Bantu/Indigenousmagic-linguistic-bitext
VecalignSentence alignment (preferred)magic-linguistic-bitext
hunalignSentence alignment (legacy)magic-linguistic-bitext

Transfer Learning & Fine-Tuning

ToolStrengthsUsed By
Unsloth2× faster QLoRA, single-GPUmagic-linguistic-transfer
LLaMA-FactoryMulti-GPU + complex samplingmagic-linguistic-transfer
AxolotlYAML-config middle groundmagic-linguistic-transfer
HuggingFace PEFTFlexible LoRA/QLoRA/DoRAmagic-linguistic-transfer
MAD-X / BAD-X adaptersLanguage + task adapter stacksmagic-linguistic-transfer

Morphological Analysis

ToolPurposeUsed By
UniMorphGold paradigms, 100+ languagesmagic-linguistic-morph
SIGMORPHON 2022/2023Unsupervised segmentersmagic-linguistic-morph
HFSTRule-based FST analyzermagic-linguistic-morph
fomaAlternative FST toolkitmagic-linguistic-morph
MorfessorStatistical segmentermagic-linguistic-morph

Syntax & Parsing

ToolStrengthsUsed By
Trankit (2021)Best low-resource UD quality (XLM-R)magic-linguistic-syntax
stanza (Stanford 2020)70+ languages, fastmagic-linguistic-syntax
UDify (2019)Single multilingual modelmagic-linguistic-syntax
UD treebank corpus100+ languagesmagic-linguistic-syntax

Semantics & Lexical Resources

ToolPurposeUsed By
Open Multilingual WordNet (OMW)Synsets for 100+ languagesmagic-linguistic-semantics
PARSEMEMWE annotation datasetsmagic-linguistic-semantics
COMET-22Learned MT quality metricmagic-linguistic-semantics, magic-linguistic-eval
LaBSE / SONARCross-lingual embeddingsmagic-linguistic-semantics, magic-linguistic-eval

Speech & Audio

ToolCoverageUsed By
MMS (Meta Massively Multilingual Speech)1,107 languagesmagic-linguistic-speech
Whisper (OpenAI)~99 languagesmagic-linguistic-speech
LhotseAudio pipeline (CutSet standard)magic-linguistic-speech
ELANField annotation formatmagic-linguistic-speech
Praat (TextGrid)Phonetic annotationmagic-linguistic-speech
FLEx FieldWorksLexicographic field datamagic-linguistic-speech
WikiPronG2P / IPA crowd-sourced datamagic-linguistic-speech
VITS / Tacotron2Low-resource TTSmagic-linguistic-speech

Annotation Tools

ToolPurposeUsed By
Label StudioGeneral annotation UImagic-linguistic-annotate
ProdigyActive-learning annotationmagic-linguistic-annotate
INCEpTIONNLP annotation with IAAmagic-linguistic-annotate
bratLightweight web annotationmagic-linguistic-annotate

Evaluation & Metrics

ToolMetricUsed By
sacrebleuchrF++, spBLEU, BLEUmagic-linguistic-eval
COMET / xCOMETLearned MT qualitymagic-linguistic-eval
GEMBA-MQMLLM-judge MQM rubricmagic-linguistic-eval
AfroBenchAfrican language benchmarksmagic-linguistic-eval
IndicXTREMEIndic language benchmarksmagic-linguistic-eval
SEACrowdSoutheast Asian benchmarksmagic-linguistic-eval

Typological Databases

DatabaseCoverageUsed By
WALS (World Atlas of Language Structures)2,662 languagesmagic-linguistic-scope
Grambank2,467 languagesmagic-linguistic-scope
URIEL / lang2vecTypological distance vectorsmagic-linguistic-scope, magic-linguistic-transfer
GlottologLanguage catalog + genealogymagic-linguistic-scope
Was this page helpful?

Last updated on

Installation

How to install and configure the Computational Linguistics suite

Quick Start

First language analysis walkthrough using magic-linguistic-scope and magic-linguistic-scripts

On this page

Supported ToolsLanguage IdentificationUnicode & Script ToolsTokenization & Vocab ExtensionData Sources & CorporaBitext Mining & AlignmentTransfer Learning & Fine-TuningMorphological AnalysisSyntax & ParsingSemantics & Lexical ResourcesSpeech & AudioAnnotation ToolsEvaluation & MetricsTypological Databases