Alauda AI

Model Repository

The core definition of the model repository feature is to provide a Git-based version-controlled storage system for machine learning models, enabling teams to manage model files, track versions, and collaborate across tenants. It leverages Git LFS for large file storage and integrates with MLOps workflows to bridge model development and deployment.

TOC

Advantages

Git-native Version Control

  • Track model changes via commits/branches/tags, ensuring reproducibility.

High-Speed Transfers

  • CLI/Notebook uploads leverage internal network bandwidth.

Cross-Tenant Sharing

  • Shared models can be accessed across namespaces (e.g., public as a model marketplace).

Seamless Integration

  • Directly deploy models from the repository to inference services.

Core Features

Model Repository Creation & Deletion

  • Create an empty Git-backed repository with metadata (name/description/visibility).
  • Delete models after dependency checks (e.g., ensure no active inference services).

File Management

  1. CLI/Git LFS
    • Use git lfs track for large files (e.g., *.bin, *.h5).
    • Example: 
      git clone <model_repo_url>
git lfs install
cp ~/local_model/* . && git add . && git commit -m "v1.0" && git push

Version Control

  1. Branching
    • Maintain parallel versions (e.g., experimental vs main branches).
  2. Tagging
    • Mark releases via UI/CLI (e.g., git tag -a v2.0 -m "Stable release").
  3. Metadata Sync
    • Auto-read README.md from the default branch for model descriptions.

Cross-Tenant Sharing

  1. Shared Models
    • Set visibility to "Shared" during creation for inter-tenant access.
  2. Public Marketplace
    • Use public namespace to publish open-source models (e.g., HuggingFace conversions).

Integration with MLOps

  1. Deployment Ready
    • One-click inference service launch from tagged model versions.
  2. Notebook Integration
    • Pull models directly into AML Notebooks for testing: 
      !git clone https://aml-public/resnet50.git

Technical Notes

  1. Git LFS Requirement
    • Must include .gitattributes to specify LFS-tracked files (e.g., *.zip filter=lfs diff=lfs merge=lfs).
  2. Default Branch Rules
    • Misconfigured README.md metadata may block inference deployment.

Create Model Repository

WARNING

Before you begin, make sure you have Git and Git LFS installed on your system.

$ git lfs version

Step 1: Clone or initialize a repository

If you already have a remote repository, clone it. Otherwise, you can initialize a new Git repository locally.

# Method 1: Clone an existing repository
git clone <repository-url>
cd <repo-name>

# Method 2: Initialize a new repository
mkdir <your-repo-name>
cd <your-repo-name>
git init
git checkout -b main # Create and switch to the main branch

Step 2: Initializing Git LFS

Initializing Git LFS

git lfs install

Step 3: Copy the model file to the repository directory (optional)

# If you have local model files that need to be added to the repository, copy them to the current directory.

cp -r /path/to/your/model/files/* .

Step 4: Configure Git LFS tracking mode (choose one of the two)

  1. You can choose to proactively create a .gitattributes file to specify the file types to track.

  2. Alternatively, use the git lfs migrate command to automatically check and migrate large files.

  • Option 1: Active tracking via .gitattributes (recommended for new projects)

    # This method tells Git LFS which files to track by explicitly specifying the file types in the `.gitattributes` file.
# This is useful for new projects or when you know exactly what file types you want to track.

cat >.gitattributes <<EOL
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text
EOL

# You can also manually add or modify the `.gitattributes` file, for example:
git lfs track "*.h5" "*.bin" "*.pt" # Track files with the specified suffix
    TIP

    If you already have files in your repository that you want to be tracked by Git LFS, you may want to run the following command after committing your .gitattributes changes to ensure Git reprocesses those files and converts them to LFS pointers:

    git add --renormalize .

  • Option 2: Use git lfs migrate to check and migrate (recommended for existing repositories or historical files)

    The git lfs migrate command can find and migrate large files that already exist in your Git history (but are not tracked by LFS). This rewrites history—coordinate with collaborators. When pushing rewritten history, prefer --force-with-lease.

    # Check for files that need to be migrated
```bash
# Check for files that need to be migrated

git status # Ensure the working tree is clean
git lfs migrate info

# Migrate existing large files to LFS
# The following command will migrate all files larger than 100MB to Git LFS. This 100MB limit is based on GitHub's recommended file size limit for optimal performance.

git lfs migrate import --above 100MB
    TIP

    For shared repositories: notify collaborators before migration, and when pushing rewritten history use: git push --force-with-lease

Step 5: Add, commit, and push

Once you have configured LFS, add your files to the staging area, commit them, and push them to the remote repository.

# Add all changes, including the `.gitattributes` file (if created) and the model files.
git add .
git add --renormalize . # Ensure all files that conform to LFS rules are correctly tagged.

# Check the list of files currently tracked by LFS (optional)
git lfs ls-files -n

# Commit the changes
# It is recommended to configure your username and email address, or make sure you have them configured globally.
# git config --global user.name "Your Name"
# git config --global user.email "your.email@example.com"
git commit -am "Add LLM model files with Git LFS"
git push -u origin main