Git Integration

Nyl provides built-in Git repository management for fetching Helm charts and scanning application manifests from Git repositories. This enables declarative infrastructure management without requiring manual repository cloning.

Features

Bare repositories: Minimal disk usage with shared object store
Worktrees: Isolated checkouts for different refs (branches, tags, commits)
Lazy fetching: Refs fetched first, objects downloaded on-demand
Automatic caching: Repositories cached locally for fast subsequent access
Concurrent access: Multiple refs from the same repository can be checked out simultaneously

Cache Directory

Nyl stores Git repositories in a cache directory to avoid redundant clones and improve performance.

Configuration

The cache directory is determined by:

NYL_CACHE_DIR environment variable (preferred)
.nyl/cache/ in the current directory (fallback)

Example:

export NYL_CACHE_DIR=/var/cache/nyl
nyl render app.yaml

Cache Structure

$NYL_CACHE_DIR/git/
├── bare/
│   ├── {url_hash}-{repo_name}/          # Bare repository
│   │   ├── objects/
│   │   ├── refs/
│   │   └── ...
│   └── {url_hash2}-{repo_name2}/
└── worktrees/
    ├── {url_hash}-{ref_hash}/            # Worktree checkout
    ├── {url_hash}-{ref_hash2}/           # Another ref from same repo
    └── {url_hash2}-{ref_hash}/           # Different repo

Key points:

One bare repository per Git URL (shared object store)
One worktree per unique URL + ref combination
Worktrees share objects from bare repo (disk-efficient)
URL and ref hashes ensure uniqueness and avoid conflicts

Using Git with HelmChart

HelmChart resources can reference Helm charts stored in Git repositories.

Basic Example

apiVersion: nyl.niklasrosenstein.github.com/v1
kind: HelmChart
metadata:
  name: nginx
spec:
  chart:
    git: https://github.com/bitnami/charts.git
    git_ref: main
    path: bitnami/nginx
  release:
    name: nginx
    namespace: default

Parameters

git: Git repository URL (required)
git_ref: Branch, tag, or commit (optional, defaults to HEAD)
path: Subdirectory within repository (optional)

Supported Ref Types

You can reference different types of Git refs:

Branch:

spec:
  chart:
    git: https://github.com/example/charts.git
    git_ref: main

Tag:

spec:
  chart:
    git: https://github.com/example/charts.git
    git_ref: v2.1.0

Commit SHA:

spec:
  chart:
    git: https://github.com/example/charts.git
    git_ref: abc123def456

HEAD (default):

spec:
  chart:
    git: https://github.com/example/charts.git
    # git_ref defaults to HEAD

Using Git with ApplicationGenerator

ApplicationGenerator resources scan Git repositories for Nyl manifests and automatically generate ArgoCD Applications.

Example

apiVersion: argocd.nyl.niklasrosenstein.github.com/v1
kind: ApplicationGenerator
metadata:
  name: cluster-apps
  namespace: argocd
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  source:
    repoURL: https://github.com/example/gitops-demo.git
    targetRevision: main
    path: apps
    include:
      - "*.yaml"
      - "*.yml"
    exclude:
      - ".*"
      - "_*"
  project: default

When rendered, Nyl will:

Clone the Git repository to cache
Check out the main branch
Navigate to the apps/ directory
Scan for YAML files matching the include/exclude patterns
Generate ArgoCD Application manifests for each NylRelease found

Performance Characteristics

Initial Clone

Lazy fetching: Only refs are fetched initially (~KB), not objects (~MB/GB)
On-demand objects: Commit objects fetched only when needed
Bandwidth efficient: Minimal initial download

Subsequent Access

Cache hit: Near-instant if ref already checked out
Ref update: Only fetch new refs if branch updated
Object reuse: Worktrees share objects from bare repo

Disk Usage

Bare repo: One copy of objects for all refs
Worktrees: Only working directory files (no .git directory)
Efficient: Much smaller than full clones for each ref

Example disk usage for a 100MB repository with 3 refs:

Traditional approach: 3 × 100MB = 300MB
Nyl approach: 100MB (bare) + 3 × ~10MB (worktrees) = ~130MB

Authentication

Nyl supports both public and private Git repositories through multiple authentication methods.

Automatic Credential Discovery

When running inside a Kubernetes cluster with access to ArgoCD secrets, Nyl automatically discovers repository credentials from ArgoCD repository secrets. No additional configuration required!

Example: Using a private Helm chart

apiVersion: nyl.niklasrosenstein.github.com/v1
kind: HelmChart
metadata:
  name: private-app
spec:
  chart:
    git: git@github.com:myorg/private-charts.git
    git_ref: main
    path: charts/app
  release:
    name: private-app
    namespace: default

If an ArgoCD repository secret exists for github.com, Nyl will automatically use those credentials.

Authentication Methods

ArgoCD Repository Secrets (Recommended)
- Credentials automatically discovered from argocd namespace
- Supports both SSH keys and HTTPS tokens
- Zero configuration required
- See Repository Secrets for details
SSH Agent
- Fallback for SSH URLs when no secret found
- Works with local development workflows
- Requires SSH agent running with key loaded
Public Repositories
- No authentication needed
- Works out of the box

Supported Credential Types

SSH Key Authentication:

Private key stored in ArgoCD secret
Recommended for production use
Better security than HTTPS tokens

HTTPS Token Authentication:

Personal access tokens or passwords
Useful for HTTPS-only repositories
Stored in ArgoCD secret

Example: Creating Repository Secret

# SSH authentication
kubectl create secret generic github-private \
  -n argocd \
  --from-literal=url=git@github.com:myorg/charts.git \
  --from-file=sshPrivateKey=$HOME/.ssh/id_rsa

kubectl label secret github-private \
  -n argocd \
  argocd.argoproj.io/secret-type=repository

# HTTPS authentication
kubectl create secret generic github-https \
  -n argocd \
  --from-literal=url=https://github.com/myorg/charts.git \
  --from-literal=username=myuser \
  --from-literal=password=ghp_token123

kubectl label secret github-https \
  -n argocd \
  argocd.argoproj.io/secret-type=repository

For complete documentation on authentication, see Repository Secrets.

Limitations

Shallow clones not supported: libgit2 (the underlying library) doesn’t support shallow clones. Full repository history is fetched.
Force checkout: When reusing worktrees, local changes are discarded. Worktrees are treated as read-only checkouts.

Troubleshooting

Cache directory permissions

Problem: Permission denied when creating cache directory

Solution: Set NYL_CACHE_DIR to a writable location:

export NYL_CACHE_DIR=$HOME/.cache/nyl

Large repository performance

Problem: Initial clone is slow for large repositories

Explanation: Nyl fetches the full repository (no shallow clone support)

Workaround: Use a specific tag/commit to avoid fetching all branches:

spec:
  chart:
    git: https://github.com/large/repo.git
    git_ref: v1.2.3  # Specific tag, not a branch

Stale cache

Problem: Git repository not updating with latest changes

Solution: Clear the cache directory:

rm -rf $NYL_CACHE_DIR/git/

Or for a specific repository:

# Find the cached repo
ls $NYL_CACHE_DIR/git/bare/
# Remove it
rm -rf $NYL_CACHE_DIR/git/bare/{hash}-{repo-name}
rm -rf $NYL_CACHE_DIR/git/worktrees/{hash}-*

apiVersion: nyl.niklasrosenstein.github.com/v1
kind: HelmChart
metadata:
  name: app-production
spec:
  chart:
    git: https://github.com/company/charts.git
    git_ref: stable
    path: applications/myapp
  release:
    name: myapp
    namespace: production
  values:
    environment: production
    replicas: 5

Development Branch

apiVersion: nyl.niklasrosenstein.github.com/v1
kind: HelmChart
metadata:
  name: app-development
spec:
  chart:
    git: https://github.com/company/charts.git
    git_ref: develop
    path: applications/myapp
  release:
    name: myapp
    namespace: development
  values:
    environment: development
    replicas: 1

Specific Version

apiVersion: nyl.niklasrosenstein.github.com/v1
kind: HelmChart
metadata:
  name: app-stable
spec:
  chart:
    git: https://github.com/company/charts.git
    git_ref: v2.1.0
    path: applications/myapp
  release:
    name: myapp
    namespace: staging

ApplicationGenerator with Filtering

apiVersion: argocd.nyl.niklasrosenstein.github.com/v1
kind: ApplicationGenerator
metadata:
  name: monitoring-apps
  namespace: argocd
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  source:
    repoURL: https://github.com/company/infrastructure.git
    targetRevision: main
    path: monitoring
    include:
      - "*.yaml"
      - "*.yml"
    exclude:
      - ".*"              # Hidden files
      - "_*"              # Files starting with underscore
      - "test_*"          # Test files
  project: monitoring
  labels:
    team: platform
    category: monitoring

Best Practices

Pin versions in production: Use tags or commit SHAs for production deployments:
```
git_ref: v1.2.3  # Tag
# or
git_ref: abc123  # Commit SHA
```
Use branches for development: Use branch refs for development environments:
```
git_ref: develop  # Branch name
```
Set cache directory: Configure NYL_CACHE_DIR in CI/CD environments:
```
export NYL_CACHE_DIR=/cache/nyl
```
Monitor cache size: Periodically clean up old worktrees if disk space is limited:
```
find $NYL_CACHE_DIR/git/worktrees -mtime +30 -delete
```

Use subpaths: Keep charts in subdirectories for better organization:

spec:
  chart:
    git: https://github.com/company/charts.git
    path: charts/applications/myapp

Nyl Documentation