Skip to Content
V2 (beta)Asset Pipeline & CLI/SDK

Asset Pipeline & CLI/SDK

Scrollsequence includes a high-speed Asset Pipeline designed to convert raw video files (.mp4, .webm) into web-ready, multi-resolution image bundles with hardware-accelerated playback.

The pipeline outputs an industry-standard scrollsequence-manifest.json file and tiered image folders that are consumed identically across all platforms: React, Vanilla JavaScript, and WordPress.


1. Overview & Architecture

Raw Video (.mp4, .webm) │ ▼ ┌──────────────────────────────┐ │ CLI Wizard (`vid2ssq`) │ <-- Interactive terminal or CI/CD script │ or Programmatic SDK │ <-- Custom Node.js or browser apps └──────────────┬───────────────┘ │ ┌─────────┴─────────┐ ▼ ▼ Multi-Res Image Tiers `scrollsequence-manifest.json` (720p, 1080p, 1920p) + Grayscale Depth Maps + Tracking │ │ └─────────┬─────────┘ ▼ Loaded by React (<Canvas2d>) or Vanilla JS (.scrollsequence-canvas2d)

You can run the asset pipeline in two ways:

  1. Interactive CLI Wizard (npx scrollsequence vid2ssq): The quickest way to prepare sequences directly from your terminal.
  2. Programmatic SDK (scrollsequence/sdk): Automated asset transformation in build pipelines, Node.js scripts, or in-browser apps.

2. Interactive CLI Wizard (vid2ssq)

The fastest way to transform any video into a production-ready Scrollsequence project is using the interactive CLI wizard:

# Run wizard directly via URL: npx https://docs.scrollsequence.com/scrollsequence-2.0.0-beta.53.tgz vid2ssq # Or if scrollsequence is already installed in your project: npx scrollsequence vid2ssq

No complex flags needed — the wizard interactively guides you through:

  1. Source Selection & Early Detection: Pick a local video file (.mp4, .webm). The wizard immediately probes the container and prints the detected resolution.
  2. Project Destination: Choose a project slug (automatically routes to public/<name> if run inside Next.js, Vite, or Astro).
  3. Format Selection: jpg (fastest hardware decoding) or webp (with alpha transparency support for transparent WebM).
  4. Compression & Sizing: Set image quality, frame step rate, and responsive width tiers (e.g. 1280,1920).
  5. AI Features: Optionally generate 3D depth maps, bounding tracking, and vision scene understanding.

Zero Configuration in Next.js & Vite: When run inside a directory containing ./public, the CLI automatically writes generated frames and the manifest to public/<project-name>, making them immediately accessible via your local dev server.

Non-Interactive & CI/CD Usage

You can automate asset generation in build scripts or CI pipelines by passing arguments directly:

# Standard local processing (100% free, local FFmpeg via WebAssembly / Node) npx scrollsequence vid2ssq input.mp4 --name hero-sequence --sizes 1280,1920 --format jpg --quality 80 # Advanced AI processing with 3D Depth Maps & Subject Tracking (Requires API Key) npx scrollsequence vid2ssq input.mp4 --name car-reveal --sizes 1280,1920 --depth --track "sports car"

CLI Options Reference

OptionFlagDescriptionDefault
[input]ArgumentPath to input video (.mp4, .webm).Prompted
--name-nOutput project name (routes to public/<name> if public/ exists).scrollsequence-project
--sizesComma-separated target widths in pixels.1280,1920
--formatImage format: jpg (fastest decoding) or webp (supports alpha).jpg
--qualityImage quality between 1 and 100.80
--step-sProcess every Nth frame (e.g. 2 skips every other frame).1
--depthGenerate 3D depth maps for WebGL displacement effects (Requires API key).false
--depth-widthResolution width for depth map sequence in pixels.720
--track-pText prompt for AI subject tracking across frames (Requires API key).undefined
--vision-vRun AI vision motion & scene understanding analysis (Requires API key).false
--zipPackage generated assets into a .zip archive upon completion.false

3. Programmatic SDK (scrollsequence/sdk)

You can run the asset processing pipeline programmatically within Node.js automation scripts, build tools, or custom browser applications using scrollsequence/sdk.

Installation

⚠️

Public Beta Notice: While our official NPM registry listing is pending trademark finalization, install the SDK via the direct archive URL or download scrollsequence-2.0.0-beta.53.tgz .

npm install scrollsequence@https://docs.scrollsequence.com/scrollsequence-2.0.0-beta.53.tgz

Usage Example

import { AssetPipeline, NodeDriver } from 'scrollsequence/sdk'; const pipeline = new AssetPipeline({ apiKey: process.env.SSQ_API, // Optional: for Cloud AI features driver: new NodeDriver(), // Or BrowserDriver for client-side processing onProgress: (p) => { console.log(`[${p.step}] ${p.message} (${Math.round(p.progress * 100)}%)`); }, }); // Run video-to-scrollsequence transformation const result = await pipeline.vid2ssq({ input: './source-video.mp4', name: 'product-reveal', sizes: [1280, 1920], depth: true, outputZip: true, }); console.log('Generated Manifest:', result.manifest); console.log('Output Path:', result.outputPath);

AssetPipeline Options

OptionTypeRequiredDescription
apiKeystringOptionalScrollsequence Cloud API key for AI features (SSQ_API).
driverPipelineDriverYesNodeDriver (Node.js/native FFmpeg) or BrowserDriver (in-browser Wasm FFmpeg).
onProgressFunctionOptionalCallback hook for tracking progress percentage and current step.

4. Manifest Specification (scrollsequence-manifest.json)

The Asset Pipeline generates a standard JSON manifest connecting your processed images, resolution tiers, and depth data directly to the rendering engine.

Complete Specification Example

{ "version": "2.0.0", "input": { "path": "input.mp4", "width": 1920, "height": 1080, "frameCount": 240 }, "imageSizes": { "1280": { "width": 1280, "height": 720, "files": { "i": "1280/%04d.jpg", "d": "depthmaps/%04d.jpg" }, "frameCount": 240 }, "1920": { "width": 1920, "height": 1080, "files": { "i": "1920/%04d.jpg", "d": "depthmaps/%04d.jpg" }, "frameCount": 240 } }, "tracking": { "main": [ { "frame": 0, "x": 0.5, "y": 0.5, "scale": 0.1 } ] }, "vision": { "camera": { "movement": "zoom-in", "shot_type": "medium" }, "subjects": [ { "subject_id": "car", "description": "sports car" } ] } }

Manifest Fields Reference

input

Metadata describing the original raw video source prior to conversion:

  • path: Original source filename.
  • width / height: Source video resolution in pixels.
  • frameCount: Total number of frames extracted.

imageSizes

Keyed dictionary containing each generated resolution tier (e.g. "1280", "1920"):

  • width / height: Output resolution dimensions for this tier.
  • frameCount: Number of frames present in this tier.
  • files.i: Zero-padded printf pattern (e.g. 1280/%04d.jpg formats to 1280/0000.jpg, 1280/0001.jpg, etc.).
  • files.d: Relative path pattern pointing to corresponding 3D grayscale depth maps (e.g. depthmaps/%04d.jpg).

tracking (Optional, Cloud AI)

Bounding box and center-point coordinates for subject tracking across frames. Allows anchoring dynamic HTML callouts directly to moving objects.

vision (Optional, Cloud AI)

Structured video understanding data generated by AI vision analysis:

  • camera: Inferred camera motion (pan, tilt, zoom, stationary).
  • subjects: List of detected primary subjects and semantic descriptions.

5. Adaptive Multi-Resolution & Responsive Strategy

Serving full 4K or 1080p uncompressed image sequences to mobile phones causes excessive network bandwidth consumption and memory exhaustion. Conversely, serving 720p images on a 5K display looks blurry.

Scrollsequence solves this with Adaptive Multi-Resolution Loading.

How It Works

When your application initializes a <Canvas2d> element with a manifest, the engine calculates the required rendering resolution based on two factors:

  1. Current Canvas Element Width (in CSS pixels).
  2. Device Pixel Ratio (DPR) (e.g., Retina displays with window.devicePixelRatio = 2 or 3).
Target Width = Math.round(canvasWidth * window.devicePixelRatio)

The engine matches this against the available keys in your manifest’s imageSizes dictionary and automatically selects the closest optimal tier that does not exceed memory constraints.

For standard web projects, we recommend generating two or three resolution tiers:

  • 1280 px width: Ideal for mobile phones, tablets, and low-DPR laptops. Lightweight file size ensures instantaneous preloading.
  • 1920 px width: Standard full-HD tier, delivering crisp visuals on standard desktop monitors and high-DPR laptops.
  • 2560 px width (Optional): Ultra-HD tier for hero animations on wide 4K/5K desktop screens.

Zero-Flicker Preloading

When a user resizes their browser window or changes orientations, the Scrollsequence manager dynamically switches to the new optimal tier without tearing or dropping scroll frames. Frames already loaded in memory remain visible while new resolution frames stream in the background.


6. Transparent Videos & Alpha Channels (webp)

Scrollsequence natively supports transparent video animations—enabling subjects, actors, 3D products, and motion graphics to seamlessly float over web pages, gradients, and dynamic background typography.

Supported Video Formats for Transparency

The pipeline extracts 32-bit RGBA frames with native alpha channels from:

  • WebM (.webm): Encoded with VP8 or VP9 with alpha (ALPHA_MODE: 1), such as videos processed with background removal tools.
# Extract with transparent WebP output npx scrollsequence vid2ssq transparent-subject.webm --format webp --sizes 1280

Why WebP? While JPG does not support alpha channels, WebP provides 24-bit RGB plus an 8-bit alpha channel with state-of-the-art compression. Transparent WebP frames are ~30% smaller than PNG while decoding with hardware acceleration at 60+ FPS in modern browsers.


7. 3D Depth Maps & WebGL Displacement

Scrollsequence features built-in support for monocular 3D depth maps. By pairing each color frame with a corresponding grayscale depth image, Scrollsequence creates real-time 2.5D parallax, mouse tilt, and camera displacement effects using GPU-accelerated WebGL shaders.

How Depth Displacement Works

  1. Grayscale Encoding:
    • In each depth map frame, pixel brightness represents spatial depth from the camera lens.
    • White (#ffffff): Represents elements closest to the foreground.
    • Black (#000000): Represents elements furthest in the background.
  2. Real-Time Shader Displacement:
    • The WebGL rendering engine offsets pixels along the X and Y axes in proportion to their depth value and user interaction (e.g. mouse cursor movements, gyroscope device orientation, or scroll acceleration).
    • Foreground objects shift more than background objects, producing a lifelike 3D stereoscopic illusion.

Shared Depth Sequences

Because depth maps represent low-frequency geometry rather than high-frequency color detail, depth maps can be rendered at lower resolutions (e.g. 720px wide) and shared across all color tiers (1280, 1920, etc.):

"files": { "i": "1920/%04d.jpg", "d": "depthmaps/%04d.jpg" }

This drastically reduces total file size while preserving high visual fidelity.


8. Cloud AI Services & API Keys

While basic video processing, frame extraction, responsive resizing, and manifest generation are 100% free and run locally, Scrollsequence provides advanced Cloud AI services to elevate your interactive animations.

Available AI Features

  1. 3D Depth Estimation (--depth): Uses modern vision neural networks to predict monocular depth geometry frame-by-frame, outputting WebGL-ready grayscale displacement maps without requiring LiDAR or dual cameras.
  2. Subject & Object Tracking (--track "<prompt>"): Tracks moving objects or actors across frames (e.g. --track "sports car"). The pipeline outputs normalized 2D coordinates and bounding boxes in scrollsequence-manifest.json, making it easy to pin dynamic HTML labels, badges, or tooltips to moving elements.
  3. Scene Vision Analysis (--vision): Extracts semantic scene descriptions, detected camera motion (pan, tilt, zoom, static), and key focus regions to assist with automated animation triggers and content synchronization.

Configuring Your API Key

To enable Cloud AI features during CLI runs or within SDK automation:

  1. Obtain an API key from the Scrollsequence Dashboard .
  2. Export the key into your shell or configure it in your project’s .env file:
SSQ_API=ssq_api_xxxxxxxxxxxxxxxx

When SSQ_API is set, the CLI automatically authenticates and routes AI tasks to the high-speed GPU processing cluster.


9. Package Exports Quick Reference

The scrollsequence npm package distributes multiple specialized entry points:

SubpathTypesTarget EnvironmentPurpose
scrollsequence./dist/types/vanilla/index.d.tsBrowser (Vanilla / CMS)Default export: ScrollsequenceManager engine.
scrollsequence/react./dist/types/react/index.d.tsReact / Next.js / AstroReact components: <Scene>, <Layer>, <Canvas2d> (styles auto-injected).
scrollsequence/sdk./dist/types/sdk/index.d.tsNode.js / BrowserPipeline SDK: AssetPipeline, BrowserDriver, NodeDriver.
scrollsequence/scrollsequence.css—Global CSSCompiled stylesheet for non-React projects.
Last updated on
Feedback