This repository contains the working Astro project used to generate the Spectral Codex website.
- All content authored in MDX using the Content Layer API
- Quality scoring system (0-5 scale) drives content prioritization site-wide
- Comprehensive validation ensuring data quality across collections (frontmatter fields, GPS coordinate de-duplication)
- Automated excerpt generation for previews and listings
- Metadata index with automatic backlinks discovery from internal links
- Content linting and formatting via mdxlint with remark plugins
- Link checker that scans external URLs across collections, tracks status in a SQLite DB, and supports rechecking and filtering to reduce linkrot
Content Layer Integration
- Experimental image loader treating individual images as first-class content with metadata
- Automatic extraction of camera settings, GPS coordinates, and other EXIF data from images
- Automatic generation of data URI-encoded low-quality image placeholders (LQIPs)
- Custom remark plugin for advanced image layout (groups, carousels, aspect ratio handling)
- Hero image support with optional CSS-only image carousels
External Image Server
Astro's built-in image optimization works well for smaller sites, but this project has 8,000+ high-resolution source images. Processing them all during build leads to memory exhaustion and long build times. The solution: delegate image processing to an external service.
- @unjs/ipx-based image server handles on-demand resizing, format conversion, and quality adjustment
- Nginx reverse proxy with aggressive caching ensures images are only processed once
- URL-based transformations (e.g.,
/q_80,f_webp,s_1200x800/path/to/image.jpg) allow flexible sizing without pre-generating variants - Rate limiting and request validation protect against cache-busting attacks
- Warming script pre-populates the cache after deployment; this runs on the server, making internal requests on localhost
- Docker Compose orchestration for easy deployment and updates
- React-based map component built with MapLibre, react-map-gl, and Protomaps
- Custom filter controls for adjusting what points are visible on the map
- Popups, clustering, filtering by objectives, and responsive design
- Administrative boundaries sourced from Overture Maps and converted to FlatGeobuf files for rendering on region maps
- Persistent storage of map data via IndexedDB
- Distance-based discovery via nearby locations, powered by kdbush spatial indexing for fast nearest-neighbor queries
- Integrated Pagefind via astro-pagefind for client-side full-text search across all content
- Related content recommendations via vector similarity (Transformers.js embeddings) with hybrid semantic + metadata ranking
- Automatic content relationship discovery via backlinks
- Hierarchical navigation through regions, themes, and series
- Chronological content browsing with yearly, monthly, and daily views
- Intelligent content deduplication across time periods based on created, updated, and visited dates
- Automatic highlight selection using quality scores
- Quality-based filtering with different thresholds for overview vs. detailed views
- Native web components for interactive elements (dark mode toggle, progress bars, image carousels)
- Dark/light mode toggle with system preference detection and localStorage persistence
- Visual reading progress indicator for long-form content
- Loading progress bar during navigation
- Custom CJK character handling and language-specific styling
- Not fully internationalized; the goal of the project is to display multiple scripts on the same page without compromising aesthetics
- Programmatic OG image generation via Satori and Sharp; per-page images with featured image backgrounds, multilingual title rendering (CJK/Thai), and luminance-aware adaptive text color
- Deterministic fallback system (theme/region/country) with blurred visual distinction for entries without a featured image
- Digest-based caching; only regenerates when content or source image changes
- Comprehensive meta tags and structured data
- Dynamic sitemap via @astrojs/sitemap
- Full RSS feeds with server-side rendered MDX content via Astro's Container API
Standard Astro commands apply:
pnpm dev- start the development server (includes local image serving)pnpm build- generate a production buildpnpm preview- preview the production build locally
Deployment is handled by custom scripts. These are specific to this project's infrastructure but demonstrate some useful patterns. The full pipeline runs:
- Content sync and validation
- Redirect generation from former content slugs
- Related content generation (semantic similarity)
- OG image generation with Satori and Sharp
- Astro production build
- E2E smoke tests
- Cache manifest generation from built HTML
- Media sync to remote storage
- Static file transfer via rsync
- OG image deployment
- Caddy config and TLS cert sync with reload
- CDN cache purge (Cloudflare)
- New image cache warming
The image server is deployed separately and manually; it is only needed when image server code or Docker config changes.
tsconfigshould also specify the remark plugin toolchain Astro uses (and the user may modify) to lint Markdown and MDX files- mdxlint is used to lint and format MDX files (
pnpm check-content/pnpm format-content)
Keep original image assets in the media folder specified in .env. High-quality JPG or lossless PNG format images at 2400+ pixels on the long edge are recommended. Current standard is mostly based on 3,600 pixel JPGs saved at maximum quality in Lightroom.
./deploy: Deployment configuration split intoinfra/(Caddy, analytics) andsite/(image server)./public: contains a favicon, fallback Open Graph images, and map division data./src: primary project source files./src/components: Astro components organized by functionality./src/layouts: layouts for different content types and page structures./src/lib: TypeScript utility code organized by feature area./src/lib/collections: Content Layer API configuration and data handling./src/pages: Astro routes including static API endpoints./src/styles: CSS files including Tailwind configuration and custom styles
./packages/astro-build-logger: Astro integration that logs build timestamps and durations./packages/content: primary content collection (private, not included in repo)./packages/content-demo: example content for testing and demonstration purposes./packages/image-loader: experimental image loader; treats image files as actual content and optionally reads EXIF metadata and generates low-quality image placeholders (LQIPs)./packages/react-map-component: interactive map component built with MapLibre and react-map-gl./packages/remark-img-group: Remark plugin for handling image groups in MDX./packages/scripts: build tooling for content validation, semantic similarity generation, OpenGraph image generation (Satori), map division processing, link checking, and deployment orchestration./packages/shared: shared utilities including Keyv-based caching (SQLite and file backends) for build-time data persistence, common map types and schemas
./.astro: automatically generated types and schemas./.cache: cache folder for generated files, delete this anytime./dist: output folder, automatically generated; delete this anytime./public/divisions: FlatGeobuf files for geographic divisions used by the map component./public/icons: icon sprites and JSON files for the map component./temp: temporary storage for generated files, delete anytime
The site is proxied through Cloudflare. HTML caching requires these dashboard settings:
- Caching > Configuration: Browser Cache TTL set to "Respect Existing Headers"
- Caching > Cache Rules: "Cache HTML" rule with hostname
spectralcodex.com, eligible for cache, edge TTL respects origin headers - Cache purge: API credentials in
deploy/.env(CLOUDFLARE_ZONE_ID,CLOUDFLARE_API_TOKEN), runs duringpnpm deploy-site
This project is licensed under the MIT License. Feel free to use and adapt the code (but not the personal content specific to the project) for your own projects.