Generating 500+ SEO pages with Python
Templates, data pipelines, schema injection, and zero broken links — the exact Python workflow I use to generate hundreds of verified pages at scale, without sacrificing quality or crawlability.
Ravi Srivastava
Full Stack Developer & Technical SEO Expert · ravisrivastava.in
Why programmatic SEO is worth the engineering effort
For the NEET education platform I work on, we have hundreds of colleges, thousands of rank-cutoff combinations, and dozens of specialisations — each a genuine search query someone types into Google. Writing those pages by hand is impossible. Writing them badly with a naive script produces thin content that gets deindexed. The engineering challenge is generating pages that are unique enough to rank, structured enough to parse, and verified enough that none of them ship with broken links, missing schema, or empty template variables.
This is the system I built to do that. It's Python all the way through, it runs in CI, and it has caught thousands of errors before they ever reached production.
Programmatic SEO only works if the generated pages are genuinely useful. The pipeline's job isn't to produce volume — it's to ensure that every page that ships answers a real question with real data, no exceptions.
Step 1: The data layer — one source of truth
Every generated page is downstream of a structured data source. For the NEET platform, that's a set of CSV files exported from a Google Sheet — one for colleges, one for rank cutoffs by category and year, one for fee structures. The first rule of the pipeline: never hardcode data in templates. If a value appears in a page, it must come from the data layer, so it can be updated in one place and regenerated everywhere.
import pandas as pd
colleges = pd.read_csv('data/colleges.csv')
cutoffs = pd.read_csv('data/cutoffs.csv')
fees = pd.read_csv('data/fees.csv')
# Merge into one flat record per page target
pages = colleges.merge(cutoffs, on='college_id').merge(fees, on='college_id')
print(f"{len(pages)} page targets loaded")The merge step is deliberate. A page with missing cutoff data should fail loudly at the data layer, not silently produce a page with an empty table. After the merge, any row with null values in required fields gets dropped and logged — we know exactly which colleges have incomplete data before a single HTML file is written.
Step 2: The template — Jinja2 with strict mode
I use Jinja2 for templating with undefined=StrictUndefined. This is the single most important configuration choice in the entire pipeline. Strict mode means any template variable that isn't provided raises an exception immediately, rather than rendering an empty string silently. In a run of 500 pages, a silent empty string in a title tag or schema field is a production bug you won't discover until a crawler audit weeks later.
from jinja2 import Environment, FileSystemLoader, StrictUndefined
env = Environment(
loader=FileSystemLoader('templates'),
undefined=StrictUndefined,
autoescape=True
)
template = env.get_template('college-page.html')The template itself is a full HTML file with placeholder variables for every dynamic field: the page title, meta description, canonical URL, the college name in headings, the cutoff table rows, and the JSON-LD schema block. Every variable is named explicitly. There are no computed values inside the template — all logic lives in the Python layer that builds the context dict.
Step 3: Schema injection — structured data at scale
Every generated page gets a JSON-LD block injected into the <head>. For college pages, that's an EducationalOrganization schema with the college name, location, and website. For cutoff pages, it's a Dataset schema pointing at the structured table data. The schema is built in Python as a dict, serialised with json.dumps, and injected into the template context — never written by hand inside the template, which would make it impossible to validate programmatically.
import json
def build_schema(row):
return {
"@context": "https://schema.org",
"@type": "EducationalOrganization",
"name": row['college_name'],
"address": {
"@type": "PostalAddress",
"addressLocality": row['city'],
"addressRegion": row['state'],
"addressCountry": "IN"
},
"url": row['official_url']
}
context = {
'schema_json': json.dumps(build_schema(row), indent=2),
# ... other variables
}Step 4: The verification pass — zero broken links
After all pages are generated but before anything is written to the output directory, the pipeline runs a verification pass over every generated HTML string. This catches four categories of error that appear regularly in programmatic generation: unfilled template variables (any remaining {{ in the output), internal links pointing to pages that weren't generated in this run, canonical URLs that don't match the actual output path, and schema blocks that fail JSON parsing.
import re
def verify_page(html, output_path, all_output_paths):
errors = []
# Catch unfilled Jinja variables that slipped through
if re.search(r'\{\{.*?\}\}', html):
errors.append("Unfilled template variable detected")
# Validate JSON-LD is parseable
schema_matches = re.findall(r'', html, re.DOTALL)
for schema in schema_matches:
try:
json.loads(schema)
except json.JSONDecodeError as e:
errors.append(f"Invalid JSON-LD: {e}")
return errorsAny page that fails verification is written to an error log with the specific failure reason, and the pipeline exits with a non-zero status code. In CI, this means the deploy never runs if verification fails. In practice, the most common errors caught are missing data fields that the merge step didn't filter and canonical URL mismatches from slug generation edge cases — both caught before they reach production.
Step 5: Sitemap and llms.txt generation
The final step of the pipeline generates a sitemap.xml and an llms.txt file from the same list of output paths used in the verification step. This is trivial to add once you have a structured list of every generated page — and it ensures the sitemap is always in sync with what was actually generated, rather than maintained separately and drifting over time.
from datetime import date
def write_sitemap(output_paths, base_url, out_file):
today = date.today().isoformat()
urls = '\n'.join(
f" {base_url}/{p} {today} \n{urls}\n '
with open(out_file, 'w') as f:
f.write(xml)Running it in CI
The entire pipeline — data load, template render, verification, sitemap generation — runs in a GitHub Actions workflow on every push to main. Total runtime for 500 pages is under 90 seconds. Failed verifications block the deploy at the workflow level. The output directory is committed back to the repo and served via Cloudflare Pages, which picks up the new files automatically. There's no manual step anywhere in the chain from data update to live page.
If you want to set up a programmatic SEO pipeline for your own site — or audit an existing one for thin content, broken schema, or crawl issues — get in touch. I'm available for consulting and project work.
More from the blog
